Configure Piwik for speed
- By default, Piwik works in real time. But when tracking a large website, It is highly recommended to disable “real time” reports and only process reports for “today” every 1 or 6 hours for example. This way, access to the user interface will only read pre-processed reports.
Note: the Real Time Visitors widget and the Visitors Log will still work in real time, only other reports for “today” (Best keywords, etc.) would not include the latest visits and page views. See also How long does Reports pre-processing take for high traffic websites?
- Update Piwik to the latest version – How to for high traffic Piwik servers. Sometimes, new Piwik releases contain SQL schema updates on very big mysql tables (when adding a new field, new indexes, etc.). The schema updates can take minutes or hours to complete: it helps to run the Piwik updater via the shell/command line.
Configure your server for speed
Use case: a Piwik user reported to us tracking 6.5 Million pages in March 2013, with PHP having only 512M of memory. The following recommendations help configure a server for maximum speed:
- Hosting Piwik on a powerful dedicated server is recommended when tracking high traffic websites.
- Piwik (php/Mysql) will benefit from significant RAM memory on the server: 1GB memory being the minimum recommended for Piwik to process reports on very large websites, the more RAM the better!
- Use PHP7, and the latest available stable version. (PHP7 brings huge performance improvements over PHP5)
- It is critical to use a PHP cache for high performance. Fortunately, PHP5 and newer versions such as PHP 7 come with this PHP cache included by default.
- It is recommended to use light web servers such as Nginx or lighttpd which can be faster and more efficient alternatives to Apache or IIS
- Do not use Piwik on a shared/cluster/network filesystem such as NFS or glusterfs. Network filesystems are slower, and will create a big overhead on top of Piwik. Please use standard filesystems for all Piwik files.
- For advanced users, you can setup Piwik to work in a load balanced environment and use multiple servers behind a load balancer.
- Tracker API: if you manage a high traffic Piwik server, you can easily setup Piwik Tracking API to use a Redis datastore for extremely fast and scalable Tracking API requests. Learn more about our QueuedTracking plugin in How do I configure Piwik to use Redis for better scalability?
- Tracker API: if the ‘Provider’ plugin is activated in your Piwik, the Internet provider by doing a reverse DNS lookup which adds a few milliseconds overhead. To track your visitors’ ISPs it will be faster to use either MaxMind’s ISP or Organization. Read more about installing them here. Alternatively, for optimal performance you may disable the ‘Provider’ plugin (the plugin is disabled by default for all new users since Piwik 2.15.0) and skip the potentially slow reverse DNS lookup.
- Learn how to setup reliable Piwik tracking for high traffic websites, using the asynchronous access logs bulk loading (advanced users only).
Configure your database server for speed
Piwik performance are heavily tied to Mysql database server performance as all Piwik data is stored and aggregated using Mysql queries.
- When possible, use Solid-state drives (SSDs) as they are often faster.
- A rough estimate of Piwik Mysql database size usage is approximately 1GB for every 5M page views. If your website tracks 100k page views per day (3M pages per month), you can expect a DB size of ~ 7GB after 1 year.
An important factor for achieving good performance is to keep the number of unique URLs tracked low. Please make sure you setup ignored URL parameters you do not wish to count in the URL (such as session ID parameters). The less unique URLs tracked in Piwik the better!
- We recommend to use Mysql Tuning Primer or Mysql Tuner tools which will suggest optimal Mysql settings.
- We now use Mysql Innodb tables by default (instead of Myisam) which leads to more reliability. Performance can be improved by tuning Mysql innodb settings. You may increase innodb_buffer_pool_size option as long as it fits into the RAM, which will improve general Piwik performance. We recommend to set innodb_buffer_pool_size to at least 80% of the RAM available to MySQL (and even a higher ratio if you have more than 8G of RAM).
- In MySQL configuration you may also set innodb_flush_log_at_trx_commit=2 (reference), which will increase the tracker throughput significantly.
- See also optimizing Innodb I/O on mysql.com.
- We recommend to configure MySQL and point the
tmpdirto a filesystem in the RAM for example using tmpfs, instead using a tmp directory on the disk. As MySQL uses tmpdir to create and store temporary datasets, it brings additional speed when only the RAM needs to be used.
- If you have limited database space, it may help to delete old logs, for example visitor logs older than 6 months. It helps keep the database size more manageable.
- If your website tracks 100M+ pages per month, Piwik may start to reach php memory limits or other mysql or hardware limitations.
- There have also been a few Piwik users tracking more than a billion pageviews over several months. If you need help, get in touch with Piwik experts.
- InnoCraft can support high traffic Piwik servers up to 500 Million actions tracked per month.
This guide was written by the team of Professional Consultants – contact us for any advice or problem: we will make sure you make the most of Piwik. If you want us to take care of your Piwik, we can migrate all of your data to our Piwik Cloud service, at no additional cost.