This FAQ explains how to configure Piwik Tracking for high reliability. By configuring Piwik as explained in this FAQ, the tracker service and APIs will be more reliable. This faq is only for advanced users and sysadmins who are running Piwik at high scale and need more reliability when Mysql goes down.
If you need professional support, get in touch with the Piwik consultants who are experts in scaling and configuring Piwik. Learn more about Piwik performance, and read tips how to optimize Piwik for speed. You may also be interested in Data Limits.
How do I improve reliability of Piwik tracking as to not be dependant on the Mysql servers being up and running at all times?
Follow these steps:
- On the Piwik tracking server(s), disable synchronous tracking in the config file: in config.ini.php, below the
- At this stage, your web server(s) still receive the piwik.php requests but will directly return the 1*1 transparent gif beacon. These requests to piwik.php are still logged into your webserver access log files. They are not stored in your Piwik database yet.
- Regularly (eg. every hour or once a day), merge all these Tracking servers access log files into one file. This merged access log file will contain all requests to piwik.php including for each: the user agent, IP address, datetime, the piwik.php? URL, etc.
Sort this merged log file by date.
To import the log file, in the Piwik server running the import, edit config.ini.php, and below the
- Import this log file by running the Log Analytics tool. Use the option
This option will “replay” the piwik.php tracking requests into Piwik, effectively storing the visitors, visits, pageviews and goals data into your Mysql Database. Learn more about replaying logs in this FAQ.
- Run the archive pre-processing after the visitors were successfully imported.
- Tracker does not depend on the Mysql server being up. If Mysql is down, tracking requests are still collected in the server access log files. Piwik is more reliable.
- If you have several frontends servers collecting the piwik.php tracking pixels requests, then make sure you merge all files prior to importing them with Log Analytics.
- After merging all server log files, sort them by date to ensure a given user’s requests are always in the right order prior to import in Piwik. Sorting can be complex on very large log files.
- Check your log rotation works well. Make sure not to replay the same log data multiple times. Piwik does not de-duplicate records and will import everything thrown at the Tracking API.
- Unless you import the log files often (eg. every minute), you will lose the Real time capability of Piwik. Typically, one would import logs every hour or every day.
- Do not run the cron archiving process until after all visitors have been imported for the previous date. If you run the archive.php process too early (before visitors are imported for yesterday or past dates), then analytics data will be incomplete, and you would have to re-process the report data.
Contact Piwik PRO if you need help with high traffic Piwik setup, or if you’d like experts to take care of your Piwik setup, configuration, maintenance and security.
We hope this FAQ helps you, and we welcome your feedback or suggestions!