Docs › How to setup auto archiving of your reports?

  1. How to setup a crontab to automatically archive the reports?
  2. Tip for medium to High traffic websites
    1. Disable Piwik archiving to be triggered from the browser and limit Piwik reports to be updated every hour
    2. Increase the memory
  3. How to setup auto archiving using Windows Scheduler on Windows?
    1. Powershell Script configuration
    2. Scheduling
  4. More information about Piwik archiving

If your website has more than a few hundreds visits per day (bravo!), waiting for Piwik to process your data may take a few minutes. The best way to avoid these waiting times is to set up a cron job on your server so that your data is automatically processed every hour.

How to setup a crontab to automatically archive the reports?

In order to trigger automatically the Piwik archives automatically every hour, you can setup a crontab. A crontab is a time-based scheduling service in Unix-like server. You need SSH access to your server in order to set it up:
# crontab -e

and then add the lines:
MAILTO="youremail@example.com"
5 * * * * www-data /path/to/piwik/misc/cron/archive.sh > /dev/null

The Piwik archive script will run every hour. Generally, it completes in less than one minute. On larger websites (10,000 visits and more), Piwik archive can take up to 30 minutes.

Description of the parameters: www-data is the user the cron job will be executed under, it should generally be your webserver user. It is sometimes "apache". All error messages resulting from the cron job execution will be sent to the youremail@example.com address.

Description of the 'linux cron' utility: The cron utility uses two different types of configuration files, the system crontab and user crontabs. The only difference between these two formats is the sixth field. In the system crontab, the sixth field is the name of a user for the command to run as. This gives the system crontab the ability to run commands as any user. In a user crontab, the sixth field is the command to run, and all commands run as the user who created the crontab; this is an important security feature. If you setup your crontab as a user crontab, you would instead write:

5 * * * * /path/to/piwik/misc/cron/archive.sh > /dev/null

This cron job will trigger the day / week / month / year archiving process at 00:05AM every day. This will make sure that when you visit your Piwik Interface, the data has already been processed: Piwik will load fast.

Test the cron command: Make sure the crontab will actually work by executing the command
# sh /path/to/piwik/misc/cron/archive.sh

in the shell. You should see XML output containing your number of visits for each date.

Tip for medium to High traffic websites

Disable Piwik archiving to be triggered from the browser and limit Piwik reports to be updated every hour

We highly recommend setting up auto archiving as explained above. You can also now safely disable the automatic archiving from the browser, making sure your Piwik users don't trigger the heavy archiving process. You can go to the "General Settings" tab in your Piwik admin panel, and select

  • Allow Piwik archiving to trigger when reports are viewed from the browser: No
  • Reports for today will be processed at most every: 3600 seconds

Click save to save your changes.

Today's statistics will have a one hour lifetime, which ensures the reports are not processed too often.

Increase the memory

If you have the error Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate X bytes), you can increase the memory allocated to PHP. Update your /etc/php5/cli/php.ini file, and set for example:
memory_limit = 128M

This will only affect PHP CLI, your webserver configuration stays unchanged. The crontab archiving should now work, and Piwik should be very fast!

How to setup auto archiving using Windows Scheduler on Windows?

Piwik comes with a Powershell script (located in: piwik/misc/cron/archive.windows.ps1) that you can use to setup automatic archiving. If you don't have Powershell on your Windows server already, you can download it for Windows XP SP2, Windows Server 2003 or Windows Vista at Windows PowerShell download page.

Open the script in a text editor of your choice and adjust the first line according to your system configuration.
$PHP_INI = "C:\Windows\php.ini"
The value of $PHP_INI must reflect the "'full"' path to your php.ini.

Powershell Script configuration

You also have to make sure that either php5.exe or php.exe lie in a path that is specified in the %PATH% environment variable. Select "'Start"' -> "'Run"' and type "cmd" to open up a command line. Then type "'php.exe -version"' and hit enter to check whether PHP is available or not. If you have to change the %PATH% variable have a look at How To Manage Environment Variables in Windows.

Scheduling

To open the task scheduler on Windows (XP, 7, 2003/2008 Server) click All Programs, point to Accessories, point to System Tools, and then click Scheduled Tasks. Click "'Add Scheduled Task"' and name the task e.g. "Piwik Auto Archiving". Click on the tab "'Trigger"' and add a new trigger. Select to create a trigger after a timetable, to be executed daily and every hour. Confirm the settings and switch to the action tab. Add a new action and select to start a program.

We want to start "'powershell.exe"' using the additional arguments –Noninteractive –Noprofile –Command "&{}" (where <fullpath to script> is the full path to your archive.windows.ps1 script).

That's it! If you need help with the Automatic archiving on Windows using the Powershell script, please report your issues here.

More information about Piwik archiving

  • Your Piwik database size will grow over time, this is normal. Piwik will delete archives that were processed for imcomplete periods (ie. when you archived a week in the middle of this week), but will not delete other archives. You will therefore have archives for every day, every week, every month and every year in the mysql tables. They ensure very fast UI response and data access, but this requires disk space. In the future one can imagine a plugin that would delete some of the old data (for example, only keep the top 50 rows for each report).
  • At this point, archiving doesn't delete logs. In the future, these logs will either be deleted or rotated in other tables or files.
  • Archiving several times per day will only result in Today's reports being updated more often. It will not change the memory requirement for other periods: Piwik archiving is not incremental.
  • There is an issue with memory and Piwik archiving. This might be an issue on very large Piwik installations.
  • If you don't setup archiving to run automatically, archiving will occur when a user requests a Piwik report. This will often be slow and a bad user experience (users would have to wait N seconds) hence why we recommend to setup auto archiving for medium to large websites as explained above.

You may also be interested in...

Entries (RSS)