What is Piwik Server Log Analytics?

The Piwik Server Logs Analytics script described on this page is a Python script that will parse server log files and track the data in the Piwik MySQL database. It will then let you visualize useful and interesting reports from the data mined from your logs. The standard Piwik user interface is available when you visualize your website reports (visits, page views, downloads, keywords, referring websites, etc.) .

Using Piwik to track and report your server logs means you have access to many useful features such as: 30+ web statistic reports, IP filters and exclusion, real time reports, high performance, and so much more!

When Should I use the Log Analytics Python Script?

There are many cases in which it can be useful to use the information contained in your server access logs and import this data into Piwik.

For example, in the following use cases, analyzing server logs with Piwik is desired:

  • if you manage a web app or a desktop app but cannot add direct Javascript,
  • if you want to track and report the activity on a particular server or set of servers (for system administration purposes, QA, debugging, dealing with spammers, etc
  • if you wish to compare the data from the server logs and the standard Piwik reports using the Javascript code,
  • if you are a web hosting provider and you can’t use the Piwik JavaScript code to track visits (eg. if you do not control a website source code directly — but you control the web server)
  • if you have weeks or months of server logs that you wish to import for historical analysis: Piwik will let you import months of server access logs that you can then visualize

And more! Please let us know if you use this script with another use case and we can add it to the list.

Pros and Cons of Using Log files Instead of JavaScript Tracking

There are advantages and specificities to each tracking method.

  • Advantages of Using Log Files over Javascript
    Log files are already available so it makes sense to make the most of this information. Log files contain search engine bot & spam bot information as well as static image/css information.
  • Advantages of Using the JavaScript Tracker over Log Analytics
    JS tracking is very customizable, it executes only once per page ensuring that the data is as accurate as possible, whereas log file analysis tends to “over-report” data. Javascript tracking is often easier to setup than log analytics.
    Using JavaScript Tracking or using the Tracking API directly also allows for the more advanced use cases: Custom Variables, Ecommerce Analytics.

In the future, we hope to improve the tracking of data so that it is possible to “merge” both techniques, use the JavaScript tracking by default, and “enrich” the data using log files (for visitors that did not execute the JavaScript or for parts of the website that do not support JS).

Can I import data Using Piwik Server Log Analytics and using the standard Piwik JS at the same time?

Yes, you can use Javascript tracking, direct calls to the Tracking API and server log file analytics at the same time- if you record data for each technique in a separate website in Piwik.

For example, if you already use the JavaScript code on your website and you wish to import data from the log file as well, it is important to import server log file data in a different website to keep things in order. If you import data from the logs in the same website as the JavaScript tracker, you will have “double counts” of visits, pages, etc. In this case we recommend the following steps:

  1. create a new website in Piwik, eg. with a name “Example.org (log files)”.
  2. note the idsite of this new website. You will use this website ID to import your log file data into.
  3. in the command line, force all requests from the log files to be recorded in a specific website ID via
    --idsite=X