In this post, we will explain how the Piwik project uses continuous integration to deliver a quality software platform to dozens of thousands of users worldwide. Read this post if you are interested in Piwik project, Quality Assurance or Automated testing.

Why do we care about tests?

Continuous Integration brings us agility and peace of mind. From the very beginning of the Piwik project, it was clear to us that writing and maintaining automated tests was a necessity, in order to create a successful open source software platform.

Over the years we have invested a lot of time into writing and maintaining our tests suites. This work has paid off in so many ways! Piwik platform has fewer bugs, fewer regressions, and we are able to release new minor and major versions frequently.

Which parts of Piwik software are automatically tested?

  • Piwik back-end in PHP5: we use PHPUnit to write and run our PHP tests: unit tests, integration tests, and plugin tests.
  • piwik.js Tracker: the JS tracker is included into all websites that use Piwik. For this reason, it is critical that piwik.js JavaScript tracker always works without any issue or regression. Our Javascript Tracker tests includes both unit and integration tests.
  • Piwik front-end: more recently we’ve started to write JavaScript tests for the user interface partially written in AngularJS.
  • Piwik front-end screenshots tests: after each change to Piwik, more than 150 different screenshots are automatically taken. For example, we take screenshots of each of the 8-step installation process, we take screenshots of the password reset workflow, etc. Each of these screenshot is then compared pixel by pixel, with the “expected” screenshot, and we can automatically detect whether the last code change has introduced an undesired visual change. Learn more about Piwik screenshot tests.

How often do we run the tests?

The tests are executed by Travis CI after each change to the Piwik source code. On average all our tests run 20 times per day. Whenever a Piwik developer pushes some code to Github, or when a community member issues a Pull request, Travis CI automatically runs the tests. In case some of the automated tests started failing after a change, the developer that has made the change is notified by email.

Should I use Travis CI?

Over the last six years, we have used various Continuous Integration servers such as Bamboo, Hudson, Jenkins… and have found that the Travis CI is the ideal continuous integration service for open source projects that are hosted on Github. Travis CI is free for open source projects and the Travis CI team is very friendly and reactive! If you work on commercial closed source software, you may also use Travis by signing up to Travis CI Pro.

Summary

Tests make the Piwik analytics platform better. Writing tests make Piwik contributors better developers. We save a lot of time and effort, and we are not afraid of change!

Here is the current status of our builds:
Main build:
Screenshot tests build:

PS: If you are a developer looking for a challenge, Piwik is hiring a software developer to join our engineering team in New Zealand or Poland.


Matthieu Aubry

While studying in India in 2006, Matthieu had the idea of creating an open source web analytics alternative to Google Analytics. In 2007 he released the first public beta version of Piwik. Matthieu now leads the project from New Zealand, where he likes to spend his days coding new features from the Piwik Roadmap, and helping users and customers use Piwik effectively. Find him on twitter or github.