Jenkins is CiviCRM's continuous integration server.
civilint before running unit tests.
For new (unrecognised by Jenkins) contributors, Jenkins will automatically respond "can an admin verify this patch?", and a Github user with admin permissions may approve running the tests by commenting on the PR "ok to test".
If the user is trusted, CiviCRM administrators can add the person to the whitelist by commenting "add to whitelist".
PR Test Jobs¶
Jenkins runs a "PR Test" job which is triggered whenever a pull request is created or updated in GitHub. PR test jobs can take anywhere from 5 - 90 min to complete. This works for the following repos:
To run all the tests in one of the suites locally, you can use civi-test-run.
The tests that are run are variable depending on the repository that is triggered.
civicrm-core: Upgrade, karma, CRM, api_v3, E2E and Civi tests
civicrm-packages: same as civicrm-core
civicrm-drupal: Upgrade, Karma and E2E tests
civicrm-backdrop: same as civicrm-drupal
For PR test jobs against
civicrm-packages, Jenkins only builds a Drupal site to run the tests against. If you are fixing an issue with another CMS you may need to build yourself a local test environment with that CMS
If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR "Jenkins, test this please"
Matrix Test Jobs¶
The other type of Job that jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations.
The main difference between the
CiviCRM-Core-Matrix job and a PR test job is that it runs more upgrade tests than the
The other matrix job is one that runs the webtests.
The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and a PHP7.0 + MySQL5.7 test seerver.
Due to the size of the test matrix, jobs can take from 2 to 24 hours to complete depending on the job.
Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase.
Code-style is not checked on this build, but all upgrade and web tests are run.