To prevent defects from entering the system or remaining in the system, tests are executed automatically by the Jenkins continuous integration server.
Automated tests are executed in a few different circumstances:
- Proposal testing for core provides feedback when reviewing specific proposals in civicrm-core and other key repositories. This runs as soon as someone submits (or revises) a proposal in Github.
- Proposal testing for extensions provides feedback when reviewing proposals for extensions like api4 and flexmailer. This serves a similar purpose, but the mechanics are different.
- Full matrix testing are the most comprehensive (and resource-intensive) tests. They run every 8-24 hours and feed into decision-making about releases and code-freezes.
The results of the tests are published at test.civicrm.org and (where feasible) linked into the Github UI.
Proposal Testing for Core¶
Whenever a pull-request (PR) is submitted or revised on Github, Jenkins runs a set of tests. PR test jobs can take anywhere from 5 min to 2 hours to complete.
The CiviCRM system is composed of several
git repositories and several test-suites. To provide quicker results, tests are selected for relevance -- for example, a patch to
civicrm-core will trigger almost all tests (because
core code can be referenced directly or indirectly by any test). However, a patch to the
civicrm-backdrop integration will only trigger end-to-end tests (because the Backdrop integration code is relevant in E2E tests) -- it will skip the APIv3 tests (because the APIv3 tests use a mock CMS which is independent of Backdrop).
||APIv3, Civi, CRM, E2E|
||APIv3, Civi, CRM, E2E|
|See also||PHPUnit Tests|
Quick results for code style
To run all the tests in one of the suites locally, you can use civi-test-run.
Alternate CMS testing
For PR test jobs against
civicrm-packages, Jenkins uses the
drupal-clean build type. If you are fixing an issue with another CMS, you may need to build yourself a local test environment with that CMS.
To prevent abuse of the testbot, Jenkins maintains a whitelist of Github contributors. It only runs automated tests if the contributor is recognized.
For new contributors, Jenkins will automatically post a comment:
can an admin verify this patch?
A Jenkins administrator should skim the PR. They will then comment:
jenkins, ok to test
After a new contributor has submitted a few good PRs, the administrator can add them to the whitelist with another comment:
jenkins, add to whitelist
Thereafter, subsequent PRs will be tested automatically.
If the tests have failed for something that we suspect is a random failure, we can ask Jenkins to run the tests again by commenting in the PR:
jenkins, test this please
Proposal Testing for Extensions¶
Work in progress
This describes a service which is currently in alpha.
The basic ideas are the same -- whenever a pull-request (PR) is submitted or revised on Github, Jenkins will be notified. There's a whitelist that determines which repos/PRs/authors will be tested. The test outcome is posted to Github.
However, unlike core testing, the extension testing uses a "convention over configuration" philosophy. Tests can be enabled with a few clicks as long as the extension follows these conventions:
- The extension lives in its own public git repository.
- The root folder includes
- If the extension requires any other extensions, these are listed in
info.xmland published in the Extension Directory.
- The PHPUnit tests are organized into two
- The extension is compatible with the
drupal-cleanbuild type in civibuild.
The civix code-generator produces compliant code by default.
The test bot supports extensions in the official
civicrm organization and some paid partner projects. If you'd like to use it on other projects, please consult with the core team.
Registration for Github repositories
To enable testing on a Github repository, visit https://github.com/apps/civibot and proceed to "Configure". You can install
civibot for any user/organization -- and then authorize it to access some repositories:
You may enable for "All repositories" (current and future), even if you have unrelated repositories. Civibot autodetects extensions and ignores other projects.
Registration for Gitlab repositories
At time of writing, we have not yet setup a registration process for Gitlab repositories. However, it has an actively considered requirement.
To prevent abuse of the testbot,
civibot checks if you are an owner/collaborator/member of the Github repository:
- If you have access, it will autotest your pull-requests.
- If you have access, it will respect manual commands like
- If you do not have access, then you must wait for someone else to issue the
When reviewing a PR, you may request a new test-run with the command
/test. The command may be interleaved with other discussion as long as it fits on its own line. For example:
We should re-test this PR because it's been sitting around for a month.
Full Matrix Testing¶
The other type of Job that jenkins runs is what is described as a "matrix" job. This is a much more extended version of the PR job and is usually run against multiple different webserver configurations.
The main difference between the
CiviCRM-Core-Matrix job and a PR test job is that it runs more upgrade tests than the
The other matrix job is one that runs the webtests.
The matrix jobs operate on two main combinations, a PHP5.5 + MySQL5.5 server and a PHP7.0 + MySQL5.7 test seerver.
Due to the size of the test matrix, jobs can take from 2 to 24 hours to complete depending on the job.
Jenkins periodically runs the full suite (once every 4-24hr) on the official codebase.
Code-style is not checked on this build, but all upgrade and web tests are run.