Skip to content

Jenkins continuous integration

Pull-requests are tested automatically with build-bot software called Jenkins which runs on test.civicrm.org. Key things to know:

  • If you are a new contributor, the tests may be placed on hold pending a cursory review. One of the administrators will post a comment like jenkins, ok to test or jenkins, add to whitelist.
  • The pull-request will have a colored dot indicating its status:
    • Yellow: The automated tests are running.
    • Red: The automated tests have failed.
    • Green: The automated tests have passed.
  • If the automated test fails, click on the red dot to investigate details. Check for information in:
    • The initial summary. Ordinarily, this will list test failures and error messages.
    • The console output. If the test-suite encountered a significant error (such as a PHP crash), the key details will only appear in the console.
      • Tip: Sometimes, the console output is pretty long (several hundred KB). This usually means that a majority of tests ran, but there's a failure mixed in somewhere. View the full log and search for the word EXITCODE; this should normally appear as blank or 0. The first non-empty EXITCODE should be close to the problem.
      • Tip: Sometimes, the console output is relatively short (a page or two). You might look for evidence of one of these typical problems:
        • A network service (such as github.com or packagist.org) was unavailable. These are usually corrected quickly without any action. Try running the test again.
        • The pull-request is based on an old version of the codebase, and it cannot be applied cleanly with git apply or git scan am. Rebasing the PR branch should resolve this.
        • On the test node, a local resource (such as disk-space, RAM, or inode count) was exhausted. If the failure was recent (past few hours), seek help on the infrastructure channel. If the failure occurred a few hours or days ago, try running the test again.
        • Some part of the build/test toolchain needs attention. For example, a test script may have been changed without supporting an edge-case; or a tool like bower or npm may need to be upgraded. Seek help on infrastructure.
        • If the test appears to have failed randomly and you have been added to the whilelist by an admin, you can comment test this please to restart the test.
  • Code-style tests are executed first. If the code-style in this patch is inconsistent, the remaining tests will be skipped.
  • The primary tests may take 20-120 min to execute. This includes the following suites: api_v3_AllTests, CRM_AllTests, Civi\AllTests, civicrm-upgrade-test, and karma
  • There are a handful of unit tests which are time-sensitive and which fail sporadically. See: https://forum.civicrm.org/index.php?topic=36964.0
  • If needed, you can also manually generate a test site with the PHP and MYSQL versions, CMS and proposed patch you need or you can manually run the test suites with the same options.
  • The web test suite (WebTest_AllTests) takes several hours to execute. It runs separately -- after the PR has been merged.

For detailed discussion about automated tests, see Testing