PHPUnit Tests

Setup

The test suites require a small amount of setup. If your system was created via buildkit and civibuild, then it was handled automatically.

PHPUnit tests ensure that CiviCRM's PHP logic is working as expected — for example, ensuring that the Contact.create API actually creates a contact.

Command name

PHPUnit is a command-line tool, but the command name varies depending on how it was installed. For example:

• In buildkit, this command is named phpunit8 or phpunit9, according to the desired version of PHPUnit.
• In other environments, it might be phpunit or phpunit.phar or phpunit.bat.

For the following examples, we'll use phpunit8.

Suites

PHPUnit tests are grouped together into suites. For example, the CRM suite includes the tests CRM_Core_RegionTest, CRM_Import_Datasource_CsvTest, and many others.

Each suite has its own coding conventions. For example, all tests in the CRM suite extend the base class CiviUnitTestCase and execute on the headless database. They require a special environment variable (CIVICRM_UF).

You'll find suites in many places, such as civicrm-core, civicrm-drupal, and various extensions. In civicrm-core, the main suites are:

Suite	Type	CMS	Typical Base Class	Comment
api_v3	Headless	Agnostic	CiviUnitTestCase	Requires CIVICRM_UF=UnitTests
Civi	Headless	Agnostic	CiviUnitTestCase	Requires CIVICRM_UF=UnitTests
CRM	Headless	Agnostic	CiviUnitTestCase	Requires CIVICRM_UF=UnitTests
E2E	E2E	Agnostic	CiviEndToEndTestCase	Useful for command-line scripts and web-services
WebTest	E2E	Drupal-only	CiviSeleniumTestCase	Useful for tests which require a full web-browser

Running tests

To run any PHPUnit test, use a command like this:

$ cd /path/to/my/project
$ phpunit8 ./tests/MyTest.php

Note how the command involves a few elements, such as the base-path of the project, the name of the PHPUnit binary, and the relative path of the test.

Let's apply this to a more realistic example. Suppose we used civibuild to create a Drupal 7 site with a copy of civicrm-core in the typical folder, sites/all/modules/civicrm. To run a typical test file like tests/phpunit/CRM/Core/RegionTest.php, you might execute:

$ cd ~/buildkit/build/dmaster/sites/all/modules/civicrm
$ phpunit8 ./tests/phpunit/CRM/Core/RegionTest.php

This command ought to work. It's well-formed. It would work in many cases -- but here it produces an error:

PHPUnit 4.8.21 by Sebastian Bergmann and contributors.

EEEEEEEEE

Time: 450 ms, Memory: 17.75Mb

There were 9 errors:

1) CRM_Core_RegionTest
exception 'RuntimeException' with message '_populateDB requires CIVICRM_UF=UnitTests'...

What's going on? The CRM suite (and its siblings, api_v3 and Civi) has a special requirement: set the environment variable CIVICRM_UF. This revised command should correct the issue:

$ cd ~/buildkit/build/dmaster/sites/all/modules/civicrm
$ env CIVICRM_UF=UnitTests phpunit8 ./tests/phpunit/CRM/Core/RegionTest.php

Using PhpStorm for local debugging

PhpStorm is an IDE which provides built-in support for executing tests with a debugger -- you can set breakpoints and inspect variables while the tests run.

Once you've successfully run a test on the command-line, you can take it to the next level and run the tests within PhpStorm.

Using civi-test-run for continuous integration

In continuous-integration, one frequently executes a large number of tests from many suites. civi-test-run is a grand unified wrapper which runs all CiviCRM test suites, and it is more convenient for use in CI scripts.

Selecting tests with AllTests.php

In civicrm-core, there are several suites (CRM, api_v3_, etc). Each suite has a file named AllTests.php which can be used as follows:

$ cd /path/to/civicrm
$ env CIVICRM_UF=UnitTests phpunit5 ./tests/phpunit/CRM/AllTests.php

Selecting tests with --filter, --group, etc

The PHPUnit CLI supports a number of filtering options. For example, execute a single test function, you can pass --filter, as in:

$ env CIVICRM_UF=UnitTests phpunit5 ./tests/phpunit/CRM/Core/RegionTest.php --filter testOverride

Selecting tests with PHPUNIT_TESTS

If you want to hand-pick a mix of tests to execute, set the environment variable PHPUNIT_TESTS. This a space-delimited list of classes and functions. For example:

$ env PHPUNIT_TESTS="MyFirstTest::testFoo MySecondTest" CIVICRM_UF=UnitTests phpunit5 ./tests/phpunit/EnvTests.php

Writing tests for core

As we mentioned in "Suites" (above), the coding conventions vary depending on the suite. We'll consider a few different ways to write tests.

CiviUnitTestCase

CiviUnitTestCase forms the basis of headless testing and unit testing in civicrm-core. In the three main test suites (CRM, Civi, and API), the vast majority of tests extend CiviUnitTestCase. This base-class is generally appropriate for writing civicrm-core tests which execute against a headless database and the standard, baseline schema. Subclasses follow a naming convention which parallels the primary core code.

For example, if you were writing a test for CRM/Foo/Bar.php, then you would create tests/phpunit/CRM/Foo/BarTest.php:

/**
 * @group headless
 */
class CRM_Foo_BarTest extends CiviUnitTestCase {
  public function testSomething() {
    $fooBar = new CRM_Foo_Bar();
    $this->assertEquals(1234, $fooBar->getOneTwoThreeFour());
  }
}

Tests based on CiviUnitTestCase have a few distinctive features:

• When you first start running the tests, they reset the headless database to a standard baseline. The DB reset generally runs once for each test-class; it does not run for each test-function.
• If you define a setUp() or tearDown() function, be sure to call the parent::setUp() or parent::tearDown().
• In the setUp() function, you can call $this->useTransaction(). This will wrap all your test functions with a MySQL transaction (BEGIN/ROLLBACK); any test data you create will be automatically cleaned up.
  • Caveat: Some SQL statements implicitly terminate a transaction -- e.g. CREATE TABLE, ALTER TABLE, and TRUNCATE. Consequently, you should only use useTransaction() if the tests perform basic data manipulation (SELECT, INSERT, UPDATE, DELETE).
• Executing tests based on CiviUnitTestCase requires setting an environment variable, CIVICRM_UF=UnitTests.
• The tests belong to @group headless.

CiviEndToEndTestCase

CiviEndToEndTestCase forms the basis of CMS-neutral end-to-end testing in civicrm-core.

For example, one might create an end-to-end test for a web service civicrm/my-web-service by creating tests/phpunit/E2E/My/WebServiceTest.php:

/**
 * @group e2e
 */
class E2E_My_WebServiceTest extends CiviEndToEndTestCase {
  public function testSomething() {
    $url = cv('url civicrm/my-web-service');
    list (, $content) = CRM_Utils_HttpClient::singleton()->post($url, array());
    $this->assertRegExp(';My service is working;', $content);
  }
}

Tests based on CiviEndToEndTestCase have a few distinctive features:

• You can call Civi classes and functions directly within the test process (eg CRM_Utils_HttpClient::singleton() or civicrm_api3('Contact','get', ['id'=>123])). In-process code executes with the permissions of an administrative user.
• You can perform work in a sub-process by either:
  • Sending HTTP requests back to the system -- as in CRM_Utils_HttpClient::singleton()->post(...).
  • Using cv() to run the scripting tool cv -- as in cv('url civicrm/my-web-service') or cv('api contact.get id=123').
• The global variable $_CV provides configuration data about the running system, such as example usernames and passwords. Use cv vars:show to view an example.
• The tests belong to @group e2e.
• There is no automated cleanup procedure. Write defensive code which cleans up after itself and checks that its environment is sufficiently clean.

Mixing in-process and sub-process work

End-to-end testing allows you to perform in-process work (eg civicrm_api3('Contact','get', ['id'=>123])) or sub-process work (eg cv('api contact.get id=123')). In-process calls are faster, but they're not as realistic. It's generally safest to pick one style or the other for a particular test because this categorically prevents issues with cache-coherence. Never-the-less, it is possible to mix the styles -- as in the example above.

Writing tests for extensions

civix

If you are writing an extension using civix, the quickest way to create a new test is to generate skeletal code with civix generate:test.

The generator includes templates for different styles of testing. To generate a basic unit test, headless test, or end-to-end test, specify --template. For example:

$ civix generate:test --template=phpunit CRM_Myextension_MyBasicUnitTest
$ civix generate:test --template=headless CRM_Myextension_MyHeadlessTest
$ civix generate:test --template=e2e CRM_Myextension_MyEndToEndTest

The resulting tests will extend PHPUnit_Framework_TestCase and employ various utilities, such as HeadlessInterface or Civi\Test. These are described more in the Reference.

From scratch

If you've worked with PHPUnit generally, you can build tests from first principles and incorporate CiviCRM. Although we're presenting in the context of PHPUnit and Civi extensions, the advice is more general -- it may be applied to other kinds of deliverables (such as Civi-CMS integrations and modules).

The first step -- as with any PHPUnit project -- is to create a phpunit.xml.dist file and specify a boostrap script.

<?xml version="1.0"?>
<phpunit bootstrap="tests/phpunit/bootstrap.php" ...>
  <testsuites>
    <testsuite name="My Test Suite">
      <directory>./tests/phpunit</directory>
    </testsuite>
  </testsuites>
  <filter>
    <whitelist>
      <directory suffix=".php">./</directory>
    </whitelist>
  </filter>
</phpunit>

At a minimum, the bootstrap.php script should register CiviCRM's class-loader. One might initially write:

require_once '/var/www/sites/all/modules/civicrm/CRM/Core/ClassLoader.php';
CRM_Core_ClassLoader()::singleton()->register();

However, this faces several problems:

• If you want to test the extension in a different environment (different server, CMS, file-structure, etc), then you have to patch the test files.
• There is no simple, portable formula for the file-path. Between various CMS configuration options and Civi configuration options, it can be quite difficult to predict the file paths (whether using absolute or relative paths).
• It only sets up the classloader. For many tests, you'll also want to bootstrap a CMS (or pseudo-CMS), setup database credentials, etc.

The simplest way to bootstrap Civi is to use cv. cv scans the directory tree to autodetect the Civi+CMS environment. The scan works in many stock environments; for more difficult environments, you can set environment variables to configure bootstrap.

The bootstrap.php file just needs one line:

eval(`cv php:boot --level=classloader`);

You can change the parameters to cv php:boot and specify different bootstrap behaviors, e.g.

• cv php:boot --level=settings -- Load CiviCRM and its settings files, but do not bootstrap Civi services or the CMS.
• cv php:boot --level=full -- Bootstrap the full CiviCRM+CMS. (This is appropriate for end-to-end testing.)
• cv php:boot --level=full --test -- Bootstrap CiviCRM and fake CMS in a headless test environment. (This is appropriate for headless testing.)

Add your own PHPUnit helpers to the bootstrap.php

There are a few PHPUnit helpers provided by civicrm-core (e.g. base-classes, traits), but you'll probably want to write some of your own. Load these files explicitly in bootstrap.php -- or add a class-loader which can handle them.

Once you have a bootstrap file, create a basic test class, tests/phpunit/MyTest.php:

class MyTest extends PHPUnit_Framework_TestCase {
  public function testSomething() { ... }
}

If you aim to write pure, basic unit-tests, then you're ready to go -- the test function has access to any CiviCRM classes. (And, if you fully bootstrapped, then it also has access to a working database environment.)

However, pure, basic unit-tests usually don't get very far in testing Civi -- because a large number of services involve constants, globals, or singletons which are difficult to mock. Most tests are headless or end-to-end, and a couple of tricks will help build those:

• It helps to establish a starting environment -- what mix of database tables and extensions should be activated as the test starts? Creating this environment can be resource-intensive, so be tactical: only do the expensive stuff when you really need to.
• For in-process, headless tests, it helps if each test-run resets the in-process state. Call Civi::reset() and/or CRM_Core_Config::singleton(TRUE,TRUE).
• For multi-process, end-to-end tests, it helps to have utility functions for launching sub-processes. For example, you might have utilities for sending HTTP requests or invoking cv.

Of course, these are recurring problems for developers in the Civi community. The Reference below describes some utilities and techniques. The civix templates make heavy use of these, but you can also assemble these pieces yourself.

Reference

\Civi\Test

Civi\Test::headless() and Civi\Test:e2e() help you to define a baseline environment -- by installing extensions, loading SQL files, etc. Consider a few examples:

// Use the stock schema and stock data in the headless DB.
Civi\Test::headless()->apply();

// Use the stock schema and install this extension (i.e. the
// extension which contains __DIR__).
Civi\Test::headless()
      ->installMe(__DIR__)
      ->apply();

// Use the stock schema, as well as some special SQL statements
// and extensions.
Civi\Test::headless()
      ->sqlFile(__DIR__ . '/../example.sql')
      ->install(array('org.civicrm.foo', 'org.civicrm.bar'))
      ->apply();

// Use the existing Civi+CMS stack, and also install this
// extension.
Civi\Test::e2e()
      ->installMe(__DIR__)
      ->apply();

// Use the existing Civi+CMS stack, and do a lot of
// crazy stuff
Civi\Test::e2e()->
      ->uninstall('*')
      ->sqlFile(__DIR__ . '/../example.sql')
      ->installMe(__DIR__)
      ->callback(function(){
        civicrm_api3('Widget', 'frobnicate', array());
      }, 'mycallback')
      ->apply();

A few things to note:

• Civi\Test::headless() and Civi\Test::e2e() are similar -- both allow you to declare a sequence of setup steps. Differences:
  • headless() only runs on a headless DB, and it can be very aggressive about resetting the system. For example, it may casually reset all your option-groups, drop all custom-data, and uninstall all extensions.
  • e2e() only runs with a live CMS (Drupal/WordPress/etc), and it has a lighter touch. It tends to leave things in-place unless you specifically instruct otherwise.
• Civi\Test is lazy (in a good way). It keeps track of which configuration steps it has run, and it avoids re-running them unless you tell it to.
  • Ex: If you call Civi\Test as part of setUp(), it will be executed several times (for every test). However, it will usually be a null-op. It will only incur a notable performance penalty when you call with different configurations.
  • How: Everytime you run apply(), it computes a signature for the requested steps. If the signature is already stored (table civitest_revs), then it does nothing. If the signature is new/changed, then it runs.
  • Caution: Civi\Test is intended to help set up the database fixture for your test class. It is not a good place to put steps which set up test class variables or other non-persistent data which need to be initialized every time, because if the signature is unchanged it will not run these steps.
• Civi\Test is stupid. It only knows what you tell it.
  • Ex: If you independently executed INSERT INTO civicrm_contact or TRUNCATE civicrm_option_value, it won't reset automatically.
  • Tip: If you know that your test cases are particularly dirty, you can force Civi\Test to execute by calling apply(TRUE) (aka apply($force === TRUE)). This may incur a significant performance penalty for the overall suite.
• PATCHWELCOME: If you need to test with custom-data, consider adding more helper functions to Civi\Test. Handling custom-data at this level (rather than the test body) should reduce the amount of work spent on tearing-down/re-creating custom data schema, and it should allow better use of transactions.

\Civi\Test\Api3TestTrait

Many CiviCRM tests focus on APIv3 or call APIv3 incidentally. This can be as simple as:

public function testContactGet() {
  $results = civicrm_api3('Contact', 'get', array('id' => 1));
  $this->assertEquals(1, $results['values'][1]['contact_id'])
}

This is pretty intuitive. If there's an error while running the API call, it will throw an exception.

However, the exceptions aren't always easy to read. The Api3TestTrait (CivCRM v5.1+) provides helper functions which report API failures in a more presentable fashion. For example, one would typically say:

use \Civi\Test\Api3TestTrait;

public function testContactGet() {
  $results = $this->callApiSuccess('Contact', 'get', array('id' => 1));
  $this->assertEquals(1, $results['values'][1]['contact_id'])
}

For a more complete listing of callApi*() and assertApi*() functions, inspect the trait directly.

\Civi\Test\CiviTestListener

The CiviTestListener is a PHPUnit plugin which allows you to mix-in common test behaviors. You can enable it in phpunit.xml.dist:

<phpunit bootstrap="tests/phpunit/bootstrap.php" ...>
  <!-- ... -->
  <listeners>
    <listener class="Civi\Test\CiviTestListener">
      <arguments/>
    </listener>
  </listeners>
  <!-- ... -->
</phpunit>

Note that the bootstrap.php script activates the CiviCRM classloader (e.g. cv php:boot --level=classloader), and the <listener> tag activates CiviTestListener.

Now, in your test classes, you can enable new behaviors by using the interfaces. This example enables several behaviors for MyFancyTest:

class MyFancyTest extends PHPUnit_Framework_TestCase
  implements HeadlessInterface, HookInterface, TransactionalInterface {

Let's consider each interface that's available.

EndToEndInterface

The \Civi\Test\EndToEndInterface marks a test-class as end-to-end, which means:

• The test will only run on a live environment (CIVICRM_UF=Drupal, CIVICRM_UF=WordPress, et al). If you try to run in a headless environment, it will throw an exception.
• The test will automatically bootstrap a live environment (if you haven't already booted).
• The test must be flagged with a PHPUnit annotation, @group e2e.
• CiviCRM errors will generally be converted to PHP exceptions.

HeadlessInterface

The \Civi\Test\HeadlessInterface marks a test-class as headless, which means:

• The test will only run on a headless environment (CIVICRM_UF=UnitTests). If you try to run in any other environment, it will throw an exception.
• The test will automatically bootstrap a headless environment (if you haven't already booted).
• The test will automatically reset common global/static variables at the start of each test function.
• The test must be flagged with a PHPUnit annotation, @group headless.
• In addition to setUp() and setUpBeforeClass(), one may implement the function setUpHeadless(). This is usually used to call Civi\Test::headless(). Like setUpBeforeClass(), setUpHeadless() runs one time, before the first test of your test case class is run.
• CiviCRM errors will generally be converted to PHP exceptions.

HookInterface

The \Civi\Core\HookInterface simplifies testing of CiviCRM hooks and events. Your test may listen to a hook by adding an eponymous function. For example, this listens to hook_civicrm_post:

class MyTest extends PHPUnit_Framework_TestCase
  implements HeadlessInterface, HookInterface {

  public function testSomething() {
    civicrm_api3('Contact', 'create', [...]);
  }

  public function hook_civicrm_post($op, $objectName, $objectId, &$objectRef) {
    // listen to hook_civicrm_post
  }
}

Similarly, to listen for a Symfony-style event (e.g. civi.api.resolve), use the on_*() prefix:

  public function on_civi_api_resolve(ResolveEvent $event) {}

These hooks only run within headless tests.

Hooks registered through a unit-test will only run within the current PHP process (while executing the test). The hooks would not work when calling out to multiple PHP processes (via HTTP or cv). Consequently, this technique does not work with E2E testing -- it only works with headless testing.

See also: Hooks in Symfony: HookInterface.

EventSubscriberInterface

The \Symfony\Component\EventDispatcher\EventSubscriberInterface also helps with testing hooks and events. It is slightly more verbose than HookInterface, but it allows more fine-tuning -- e.g. specifying listener priorities, unconventional event-names, and multiple listeners.

These hooks only run within headless tests.

Hooks registered through a unit-test will only run within the current PHP process (while executing the test). The hooks would not work when calling out to multiple PHP processes (via HTTP or cv). Consequently, this technique does not work with E2E testing -- it only works with headless testing.

See also: Hooks in Symfony: EventSubscriberInterface.

TransactionalInterface

The \Civi\Test\TransactionalInterface simplifies data-cleanup. At the start of each test-function, it will issue a MySQL BEGIN; and, at the end of each test-function, it will issue a MySQL ROLLBACK. The test is free to INSERT, UPDATE, and DELETE data -- and those changes will be automatically undone. This ensures that subsequent test-functions run in the same clean, baseline environment.

However, there are a few caveats:

• Some SQL statements implicitly terminate a transaction -- e.g. CREATE TABLE, ALTER TABLE, and TRUNCATE. If you need these, then don't use TransactionalInterface.
• MySQL transactions can only be enforced if all work focuses on one MySQL database using one PHP process. If you have other databases (e.g. Drupal/WP) or other multiple PHP processes (HTTP/cv), then they won't work. Consequently, TransactionalInterface is only compatible with headless testing -- not with E2E testing.

\Civi\Test\Invasive

In a PHP class, the methods and properties are declared with visibility of private, protected, or public. For example:

class WidgetList {
  private $widgets = [];
  private $tagIndex = [];

  public function add(Widget $w) {
    $this->widgets[] = $w;
    foreach ($w->getTags() as $tag) {
      $this->tagIndex[$tag][] = $widget;
    }
  }
}

The upshot: as a maintainer, you are free to rethink/rework private members ($widgets and $tagIndex) without fear of affecting downstream consumers.

However, this also means that a test-class cannot do any assertions about private or protected members. This example would fail in the last line:

class WidgetListTest extends \PHPUnit\Framework\TestCase {
  public function testAdd() {
    $widget = new Widget(['tags' => ['apple', 'banana']]);
    $list = new WidgetList();
    $list->add($widget);
    $this->assertTrue(in_array($widget, $list->tagIndex['banana']));
  }
}

The last line fails because WidgetListTest cannot access private members of WidgetList ($list->tagIndex).

PHP's Reflection API provides a work-around, but usage is a bit verbose. For a cleaner notation, use the helper \Civi\Test\Invasive (v5.34+):

use Civi\Test\Invasive;

// Read a private or protected property
Invasive::get([$list, 'tagIndex'])

// Update a private or protected property
Invasive::set([$list, 'tagIndex'], $newValue)

// Call a private or protected method
Invasive::call([$list, 'doSomething'])

// Call a private or protected method
Invasive::call([$list, 'doSomething'], [$arg1, &$arg2, ...])

Note that the syntax resembles PHP callback notation, i.e.

Goal	Notation
Call a regular object method	[$myObject, 'myMethod']
Call a static method	['MyClass', 'myMethod']
Call a static method	'MyClass::myMethod'