Depending on the scope of the test and the relevant language, one chooses among these tools:
Basic testing of a single function or class
Trade-off: Fast but minimal/synthetic environment
Good for testing: Helpers, libraries
Mid-level testing with headless database
Trade-off: Mid-level speed and realism
Good for testing: Data management APIs, portable backend services
|End-to-End Test (E2E)||
Full-stack testing with CMS, HTTPd, etc
Trade-off: Slow but realistic environment
Good for testing: Screens, pageflows, integrations
Be mindful of test types
Each style of testing (unit, headless, E2E) has distinctive practices. For example, headless tests periodically reset the entire database, but E2E tests don't. These concepts are discussed in more detail below.
Many test suites require information about your local development environment. For example, headless tests may require credentials for an extra MySQL database, and end-to-end tests may require credentials for a CMS.
The test tools obtain this information via cv. If your build was created by buildkit and
cv can fetch all the information automatically. Other builds may require some manual configuration.
To inspect the configuration, run:
$ cd /path/to/civicrm $ cv vars:show
This should display a number of properties, including:
- Credentials for an empty test database (
- (The database user requires SUPER privilege in order to set innodb_flush_log_at_trx_commit.)
- Credentials for an administrative CMS user (
- Credentials for a non-administrative CMS user (
If these are missing or blank, then you need to fill them in. Initialize and edit the configuration file:
$ cv vars:fill $ vi ~/.cv.json
Tip: Database snapshots
Many tests interact with the database. In case they mess up the database, you should retain a snapshot of your baseline DB.
If you used
civibuild, it automatically retained a DB snapshot when you last (re)installed the site.
See civibuild restore for more information.
Tip: Multi-user systems
cv uses the configuration file
~/.cv.json. However, if this build is accessed by many user accounts, then
you can use a shared configuration file. Run
export CV_CONFIG=/path/to/shared/file.json and then call
Most automated tests require access to some mix of resources -- such as source-code files, databases, or URLs. To access these resources, test frameworks should build on the cv command.
There are three types of tests. Each handles these resources differently.
Minimal unit test¶
A minimal unit test focuses on testing a discrete technical artifact, such as a function, class, or file. Minimal unit tests are loosely coupled, and they generally shouldn't require an external service (such as a database or web-server). The narrow scope and loose-coupling makes these fast -- so you can execute a large suite of tests in a short time. However, it also makes them less representative. This type of testing is ideal for helpers, utilities, and libraries which have innately low coupling. They're also useful if you need to test a large range of possible inputs/permutations.
Unit tests and safety
In a minimal unit-test, running the test should have no side-effects -- ie it should not change data-files or database tables.
Unit tests and bootstrap
In a framework like
karma, a minimal unit test might call
cv php:boot --level=classloader
cv path -d [civicrm.root] to gain access to Civi's source code.
The most common kind of automated test in
civicrm-core is a headless test. Headless tests use a nearly complete CiviCRM environment;
however, there is no CMS or web-server, and all data is stored on a private, headless database. This is ideally suited to testing data-management
APIs (where the DBMS is an important part of the system) and other portable services (where you wouldn't expect the CMS to influence behavior).
Headless tests and safety
In a headless unit-test, the test takes ownership over a private, test-only, headless copy of CiviCRM. Headless tests commonly take steps to keep the database at a clean baseline, such as (a) rolling back a SQL transaction or (b) truncating a few data tables or (c) resetting the entire database.
Headless tests and bootstrap
In a framework like
codeception, a headless test might call
cv php:boot --level=full. To avoid bootstrapping a full CMS, it
would need an environment variable
An end-to-end test (E2E) focuses on testing a full stack, including some combination of CRM, CMS, web-server, file-system, database, and web-browser. These tests are the most representative of real-world usage, because they tie together so many components. However, this high coupling also makes them brittle (because a design-change or fault in any part of the system can disrupt the expected flow). This type of testing is ideal for screens, pageflows, and integrations which have innately high coupling.
End-to-end tests and safety
In an E2E test, the test runs against an active, local CiviCRM installation. By default, each test makes persistent changes, which can create side-effects for other tests. Be conscientious about (a) checking pre-conditions (in case other tests have side-effects) and (b) cleaning up to prevent side-effects (that might break other tests).
End-to-end tests and bootstrap
In a framework like
codeception, an E2E test might call
cv php:boot --level=full. However, it would not need to explicitly
CIVICRM_UF. Depending on the active CMS, this value is determined automatically.