This document discusses upgrade conventions used in CiviCRM core code. For extensions and modules which integrate with CiviCRM, upgrade conventions are different. See Create a Module Extension for more discussion of the module-focused upgrader.
Within CiviCRM Core code the upgrader classes handle making any necessary database changes either through PHP or SQL code, printing out any messages before or after the upgrade happens.
||Upgrade helper functions and infrastructure code. (Note: The name is misleading; today, the class has no responsibility for processing forms.)|
||Primary entry point for running upgrades through command line tools e.g. wp-cli, drush|
||Primary entry point for running upgrades through the User Interface|
||One file and one class for each major release (e.g. 4.7, 4.6), For each minor release (e.g. 4.7.12, 4.6.31) there is one function|
||One Smarty SQL file for each minor release|
Order of Upgrade Steps¶
- Incremental upgrades are executed in order of versions
- Pre-upgrade message is generated before doing any upgrades by examining the version in the civicrm-version file
- For a given major-version the
php verifyPreDBState()runs before any changes are made
- For a given minor-version there may be a php function and some SQL code in the
.tplfile. If there is a php function for the version it will determine when the SQL is run
- For a given minor-version the "Post Upgrade" message is generated immediately after calling the php
Incremental PHP Files¶
Incremental PHP-SQL steps are preferred over the Smarty-SQL where possible. This is because it gives us a chance to query the database before running upgrades and makes upgrades safer.
Each class in the
CRM_Upgrade_Incremental_php corresponds to a major release of CiviCRM (e.g. 4.1, 4.2, 4.3). The key functions that can be defined are:
||Called at the beginning of the upgrade process before any changes have been made. This function should generate any notices that site-administrators should see before running the upgrade. (This is called several times – once for each incremental upgrade.)|
||Called immediately before executing a major upgrade (e.g. 4.2.4 => 4.3.0). (This is not called for each incremental upgrade.)|
||Perform any changes in a specific queue order for version X.Y.Z. During this function you MUST call
||Called after each incremental upgrade. Any messages generated by this function will be displayed to site-administrators after the upgrade has completed. (This is called several times – once for each incremental upgrade.)|
Upgrading message templates¶
Message templates that need upgrading should be declared under
CRM_Upgrade_Incremental_MessageTemplates::getTemplateUpdates(). The upgrade script will pick them up from there (5.4+)
Tips: Prefer simple SQL semantics over API/BAO/DAO¶
When writing upgrade steps in PHP, it's preferable to use lower-level SQL semantics rather than higher-level services (eg metadata, public APIs, BAOs, DAOs). Higher-level services are more complex and often have indirect dependencies; thus, higher-level services are more likely to fail in future upgrade environments. Lower-level SQL tends to provide a more reliable upgrade path.
The upgrader may use basic SQL APIs (eg
CRM_Utils_SQL*). Additionally, there are several SQL-oriented helper functions specifically for common upgrade tasks, e.g.
Compared to raw SQL, these SQL helper functions reduce unnecessary upgrade failures for early-adopters who previously applied a backport or pre-release patch.
Tip: Write upgrades in small chunks¶
When updating sometimes it can be found that upgrades steps speed can vary depending on the size of the database. Sometimes some upgrades steps may take longer than the websever timeout (e.g. Apache, PHP, nginx). To prevent timeouts it's recommended to write upgrade steps in smaller steps. This also helps in debugging as the failure can be pin pointed to a smaller amount of code.
For a working example, see
addTask()– each task will be added to the queue of pending upgrade tasks and executed in a separate HTTP request.
addTask()accepts the name of a function as an argument. Optionally, it also accepts a list of parameters to pass to the function.
- The task function is static; its first argument is an instance of
CRM_Queue_TaskContext; and its return value is a boolean (
Incremental SQL Files¶
Writing upgrade steps in incremental PHP is preferred to incremental SQL, since the resulting upgrade code is typically clearer, and you are better able to verify the expected DB state beforehand. For example, it is not possible in all versions of MySQL to only create DB tables or index if they don't already exist, and a failed SQL statement in the upgrade process can leave the DB in a state which requires manual correction.
CRM/Upgrade/Incremental/sql stores SQL scripts for upgrading to each point release. Some important notes:
- The scripts are not executed directly by MySQL
- Scripts are preprocessed by Smarty which allows for the use of Smarty functions like
- Scripts are evaluated by PHP and therefore do Not support Delimiter Notion or inline comments (Comments need to be placed on their own line with two dashes at the front)
SQL Files are required
Creating a SQL file is required for each incremental release – even if you don't put any content in them. Why? Because the upgrade system constructs a list of incremental releases by scanning the "sql" directory. If an incremental release doesn't appear in the SQL directory, then the upgrader may skip other important steps (such as executing incremental PHP updates).
Tip: Smarty Preprocessing SQL Files¶
All Smarty tags are evaluated before the sql is run. Commonly used Smarty variables and functions:
||The escape param is needed for escaping special characters that may be in the translated string (even if it is not in the source string)|
||Current domain (the one the upgrade is running from). This should not be used in most cases, since usually every domain needs upgrading, not just one. See best practices section.|
||Translate the following text as per the relevant locale|
||Is this a multilingual install? If so you may need to iterate through each one i.e.:|
When testing upgrades, it's important to run the upgrades against several different databases (representing different versions of CiviCRM and different upgrade paths). The tool "civicrm-upgrade-test" can automate this process. For instructions:
CiviBuild Upgrade test¶
When one makes a schema change, one must also prepare and test an upgrade script. The basic cycle is:
- Modify the upgrade script (
- Load a DB snapshot from an older version (e.g. CiviCRM 4.3.0)
- Execute the upgrade script
- Repeat until the upgrade works as expected
You can do these steps manually. Of course, it's a bit tedious to generate and track the DB snapshots while reloading them and rerunning the upgrade logic. If you're particularly impatient/mindless, you can use the command:
$ civibuild upgrade-test BUILDNAME SQLFILE
$ cd build/drupal-demo/sites/all/modules/civicrm $ vi CRM/Upgrade/Incremental/php/FourFive.php $ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2 # Uhoh, that didn't work! Try again... $ vi CRM/Upgrade/Incremental/php/FourFive.php $ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2 # Hooray! It worked.
4.3.0-setupsh.sql.bz2 is a standard DB snapshot bundled with buildkit -- it contains a database from CiviCRM 4.3.0 with randomly-generated data. The
upgrade-test command will load
4.3.0-setupsh.sql.bz2, execute a headless upgrade, and write any errors to the log. (See console output for details.)
Of course, it's fairly common to encounter different upgrade issues depending on the original DB -- an upgrade script might work if the original DB is v4.3 but fail for v4.2. It's a good idea to test your upgrade logic against multiple versions:
$ civibuild upgrade-test drupal-demo '4.2.*' '4.3.*' '4.4.*'
All of the tests above use standard DB snapshots with randomly-generated data. If you want to test something more specific, then create your own DB snapshot and use it, eg:
# Make your own DB with weird data; save a snapshot $ echo "update civicrm_contact set weird=data" | mysql exampledbname $ mysqldump exampledbname | gzip > /tmp/my-snapshot.sql.gz # Write some upgrade logic & try it $ vi CRM/Upgrade/Incremental/php/FourFive.php $ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz # Uhoh, that didn't work! Try again... $ vi CRM/Upgrade/Incremental/php/FourFive.php $ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz # Hooray! It worked.
$ civibuild restore drupal-demo
- If using your own processes see https://github.com/civicrm/civicrm-upgrade-test