Upgrade Reference¶

Scope

This document discusses upgrade conventions used in CiviCRM core code. For extensions and modules which integrate with CiviCRM, upgrade conventions are different. See Create a Module Extension for more discussion of the module-focused upgrader.

Introduction¶

Within CiviCRM Core code the upgrader classes handle making any necessary database changes either through PHP or SQL code, printing out any messages before or after the upgrade happens.

Directory Structure¶

Path	Description
`CRM/Upgrade/Form.php`	Upgrade helper functions and infrastructure code. (Note: The name is misleading; today, the class has no responsibility for processing forms.)
`CRM/Upgrade/Headless.php`	Primary entry point for running upgrades through command line tools e.g. wp-cli, drush
`CRM/Upgrade/Page/Upgrade.php`	Primary entry point for running upgrades through the User Interface
`CRM/Upgrade/Incremental/*.php`	One file and one class for each major release (e.g. 4.7, 4.6), For each minor release (e.g. 4.7.12, 4.6.31) there is one function
`CRM/Upgrade/Incremental/*.tpl`	One Smarty SQL file for each minor release

Order of Upgrade Steps¶

Incremental upgrades are executed in order of versions
Pre-upgrade message is generated before doing any upgrades by examining the version in the civicrm-version file
For a given minor-version there may be a php function and some SQL code in the .tpl file. If there is a php function for the version it will determine when the SQL is run
For a given minor-version the "Post Upgrade" message is generated immediately after calling the php

Incremental PHP Files¶

Incremental PHP-SQL steps are preferred over the Smarty-SQL where possible. This is because it gives us a chance to query the database before running upgrades and makes upgrades safer.

Each class in the CRM_Upgrade_Incremental_php corresponds to a major release of CiviCRM (e.g. 4.1, 4.2, 4.3). The key functions that can be defined are:

Function	Description
`php public function setPreUpgradeMessage()`	Called at the beginning of the upgrade process before any changes have been made. This function should generate any notices that site-administrators should see before running the upgrade. (This is called several times – once for each incremental upgrade.)
`php public function upgrade_X_Y_Z()`	Perform any changes in a specific queue order for version X.Y.Z. During this function you MUST call `php $upgrade->processSQL($rev)`. Without calling this function the X.Y.Z.sql.tpl file will not be executed
`php public function setPostUpgradeMessage()`	Called after each incremental upgrade. Any messages generated by this function will be displayed to site-administrators after the upgrade has completed. (This is called several times – once for each incremental upgrade.)

Upgrading message templates¶

Message templates that need upgrading should be declared under CRM_Upgrade_Incremental_MessageTemplates::getTemplateUpdates(). The upgrade script will pick them up from there (5.4+)

Tips: Prefer simple SQL semantics over API/BAO/DAO¶

When writing upgrade steps in PHP, it's preferable to use lower-level SQL semantics rather than higher-level services (eg metadata, public APIs, BAOs, DAOs). Higher-level services are more complex and often have indirect dependencies; thus, higher-level services are more likely to fail in future upgrade environments. Lower-level SQL tends to provide a more reliable upgrade path.

The upgrader may use basic SQL APIs (eg CRM_Core_DAO::executeQuery() and CRM_Utils_SQL*). Additionally, there are several SQL-oriented helper functions specifically for common upgrade tasks, e.g.

CRM_Upgrade_Form::addColumn()
CRM_Upgrade_Form::dropColumn()
CRM_Core_BAO_SchemaHandler::dropIndexIfExists()

Compared to raw SQL, these SQL helper functions reduce unnecessary upgrade failures for early-adopters who previously applied a backport or pre-release patch.

Tip: Write upgrades in small chunks¶

When updating sometimes it can be found that upgrades steps speed can vary depending on the size of the database. Sometimes some upgrades steps may take longer than the websever timeout (e.g. Apache, PHP, nginx). To prevent timeouts it's recommended to write upgrade steps in smaller steps. This also helps in debugging as the failure can be pin pointed to a smaller amount of code.

For a working example, see CRM_Upgrade_Incremental_php_FourTwo::upgrade_4_2_beta3(). Notice:

The upgrade_X_Y_Z() function calls addTask() – each task will be added to the queue of pending upgrade tasks and executed in a separate HTTP request.
addTask() accepts the name of a function as an argument. Optionally, it also accepts a list of parameters to pass to the function.
The task function is static; its first argument is an instance of CRM_Queue_TaskContext; and its return value is a boolean (TRUE for success).

Incremental SQL Files¶

Writing upgrade steps in incremental PHP is preferred to incremental SQL, since the resulting upgrade code is typically clearer, and you are better able to verify the expected DB state beforehand. For example, it is not possible in all versions of MySQL to only create DB tables or index if they don't already exist, and a failed SQL statement in the upgrade process can leave the DB in a state which requires manual correction.

CRM/Upgrade/Incremental/sql stores SQL scripts for upgrading to each point release. Some important notes:

The scripts are not executed directly by MySQL
Scripts are preprocessed by Smarty which allows for the use of Smarty functions like {ts} and {localize}
Scripts are evaluated by PHP and therefore do Not support Delimiter Notion or inline comments (Comments need to be placed on their own line with two dashes at the front)

For more details on how to write upgrade steps for incremental SQL scripts, see the Incremental SQL Readme.

SQL Files are required

Creating a SQL file is required for each incremental release – even if you don't put any content in them. Why? Because the upgrade system constructs a list of incremental releases by scanning the "sql" directory. If an incremental release doesn't appear in the SQL directory, then the upgrader may skip other important steps (such as executing incremental PHP updates).

Tip: Smarty Preprocessing SQL Files¶

All Smarty tags are evaluated before the sql is run. Commonly used Smarty variables and functions:

Tag	Description
`{ts escape='sql'}`	The escape param is needed for escaping special characters that may be in the translated string (even if it is not in the source string)
`{$domainID}`	Current domain (the one the upgrade is running from). This should not be used in most cases, since usually every domain needs upgrading, not just one. See best practices section.
`{localize}`	Translate the following text as per the relevant locale
`{$multilingual}`	Is this a multilingual install? If so you may need to iterate through each one i.e.:
`{$locales}`	`{if $multilingual}{foreach from=$locales item=locale} SQL goes here per locale {/foreach} {else} Single locale SQL here {/if}`

Testing¶

When testing upgrades, it's important to run the upgrades against several different databases (representing different versions of CiviCRM and different upgrade paths). The tool "civicrm-upgrade-test" can automate this process. For instructions:

CiviBuild Upgrade test¶

When one makes a schema change, one must also prepare and test an upgrade script. The basic cycle is:

Modify the upgrade script (*.mysql or *.php -- eg CRM/Upgrade/Incremental/php/FourFive.php)
Load a DB snapshot from an older version (e.g. CiviCRM 4.3.0)
Execute the upgrade script
Repeat until the upgrade works as expected

You can do these steps manually. Of course, it's a bit tedious to generate and track the DB snapshots while reloading them and rerunning the upgrade logic. If you're particularly impatient/mindless, you can use the command:

$ civibuild upgrade-test BUILDNAME SQLFILE

For example:

$ cd build/drupal-demo/sites/all/modules/civicrm
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2
# Uhoh, that didn't work! Try again...
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo 4.3.0-setupsh.sql.bz2
# Hooray! It worked.

The file 4.3.0-setupsh.sql.bz2 is a standard DB snapshot bundled with buildkit -- it contains a database from CiviCRM 4.3.0 with randomly-generated data. The upgrade-test command will load 4.3.0-setupsh.sql.bz2, execute a headless upgrade, and write any errors to the log. (See console output for details.) You can find the DB snapshots in buildkit\vendor\civicrm\upgrade-test\databases.

Of course, it's fairly common to encounter different upgrade issues depending on the original DB -- an upgrade script might work if the original DB is v4.3 but fail for v4.2. It's a good idea to test your upgrade logic against multiple versions:

$ civibuild upgrade-test drupal-demo '4.2.*' '4.3.*' '4.4.*'

All of the tests above use standard DB snapshots with randomly-generated data. If you want to test something more specific, then create your own DB snapshot and use it, eg:

# Make your own DB with weird data; save a snapshot
$ echo "update civicrm_contact set weird=data" | mysql exampledbname
$ mysqldump exampledbname | gzip > /tmp/my-snapshot.sql.gz
# Write some upgrade logic & try it
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz
# Uhoh, that didn't work! Try again...
$ vi CRM/Upgrade/Incremental/php/FourFive.php
$ civibuild upgrade-test drupal-demo /tmp/my-snapshot.sql.gz
# Hooray! It worked.

If at any point you need to backout and load a "known-working" database, then use the DB created by the original build:

$ civibuild restore drupal-demo

If using your own processes see https://github.com/civicrm/civicrm-upgrade-test