Upgrade ServiceControl from version 1 to version 2

Upgrading ServiceControl from version 1 to version 2 is a major upgrade and requires careful planning. During the upgrade process, the instance of ServiceControl that is being upgraded will no longer be available and will not ingest any messages. The upgrade process is not reversible and cannot be backed out once started.

Prerequisites

Before upgrading to ServiceControl version 2 the instance being upgraded must be upgraded to at least version 1.41.3. If the instance being upgraded is lower than 1.41.3, the upgrade process could fail.

New settings added to ServiceControl

A new setting has been added to ServiceControl version 2. This setting is the maintenance port and is the port that the internal ServiceControl database is exposed over when ServiceControl has been switched to maintenance mode. The value for this setting must be an unused port. For new instances of ServiceControl, this value defaults to 33334.

Upgrading ServiceControl

There are two ways to upgrade to version 2 of ServiceControl. The first is to perform an in-place upgrade, and the second is to deploy a separate instance of ServiceControl version 2 while leaving the current version 1 instance running in production. Both options are discussed in more detail below.

In-place upgrade to version 2

The upgrade is triggered by the ServiceControl Management application. ServiceControl Management will display the instances of the ServiceControl service that are installed. If the version of the binaries used by an instance are older than those shipped with ServiceControl Management, an upgrade link will be shown next to the version label.

To upgrade the service, click the upgrade link next to the service name.

Clicking the upgrade link will:

• Prompt for additional required information, i.e. values for new mandatory settings introduced in version 2
• Stop the service
• Remove the old binaries for ServiceControl and the configured transport
• Run the new binaries to create the required queues
• Start the service

Side-by-side upgrade process

A side-by-side upgrade process minimizes downtime in ServiceControl and allows for a controlled migration between versions. Instead of upgrading the existing version 1 instance of ServiceControl, the version 1 instance is left running but is configured so that it no longer ingest any messages. A new instance of ServiceControl version 2 is installed and configured to ingest messages from the audit and error queues that the version 1 instance was previously configured to ingest from. The new instance is also configured as a master instance of the previous instance. This results in two instances of ServiceControl, where the new instance will ingest any future messages and the version 1 instance will contain all messages previously ingested but will no longer process new messages. All ServiceInsight instances must then be configured to connect to the new instance of ServiceControl. Since the new instance is the master instance for the version 1 instance, ServiceInsight will show audited messages from both the version 1 instance as well as the new instance of ServiceControl.

The process for a side-by-side upgrade is as follows:

1. Configure the existing version 1 instance of ServiceControl to ingest audit and error messages from a queue named !disable

• This will ensure that the version 1 instance of ServiceControl no longer ingests new messages

2. Add a new version 2 instance of ServiceControl, with a name different to the version 1 instance name, and configure it to ingest messages from the error and audit queues that the version 1 instance was previously configured to ingest messages from.
3. Configure the version 1 instance of ServiceControl as a secondary instance to the new instance by adding the version 1 instance to the RemoteInstances setting of the new instance's app.config

<configuration>
  <appSettings>
    <add key="ServiceControl/RemoteInstances" value="[{'api_uri':'http://localhost:33333/api', 'queue_address':'Particular.ServiceControl'}]'"/>
    <!-- The URI and queue address here are the URI and queue of the version 1 instance of ServiceControl -->
  </appSettings>/
</configuration>

4. Start both instances of ServiceControl
5. Configure ServiceInsight to connect to the new instance of ServiceControl instead of the version 1 instance

At this stage, new messages will be ingested by the version 2 instance of ServiceControl, while ServiceInsight will show data from both instances. The next step is to migrate error messages from the version 1 instance to the new instance. To do this, retry all of the failed messages back into production. If the messages fail again, they will be sent back to the error queue which means they will then be ingested by the new instance of ServiceControl again.

6. Using ServicePulse, retry all of the failed messages currently in the version 1 instance of ServiceControl
7. Configure ServicePulse to connect to the new instance of ServiceControl

Once these steps have been completed, the version 1 instance of ServiceControl will contain only audit messages and the new instance of ServiceControl will contain the error messages and will ingest any new messages. The version 1 instance will slowly clean up its audit messages over time based on the audit message retention period which will result in an empty database after all audit messages have been evicted. Once the audit retention period has elapsed, it is safe to remove the version 1 instance of ServiceControl as there will no longer be any audit messages in that database.

Caveats

If it is not possible to retry the failed messages back into production, the second instance of ServicePulse must be installed so that the first instance of ServicePulse communicates with the previous instance of ServiceControl and the second instance of ServicePulse communicates with the new instance of ServiceControl.