Upgrade Tips

1. Ensure the machine that hosts ServiceControl has sufficient CPU and RAM resources during upgrades. Consider temporarily adding more vCPU and RAM until the upgrade completes and afterwards trim down the resources.
2. Updating ServiceControl will stop ingesting error/audit queue messages but does not affect the sending of messages between endpoints. While an update is in progress, it's not possible to trigger retries manually via ServicePulse. Consider multiple (remote) instances if uptime of ServicePulse is important.
3. Compacting the database will not make the update run faster. Compacting should only be considered after upgrading.
4. In-place updates are recommended over side-by-side updates as they have less complexity. Consider side-by-side if ServicePulse downtime must be minimized.
5. Backup the database before upgrading.
6. Account for downtime of ServicePulse as failed messages cannot be retried when ServiceControl is not running.
7. Account for queue storage size. While ServiceControl is down, messages are not ingested but they will still build up in the queue. If the system has a high message throughput the queues act as a buffer. Ensure queues will not run out of storage (quota) space as when queues are full no new messages can be added which will cause application outage.
8. Analyze the storage size for each ServiceControl instance and update starting from the smallest to the largest. This helps understand how smaller migrations run and can be used to extrapolate how long updating the larger instances will take which can help decided whether a side-by-side upgrade is necessary.

Upgrades that rebuild indexes

Some ServiceControl versions change the definition of one or more RavenDB indexes. The upgrade guide for the version notes when this happens. At the first start-up after the upgrade, ServiceControl deploys the new definition and RavenDB builds it as a side-by-side replacement index, listed as ReplacementOf/<index> in RavenDB Studio. The existing index keeps serving queries while the replacement is built. Once the replacement has caught up with all documents and is no longer stale, RavenDB swaps it in and deletes the old index.

The rebuild has to process every document in the database. On large databases, it can take a long time, and it competes with message ingestion for CPU, RAM, storage I/O, and space. While ingestion continues, the rebuild takes longer, ingestion and indexing rates are degraded, and the replacement index might never be reported as non-stale, which prevents the automatic swap.

Upgrading audit instances one at a time

When audit messages are spread across multiple audit instances, upgrade them one at a time so that the rebuild does not have to compete with ingestion and the environment keeps running:

1. Pick an audit instance and disable audit message ingestion by setting ServiceControl.Audit/IngestAuditMessages to false. The instance keeps serving its data to ServicePulse but no longer adds documents to its database and indexes. With competing consumers, the other audit instances keep ingesting from the audit queue.
2. Upgrade the instance. At start-up, RavenDB starts building the replacement indexes.
3. Wait until the rebuild completes. In RavenDB Studio, the List of Indexes must no longer show ReplacementOf/<index> entries, and the stale indexes custom check must pass. See accessing the ServiceControl database for how to open RavenDB Studio.
4. Enable ingestion again by setting ServiceControl.Audit/IngestAuditMessages to true, or by removing the setting, and restart the instance.
5. Repeat with the next audit instance.

Start with the smallest instance to estimate how long the rebuild takes on the larger ones.

Reducing the database size before rebuilding a locked index

When the index is locked, the upgrade does not apply the new definition and the index is not rebuilt until it is unlocked. As an alternative to the procedure above, the rebuild can be postponed until a large part of the database has been purged, which makes the rebuild shorter. For each audit instance, one at a time:

1. Disable audit message ingestion by setting ServiceControl.Audit/IngestAuditMessages to false, and upgrade the instance.
2. Wait until a significant amount of audit messages has been purged because their retention period expired. Expired messages are deleted while ingestion is disabled.
3. Unlock the index in RavenDB Studio by changing 🔒 Locked (ignore) to 🔓 Unlocked.
4. Restart the instance. At start-up, RavenDB starts building the replacement index.
5. Wait until the rebuild completes, as described in step 3 of the procedure above.
6. Enable ingestion again by setting ServiceControl.Audit/IngestAuditMessages to true, or by removing the setting, and restart the instance.
7. Repeat with the next audit instance.

The instance is not ingesting for a longer time, up to the full retention period. Messages that have been purged are no longer available in ServicePulse. On versions 6.20.0 and 6.21.0, the rebuild resets an index that was migrated to Lucene to Corax, see Index reset to Corax after unlocking.

For error instances, the equivalent setting is ServiceControl/IngestErrorMessages. While error ingestion is disabled, failed messages build up in the error queue.

Downgrading ServiceControl

This section contains information about downgrading ServiceControl instances and the ServiceControl Management Utility.

Instances

ServiceControl instances cannot be downgraded with the ServiceControl Management Utility. This is because the storage schema could have changed during a previous upgrade.

1. Open the ServiceControl Management Utility
2. Scroll to the instance that requires downgrading
3. Open the "Installation path"
4. Copy the ServiceControl.exe.config and store the identical instance name
5. Delete the instance but keep the database files
6. Close ServiceControl Management Utility
7. Downgrade the installer
8. Add a new instance using the same identical name and configuration as the deleted instance

Installer

There may be a need to setup a ServiceControl instance for an older version of ServiceControl, in which case follow these steps:

1. Uninstall the current (newer) version from Windows in "Add/remove programs"
2. Install the previous version

The ServiceControl Management Utility (SCMU) can now be used again to add instances based on the older version.