Zero downtime upgrades

The ServiceControl remotes feature can be used to upgrade a ServiceControl installation without taking it offline.

Audit instances

The process follows these steps:

1. Add a new audit instance as a remote
2. Disable audit queue management on the old audit instance
3. Decommission the old audit instance, when it is empty

Initial state

Before doing anything, the deployment looks like this:

Add a new audit instance

Create a new ServiceControl Audit instance, and configure it as a remote instance of the ServiceControl Error instance.

On the audit instance machine:

$auditInstance = New-ServiceControlAuditInstance `
  -Name Particular.ServiceControl.NewAudit `
  -InstallPath C:\ServiceControl.NewAudit\Bin `
  -DBPath C:\ServiceControl.NewAudit\DB `
  -LogPath C:\ServiceControl.NewAudit\Logs `
  -Port 44446 `
  -DatabaseMaintenancePort 44447 `
  -Transport MSMQ `
  -AuditQueue audit `
  -AuditRetentionPeriod 10:00:00:00 `
  -ForwardAuditMessages:$false `
  -ServiceControlQueueAddress "Particular.ServiceControl"

On the ServiceControl Error instance machine:

Add-ServiceControlRemote `
  -Name "Particular.ServiceControl" `
  -RemoteInstanceAddress "http://localhost:44446/api"

After this step the installation looks like this:

Although both ServiceControl Audit instances ingest messages from the audit queue, each message only ends up in a single instance. The ServiceControl Error instance queries both transparently.

Disable audit queue ingestion on the old instance

Update the audit queue configuration on the original Audit instance and add the setting key ServiceControl/IngestAuditMessages with value false, save, and restart the instance.

On the original audit instance machine:

$originalAuditInstanceName = "Particular.ServiceControl.Audit"
$auditInstance = (Get-ServiceControlAuditInstances | where Name -eq $originalAuditInstanceName)[0]

# Stop instance
Stop-Service $originalAuditInstanceName

# Update configuration
$configPath = Join-Path $auditInstance.InstallPath "ServiceControl.Audit.exe.config"
[xml]$configDoc = Get-Content $configPath
$element = $configDoc.SelectSingleNode("//configuration/appSettings/add[@key='ServiceControl/IngestAuditMessages']")
$element.value = "false"
$configDoc.Save($configPath)

# Start instance
Start-Service $originalAuditInstanceName

After this step the installation looks like this:

The ServiceControl Error instance continues to query both instances but the original Audit instance no longer reads new messages.

Decommission the old audit instance, when it is empty

As the original audit instance is no longer ingesting messages, it will be empty after the audit retention period has elapsed and can be removed. The following steps describe how to determine when an audit instance is empty:

1. Put the audit instance in maintenance mode.
2. Launch RavenDB Management Studio with a browser.
3. If the instance is using RavenDB 3.5 for persistence, go to the <system> database. If the instance is using RavenDB 5, go to the audit database.
4. Check the documents count in the ProcessedMessages collection.

When the ProcessedMessages collection is empty, the audit instance can be decomissioned.

On the ServiceControl Error instance machine:

Remove-ServiceControlRemote `
  -Name "Particular.ServiceControl" `
  -RemoteInstanceAddress "http://localhost:44444/api"

On the original audit instance machine:

Remove-ServiceControlAuditInstance `
  -Name "Particular.ServiceControl.OriginalAudit" `
  -RemoveDB `
  -RemoveLogs

After this step the installation looks like this: