ServiceControl remote instances

ServiceInsight and ServicePulse retrieve data from a ServiceControl instance using an HTTP API. In some installations, that data may reside in multiple ServiceControl instances. The ServiceControl Remotes features allows a ServiceControl instance to aggregate data from other ServiceControl instances, providing a unified experience in ServiceInsight and ServicePulse.

Overview

One ServiceControl Error instance is designated as the primary instance. All other ServiceControl instances are remote instances. The HTTP API of the primary instance aggregates data from the primary instance and from all the remote instances. ServiceInsight and ServicePulse are configured to connect to the primary instance.

In ServiceControl version 4 and later, a ServiceControl Error instance can be configured with remote instances that are also ServiceControl Error instances or are ServiceControl Audit instances. ServiceControl Audit instances cannot be configured as primary instances.

Default deployment

In ServiceControl version 4 and above, the ServiceControl Management utility creates a primary ServiceControl Error instance and a remote ServiceControl Audit instance.

Sharding audit messages with competing consumers

Two ServiceControl Audit instances ingest messages from the same audit queue. This approach can be used to scale out the ingestion of messages from high-volume audit queues.

Sharding audit messages with split audit queues

Endpoints are partitioned into groups. Each group sends messages to its own a different audit queue. Each audit queue is managed by a different ServiceControl Audit instance. This approach is useful if different audit retention periods are required for specific groups of endpoints.

Multi-region deployments

It is possible to create a multi-region deployment using remotes.

In this deployment, each region has a full ServiceControl installation with a primary Error instance and an Audit instance. Each region can be managed and controlled via a dedicated ServicePulse.

A new cross-region primary instance is added to allow ServiceInsight to show messages from both regions. This cross-region instance includes each region-specific primary Error instance as a remote allowing it to query messages from both. The cross-region instance should disable error queue management by configuring the error queue with the value !disable.

Zero downtime upgrades

The remotes feature can be used to perform zero downtime upgrades.

Configuration

Remote instances are listed in the ServiceControl/RemoteInstances app setting in the primary instance configuration file. The value of this setting is a JSON array of remote instances. Each entry requires an api_url property specifying the API URL of the remote instance. For ServiceControl version 3 and earlier, each entry requires a queue_address property specifying the queue address of the remote instance.

Version 4 and later

<configuration>
  <appSettings>
    <add key="ServiceControl/RemoteInstances" value="[{'api_uri':'http://localhost:33334/api'}]"/>
  </appSettings>/
</configuration>

Version 3 and earlier

<configuration>
  <appSettings>
    <add key="ServiceControl/RemoteInstances" value="[{'api_uri':'http://localhost:33334/api', 'queue_address':'Particular.ServiceControl.Remote'}]"/>
  </appSettings>/
</configuration>

Managing remote instances using PowerShell

The following cmdlets are available in ServiceControl version 4 and above, for the management of remote instances:

Add a remote instance

Add-ServiceControlRemote adds a remote instance to a primary instance.

Add-ServiceControlRemote -Name Particular.ServiceControl -RemoteInstanceAddress "http://localhost:44444/api"

Remove a remote instance

Remove-ServiceControlRemote removes a remote instance from a primary instance.

Remove-ServiceControlRemote -Name Particular.ServiceControl -RemoteInstanceAddress "http://localhost:44444/api"

List remote instances

Get-ServiceControlRemotes gets a list of remote instances from a primary instance.

Get-ServiceControlRemotes -Name Particular.ServiceControl

Changing the address of a remote instance

To change the address of a remote instance to a new host and/or port number:

1. Remove the current address from the list of remote instances:
   • Remove-ServiceControlRemote -Name $primaryServiceControl.Name -RemoteInstanceAddress $currentAddress
2. Restart the primary instance to refresh the list of remote instances
3. Stop the remote instance
4. Change the host and/or port number of the remote instance using the ServiceControl Management utility
5. Start the remote instance at its new address
6. Add the new address to the list of remote instances:
   • Add-ServiceControlRemote -Name $primaryServiceControl.Name -RemoteInstanceAddress $newAddress
7. Restart the primary instance to refresh the list of remote instances

Considerations

• Pagination in ServiceInsight may not work as expected. For example, each page may contain a different number of items, depending on how those items are distributed across the various ServiceControl instances.
• If the primary instance cannot contact a given remote instance, data from that remote instance will not be included in any views in either ServiceInsight or ServicePulse.
• Multi-instance configuration is not possible the ServiceControl Management utility.
• Incorrect configuration may cause cyclical dependencies. For example, instance A may attempt to get data from instance B, and instance B may attempt to get data from instance A.
• It is recommended to run only one primary instance. Multiple primary instances are not recommended.