Monitor ServiceControl events with Azure Application Insights

This sample shows how to monitor a running NServiceBus system with ServiceControl and ServicePulse as well as how to integrate with existing monitoring solutions. The sample uses the learning transport and a portable version of the Particular Service Platform tools. Installing ServiceControl is not required.

Connect to Application Insights Azure Monitor

To connect the sample code to Application Insights, the instrumentation key must be provided. The key is loaded from the ApplicationInsightKey environment variable.

The instrumentation key can be retrieved from the Azure Portal by locating the Application Insights instance, then navigating to the Properties view.

var envInstrumentationKey = "ApplicationInsightKey";
var instrumentationKey = Environment.GetEnvironmentVariable(envInstrumentationKey);

if (string.IsNullOrEmpty(instrumentationKey))
{
    throw new Exception($"Environment variable '{envInstrumentationKey}' required.");
}

Console.WriteLine("Using application insights application key: {0}", instrumentationKey);

var telemetryConfiguration = new TelemetryConfiguration(instrumentationKey);
var telemetryClient = new TelemetryClient(telemetryConfiguration);

endpointConfiguration.RegisterComponents(cc => cc.RegisterSingleton(telemetryClient));

Running the project

Running the project shows three console windows:

1. NServiceBusEndpoint: The endpoint that represents the system being monitored.
2. AzureMonitorConnector: The endpoint that subscribes to ServiceControl notification events and pushes them to Application Insights as custom telemetry events.
3. PlatformLauncher: Runs an in-process version of ServiceControl and ServicePulse. When the ServiceControl instance is ready, a browser window is launched displaying the ServicePulse dashboard.

The samples triggers two types of events:

Message failures

A MessageFailed event is emitted when processing a message fails and the message is moved to the error queue.

To observe this in action, press Enter in the NServiceBusEndpointconsole window. The application will generate a new SimpleMessage event that fails when processed.

When a MessageFailed event is received, the AzureMonitorConnector prints the following message in its console window:

> Received ServiceControl 'MessageFailed' event for a SimpleMessage with ID 42f25e40-a673-61f3-a505-c8dee6d16f8a

The failed message can also be viewed in the ServicePulse browser window. Navigating to the failed message shows more details about the message failure.

Heartbeat statuses

The HeartbeatStopped event is published whenever an endpoint fails to send a control message within an expected interval. The HeartbeatRestored event is published whenever the endpoint successfully sends a control message again.

To observe this in action, stop the NServiceBusEndpoint process and wait up to 30 seconds. When a HeartbeatStopped event is received, the AzureMonitorConnector prints the following message to the console window:

Heartbeat from NServiceBusEndpoint stopped.

Next, restart the NServiceBusEndpoint application and wait up to 30 seconds. When a HeartbeatRestored event is received, the AzureMonitorConnector prints the following message in its console window:

Heartbeat from EndpointsMonitoring.NServiceBusEndpoint restored.

Code walk-through

NServiceBusEndpoint

Retries are disabled in the sample for simplicity; messages are immediately moved to the error queue after a processing failure:

var recoverability = endpointConfiguration.Recoverability();

recoverability.Delayed(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });
recoverability.Immediate(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });

The MessageFailed event is published whenever ServiceControl detects a new message in the error queue.

In order to receive HeartbeatStopped and HeartbeatRestored events, the endpoint must use the heartbeats plugin.

AzureMonitorConnector

In order to get notifications when the exposed ServiceControl events occur, create an NServiceBus endpoint. Next, reference the ServiceControl.Contracts NuGet package and implement a handler which handles specific ServiceControl events:

public class MessageFailedHandler :
    IHandleMessages<MessageFailed>
{
    readonly TelemetryClient telemetryClient;
    static ILog log = LogManager.GetLogger<CustomEventsHandler>();

    public MessageFailedHandler(TelemetryClient telemetryClient)
    {
        this.telemetryClient = telemetryClient;
    }

    public Task Handle(MessageFailed message, IMessageHandlerContext context)
    {
        
        telemetryClient.TrackEvent("Message Failed", new Dictionary<string, string>
        {
            {"MessageId", message.FailedMessageId},
        });

        log.Error($"Received ServiceControl 'MessageFailed' event for a {message.MessageType} with ID {message.FailedMessageId}.");
        return Task.CompletedTask;
    }
}

The handler creates a custom telemetry event and pushes it to Application Insights.

Notes on other transports

This sample uses the learning transport in order to be portable with no transport dependencies.

When adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl. See the configuration settings.

The same applies to the Azure Storage Queues name sanitization strategy