Monitor with ServiceControl events

This sample shows how to monitor heartbeat and failed message events in ServiceControl, as well as observing the same activity in ServicePulse. The sample uses the Learning Transport and a portable version of the Particular Service Platform tools. Installing ServiceControl is not required.

Running the project

Running the project will result in 3 console windows:

1. NServiceBusEndpoint: The endpoint that represents the system being monitored.
2. EndpointsMonitor: The endpoint that subscribes to ServiceControl heartbeat and failed message events.
3. PlatformLauncher: Runs an in-process version of ServiceControl and ServicePulse. When the ServiceControl instance is ready, a browser window will be launched displaying the ServicePulse Dashboard.

The project handles two kinds of events:

MessageFailed event

A MessageFailed event is emitted when processing a message fails and the message is moved to the error queue.

To observe this in action, press Enter in the NServiceBusEndpoint console window to send a new SimpleMessage event. Processing of the message fails every time.

When a MessageFailed event is received, the EndpointsMonitor prints the following message in its console window:

> Received ServiceControl 'MessageFailed' event for a SimpleMessage with ID 42f25e40-a673-61f3-a505-c8dee6d16f8a

Using the details in the MessageFailed message, handler code can be written to notify operations or development staff by email or other method.

The failed message can also be viewed in the ServicePulse browser window. Navigating to the failed message allows viewing more details about the message failure.

HeartbeatStopped and HeartbeatRestored events

The HeartbeatStopped event is emitted whenever an endpoint fails to send a control message within the expected interval. The HeartbeatRestored event is emitted whenever the endpoint successfully sends a control message again.

To observe this in action, stop the NServiceBusEndpoint application and wait up to 30 seconds. When a HeartbeatStopped event is received, the EndpointsMonitor prints the following message in its console window:

Heartbeat from NServiceBusEndpoint stopped.

Next, restart the NServiceBusEndpoint application and wait up to 30 seconds. When a HeartbeatRestored event is received, the EndpointsMonitor prints the following message in its console window:

Heartbeat from EndpointsMonitoring.NServiceBusEndpoint restored.

Code walk-through

NServiceBusEndpoint

Retries are disabled in the sample for simplicity; messages are immediately moved to the error queue after a processing failure:

var recoverability = endpointConfiguration.Recoverability();

recoverability.Delayed(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });
recoverability.Immediate(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });

The MessageFailed event is published whenever ServiceControl detects a new message in the error queue.

In order to receive HeartbeatStopped and HeartbeatRestored events, the endpoint must use the heartbeats plugin.

EndpointsMonitor

In order to get notifications when the exposed ServiceControl events occur, create an NServiceBus endpoint. Next, reference the ServiceControl.Contracts NuGet package and implement a handler which handles ServiceControl events:

public class CustomEventsHandler :
    IHandleMessages<MessageFailed>,
    IHandleMessages<HeartbeatStopped>,
    IHandleMessages<HeartbeatRestored>,
    IHandleMessages<FailedMessagesArchived>,
    IHandleMessages<FailedMessagesUnArchived>,
    IHandleMessages<MessageFailureResolvedByRetry>,
    IHandleMessages<MessageFailureResolvedManually>
{
    static ILog log = LogManager.GetLogger<CustomEventsHandler>();

    public Task Handle(MessageFailed message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'MessageFailed' event for a {message.MessageType} with ID {message.FailedMessageId}.");
        return Task.CompletedTask;
    }

    public Task Handle(HeartbeatStopped message, IMessageHandlerContext context)
    {
        log.Warn($"Heartbeats from {message.EndpointName} have stopped.");
        return Task.CompletedTask;
    }

    public Task Handle(HeartbeatRestored message, IMessageHandlerContext context)
    {
        log.Info($"Heartbeats from {message.EndpointName} have been restored.");
        return Task.CompletedTask;
    }

    public Task Handle(FailedMessagesArchived message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'FailedMessageArchived' with ID {message.FailedMessagesIds.FirstOrDefault()}.");
        return Task.CompletedTask;
    }
    
    public Task Handle(FailedMessagesUnArchived message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'FailedMessagesUnArchived' MessagesCount: {message.FailedMessagesIds.Length}.");
        return Task.CompletedTask;
    }   
    
    public Task Handle(MessageFailureResolvedByRetry message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'MessageFailureResolvedByRetry' with ID {message.FailedMessageId}.");
        return Task.CompletedTask;
    }

    public Task Handle(MessageFailureResolvedManually message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'MessageFailureResolvedManually'  with ID {message.FailedMessageId}.");
        return Task.CompletedTask;
    }
}

Notes on other transports

This sample uses the Learning Transport in order to be portable with no transport dependencies.

When changing this sample to use Azure Service Bus as a transport, note that the subscribing endpoint must not override the default name shortening strategy. The entity creation configuration settings of Azure Service Bus Transport should not be changed.

The same applies to Azure Storage Queues' name sanitization strategy.