Monitor with ServiceControl events

This sample shows how to monitor heartbeat and failed message events in ServiceControl, as well as observing the same activity in ServicePulse. The sample uses the Learning Transport and a portable version of the Particular Service Platform tools. Installing ServiceControl is not required.

This sample requires a Windows environment in order to run the portable version of the Particular Service Platform tools.

Running the project

Running the project will result in 3 console windows:

  1. NServiceBusEndpoint: The endpoint that represents the system being monitored.
  2. EndpointsMonitor: The endpoint that subscribes to ServiceControl heartbeat and failed message events.
  3. PlatformLauncher: Runs an in-process version of ServiceControl and ServicePulse. When the ServiceControl instance is ready, a browser window will be launched displaying the ServicePulse Dashboard.

The project handles two kinds of events:

MessageFailed event

A MessageFailed event is emitted when processing a message fails and the message is moved to the error queue.

To observe this in action, press Enter in the NServiceBusEndpoint console window to send a new SimpleMessage event. Processing of the message fails every time.

The exception will cause the debugger to enter a breakpoint. It may be preferable to detach the debugger in order to better observe what's going on.

When a MessageFailed event is received, the EndpointsMonitor prints the following message in its console window:

> Received ServiceControl 'MessageFailed' event for a SimpleMessage with ID 42f25e40-a673-61f3-a505-c8dee6d16f8a

Using the details in the MessageFailed message, handler code can be written to notify operations or development staff by email or other method.

The failed message can also be viewed in the ServicePulse browser window. Navigating to the failed message allows viewing more details about the message failure.

HeartbeatStopped and HeartbeatRestored events

The HeartbeatStopped event is emitted whenever an endpoint fails to send a control message at an expected interval. The HeartbeatRestored event is emitted whenever the endpoint successfully sends a control message again.

The monitor must receive at least one control message before it can observe that the endpoint stopped responding.

To observe this in action, stop the NServiceBusEndpoint application and wait up to 30 seconds. When a HeartbeatStopped event is received, the EndpointsMonitor prints the following message in its console window:

Heartbeat from NServiceBusEndpoint stopped.

Next, restart the NServiceBusEndpoint application and wait up to 30 seconds. When a HeartbeatRestored event is received, the EndpointsMonitor prints the following message in its console window:

Heartbeat from EndpointsMonitoring.NServiceBusEndpoint restored.

Code walk-through

NServiceBusEndpoint

Retries are disabled in the sample for simplicity; messages are immediately moved to the error queue after a processing failure:

var recoverability = endpointConfiguration.Recoverability();

recoverability.Delayed(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });
recoverability.Immediate(
    customizations: retriesSettings =>
    {
        retriesSettings.NumberOfRetries(0);
    });

The MessageFailed event is published for any standard NServiceBus endpoint that is monitored by ServiceControl.

In order to receive HeartbeatStopped and HeartbeatRestored events, the endpoint must use the heartbeats plugin.

Heartbeat control messages are sent every 30 seconds by default so there will be up to a 30 second delay before ServiceControl realizes that it lost or restored connection with the endpoint.

EndpointsMonitor

In order to get notifications when the exposed ServiceControl events occur, create an NServiceBus endpoint. Next, reference the ServiceControl.Contracts NuGet package and implement a handler which handles ServiceControl events:

public class CustomEventsHandler :
    IHandleMessages<MessageFailed>,
    IHandleMessages<HeartbeatStopped>,
    IHandleMessages<HeartbeatRestored>
{
    static ILog log = LogManager.GetLogger<CustomEventsHandler>();

    public Task Handle(MessageFailed message, IMessageHandlerContext context)
    {
        log.Error($"Received ServiceControl 'MessageFailed' event for a {message.MessageType} with ID {message.FailedMessageId}.");
        return Task.CompletedTask;
    }

    public Task Handle(HeartbeatStopped message, IMessageHandlerContext context)
    {
        log.Warn($"Heartbeats from {message.EndpointName} have stopped.");
        return Task.CompletedTask;
    }

    public Task Handle(HeartbeatRestored message, IMessageHandlerContext context)
    {
        log.Info($"Heartbeats from {message.EndpointName} have been restored.");
        return Task.CompletedTask;
    }
}

Notes on other transports

This sample uses the Learning Transport in order to be portable with no transport dependencies.

If adjusting this sample to use the Azure Service Bus transport, note that the subscribing endpoint must also use the same name shortening strategy as ServiceControl. See the configuration settings, or if using the legacy Azure Service Bus transport, see its sanitization strategy documentation

Samples

Related Articles


Last modified