Using ServiceControl Events

ServiceControl publishes events that enable the creation of custom notifications and integrations to monitor the health of the system. ServiceControl processes messages from the error queue as well as data sent by endpoints using the NServiceBus.Heartbeat and NServiceBus.CustomChecks packages. When messages fail, heartbeats stop arriving, or custom checks fail, ServiceControl publishes events that any subscribing endpoint can react to.

See Monitor with ServiceControl events for a sample.

MessageFailed events

Once a message arrives in the error queue, ServiceControl will publish a MessageFailed event. This message contains:

• The endpoint that sent the message
• The endpoint that received the message
• The cause of the failure (i.e. the exception type and message)
• The original message headers
• The original message body (only if it is non-binary, smaller than 85 KB and full-text body indexing is enabled)

Subscribe

It is possible to subscribe to this event type and act on the messages, for example: by sending an email or triggering a text message.

To subscribe to the MessageFailed event:

1. Create an NServiceBus endpoint.
2. Install the ServiceControl.Contracts NuGet package.
3. Configure the endpoint to use SystemJsonSerializer as the message published by ServiceControl uses JSON serialization. Configure the endpoint with the following conventions, as the events published by ServiceControl do not derive from IEvent.

endpointConfiguration.UseSerialization<SystemJsonSerializer>();
var conventions = endpointConfiguration.Conventions();
conventions.DefiningEventsAs(
    type =>
    {
        return typeof(IEvent).IsAssignableFrom(type) ||
               // include ServiceControl events
               type.Namespace != null &&
               type.Namespace.StartsWith("ServiceControl.Contracts");
    });

4. Add a message handler for the MessageFailed event in the endpoint. In the following example, there is a simple HTTP call to show how to integrate with a third-party system to provide notification of the error.

class MessageFailedHandler :
    IHandleMessages<MessageFailed>
{
    public Task Handle(MessageFailed message, IMessageHandlerContext context)
    {
        var failedMessageId = message.FailedMessageId;
        var exceptionMessage = message.FailureDetails.Exception.Message;

        var chatMessage = $@"Message with id: {failedMessageId} failed.
Reason: '{exceptionMessage}'.
Open in ServicePulse: {GetServicePulseUri(failedMessageId)}";

        using (var client = new ChatClient())
        {
            client.PostChatMessage(chatMessage);
        }
        return Task.CompletedTask;
    }

Registering the publisher for message-driven publish/subscribe

Transports that use message-driven publish-subscribe must have the ServiceControl instance registered as the publisher of the MessageFailed event.

The routing config code API can be used:

var routing = transport.Routing();
routing.RegisterPublisher(typeof(ServiceControl.Contracts.MessageFailed).Assembly, "Particular.ServiceControl");

Monitoring events

ServiceControl will also publish events based on collected monitoring data.

See Heartbeat Notification Events and Custom Check Notification Events for a description of these events.

Other events

ServiceControl will also publish events related to archiving and retrying messages:

• FailedMessagesArchived: Event emitted for failed messages that were archived, indicating they won’t be retried
• FailedMessagesUnArchived: Event emitted for failed messages that were un-archived (restored from the archive), making them eligible for retry or further action
• MessageFailureResolvedByRetry: Event emitted by ServiceControl for each failed message that succeeded after retrying
• MessageFailureResolvedManually: Event emitted by ServiceControl for each failed message that was manually marked as resolved, typically via the "Resolve" or "Resolve All" actions in ServicePulse
• MessageEditedAndRetried: Event emitted by ServiceControl every time that a failed message was edited and retried

Decommissioning subscribers to ServiceControl events

ServiceControl uses event publishing to expose information to subscribers. When using a persistence-based transport an internal reference will be kept to each subscriber. If a subscriber for an event cannot be contacted then a log entry will be written with the following error:

Failed dispatching external integration event

An event will also be published and displayed in the ServicePulse dashboard with the following text:

'EVENTTYPE' failed to be published to other integration points. Reason for failure: REASON.

To avoid this situation, it is important to properly decommission an endpoint that subscribes to ServiceControl events. To do this, disable auto-subscription and then unsubscribe from all events.

Disabling integration

In systems that have significant load and don't have any subscribers for ServiceControl events, it can be beneficial to disable publishing the events to prevent unneeded traffic to the broker. This can be disabled by setting DisableExternalIntegrationsPublishing to True on the ServiceControl configuration.