Custom notification and alerting using ServiceControl events
ServiceControl events enable the construction of custom notifications and integration that will alert when something goes wrong in the system.
ServiceControl endpoint plugins collect information from monitored NServiceBus endpoints.
Alerting on FailedMessages events
Once a message ends up in the error queue, ServiceControl will publish a MessageFailed
event. The message contains:
- The processing of the message (i.e. the endpoint that sent and received the message).
- The failure cause (i.e. the exception type and message).
- The message itself (i.e. the headers and, if using non-binary serialization, the body).
Subscribing to ServiceControl events
ServiceControl publishes a MessageFailed
event when a message gets to the error queue. It is possible to subscribe to these events and act on them (e.g. send an email, trigger a text message, etc.).
To subscribe to the MessageFailed
event:
- Create an NServiceBus endpoint.
- Install the ServiceControl.Contracts NuGet package.
- Customize the endpoint configuration to use
JsonSerializer
as the message published by ServiceControl uses JSON serialization. - Customize the endpoint configuration so that the following conventions are used, as the events published by ServiceControl do not derive from
IEvent
.
This code sample illustrates how to do this customization:
endpointConfiguration.UseSerialization<JsonSerializer>();
var conventions = endpointConfiguration.Conventions();
conventions.DefiningEventsAs(
type =>
{
return typeof(IEvent).IsAssignableFrom(type) ||
// include ServiceControl events
type.Namespace != null &&
type.Namespace.StartsWith("ServiceControl.Contracts");
});
- The endpoint will also need a message handler that handles the
MessageFailed
event. In the following example, there is a simple HTTP call to HipChat's API to show how to integrate with a third-party system to provide notification of the error.
class MessageFailedHandler :
IHandleMessages<MessageFailed>
{
public Task Handle(MessageFailed message, IMessageHandlerContext context)
{
var failedMessageId = message.FailedMessageId;
var exceptionMessage = message.FailureDetails.Exception.Message;
var chatMessage = $@"Message with id: {failedMessageId} failed.
Reason: '{exceptionMessage}'.
Open in ServiceInsight: {GetServiceInsightUri(failedMessageId)}";
using (var client = new HipchatClient())
{
client.PostChatMessage(chatMessage);
}
return Task.CompletedTask;
}
error
and audit
queues as other endpoints. Using the same error
queue could cause an infinite feedback loop if processing a MessageFailed
message failed. Using the same audit
queue will cause the processing of the MessageFailed
messages to be included in the ServiceInsight search results. This could confuse users searching for a given failure since both the failure and the failure notification will be shown. See also: Recoverability and Audit Queue Settings.Registering the publisher for message-driven publish/subscribe
Transports that use message-driven publish-subscribe must have the ServiceControl instance registered as the publisher of the MessageFailed
event.
For NServiceBus version 6 and higher, the routing config code API can be used:
var routing = transport.Routing();
routing.RegisterPublisher(typeof(ServiceControl.Contracts.MessageFailed).Assembly, "Particular.ServiceControl");
For NServiceBus version 5 and below, add the message mapping in the UnicastBusConfig
section of the endpoint's app.config :
<UnicastBusConfig>
<MessageEndpointMappings >
<add Assembly="ServiceControl.Contracts"
Endpoint="Particular.ServiceControl" />
</MessageEndpointMappings>
</UnicastBusConfig>
Monitoring events
ServiceControl will also publish events based on collected monitoring data.
See Heartbeat Notification Events and Custom Check Notification Events for a description of these events.
Decommissioning subscribers to ServiceControl events
ServiceControl uses event publishing to expose information to subscribers. When using a persistence-based transport an internal reference will be kept to each subscriber. If a subscriber for an event cannot be contacted then a log entry will be written with the following error:
Failed dispatching external integration event
An event will also be published and displayed in the ServicePulse dashboard with the following text:
'EVENTTYPE' failed to be published to other integration points. Reason for failure: REASON.
To avoid this situation it is important to properly decommission an endpoint that subscribes to ServiceControl events. To do this, disable auto-subscription and then unsubscribe from all events.