Report metrics to Azure Application Insights

Component: Metrics
NuGet Package NServiceBus.Metrics (2-pre)
Target NServiceBus Version: 7.x
This page targets a pre-release version and is subject to change prior to the final release.

Azure Application Insights provides monitoring and alerting capabilities that can be leveraged to monitor the health of NServiceBus endpoints. This sample demonstrates the ability to report metric data to Azure Application Insights and present it graphically:

example graph events

This sample reports the following metrics to Application Insights:

  • Fetched messages per second
  • Failed messages per second
  • Successful messages per second

The sample also shows data like Average message processing time in milliseconds and Maximum critical time in milliseconds. For a detailed explanation of these metrics refer to the metrics captured section in the metrics documentation.

Prerequisites

To run this sample, an Application Insights account and instrumentation key are required.

Although the sample uses Azure Application Insights, the solution itself does not have to run on an Azure message transport. This example uses the Learning Transport but could be modified to run on any transport.

Code overview

The sample simulates messages load with a random 10% failure rate using the LoadSimulator class:

var simulator = new LoadSimulator(endpointInstance, TimeSpan.Zero, TimeSpan.FromSeconds(10));
await simulator.Start()
    .ConfigureAwait(false);

Connect to Azure Application Insights

To connect the sample code to Application Insights, the instrumentation key needs to be provided.

By default the sample code loads this key from an environment variable called ApplicationInsightKey. Either set this environment variable or paste the instrumentation key in the following section.

TelemetryConfiguration.Active.InstrumentationKey = instrumentationKey;

The instrumentation key can be obtained from the Azure Portal by locating the Application Insights instance, and then navigating to the Properties view.

Use NServiceBus.Metrics to capture metric values

To capture the metric values generated by NServiceBus, a class called ApplicationInsightsFeature is added to the sample. It is responsible for enabling the NServiceBus metrics feature.

metrics = settings.EnableMetrics();

ProbeCollector ensures that the telemetry probe is subscribed to the event stream. During this stage ProbeCollector is configured and wired. Information such as endpoint name and input queue are passed to allow advanced data analysis in Application Insights.

var logicalAddress = settings.LogicalAddress();
var discriminator = logicalAddress.EndpointInstance.Discriminator;
var instance = Guid.NewGuid().ToString("N");

var endpoint = settings.EndpointName();
var queue = settings.LocalAddress();

collector = new ProbeCollector(
    endpoint,
    discriminator,
    instance,
    queue
);

metrics.RegisterObservers(collector.RegisterProbes);

Events and metrics are buffered and the Application Insights telemetry client does this every minute. If the endpoint instance would gracefully stop it needs to ensure that any pending events are flushed to Application Insight.

class CleanupAtStop : FeatureStartupTask
{
    public CleanupAtStop(ApplicationInsightsFeature instance)
    {
        this.instance = instance;
    }

    protected override Task OnStart(IMessageSession session)
    {
        return Task.CompletedTask;
    }

    protected override Task OnStop(IMessageSession session)
    {
        instance.collector.Flush();
        return Task.CompletedTask;
    }

    ApplicationInsightsFeature instance;
}

Sending data to Application Insights

The ProbeCollector is used to pass NServiceBus captured metric data to the Application Insights instance configured earlier. Information passed is added as telemetry context properties, which will be attached to every metric (NServiceBus signal probe) and event (NServiceBus duration probe) captured via the telemetry client. The telemetry client is used to communicate with Application Insights.

telemetryClient = new TelemetryClient();
var properties = telemetryClient.Context.Properties;
properties.Add("Endpoint", endpointName);
properties.Add("EndpointInstance", instanceIdentifier);
properties.Add("MachineName", Environment.MachineName);
properties.Add("HostName", Dns.GetHostName());
properties.Add("EndpointDiscriminator", discriminator);
properties.Add("EndpointQueue", queue);

When the ProbeCollector is registered with the NServiceBus metrics API, it registers observer callbacks to receive probe names, signals and durations. Signals are converted into Application Insights custom events. Durations are converted into Application Insights metrics.

foreach (var duration in context.Durations)
{
    if (!probeNameToAiNameMap.TryGetValue(duration.Name, out var name))
    {
        continue;
    }
    duration.Register(
        observer: (ref DurationEvent @event) =>
        {
            var milliseconds = @event.Duration.TotalMilliseconds;
            var telemetry = new MetricTelemetry(name, milliseconds);
            telemetryClient.TrackMetric(telemetry);
        });
}

foreach (var signal in context.Signals)
{
    if (!probeNameToAiNameMap.TryGetValue(signal.Name, out var name))
    {
        continue;
    }
    signal.Register(
        observer: (ref SignalEvent @event) =>
        {
            telemetryClient.TrackEvent(new EventTelemetry(name));
        });
}

Creating charts

To create the charts, metrics provided by NServiceBus, signals, and durations need to be converted into Application Insights metrics and events.

The easiest way to setup graphs for metrics in Application Insights is to go to the Metrics Explorer, add a new graph selecting the custom metric of choice and then use the push pin symbol to add it to a dashboard.

The following graphs show the following metrics:

  • Maximum critical time & critical time grouped by instance id.
  • Average processing time, not grouped.
  • Minimum hours countdown until Service Level Agreement violation.

example graph metrics

Similar for events, the following graphs shows all events grouped by event name. This graph can be further filtered, for example, to show failures only.

example graph events

Refer to the Microsoft documentation for more information on creating dashboards from the metrics explorer.

Using the metrics explorer is an easy way to get started, but to create truly advanced graphs Application Insights offers an advanced analytics environment.


Last modified