Distributed tracing with Azure Monitor Application Insights

.NET Core 3.1 introduced support for distributed tracing. Types in the System.Diagnostics namespace may be used to expose and correlate execution data from a distributed system. This sample shows how to extend the NServiceBus pipeline with custom behaviors that publish trace data and how to send that data to Azure Monitor Application Insights for analysis and visualization.

Running the sample

Running the sample requires an Application Insights resource. The Instrumentation Key must be provided by setting the ApplicationInsightsKey environment variable.

After starting the sample, pressing Enter sends an InitialCommand. When that command is handled, a FollowUpEvent is published. Trace data for both actions is pushed to Application Insights and available on the transaction diagnostics pane:

Code overview

Pipeline behaviors

Trace data is exposed by custom pipeline behaviors. Each behavior defines a DiagnosticSource.

static DiagnosticSource diagnosticSource = new DiagnosticListener(MessageTracing.SendMessage);

When a behavior is executed, if there is any listener subscribed to the source, an instance of Activity is created and started with the diagnostic source.

if (diagnosticSource.IsEnabled(MessageTracing.SendMessage))
{
    activity = new Activity(MessageTracing.SendMessage);

    foreach (var kvp in context.Headers)
    {
        activity.AddTag(kvp.Key, kvp.Value);
    }

    diagnosticSource.StartActivity(activity, context.Headers);

    // This must happen after StartActivity to ensure the Id is initialized
    context.Headers.Add(MessageTracing.ParentActivityIdHeaderName, activity.Id);
}

The new Activity inherits any existing tracing context (stored in Activity.Current), which initializes the ParentId property and links the new Activity to the parent Activity. Message headers are stored in the Tags collection to provide information about the specific message.

Finally, the activity ID is stored in the outgoing message header to allow the recipient to link its activities to those of the sender.

Application Insights integration

Trace data produced by the behaviors is consumed by tracing listeners registered in NServiceBusInsightsModule:

if (value.Name == MessageTracing.ReceiveMessage)
{
    var listener = new NServiceBusTracingListener(client);
    var subscription = value.Subscribe(listener);

    subscriptions.Add(subscription);
    listeners.Add(listener);
}

When an activity is started by a behavior, a listener is notified by its OnNext method. The activity is pushed to Application Insights using the TelemetryClient API:

var key = value.Key;
var activity = Activity.Current;

if (key.EndsWith("Start"))
{
    var operation = client.StartOperation<DependencyTelemetry>(activity);
    operation.Telemetry.Type = "NServiceBus";

    inProgressOperations.AddOrUpdate(activity.Id, _ => operation, (_, __) => operation);

}
else if (key.EndsWith("Stop"))
{
    if (inProgressOperations.TryRemove(activity.Id, out var operation))
    {
        operation.Telemetry.Success = (bool) value.Value;

        client.StopOperation(operation);

        operation.Dispose();
    }
}