OpenTelemetry

NServiceBus version 8 and above supports OpenTelemetry through traces, metrics, and logging.

Enable OpenTelemetry instrumentation in NServiceBus:

endpointConfiguration.EnableOpenTelemetry();

With OpenTelemetry instrumentation enabled, tracing, metrics, and logging can be individually configured via the OpenTelemetry API itself.

Traces

NServiceBus endpoints generate OpenTelemetry traces for incoming and outgoing messages. To capture trace information, add the NServiceBus.Core activity source to the OpenTelemetry configuration:

var tracingProviderBuilder = Sdk.CreateTracerProviderBuilder()
    .AddSource("NServiceBus.Core")
    // ... Add other trace sources
    // ... Add exporters
    .Build();

Emitted span structure

Send operations

A span is emitted for each message sent by an NServiceBus endpoint. When the message is received, a receive span is created as a child to the send span.

To force the creation of a new trace when receiving the message, the SendOptions-API can be used as follows:

var options = new SendOptions();
options.StartNewTraceOnReceive();
var message = new MyMessage();
await context.Send(message, options);

This ensures a new trace is created, and links the send and receive spans, which looks as follows:

Publish operations

A span is emitted for each message published by an NServiceBus endpoint. When the message is processed by a subscriber, a process span is created in a new trace, which is linked to the publish span.

To force the continuation of the existing trace when receiving the message, the PublishOptions-API can be used as follows:

var options = new PublishOptions();
options.ContinueExistingTraceOnReceive();
var message = new MyEvent();
await context.Publish(message, options);

This ensures the trace is continued and the receive span to be created as a child of the publish span, which looks as follows:

Delayed messages

In some cases, the user can choose to delay the delivery of a message to some point in the future. This is also the mechanism that's used for delayed retries. When a message is delayed, a new trace will always be created for the receive operation, as it happens at a different moment in time. Therefore, any delayed retry or delayed message, will automatically appear linked to the send or publish context.

See the OpenTelemetry samples for instructions on how to send trace information to different tools.

Meters

NServiceBus endpoints can be configured to expose metrics related to message processing. To capture meter information, add the NServiceBus.Core.Pipeline.Incoming meter source to the OpenTelemetry configuration:

var meterProviderProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter("NServiceBus.Core.Pipeline.Incoming")
    // ... Add other meters
    // ... Add exporters
    .Build();

Emitted meters

• nservicebus.messaging.successes - Total number of messages processed successfully by the endpoint
• nservicebus.messaging.fetches - Total number of messages fetched from the queue by the endpoint
• nservicebus.messaging.failures - Total number of messages processed unsuccessfully by the endpoint
• nservicebus.messaging.handler_time - The time the user handling code takes to handle a message
• nservicebus.messaging.processing_time - The time the endpoint takes to process a message
• nservicebus.messaging.critical_time - The time between when a message is sent and when it is fully processed
• nservicebus.recoverability.immediate - Total number of immediate retries requested
• nservicebus.recoverability.delayed - Total number of delayed retries requested
• nservicebus.recoverability.error - Total number of messages sent to the error queue

See the OpenTelemetry samples for instructions on how to send metric information to different tools.

Logging

NServiceBus supports logging out of the box. To collect OpenTelemetry-compatible logging in NServiceBus endpoints, it's possible to configure the endpoint to connect traces and logging when using Microsoft.Extensions.Logging package. See the Connecting OpenTelemetry traces and logs sample for more details.

Alignment of host identifier

It is recommended to align the instance identifier between NServiceBus and OpenTelemetry so all logs, metrics, traces and audit messages can be correlated by a host (instance) if needed.

NServiceBus adds a host identifier to all audit messages and this instance identifier is also used to show performance metrics for each running instance in ServicePulse. The instance identifier used for ServicePulse value can be overriden.

OpenTelemetry also allows to customize the instance id used for service.instance.id in various ways.

Consider aligning the instance ID used by OpenTelemetry and ServiceControl metrics API.

Example

// Generate instance ID shared by all components

// Generate deterministic uuid v4 via
// https://github.com/Faithlife/FaithlifeUtility/blob/master/src/Faithlife.Utility/GuidUtility.cs
var deterministicValue = "MyEndpoint@" + Dns.GetHostName();
Guid serviceInstanceId = GuidUtility.Create(deterministicValue); // or Guid.NewGuid()

// OpenTelemetry
services.AddOpenTelemetry()
    .ConfigureResource(builder =>
        builder.AddService("MyService", serviceInstanceId: serviceInstanceId.ToString()));

// NServiceBus
endpointConfiguration.UniquelyIdentifyRunningInstance()
    .UsingCustomDisplayName("original-instance")
    .UsingCustomIdentifier(serviceInstanceId);

// ServiceControl Metrics
endpointConfiguration
    .EnableMetrics()
    // Not required when already set via UsingCustomIdentifier
    .SendMetricDataToServiceControl("particular.monitoring",
        TimeSpan.FromMinutes(1), serviceInstanceId.ToString());