# OpenTelemetry


NServiceBus version 8 and above supports [OpenTelemetry](https://opentelemetry.io/docs/instrumentation/net/) through traces, metrics, and logging.

In NServiceBus version 10 and later, OpenTelemetry instrumentation is enabled by default.

## Traces

NServiceBus endpoints generate OpenTelemetry traces for incoming and outgoing messages.

### Span relationships

#### Send operations

A span is emitted for each message sent by an NServiceBus endpoint. When the message is received, a receive span is created as a child to the send span.

```mermaid
flowchart LR;
  subgraph SENDER
  direction TB
   NSBM1[NServiceBus Send span]
  end
  subgraph RECEIVER
  direction TB
  PRM1[NServiceBus Process span]

  end
  NSBM1--child--> PRM1
```

To force the creation of a new trace when receiving the message, the `SendOptions`-API can be used as follows:

<!-- snippet: opentelemetry-sendoptions-start-new-trace -->

```cs
var options = new SendOptions();
options.StartNewTraceOnReceive();
var message = new MyMessage();
await context.Send(message, options);
```

<!-- endsnippet -->

This ensures a new trace is created, and links the send and receive spans, which looks as follows:

```mermaid
flowchart LR;
  subgraph SENDER
  direction TB
   NSBM1[NServiceBus Send span]
  end
  subgraph RECEIVER
  direction TB
  PRM1[NServiceBus Receive span]

  end
  NSBM1-. link .-PRM1;
```

#### Publish operations

A span is emitted for each message published by an NServiceBus endpoint. When the message is processed by a subscriber, a process span is created in a new trace, which is linked to the publish span.


```mermaid
flowchart LR;
  subgraph PRODUCER
  direction TB
   NSBM1[NServiceBus Publish span]
  end
  subgraph CONSUMER
  direction TB
  PRM1[NServiceBus Process span]

  end
  NSBM1-. link .-PRM1;
```

To force the continuation of the existing trace when receiving the message, the `PublishOptions`-API can be used as follows:

<!-- snippet: opentelemetry-publishoptions-continue-trace -->

```cs
var options = new PublishOptions();
options.ContinueExistingTraceOnReceive();
var message = new MyEvent();
await context.Publish(message, options);
```

<!-- endsnippet -->

This ensures the trace is continued and the receive span to be created as a child of the publish span, which looks as follows:

```mermaid
flowchart LR;
  subgraph PRODUCER
  direction TB
   NSBM1[NServiceBus Publish span]
  end
  subgraph CONSUMER
  direction TB
  PRM1[NServiceBus Process span]

  end
  NSBM1--child--> PRM1
```

### Delayed messages

In some cases, the user can choose to delay the delivery of a message to some point in the future. This is also the mechanism that's used for [delayed retries](/nservicebus/recoverability/index.md#delayed-retries).
When a message is delayed, a new trace will always be created for the receive operation, as it happens at a different moment in time. Therefore, any delayed retry or delayed message, will automatically appear linked to the send or publish context.

See the [OpenTelemetry samples](/samples/open-telemetry/index.md) for instructions on how to send trace information to different tools.


## Meters

NServiceBus endpoints can be configured to expose metrics related to message processing. To capture meter information, add the appropriate meter source (e.g. `NServiceBus.Core.Pipeline.Incoming`) to the OpenTelemetry configuration:

> [!NOTE]
> The metric definitions published by NServiceBus are not yet finalized and could change in a minor release.

<!-- snippet: opentelemetry-enablemeters -->

```cs
var meterProviderProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter("NServiceBus.Core.Pipeline.Incoming")
    // ... Add other meters
    // ... Add exporters
    .Build();
```

<!-- endsnippet -->

### Emitted meters

Meter source `NServiceBus.Core.Pipeline.Incoming`:

- [`nservicebus.messaging.successes`](/monitoring/metrics/definitions.md#metrics-captured-number-of-messages-successfully-processed) - Total number of messages processed successfully by the endpoint
- [`nservicebus.messaging.fetches`](/monitoring/metrics/definitions.md#metrics-captured-number-of-messages-pulled-from-queue) - Total number of messages fetched from the queue by the endpoint
- [`nservicebus.messaging.failures`](/monitoring/metrics/definitions.md#metrics-captured-number-of-message-processing-failures) - Total number of messages processed unsuccessfully by the endpoint
- [`nservicebus.messaging.handler_time`](/monitoring/metrics/definitions.md#metrics-captured-handler-time) - The time the user handling code takes to handle a message
- [`nservicebus.messaging.processing_time`](/monitoring/metrics/definitions.md#metrics-captured-processing-time) - The time the endpoint takes to process a message
- [`nservicebus.messaging.critical_time`](/monitoring/metrics/definitions.md#metrics-captured-critical-time) - The time between when a message is sent and when it is fully processed
- [`nservicebus.recoverability.immediate`](/monitoring/metrics/definitions.md#metrics-captured-immediate-retries) - Total number of immediate retries requested
- [`nservicebus.recoverability.delayed`](/monitoring/metrics/definitions.md#metrics-captured-delayed-retries) - Total number of delayed retries requested
- [`nservicebus.recoverability.error`](/monitoring/metrics/definitions.md#metrics-captured-moved-to-error-queue) - Total number of messages sent to the error queue

Starting NServiceBus V10.1 envelope unwrapping metrics are emitted as part of the `NServiceBus.Core.Pipeline.Incoming` source:

- [`nservicebus.envelope.unwrapped`](/monitoring/metrics/definitions.md#metrics-captured-envelope-handling-metrics) - Total number of times when an envelope handler failed to unwrap an incoming message. Emitted for every unwrapping attempt

Meter source `NServiceBus.TransactionalSession`:

- [`nservicebus.transactional_session.commit.duration`](/monitoring/metrics/definitions.md#metrics-captured-transactional-session-metrics) - The time the endpoint takes to commit the session in the Transactional Session
- [`nservicebus.transactional_session.dispatch.duration`](/monitoring/metrics/definitions.md#metrics-captured-transactional-session-metrics) - The time the endpoint takes to dispatch the control message in the Transactional Session
- [`nservicebus.transactional_session.control_message.attempts`](/monitoring/metrics/definitions.md#metrics-captured-transactional-session-metrics) - Total number of attempts to process the control message in the Transactional Session
- [`nservicebus.transactional_session.control_message.transit_time`](/monitoring/metrics/definitions.md#metrics-captured-transactional-session-metrics) - The time between dispatching the control message and starting to process it in the Transactional Session

Meter source `NServiceBus.Envelope.CloudEvents`:

- [`nservicebus.envelope.cloud_events.received.unwrapping_attempt`](/monitoring/metrics/definitions.md#metrics-captured-envelope-handling-metrics-cloudevents-specific-metrics) - Total number of unwrapping attempts
- [`nservicebus.envelope.cloud_events.received.invalid_message`](/monitoring/metrics/definitions.md#metrics-captured-envelope-handling-metrics-cloudevents-specific-metrics) - Total number of received messages not conforming to the specification
- [`nservicebus.envelope.cloud_events.received.unexpected_version`](/monitoring/metrics/definitions.md#metrics-captured-envelope-handling-metrics-cloudevents-specific-metrics) - Total number of received messages with unexpected version field value

See the [OpenTelemetry samples](/samples/open-telemetry/index.md) for instructions on how to send metric information to different tools.


## Logging

NServiceBus supports logging out of the box. To collect OpenTelemetry-compatible logging in NServiceBus endpoints, it's possible to configure the endpoint to connect traces and logging when using `Microsoft.Extensions.Logging` package. See the [_Connecting OpenTelemetry traces and logs_ sample](/samples/open-telemetry/logging/index.md) for more details.


## Alignment of host identifier

It is recommended to align the instance identifier between NServiceBus and OpenTelemetry so all logs, metrics, traces and audit messages can be correlated by a host (instance) if needed.

> [!NOTE]
> The OpenTelemetry specification recommends this to be a random uuid. However, it may also be a [deterministic uuid v5](https://opentelemetry.io/docs/specs/semconv/attributes-registry/service/#service-attributes) (i.e. hash of machine name and  endpointname).

NServiceBus adds a [host identifier to all audit messages](/nservicebus/hosting/override-hostid.md) and this instance identifier is also used to show [performance metrics for each running instance in ServicePulse](/monitoring/metrics/in-servicepulse.md). The [instance identifier used for ServicePulse value can be overriden](/monitoring/metrics/install-plugin.md#configuration-instance-id).

OpenTelemetry also allows to customize the instance id used for `service.instance.id` in various ways. 

Consider aligning the instance ID used by OpenTelemetry and ServiceControl metrics API.

#### Example

<!-- snippet: opentelemetry-align-host-id -->

```cs
// Generate instance ID shared by all components

// Generate deterministic uuid v4 via
// https://github.com/Faithlife/FaithlifeUtility/blob/master/src/Faithlife.Utility/GuidUtility.cs
var deterministicValue = "MyEndpoint@" + Dns.GetHostName();
Guid serviceInstanceId = GuidUtility.Create(deterministicValue); // or Guid.NewGuid()

// OpenTelemetry
services.AddOpenTelemetry()
    .ConfigureResource(builder =>
        builder.AddService("MyService", serviceInstanceId: serviceInstanceId.ToString()));

// NServiceBus
endpointConfiguration.UniquelyIdentifyRunningInstance()
    .UsingCustomDisplayName("original-instance")
    .UsingCustomIdentifier(serviceInstanceId);

// ServiceControl Metrics
endpointConfiguration
    .EnableMetrics()
    // Not required when already set via UsingCustomIdentifier
    .SendMetricDataToServiceControl("particular.monitoring",
        TimeSpan.FromMinutes(1), serviceInstanceId.ToString());
```

<!-- endsnippet -->