# ServiceControl Audit instances


Each endpoint in the system can be [configured to send audit copies of every message that is processed into a central audit queue](/nservicebus/operations/auditing.md). ServiceControl Audit instances consume and store the messages sent to the audit queue and make them available for visualizing message flows in [ServicePulse](/servicepulse/index.md).

ServiceControl Audit can optionally forward these messages into an [audit log queue](configuration.md#transport-servicecontrol-auditforwardauditmessages) for further processing if required.

```mermaid
graph LR
  Endpoints -- Audit<br>Data --> AuditQ[Audit Queue]

  SCA[ServiceControl Audit<br>instance]
  SC[ServiceControl<br>Instance]
  AuditQ --> SCA

  ServicePulse -.-> SC
  SCA --> AuditLog[Audit.Log<br>Queue]

  SC -. HTTP Queries .-> SCA
  SCA -- Notifications --> SC
```

Data about audit messages is exposed via an HTTP API from a ServiceControl Error instance, which aggregates the data stored in [all connected ServiceControl Audit instances](/servicecontrol/servicecontrol-instances/remotes.md#overview-sharding-audit-messages-with-competing-consumers).

> [!IMPORTANT]
> Connecting ServicePulse directly to a ServiceControl Audit instance is not supported.

> [!NOTE]
> The ServiceControl HTTP API is designed for use by ServicePulse only and may change at any time. Use of this HTTP API for other purposes is discouraged.

## Persistence

Each ServiceControl Audit instance stores message data in a RavenDB database. For instances deployed using the ServiceControl Management utility or PowerShell this database is embedded with the ServiceControl Audit instance. For ServiceControl Audit instances deployed using containers the database resides in a [separate container](/servicecontrol/ravendb/containers.md).

By default, audit data is retained for 7 days. [This retention period can be customized](/servicecontrol/audit-instances/configuration.md#data-retention).

> [!WARNING]
> The ServiceControl RavenDB server and databases are used exclusively by ServiceControl and are not intended for external manipulation or modifications.

### RavenDB versions used

In ServiceControl.Audit version 4.26 and above new instances use RavenDB version 5. Instances created by version 4.25 and below use RavenDB version 3.5.

Upgrading ServiceControl.Audit instances to version 4.26 or higher does not change the database version. Instances using RavenDB version 3.5, when upgraded to the newest version, will still use RavenDB version 3.5. For more details see [upgrade guide to new persistence format](/servicecontrol/migrations/new-persistence.md)

## Notifications

Each ServiceControl Audit instance sends notification messages to a ServiceControl Error instance.

### Endpoint detection

When a ServiceControl Audit instance detects a new endpoint, it sends a notification to the ServiceControl Error instance. The Error instance keeps track of all of the endpoints in the system and can monitor them with heartbeats and custom checks.

### Health monitoring

ServiceControl includes some basic self-monitoring implemented as [custom checks](/monitoring/custom-checks/index.md). These checks are reported in ServicePulse along with other custom checks.

#### MSMQ transactional dead letter queue

A machine running MSMQ has a single transactional dead letter queue. Messages that cannot be delivered to queues located on remote machines are eventually moved to the transactional dead letter queue. ServiceControl monitors the transactional dead letter queue on the machine it is installed on. The presence of messages in this queue may indicate problems delivering messages for retries.

#### Azure Service Bus staging dead letter queue

Every Azure Service Bus queue has an associated dead letter queue. When ServiceControl sends a message for retry, it uses a staging queue. ServiceControl monitors the dead letter queue associated with the staging queue. The presence of messages in the dead letter queue indicates problems delivering messages for retries.

#### Failed imports

When ServiceControl is unable to ingest an audit or error message, an error is logged and the message is stored separately. ServiceControl monitors these messages. For more information, see [re-importing failed messages](/servicecontrol/import-failed-messages.md).

#### Error message ingestion process

When ServiceControl has difficulty connecting to the configured transport, the error message ingestion process is shut down for sixty seconds. These shutdowns are monitored. The time to wait before restarting the error ingestion process is controlled by the [ServiceControl/TimeToRestartErrorIngestionAfterFailure](/servicecontrol/servicecontrol-instances/configuration.md#recoverability-servicecontroltimetorestarterroringestionafterfailure) setting.

#### Message database storage space

ServiceControl stores messages in an embedded database. If the drive containing the database runs out of storage space, message ingestion fails and the ServiceControl instance stops. This may cause instability in the database, even after storage space is increased. The remaining storage space on the drive is monitored. The check reports a failure if the drive has less than 20% remaining of its total capacity. This threshold is controlled by the [ServiceControl/DataSpaceRemainingThreshold](/servicecontrol/servicecontrol-instances/configuration.md#troubleshooting-servicecontroldataspaceremainingthreshold) setting.

#### Critical message database storage space

This is similar to the [Message database storage space check](#notifications-health-monitoring-message-database-storage-space). However, in this case, if the drive containing the database has less than 5% remaining of its total capacity, message ingestion on the ServiceControl instance is stopped to prevent data loss, and a failure is reported. This threshold is controlled by the [ServiceControl/MinimumStorageLeftRequiredForIngestion](/servicecontrol/servicecontrol-instances/configuration.md#troubleshooting-servicecontrolminimumstorageleftrequiredforingestion) (for the error instance) and [ServiceControl.Audit/MinimumStorageLeftRequiredForIngestion](/servicecontrol/audit-instances/configuration.md#troubleshooting-servicecontrol-auditminimumstorageleftrequiredforingestion) (for the audit instance) settings.