In ServiceControl versions 4 and above, a ServiceControl Audit instance manages the audit queue. Data about audit messages is exposed via an HTTP API on a primary ServiceControl instance. This API is used by ServiceInsight for visualizing message flows.
Each endpoint in the system should be configured to send audit copies of every message that is processed into a central audit queue. A ServiceControl Audit instance reads the messages in the audit queue and makes them available for visualization in ServiceInsight. ServiceControl Audit can optionally forward these messages into an Audit Log queue for further processing if required.
Each ServiceControl Audit instance stores data in an embedded database. Audit data is retained for 30 days. This retention period can be customized.
Connected to a ServiceControl instance
When using ServiceControl Management to create a new ServiceControl instance, a connected ServiceControl Audit instance is automatically created. Using PowerShell, create the ServiceControl instance first, then the ServiceControl Audit instance.
A single ServiceControl instance can have zero or more ServiceControl Audit secondary instances attached to it. ServiceInsight connects directly to the primary ServiceControl instance which will aggregate the data stored in all connected ServiceControl Audit instances.
Connecting ServiceInsight directly to a ServiceControl Audit instance is not supported.
Notifications
Each ServiceControl Audit instance sends notification messages to a primary ServiceControl instance.
Endpoint detection
When a ServiceControl Audit instance detects a new endpoint it sends a notification to the primary ServiceControl instance. The primary instance keeps track of all of the endpoints in the system and can monitor them with heartbeats and custom checks.
Successful retry detection
When a ServiceControl Audit instance detects that an audited message is the result of a retry, it sends a notification to the primary ServiceControl instance.
Health monitoring
ServiceControl includes some basic self-monitoring implemented as custom checks. These checks will be reported in ServicePulse alongside any custom checks being reported from endpoints.
MSMQ transactional dead letter queue
MSMQ servers have a single transactional dead letter queue. Messages that cannot be delivered to queues located on remote servers will eventually be moved to the transactional dead letter queue when the MSMQ service is unable to deliver the message. ServiceControl will monitor the transactional dead letter queue on the server it is installed on as the presence of messages in this queue may indicate a problem with delivering message retries.
Azure Service Bus staging dead letter queue
Azure Service Bus queues each come with an associated dead letter queue. When ServiceControl sends a message for retry it utilizes a staging queue to do so. ServiceControl will monitor the dead letter queue of the ServiceControl staging queue as the presence of messages in this queue indicates a problem with delivering message retries.
Failed imports
When ServiceControl is unable to properly import an audit or error message, the error is logged and the message is stored separately in ServiceControl. ServiceControl will monitor these failed import stores and notify when any are found. Read more about re-importing failed messages here.
Error Message Ingestion Process
When ServiceControl has difficulty connecting to the configured transport, the error message ingestion process is shut down for sixty seconds. These shutdowns are monitored with a custom check. The time to wait before restarting the error ingestion process is configurable with the ServiceControl/TimeToRestartErrorIngestionAfterFailure setting.
Message database storage space
ServiceControl stores messages in an embedded database on the local file system. If the disk runs out of space, message ingestion will fail and the ServiceControl instance will stop. This type of failure can cause instability in the database, even after storage space has been increased. ServiceControl will monitor the remaining storage space on the drive containing the embedded message store. The check will report a failure if the hard drive has less than 20% remaining of its total capacity. This threshold can be changed with the ServiceControl/DataSpaceRemainingThreshold setting.
