Critical Errors

Component: NServiceBus
NuGet Package NServiceBus (7.x - 7.1)

NServiceBus has built-in recoverability but in certain scenarios, it is not possible to handle errors in a graceful way. The reason for this is that NServiceBus does not have enough context to make a sensible decision on how to proceed after these errors have occurred. Examples of these critical errors include:

  • An exception occurs when NServiceBus is attempting to execute the recoverability policy, including moving a message to the error queue. The context will contain a specific error "Failed to execute recoverability policy for message with native ID: ``"
  • There are repeated failures in reading information from a required storage.
  • An exception occurs reading from the input queue.

Default behavior

The default behavior is to log the exception and keep retrying indefinitely.

Custom handling

It is possible to providing a delegate that overrides the above action. When a critical error occurs the new action will be called instead of the default.

Examples of reasons to consider implementating a custom critical error action include:

  • The endpoint contains a handler which must not be executed beyond the configured recoverability policy.
  • Restarting the endpoint and resetting the transport connection may resolve underlying issues in receiving or dispatching messages.
  • To notify support personnel when the endpoint has raised a critical error.

Define a custom handler using the following code.

endpointConfiguration.DefineCriticalErrorAction(OnCriticalError);

A possible custom implementation

async Task OnCriticalError(ICriticalErrorContext context)
{
    try
    {
        // To leave the process active, stop the endpoint.
        // When it is stopped, attempts to send messages will cause an ObjectDisposedException.
        await context.Stop().ConfigureAwait(false);
        // Perform custom actions here, e.g.
        // NLog.LogManager.Shutdown();
    }
    finally
    {
        var failMessage = $"Critical error shutting down:'{context.Error}'.";
        Environment.FailFast(failMessage, context.Exception);
    }
}

Implementation Concerns

If the endpoint is stopped without exiting the process, then any Send or Publish operation will result in a KeyNotFoundException being thrown.

When implementing a custom critical error callback:

  • To exit the process use the Environment.FailFast method. In case the environment has threads running that should be completed before shutdown (e.g. non transactional operations), the Environment.Exit method can also be used.
  • The code should be wrapped in a try...finally clause. In the try block perform any custom operations; in the finally block call the method that exits the process.
  • The custom operations should include flushing any in-memory state and cached data, if normally it is persisted at a certain interval or during graceful shutdown. For example, flush appenders when using buffering or asynchronous appenders for NLog or log4net state by calling LogManager.Shutdown();.

Whenever possible rely on the environment hosting the endpoint process to automatically restart it:

  • IIS: The IIS host will automatically spawn a new instance.
  • Windows Service: The OS can restart the service after 1 minute if Windows Service Recovery is enabled.
It is important to consider the effect these defaults will have on other things hosted in the same process. For example if co-hosting NServiceBus with a web-service or website.

Raising a critical error

Any code in the endpoint can invoke the Critical Error action.

// 'criticalError' is an instance of NServiceBus.CriticalError
// This instance can be resolved from dependency injection.
criticalError.Raise(errorMessage, exception);

Heartbeat functionality

The Heartbeat functionality is configured to start pinging ServiceControl immediately after the endpoint starts. It only stops when the process exits. The only way for a critical error to result in a heartbeat failure in ServicePulse/ServiceControl is for the critical error to kill the process.


Last modified