Custom Recoverability Policy

Component: NServiceBus
NuGet Package NServiceBus (6.x)

Default Recoverability

The default Recoverability Policy is implemented in DefaultRecoverabilityPolicy class. It is publicly exposed in case the default recoverability behavior needs to be reused as part of a custom recoverability policy. The default policy takes into account the settings provided for Immediate Retries, Delayed Retries and the configured error queue.

The default policy works in the following way:

  1. If an unrecoverable exception is raised, then the MoveToError action is returned immediately with the default error queue.

  2. If the ImmediateProcessingFailures value is less or equal to the configured MaxNumberOfRetries for Immediate Retries, then the ImmediateRetry action is returned.

  3. If Immediate Retries are exceeded, then the Delayed Retries configuration is considered. If the DelayedDeliveriesPerformed is less or equal than MaxNumberOfRetries and the message hasn't reached the maximum time allowed for retries (24 hours), then the DelayedRetry action is returned. The delay is calculated according to the following formula:

    delay = DelayedRetry.TimeIncrease * (DelayedDeliveriesPerformed + 1).

  4. If MaxNumberOfRetries for both ImmediateRetries and DelayedRetries is exceeded, then the MoveToError action is returned with the default error queue.

According to the default policy, only exceptions of type MessageDeserializedException are unrecoverable. It's possible to customize the policy and instruct NServiceBus to treat other types as unrecoverable exceptions. Refer to the Unrecoverable exceptions section to learn more.

Fallback

As outlined in the Recoverability introduction Immediate and Delayed Retries can only be performed under certain conditions. If a custom Recoverability Policy returns a recoverability action which cannot be fulfilled by the infrastructure, the decision will be overridden with the MoveToError action with the default error queue.

This behavior guarantees safety in edge cases and cannot be overridden.

Recoverability Configuration

RecoverabilityConfig contains all required information to take into account when a recoverability policy is implemented. It provides the following information provided via configuration:

  • Maximum number of retries for Immediate Retries
  • Maximum number of retries for Delayed Retries
  • Time increase interval for delays in subsequent Delayed Retries executions
  • Configured error queue address
  • Exceptions that need to be treated as unrecoverable (NServiceBus Version 6.2 and above)

The information provided in the configuration is static and will not change between subsequent executions of the policy.

In cases when Immediate and/or Delayed Retry capabilities have been turned off and/or are not available, the MaxNumberOfRetries exposed to recoverability policy will be set to 0 (zero).

Error Context

ErrorContext contains all required information to take into account regarding the currently failing message. It provides the following information:

  • Exception which caused the message to fail
  • Transport transaction on which the message failed
  • Number of failed immediate processing attempts
  • Number of delayed deliveries performed
  • Message which failed

Implement a custom policy

Partial customization

Sometimes only a part of the default Recoverability Policy needs to be customized. In such situation, the DefaultRecoverabilityPolicy needs to be called in the customized Recoverability Policy delegate.

For example, this custom policy will directly move the message to an error queue without retries when certain exceptions like MyBusinessException triggered the policy:

6.x NServiceBus
RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    if (context.Exception is MyBusinessException)
    {
        return RecoverabilityAction.MoveToError(config.Failed.ErrorQueue);
    }

    return DefaultRecoverabilityPolicy.Invoke(config, context);
}
6.2 - N NServiceBus
var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();

In the following example the default Recoverability Policy is tweaked to do three Immediate Retries and three Delayed Retries with a time increase of two seconds:

6.x NServiceBus
var recoverability = endpointConfiguration.Recoverability();
recoverability.CustomPolicy(MyCustomRetryPolicy);
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        delayed.NumberOfRetries(3).TimeIncrease(TimeSpan.FromSeconds(2));
    });
6.2 - N NServiceBus
var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();
recoverability.CustomPolicy(MyCustomRetryPolicy);
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        delayed.NumberOfRetries(3).TimeIncrease(TimeSpan.FromSeconds(2));
    });

In the following example, for exceptions of type MyOtherBusinessException the default Delayed Retries time increase will be always five seconds, in all other cases the Default Recoverability Policy will be applied:

6.x NServiceBus
RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // early decisions and return before custom policy is invoked
    // i.e. MyBusinessException should always go to error
    if (context.Exception is MyBusinessException)
    {
        return RecoverabilityAction.MoveToError(config.Failed.ErrorQueue);
    }

    // invocation of default recoverability policy
    var action = DefaultRecoverabilityPolicy.Invoke(config, context);

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    var delayedRetryAction = action as DelayedRetry;
    if (delayedRetryAction != null &&
        context.Exception is MyOtherBusinessException)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    return action;
}
6.2 - N NServiceBus
RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // invocation of default recoverability policy
    var action = DefaultRecoverabilityPolicy.Invoke(config, context);

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    var delayedRetryAction = action as DelayedRetry;
    if (delayedRetryAction != null &&
        context.Exception is MyOtherBusinessException)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    return action;
}

If more control over Recoverability is needed, the Recoverability delegate can be overridden completely.

Full customization

If the Recoverability Policy is fully customized, then the DefaultRecoverabilityPolicy won't be called. In such cases it is still possible to use the recoverability high level APIs, for example:

6.x NServiceBus
var recoverability = endpointConfiguration.Recoverability();
recoverability.CustomPolicy(MyCustomRetryPolicy);
// configuration can still be tweaked on this level if desired, data will be passed into the policy
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        var retries = delayed.NumberOfRetries(3);
        retries.TimeIncrease(TimeSpan.FromSeconds(2));
    });
6.2 - N NServiceBus
var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();
recoverability.CustomPolicy(MyCustomRetryPolicy);
// configuration can still be tweaked on this level if desired, data will be passed into the policy
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        var retries = delayed.NumberOfRetries(3);
        retries.TimeIncrease(TimeSpan.FromSeconds(2));
    });

The configuration will be passed into the custom policy.

The snippet below shows a fully custom policy that:

  • for unrecoverable exceptions such as MyBusinessException immediately moves failed messages to a custom error queue
  • for MyOtherBusinessException does Delayed Retries with a constant time increase of five seconds
  • for all other cases immediately moves failed messages to the configured standard error queue.
6.x NServiceBus
RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // early decisions and return before custom policy is invoked
    // i.e. MyBusinessException should always go to customized error queue
    if (context.Exception is MyBusinessException)
    {
        return RecoverabilityAction.MoveToError("customErrorQueue");
    }

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    if (context.Exception is MyOtherBusinessException && context.DelayedDeliveriesPerformed <= config.Delayed.MaxNumberOfRetries)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    // in all other cases No Immediate or Delayed Retries, go to default error queue
    return RecoverabilityAction.MoveToError(config.Failed.ErrorQueue);
}
6.2 - N NServiceBus
RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // early decisions and return before custom policy is invoked
    // i.e. all unrecoverable exceptions should always go to customized error queue
    foreach (var exceptionType in config.Failed.UnrecoverableExceptionTypes)
    {
        if (exceptionType.IsInstanceOfType(context.Exception))
        {
            return RecoverabilityAction.MoveToError("customErrorQueue");
        }
    }

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    if (context.Exception is MyOtherBusinessException &&
        context.DelayedDeliveriesPerformed <= config.Delayed.MaxNumberOfRetries)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    // in all other cases No Immediate or Delayed Retries, go to default error queue
    return RecoverabilityAction.MoveToError(config.Failed.ErrorQueue);
}

Samples


Last modified