Custom Recoverability Policy

Default Recoverability

The default Recoverability Policy is implemented in DefaultRecoverabilityPolicy class. It is publicly exposed in case the default recoverability behavior needs to be reused as part of a custom recoverability policy. The default policy takes into account the settings provided for Immediate Retries, Delayed Retries and the configured error queue.

The default policy works in the following way:

1. If an unrecoverable exception is raised, then the MoveToError action is returned immediately with the default error queue.

2. If the ImmediateProcessingFailures value is less or equal to the configured MaxNumberOfRetries for Immediate Retries, then the ImmediateRetry action is returned.

3. If Immediate Retries are exceeded, then the Delayed Retries configuration is considered. If the DelayedDeliveriesPerformed is less than MaxNumberOfRetries and the message hasn't reached the maximum time allowed for retries (24 hours), then the DelayedRetry action is returned. The delay is calculated according to the following formula:

   delay = DelayedRetry.TimeIncrease * (DelayedDeliveriesPerformed + 1).

4. If MaxNumberOfRetries for both ImmediateRetries and DelayedRetries is exceeded, then the MoveToError action is returned with the default error queue.

Fallback

As outlined in the Recoverability introduction, Immediate and Delayed Retries can only be performed under certain conditions. If a custom Recoverability Policy returns a recoverability action which cannot be fulfilled by the infrastructure, the decision will be overridden with the MoveToError action with the default error queue.

This behavior guarantees safety in edge cases and cannot be overridden.

Recoverability Configuration

RecoverabilityConfig contains all the information required when a recoverability policy is implemented. This includes:

• Maximum number of retries for Immediate Retries
• Maximum number of retries for Delayed Retries
• Time increase interval for delays in subsequent Delayed Retries executions
• Configured error queue address
• Exceptions that need to be treated as unrecoverable (NServiceBus Version 6.2 and above)

The information provided in the configuration is static and will not change between subsequent executions of the policy.

Error Context

ErrorContext provides all the information about the currently failing message. It contains the following information:

• Exception which caused the message to fail
• Transport transaction on which the message failed
• Number of failed immediate processing attempts
• Number of delayed deliveries performed
• Message which failed

Implement a custom policy

Partial customization

Sometimes only part of the default Recoverability Policy needs to be customized. In these situations, the DefaultRecoverabilityPolicy needs to be called in the customized Recoverability Policy delegate.

For example, the following custom policy will move the message directly to an error queue without retries when a MyBusinessException triggers the policy:

var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();

In the following example the default Recoverability Policy is tweaked to do three Immediate Retries and three Delayed Retries with a time increase of two seconds:

var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();
recoverability.CustomPolicy(MyCustomRetryPolicy);
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        delayed.NumberOfRetries(3).TimeIncrease(TimeSpan.FromSeconds(2));
    });

In the following example, for exceptions of type MyOtherBusinessException the default Delayed Retries time increase will be always five seconds, in all other cases the Default Recoverability Policy will be applied:

RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // invocation of default recoverability policy
    var action = DefaultRecoverabilityPolicy.Invoke(config, context);

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    if (action is DelayedRetry delayedRetryAction &&
        context.Exception is MyOtherBusinessException)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    return action;
}

If more control over Recoverability is needed, the Recoverability delegate can be overridden completely.

Full customization

If the Recoverability Policy is fully customized, then the DefaultRecoverabilityPolicy won't be called. In this case it is still possible to use the recoverability high level APIs, for example:

var recoverability = endpointConfiguration.Recoverability();
recoverability.AddUnrecoverableException<MyBusinessException>();
recoverability.CustomPolicy(MyCustomRetryPolicy);
// configuration can be changed at this point, data will be passed to the policy
recoverability.Immediate(
    immediate =>
    {
        immediate.NumberOfRetries(3);
    });
recoverability.Delayed(
    delayed =>
    {
        var retries = delayed.NumberOfRetries(3);
        retries.TimeIncrease(TimeSpan.FromSeconds(2));
    });

The snippet below shows a fully custom policy that does the following:

• For unrecoverable exceptions, such as MyBusinessException, failed messages are immediately moved to a custom error queue.
• For exceptions where retrying is not required, such as MyBusinessTimedOutException, failed messages will be discarded as if they had not occurred. The discarded messages will neither be moved to the error queue nor forwarded to the audit queue. However, the reason for their failure will appear in the logs.
• For MyOtherBusinessException, delayed retries are performed with a constant time increase of five seconds.
• For all other cases, failed messages are immediately moved to the configured error queue.

RecoverabilityAction MyCustomRetryPolicy(RecoverabilityConfig config, ErrorContext context)
{
    // early decisions and return before custom policy is invoked
    // i.e. all unrecoverable exceptions should always go to customized error queue
    foreach (var exceptionType in config.Failed.UnrecoverableExceptionTypes)
    {
        if (exceptionType.IsInstanceOfType(context.Exception))
        {
            return RecoverabilityAction.MoveToError("customErrorQueue");
        }
    }

    // If it does not make sense to have this message around anymore
    // it can be discarded with a reason.
    if (context.Exception is MyBusinessTimedOutException)
    {
        return RecoverabilityAction.Discard("Business operation timed out.");
    }

    // override delayed retry decision for custom exception
    // i.e. MyOtherBusinessException should do fixed backoff of 5 seconds
    if (context.Exception is MyOtherBusinessException &&
        context.DelayedDeliveriesPerformed < config.Delayed.MaxNumberOfRetries)
    {
        return RecoverabilityAction.DelayedRetry(TimeSpan.FromSeconds(5));
    }

    // in all other cases No Immediate or Delayed Retries, go to default error queue
    return RecoverabilityAction.MoveToError(config.Failed.ErrorQueue);
}

Note that the RecoverabilityConfig will be passed into the custom policy so the code can be fine-tuned based on the configured values.