Automatic Retries

Sometimes processing of a message can fail. This could be due to a transient problem like a deadlock in the database, in which case retrying the message a few times might overcome this problem. Or, if the problem is more protracted, like a third party web service going down or a database being unavailable, where it might be useful to wait a little longer before retrying the message again.

For situations like these, NServiceBus offers two levels of retries:

  • First Level Retry(FLR) is for the transient errors where quick successive retries could solve the problem.
  • Second Level Retry(SLR) is when a small delay is needed between retries.
When a message cannot be deserialized, it will bypass all retry mechanisms both the FLR and SLR and the message will be moved directly to the error queue.

First Level Retries

NServiceBus automatically retries the message when an exception is thrown during message processing up to five successive times by default. This value can be configured through app.config or code.

The configured value describes the minimum number of times a message will be retried. Especially in environments with competing consumers on the same queue there is an increased chance of retrying a failing message more often across the endpoints.

Configuring FLR using app.config

In NServiceBus version 3, this configuration was available via MsmqTransportConfig.

From version 4 onward the configuration for this mechanism is implemented in the TransportConfig section. For more details on MsmqTransportConfig and TransportConfig read this article.

<configuration>
  <configSections>
    <section name="MsmqTransportConfig"
             type="NServiceBus.Config.MsmqTransportConfig , NServiceBus.Core" />
  </configSections>
  <MsmqTransportConfig ErrorQueue="error"
                       NumberOfWorkerThreads="1"
                       MaxRetries="5"/>
</configuration>
<configuration>
  <configSections>
    <section name="TransportConfig"
             type="NServiceBus.Config.TransportConfig, NServiceBus.Core"/>
  </configSections>
  <MessageForwardingInCaseOfFaultConfig ErrorQueue="error"/>
  <TransportConfig MaxRetries="2" />
</configuration>
<configuration>
  <configSections>
    <section name="TransportConfig"
             type="NServiceBus.Config.TransportConfig, NServiceBus.Core"/>
  </configSections>
  <TransportConfig MaxRetries="2" />
</configuration>
<configuration>
  <configSections>
    <section name="TransportConfig"
             type="NServiceBus.Config.TransportConfig, NServiceBus.Core"/>
  </configSections>
  <TransportConfig MaxRetries="2" />
</configuration>

Configuring FLR through IProvideConfiguration

class ProvideConfiguration : IProvideConfiguration<TransportConfig>
{
    public TransportConfig GetConfiguration()
    {
        return new TransportConfig
        {
            MaxRetries = 2
        };
    }
}
class ProvideConfiguration : IProvideConfiguration<TransportConfig>
{
    public TransportConfig GetConfiguration()
    {
        return new TransportConfig
        {
            MaxRetries = 2
        };
    }
}

Configuring FLR through ConfigurationSource

public class ConfigurationSource : IConfigurationSource
{
    public T GetConfiguration<T>() where T : class, new()
    {
        //To Provide FLR Config
        if (typeof(T) == typeof(MsmqTransportConfig))
        {
            var flrConfig = new MsmqTransportConfig
            {
                MaxRetries = 2
            };

            return flrConfig as T;
        }

        // To in app.config for other sections not defined in this method, otherwise return null.
        return ConfigurationManager.GetSection(typeof(T).Name) as T;
    }
}
public class ConfigurationSource : IConfigurationSource
{
    public T GetConfiguration<T>() where T : class, new()
    {
        //To Provide FLR Config
        if (typeof(T) == typeof(TransportConfig))
        {
            var flrConfig = new TransportConfig
            {
                MaxRetries = 2
            };

            return flrConfig as T;
        }

        // To in app.config for other sections not defined in this method, otherwise return null.
        return ConfigurationManager.GetSection(typeof(T).Name) as T;
    }
}
Configure configure = Configure.With();
configure.CustomConfigurationSource(new ConfigurationSource());
busConfiguration.CustomConfigurationSource(new ConfigurationSource());

Second Level Retries

SLR introduces another level of retrying mechanism for messages that fail processing. When using SLR, the message that causes the exception is, as before, instantly retried, but instead of being sent to the error queue, it is sent to a retries queue.

SLR then picks up the message and defers it, by default first for 10 seconds, then 20, and lastly for 30 seconds, then returns it to the original worker queue.

For example, if there is a call to an web service in your handler, but the service goes down for five seconds just at that time. Without SLR, the message is retried instantly and sent to the error queue. With SLR, the message is instantly retried, deferred for 10 seconds, and then retried again. This way, when the Web Service is available the message is processed just fine.

SLR can be configured either via code or through app.config.

Configuring SLR using app.config

To configure SLR, enable its configuration section:

<configSections>
  <section name="SecondLevelRetriesConfig"
           type="NServiceBus.Config.SecondLevelRetriesConfig, NServiceBus.Core"/>
  </configSections>
<SecondLevelRetriesConfig Enabled="true" 
                          TimeIncrease="00:00:10" 
                          NumberOfRetries="3" />
  • Enabled: Turns the feature on and off. Default: true.
  • TimeIncrease: A time span after which the time between retries increases. Default: 10 seconds (00:00:10).
  • NumberOfRetries: Number of times SLR kicks in. Default: 3.

Configuration SLR through IProvideConfiguration

class ProvideConfiguration : IProvideConfiguration<SecondLevelRetriesConfig>
{
    public SecondLevelRetriesConfig GetConfiguration()
    {
        return new SecondLevelRetriesConfig
        {
            Enabled = true,
            NumberOfRetries = 2,
            TimeIncrease = TimeSpan.FromSeconds(10)
        };
    }
}
class ProvideConfiguration : IProvideConfiguration<SecondLevelRetriesConfig>
{
    public SecondLevelRetriesConfig GetConfiguration()
    {
        return new SecondLevelRetriesConfig
        {
            Enabled = true,
            NumberOfRetries = 2,
            TimeIncrease = TimeSpan.FromSeconds(10)
        };
    }
}

Configuring SLR through ConfigurationSource

public class ConfigurationSource : IConfigurationSource
{
    public T GetConfiguration<T>() where T : class, new()
    {
        // To provide SLR Config
        if (typeof(T) == typeof(SecondLevelRetriesConfig))
        {
            var slrConfig = new SecondLevelRetriesConfig
            {
                Enabled = true,
                NumberOfRetries = 2, 
                TimeIncrease = TimeSpan.FromSeconds(10)
            };

            return slrConfig as T;
        }

        // To in app.config for other sections not defined in this method, otherwise return null.
        return ConfigurationManager.GetSection(typeof(T).Name) as T;
    }
}
Configure configure = Configure.With();
configure.CustomConfigurationSource(new ConfigurationSource());
busConfiguration.CustomConfigurationSource(new ConfigurationSource());

Disabling SLR through code

To completely disable SLR through code:

Configure configure = Configure.With();
configure.DisableSecondLevelRetries();
Configure.Features
    .Disable<NServiceBus.Features.SecondLevelRetries>();
BusConfiguration busConfiguration = new BusConfiguration();
busConfiguration.DisableFeature<SecondLevelRetries>();

Custom Retry Policy

You can change the time between retries or the number of retries in code.

Here is a sample method for handling this behavior.

TimeSpan MyCustomRetryPolicy(TransportMessage message)
{
    // retry max 3 times
    if (GetNumberOfRetries(message) >= 3)
    {
        // sending back a TimeSpan.MinValue tells the 
        // SecondLevelRetry not to retry this message
        return TimeSpan.MinValue;
    }

    return TimeSpan.FromSeconds(5);
}

static int GetNumberOfRetries(TransportMessage message)
{
    string value;
    if (message.Headers.TryGetValue(Headers.Retries, out value))
    {
        int i;
        if (int.TryParse(value, out i))
        {
            return i;
        }
    }
    return 0;
}

To plug this into NServiceBus use the following APIs.

SecondLevelRetries.RetryPolicy = MyCustomRetryPolicy;
Configure.Features.SecondLevelRetries(s => s.CustomRetryPolicy(MyCustomRetryPolicy));
SecondLevelRetriesSettings retriesSettings = busConfiguration.SecondLevelRetries();
retriesSettings.CustomRetryPolicy(MyCustomRetryPolicy);

Samples

  • Fault Tolerance
    See how NServiceBus messaging can get past all sorts of failure scenarios.


Last modified 2015-09-03 06:34:15Z