Azure Blob Storage DataBus

Component: Azure Blob Storage Databus
NuGet Package NServiceBus.DataBus.AzureBlobStorage (1.x)
Target NServiceBus Version: 6.x

Azure Blob Storage DataBus will remove the Azure storage blobs used for physical attachments after the message is processed if the TimeToBeReceived value is specified. When this value isn't provided, the physical attachments will not be removed.

Usage

endpointConfiguration.UseDataBus<AzureDataBus>();

Cleanup strategies

Discarding old Azure DataBus attachments can be done in one of the following ways:

  1. Using the built-in method (enabled by default)
  2. Using an Azure Durable Function
  3. Using the Blob Lifecycle Management policy
Be sure to disable the built-in cleanup method when using Durable Function of Blob Lifecycle Management policy options.

Using the built-in clean-up method

Specify a value for the TimeToBeReceived property. For more details on how to specify this, see Discarding Old Messages.

The built-in method uses continuous blob scanning which can add to the cost of the storage operations. It is not recommended for multiple endpoints that are scaled out. If this method is not used, be sure to disable the built-in cleanup by setting the CleanupInterval to 0. In versions 3 and above built-in cleanup is disabled by default.

Using an Azure Durable Function

Review the sample to see how to use a durable function to clean up attachments.

Using the Blob Lifecycle Management policy

Attachment blobs can be cleaned up using the Blob Storage Lifecycle feature. This method allows configuring a single policy for all DataBus-related blobs. Those blobs can be either deleted or archived. The policy does not require custom code and is deployed directly to the storage account. This feature can only be used on GPv2 and Blob storage accounts, not on GPv1 accounts.

Configuration

The following extension methods are available for changing the behavior of AzureDataBus defaults:

var dataBus = endpointConfiguration.UseDataBus<AzureDataBus>();
dataBus.ConnectionString(azureStorageConnectionString);
dataBus.Container(containerName);
dataBus.BasePath(basePathWithinContainer);
dataBus.BlockSize(blockSize);
dataBus.DefaultTTL(timeToLiveInSeconds);
dataBus.MaxRetries(maxNumberOfRetryAttempts);
dataBus.NumberOfIOThreads(numberOfIoThreads);
dataBus.BackOffInterval(backOffIntervalBetweenRetriesInSecs);
dataBus.CleanupInterval(cleanupIntervalInMilSecs);

Disabling built-in blob cleanup

Setting the CleanupInterval to 0 will disable blob cleanup.

var dataBus = endpointConfiguration.UseDataBus<AzureDataBus>();
dataBus.CleanupInterval(0);

Samples


Last modified