Azure Blob Storage Data Bus

Usage

endpointConfiguration.UseDataBus<AzureDataBus>();

Cleanup strategies

Discarding old Azure Data Bus attachments can be done in one of the following ways:

1. Using the built-in method (disabled by default)
2. Using an Azure Durable Function
3. Using the Blob Lifecycle Management policy

Using the built-in clean-up method

Specify a value for the TimeToBeReceived property. For more details on how to specify this, see Discarding Old Messages.

Using an Azure Durable Function

Review the Azure Blob Storage Data Bus cleanup with Azure Functions sample to see how to use a durable function to clean up attachments.

Using the Blob Lifecycle Management policy

Attachment blobs can be cleaned up using the Blob Storage Lifecycle feature. This method allows configuring a single policy for all data bus-related blobs. Those blobs can be either deleted or archived. The policy does not require custom code and is deployed directly to the storage account. This feature can only be used on GPv2 and Blob storage accounts, not on GPv1 accounts.

How lifecycle rules relate to Azure Blob Storage Databus settings

When creating a rule the blob prefix match filter setting should be set to the value of databus/ by default. If the Container() or BasePath() configuration options have been specified when configuring the data bus the blob prefix match filter setting must be modified to take into account the configured container and/or base path values.

Manage the Blob Lifecycle policy via Azure portal

A lifecycle management policy can be set directly on the azure storage account via the portal. Additional information on the configuration, can be found in azure blob lifecycle management policy

Manage the Blob Lifecycle policy via the Azure Command-Line Interface (CLI)

The lifecycle management policy can be set in a JSON document via the Azure CLI.

{
  "rules": [
    {
      "enabled": true,
      "name": "delete-databus-files",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "version": {
            "delete": {
              "daysAfterCreationGreaterThan": 90
            }
          },
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90,
              "daysAfterLastTierChangeGreaterThan": 7
            },
            "delete": {
              "daysAfterModificationGreaterThan": 2555
            }
          }
        },
        "filters": {
          "blobTypes": [
            "blockBlob"
          ],
          "prefixMatch": [
            "databus/"
          ]
        }
      }
    }
  ]
}

The data policy rules associated with the specified storage account can be created as follows.

az storage account management-policy create --account-name myaccount --policy @policy.json --resource-group myresourcegroup

Configuration

Behavior

The following extension methods are available for changing the behavior of AzureDataBus defaults:

var dataBus = endpointConfiguration.UseDataBus<AzureDataBus>();
dataBus.ConnectionString(azureStorageConnectionString);
dataBus.Container(containerName);
dataBus.BasePath(basePathWithinContainer);
dataBus.BlockSize(blockSize);
dataBus.DefaultTTL(timeToLiveInSeconds);
dataBus.MaxRetries(maxNumberOfRetryAttempts);
dataBus.NumberOfIOThreads(numberOfIoThreads);
dataBus.BackOffInterval(backOffIntervalBetweenRetriesInSecs);
dataBus.CleanupInterval(cleanupIntervalInMilSecs);
dataBus.AuthenticateWithManagedIdentity(storageAccountName, renewFiveMinutesBeforeTokenExpires);

• ConnectionString(): The connection string to the storage account for storing databus properties; defaults to UseDevelopmentStorage=true.
• Container(): Container name; defaults to databus.
• BasePath(): The blobs' base path in the container; defaults to an empty string.
• DefaultTTL: Time in seconds to keep a blob in storage before it is removed; defaults to Int64.MaxValue seconds.
• MaxRetries: Number of upload/download retries; defaults to 5 retries.
• NumberOfIOThreads: Number of blocks that will be simultaneously uploaded; defaults to 1 thread.
• BackOffInterval: The back-off time between retries; defaults to 30 seconds.
• BlockSize: The size of a single block for upload when the number of I/O threads is more than 1; defaults to 4MB.
• CleanupInterval: The default time interval to perform periodic clean up of blobs for expired messages with specific TTL; defaults to 5 minutes.
• BlockSize: The size of a single block for upload when the number of I/O threads is more than 1; defaults to 4MB.
• CleanupInterval: The default time interval to perform periodic clean-up of blobs for expired messages with specific TTL; disabled by default.
• AuthenticateWithManagedIdentity(storageAccountName, renewalTimeBeforeTokenExpires, endpointSuffix): Authenticate with Azure Managed Identity instead of connection string.
  • storageAccountName: The storage account name used for the data bus.
  • renewalTimeBeforeTokenExpires: How long before the current token expires a token renewal request should be be issued.
  • endpointSuffix: Endpoint suffix used for the storage account. The default is set to public Azure cloud (core.windows.net).

Azure Blob Storage Data Bus will remove the Azure storage blobs used for physical attachments after the message is processed if the TimeToBeReceived value is specified. When this value isn't provided, the physical attachments will not be removed.