DataBus

Component: NServiceBus
NuGet Package NServiceBus (3.x)
Standard support for version 3.x of NServiceBus has expired. For more information see our Support Policy.

Although messaging systems work best with small message sizes, some scenarios require sending large binary data along with a message. For this purpose, NServiceBus has a "DataBus" feature to overcome the message size limitations imposed by an underlying transport.

How it works

Instead of serializing the payload along with the rest of the message, the DataBus approach involves storing the payload in a separate location that both the sending and receiving parties can access, and then putting the reference to that location in the message.

If the location is not available upon sending, the send operation will fail. When a message is received and the payload location is not available, the receive operation will fail as well, resulting in the standard NServiceBus retry behavior, possibly resulting in the message being moved to the error queue if the error could not be resolved.

The Handling large stream properties via pipeline sample demonstrates a purely stream-based approach (rather than loading the full payload into memory) implemented by leveraging NServiceBus pipeline.

Enabling the DataBus

See the individual DataBus implementations for details on enabling and configuring the databus.

Specifying DataBus properties

There are two ways to specify the message properties to be sent using the DataBus:

  1. Using the DataBusProperty<T> type
  2. Message conventions
The properties must be of type byte[].

Using DataBusProperty

Set the type of the property to be sent over the DataBus as DataBusProperty<byte[]>:

public class MessageWithLargePayload
{
    public string SomeProperty { get; set; }
    public DataBusProperty<byte[]> LargeBlob { get; set; }
}

Using message conventions

NServiceBus also supports defining DataBus properties via convention. This allows data properties to be sent using the DataBus without using DataBusProperty<T>, thus removing the need for having a dependency on NServiceBus from the message types.

In the configuration of the endpoint include:

configuration.DefiningDataBusPropertiesAs(property =>
{
    return property.Name.EndsWith("DataBus");
});

Set the type of the property as byte[]:

public class MessageWithLargePayload
{
    public string SomeProperty { get; set; }
    public byte[] LargeBlobDataBus { get; set; }
}

Serialization

By default NServiceBus uses a BinaryFormatter to serialize and deserialize DataBus properties.

Using a custom serializer

To override the DataBus property serializer, create a class that implements IDataBusSerializer and add it to the dependency injection container.

Implementations of IDataBusSerializer should not close Stream instances that NServiceBus provides. NServiceBus manages the lifecycle of these Stream instances and may attempt to manipulate them after the custom serializer has been called.
The DataBus serializer installed in the receiving endpoint must be able to deserialize data that was generated by the serializer installed in the sending endpoint.

DataBus attachments cleanup

The various DataBus implementations each behave differently with regard to cleanup of physical attachments used to transfer data properties depending on the implementation used.

Why attachments are not removed by default

Automatically removing these attachments can cause problems in many situations. For example:

  • The supported DataBus implementations do not participate in distributed transactions. If for some reason, the message handler throws an exception and the transaction rolls back, the delete operation on the attachment cannot be rolled back. Therefore, when the message is retried, the attachment will no longer be present causing additional problems.
  • The message can be deferred so that the file will be processed later. Removing the file after deferring the message, results in a message without the corresponding file.
  • Functional requirements might dictate the message to be available for a longer duration.
  • If the DataBus feature is used when publishing an event to multiple subscribers, neither the publisher nor any specific subscribing endpoint can determine when all subscribers have successfully processed the message allowing the file to be cleaned up.
  • If message processing fails, it will be handled by the recoverability feature. This message can then be retried some period after that failure. The DataBus files need to exist for that message to be re-processed correctly.

Other considerations

Monitoring and reliability

The storage location for DataBus blobs is critical to the operation of endpoints. As such it should be as reliable as other infrastructure such as the transport or persistence. It should also be monitored for errors and be actively maintained. Since messages cannot be sent or received when the storage location is unavailable, it may be necessary to stop endpoints when maintenance tasks occur.

Auditing

The data stored in DataBus blobs may be considered part of an audit record. In these cases DataBus blobs should be archived alongside messages in for as long as the audit record is required.

Samples

Related Articles


Last modified