Topology

The topology used by the transport is composed of several AWS components.

SQS

Amazon SQS exposes queues via service endpoints that are publicly available via HTTPS. An NServiceBus endpoints can access SQS Queues whether they are deployed in AWS or not; as long as the endpoint can reach both SQS and S3 via HTTPS it can use the transport. By default, NServiceBus endpoints process messages from an SQS queue with the same name as the endpoint.

The transport initiates all network connections to SQS and S3; hence the endpoint itself does not need to be publicly accessible and can reside behind a firewall or proxy.

The transport uses SQS Standard Queues.

SQS queues support competing consumers. When an endpoint scales out to multiple instances, each instance consumes messages from the same input queue.

Publish/Subscribe

The transport is a multicast-enabled transport and provides built-in support for publish-subscribe messaging using Amazon Simple Notification Service (SNS). Publish and subscribe using SNS and SQS is achieved by publishing a message to a SNS topic to which none or more SQS queues are subscribed, with each SQS queue receiving a copy. The way the NServiceBus SQS transport leverages SNS and SQS, or topology, is by publishing events to the dedicated topics for the event type, with subscribing endpoints creating SNS subscriptions between the topic of events they handle to the endpoint's SQS queue.

Topology deployment can be automated, or manually created, using the transport CLI tool. Topics and subscriptions are created automatically by the subscribing endpoints. Publishing endpoints must create event topics manually before publishing events if there are no subscribers. Refer to the transport operations section for more information.

Message inheritance support

By default a subscriber will subscribe only to the most concrete type it knows about and a publisher will always publish the most concrete type it knows about. Inheritance at the subscriber level is not supported when using the automatically created topology.

In case a subscriber needs to subscribe to a message type that is not the most concrete type as seen by the publisher, a custom mapping is needed. For example, if a subscriber is subscribed to the IOrderAccepted event defined in the Contracts assembly it will create and subscribe to a topic named namespace-IOrderAccepted. However, if in the same system the publisher publishes the OrderAccepted message that implements IOrderAccepted from the Messages assembly it'll try to publish to the namespace-OrderAccepted topic and the message won't be delivered to the desired destination.

For the described inheritance scenario to work properly, a custom mapping must be defined at the subscriber:

var transport = new SqsTransport();

transport.MapEvent<IOrderAccepted>("namespace-OrderAccepted");

endpointConfiguration.UseTransport(transport);

The above snippet instructs the subscriber's subscription manager to create and subscribe to a topic named namespace-OrderAccepted when subscribing to the IOrderAccepted event.

More information about the custom mapping API can be found in the configuration options documentation.

Evolving an existing topology

Evolving an existing topology needs to be handled with care. The SQS Transport never deletes any resource so it might happen that messages are routed to undesirable destinations if or when message handlers are moved to a different endpoint, or completely removed.

For instance, in the following scenario:

• The system is composed by a publisher and a scaled-out subscribers (two instances, for the sake of the sample)
• An event needs to be removed as not anymore needed
• The message handler, for the removed event, needs to be removed with no subscribers downtime

The following steps needs to be put in place:

1. Change the publisher first to stop publishing the event for which the handler needs to be removed
2. Deploy the publisher changes
3. Wait for in-flight messages to be consumed
4. Change the subscriber by removing the not anymore needed handler
5. Deploy subscribers instances one-by-one with no downtime
6. Remove the SNS to SQS subscription for the specific event in AWS
7. If not anymore needed, remove the SNS topic for the specific event

S3

SQS supports a maximum message size of 256 KiB. The transport works around this size limit by using Amazon S3 to store message payloads for messages that are larger than 256 KiB. This allows the transport to send and receive messages of practically any size. Note that messages that fit within the size limit only use SQS; S3 does not come into play.

When a large message is sent, the transport uploads the message body to an S3 bucket and then sends an SQS message that contains a reference to the S3 object. On the receiving end, the transport receives the message from SQS, identifies the reference to the S3 object, downloads it, and processes the message as usual. When the message is to be deleted, the transport deletes the message from SQS and then deletes it from S3. To ensure that the message is deleted from S3, the transport applies a lifecycle policy to the S3 bucket that deletes any messages that are older than the configured retention period.