Amazon SQS Transport

Source
NuGet Package NServiceBus.AmazonSQS (2.x)
Target NServiceBus Version: 6.x
Due to a bug in the AWS SDK, messages might indicate that they are in-flight for a longer time than expected. Messages will not be lost but may end up being delivered up to 30 seconds later.

Simple Queue Service (SQS) is a message queue service provided by Amazon Web Services.

Advantages

  • Fully managed turn-key messaging infrastructure. SQS queues requires very little effort to set up, maintain and manage over time.
  • Integrates seamlessly with other services provided by AWS, for example, IAM, CloudWatch, Lambda, etc. For organizations already committed to AWS, SQS is a natural choice.
  • Can be used as a gateway between endpoints that may not have direct connectivity to each-other.

Disadvantages

  • Like other message brokers, there is no local store-and-forward mechanism available. If an endpoint cannot reach SQS, either due to network problems or if SQS is unavailable, the endpoint will not be able to send nor receive messages.
  • Can be relatively expensive when using larger volumes of messages.

Getting Started

An AWS IAM account with a pair of Access Keys is required.

The IAM account requires the following:

SQS permissions

  • CreateQueue
  • DeleteMessage
  • DeleteMessageBatch
  • GetQueueUrl
  • ReceiveMessage
  • SendMessage
  • SendMessageBatch
  • SetQueueAttributes
  • GetQueueAttributes
  • ChangeMessageVisibility
  • ChangeMessageVisibilityBatch
  • PurgeQueue

S3 permissions

  • CreateBucket
  • DeleteObject
  • GetObject
  • PutObject
  • PutLifecycleConfiguration
  • GetLifecycleConfiguration
  • ListAllMyBuckets

By default, AWS Access Key ID and AWS Secret Access Key are discovered from environment variables of the machine that is running the endpoint:

  • Access Key ID goes in AWS_ACCESS_KEY_ID
  • Secret Access Key goes in AWS_SECRET_ACCESS_KEY
var transport = endpointConfiguration.UseTransport<SqsTransport>();
transport.Region("ap-southeast-2");
// S3 bucket only required for messages larger than 256KB
transport.S3BucketForLargeMessages("myBucketName", "my/key/prefix");

For more configuration options, consult the configuration options page.

Retries and Timeouts

The SQS transport uses the default retries and timeouts values implemented by the AWS SDK for .NET:

ParameterDefault value
MaxErrorRetries4
RequestTimeout100s
ReadWriteTimeout300s
NServiceBus will perform immediate and delayed retries in addition to retries performed internally by the SQS client.

Troubleshooting

AmazonSQSException: Request is throttled

Amazon SQS can handle large, continuous throughput but if there are sudden spikes, the service may apply throttling.

When throttling happens, the following exception is logged:

2017-11-14 23:10:24,314|ERROR|18|NServiceBus.Transports.SQS.MessageDispatcher|Exception from Send.
Amazon.SQS.AmazonSQSException: Request is throttled. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

Throttling is more likely to happen when sending a large number messages concurrently. For example, using a list of tasks when using async/await.

To avoid Amazon throttling errors, limit the maximum number of concurrent sends. For example, allow only a small amount of messages to be sent concurrently as outlined in the sending large amount of messages guidelines or send messages sequentially.

Throttling can happen during any send or receive operation and can happen during the following scenarios:

  • Incoming message (receiving)
  • Sending from within a handler
  • Sending outside of a handler

Incoming message (receiving)

For incoming messages throttling errors can be safely ignored as the message pump will try to fetch the next available message again.

Sending from within a handler

Failing message sends raise an exception when throttled. The exception will be handled by the recoverability feature mechanism. An incoming message that continuously fails due to throttling errors will be moved to the error queue.

A throttling error could result in partial message delivery while the incoming message is not processed successfully and can occur regardless of using the default batched message dispatch or when using immediate dispatch.

Throttling errors are similar to any other technical error that can occur.

Sending outside of a handler

As message sending does not happen within a handler context any failures during sending will not rely or be covered by the recoverability feature mechanism. Any retry logic must be manually implemented.

When throttling occurs with no custom error logic implemented, one or more messages might not have been transmitted to Amazon SQS. The custom retry logic could either retry all messages to be sent again, including already succeeded messages or only retry individual messages that failed.

Samples

Related Articles


Last modified