Simple Queue Service (SQS) is a message queue service provided by Amazon Web Services.
Transport at a glance
Feature | |
---|---|
Transactions | None, ReceiveOnly (Message visibility timeout) |
Pub/Sub | Native (Requires SNS, supports hybrid-mode for migration purposes) |
Timeouts | Native (Requires FIFO Queues) |
Large message bodies | Native (Requires S3) |
Scale-out | Competing consumer |
Scripted Deployment | Built-in CLI, C# |
Installers | Optional |
Native integration | Supported |
Advantages
- Fully managed turn-key messaging infrastructure. SQS queues requires little effort to set up, maintain, and manage over time.
- Integrates seamlessly with other services provided by AWS, such as IAM, CloudWatch, and Lambda. For organizations already committed to AWS, SQS is a natural choice.
- Can be used as a gateway between endpoints that may not have direct connectivity to each other.
- Can send and receive large messages that exceed the queue limitations by storing large payloads in S3. For more information review the documentation for the transport topology and configuration options.
Disadvantages
- Like other message brokers, there is no local store-and-forward mechanism available. If an endpoint cannot reach SQS, either due to network problems or if SQS is unavailable, the endpoint will not be able to send nor receive messages.
- Can be expensive with large volumes of messages.
Prerequisites
An AWS IAM account with a pair of Access Keys is required.
The IAM account requires the following permissions to provision infrastructure and run the transport:
SQS permissions
- CreateQueue
- DeleteMessage
- DeleteMessageBatch
- GetQueueUrl
- ReceiveMessage
- SendMessage
- SendMessageBatch
- SetQueueAttributes
- GetQueueAttributes
- ChangeMessageVisibility
- ChangeMessageVisibilityBatch
- PurgeQueue
SNS permissions
- CreateTopic
- ListTopics
- GetTopicAttributes
- SetTopicAttributes
- SetEndpointAttributes
- Publish
- Subscribe
- Unsubscribe
- ListSubscriptions
- ListSubscriptionsByTopic
- GetSubscriptionAttributes
- SetSubscriptionAttributes
In addition to the above permissions the queue subscribing to a topic needs sqs:SendMessage
permission to enable the topics delivering messages to the subscribing queue.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SomeSid",
"Effect": "Allow",
"Principal": {
"AWS": "yourPrincipal"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:yourQueueArn",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:sns:yourTopicArn"
}
}
},
]
}
S3 permissions
- CreateBucket
- DeleteObject
- GetObject
- PutObject
- PutLifecycleConfiguration
- GetLifecycleConfiguration
- ListAllMyBuckets
Other permissions
- If using server-side encryption of SQS queues, all NServiceBus endpoints (as well as ServiceControl) will require the
kms:GenerateDataKey
permission in order to support key management.
Configuration
By default, AWS Access Key ID, AWS Secret Access Key and AWS Region Key are discovered from environment variables of the machine that is running the endpoint:
- Access Key ID goes in
AWS_ACCESS_KEY_ID
- Secret Access Key goes in
AWS_SECRET_ACCESS_KEY
- Region Key goes in
AWS_REGION
// S3 bucket only required for messages larger than 256KB
var transport = new SqsTransport
{
S3 = new S3Settings("myBucketName", "my/key/prefix")
};
endpointConfiguration.UseTransport(transport);
For more configuration options consult the configuration options page.
The existing API surface with UseTransport
is supported via a shim API to ease migration. However, it is recommended to switch to the new transport configuration API to prepare for future upgrades of NServiceBus.
Retries and timeouts
The SQS transport uses the default retry and timeout values implemented by the AWS SDK for .NET:
Parameter | Default value |
---|---|
MaxErrorRetries | 4 |
RequestTimeout | 100s |
ReadWriteTimeout | 300s |
NServiceBus will perform immediate and delayed retries in addition to retries performed internally by the SQS client.
Batching
Messages sent from within a handler are batched with up to ten messages per batch depending on the size of the message. Messages sent outside a handler are not batched.