Endpoints using the MSMQ transport are unable to use the competing consumers pattern to scale out by adding additional worker instances. Sender-side distribution is a method of scaling out an endpoint using the MSMQ transport without relying on a centralized distributor assigning messages to available workers.
When using sender-side distribution:
- Multiple endpoint instances (deployed to different servers) are capable of processing a message that requires scaled-out processing.
- A client sending a message is aware of all the endpoint instances that can process the message.
- The client sends a message to a worker endpoint instance based on round-robin distribution, or a custom distribution strategy.
Using sender-side distribution requires two parts. The first part maps message types to logical endpoints, and occurs in code. The second part maps logical endpoints to physical endpoint instances running on a specific machine.
Mapping logical endpoints
To map message types to logical endpoints, use the following config:
var routing = endpointConfiguration.UseTransport(new MsmqTransport());
routing.RouteToEndpoint(
messageType: typeof(AcceptOrder),
destination: "Sales");
routing.RouteToEndpoint(
messageType: typeof(SendOrder),
destination: "Shipping");
This creates mappings specifying that the AcceptOrder
command is handled by the Sales endpoint, while the SendOrder
command is handled by the Shipping endpoint.
Meanwhile, the logical-to-physical mappings will be configured in the instance-mapping.
file, as this information is an operational concern that must be changed for deployment to multiple machines.
UnicastBusConfig/MessageEndpointMappings
configuration section, that message cannot participate in sender-side distribution. The endpoint address specified by a message endpoint mapping is a physical address (QueueName@MachineName
, where MachineName
is assumed to be localhost
if omitted) which combines the message-to-owner-endpoint and endpoint-to-physical-address concerns in a way that can't be separated.Mapping physical endpoint instances
The routing configuration file specifies how logical endpoint names are mapped to physical queues on specific machines:
<endpoints>
<endpoint name="Sales">
<instance machine="VM-S-1"/>
<instance machine="VM-S-2"/>
<instance machine="VM-S-3"/>
</endpoint>
</endpoints>
To read more about the instance mapping, refer to MSMQ Routing.
Message distribution
Every message is always delivered to a single physical instance of the logical endpoint. When scaling out, there are multiple instances of a single logical endpoint registered in the routing system. Each outgoing message must undergo the distribution process to determine which instance is going to receive this particular message. By default, a round-robin algorithm is used to determine the destination. Routing extensions can override this behavior by registering a custom DistributionStrategy
for a given destination endpoint.
var transport = new MsmqTransport();
var routing = endpointConfiguration.UseTransport(transport);
routing.SetMessageDistributionStrategy(new RandomStrategy("Sales", DistributionStrategyScope.Send));
class RandomStrategy :
DistributionStrategy
{
static Random random = new Random();
public RandomStrategy(string endpoint, DistributionStrategyScope scope) : base(endpoint, scope)
{
}
public override string SelectDestination(DistributionContext context)
{
// access to headers, payload...
return context.ReceiverAddresses[random.Next(context.ReceiverAddresses.Length)];
}
}
To learn more about creating custom distribution strategies, see the fair distribution sample.
Events and subscriptions
Subscription requests:
Each subscriber endpoint instance will at start send a subscription message to each configured publisher instance. Each publisher instance receives a subscription requests and stores this. In most cases the subscription storage is shared.
Events:
When an event is published the event will be send to only one of the endpoint instances. Which instance depends on the distribution strategy
Limitations
Sender-side distribution does not use message processing confirmations. Therefore the sender has no feedback on the availability of workers and, by default, sends the messages in a round-robin behavior. Should one of the nodes stop processing, the messages will pile up in its input queue. As such, nodes running in sender-side distribution mode require more careful monitoring.
Decommissioning endpoint instances
For the reasons outlined above, when scaling down (removing a "target" endpoint instance from service), it is important to properly decommission the instance:
- Change the instance mapping file to remove the target endpoint instance.
- Ensure that the updated instance mapping information is distributed to all endpoint instances that might send a message to the target endpoint.
- Allow time (30 seconds by default) for all endpoints to reread the instance mapping file, and ensure no new messages are arriving in the target instance's queue.
- Allow the target endpoint instance to complete processing all messages in its queue.
- Disable the target endpoint instance.
- Check the input queue of the decommissioned instance for leftover messages and move them to other instances if necessary.