Scaling Out With Sender-side Distribution

Endpoints using the MSMQ transport are unable to use the competing consumers pattern to scale out by adding additional worker instances. Sender-side distribution is a method of scaling out an endpoint using the MSMQ transport without relying on a centralized distributor assigning messages to available workers.

When using sender-side distribution:

• Multiple endpoint instances (deployed to different servers) are capable of processing a message that requires scaled-out processing.
• A client sending a message is aware of all the endpoint instances that can process the message.
• The client sends a message to a worker endpoint instance based on round-robin distribution, or a custom distribution strategy.

Using sender-side distribution requires two parts. The first part maps message types to logical endpoints, and occurs in code. The second part maps logical endpoints to physical endpoint instances running on a specific machine.

Mapping logical endpoints

To map message types to logical endpoints, use the following config:

var routing = endpointConfiguration.UseTransport(new MsmqTransport());

routing.RouteToEndpoint(
    messageType: typeof(AcceptOrder),
    destination: "Sales");
routing.RouteToEndpoint(
    messageType: typeof(SendOrder),
    destination: "Shipping");

This creates mappings specifying that the AcceptOrder command is handled by the Sales endpoint, while the SendOrder command is handled by the Shipping endpoint.

Meanwhile, the logical-to-physical mappings will be configured in the instance-mapping.xml file, as this information is an operational concern that must be changed for deployment to multiple machines.

Mapping physical endpoint instances

The routing configuration file specifies how logical endpoint names are mapped to physical queues on specific machines:

<endpoints>
  <endpoint name="Sales">
    <instance machine="VM-S-1"/>
    <instance machine="VM-S-2"/>
    <instance machine="VM-S-3"/>
  </endpoint>
</endpoints>

To read more about the instance mapping, refer to MSMQ Routing.

Message distribution

Every message is always delivered to a single physical instance of the logical endpoint. When scaling out, there are multiple instances of a single logical endpoint registered in the routing system. Each outgoing message must undergo the distribution process to determine which instance is going to receive this particular message. By default, a round-robin algorithm is used to determine the destination. Routing extensions can override this behavior by registering a custom DistributionStrategy for a given destination endpoint.

var transport = new MsmqTransport();
var routing = endpointConfiguration.UseTransport(transport);
routing.SetMessageDistributionStrategy(new RandomStrategy("Sales", DistributionStrategyScope.Send));

class RandomStrategy :
    DistributionStrategy
{
    static Random random = new Random();

    public RandomStrategy(string endpoint, DistributionStrategyScope scope) : base(endpoint, scope)
    {
    }

    public override string SelectDestination(DistributionContext context)
    {
        // access to headers, payload...
        return context.ReceiverAddresses[random.Next(context.ReceiverAddresses.Length)];
    }
}

To learn more about creating custom distribution strategies, see the fair distribution sample.

Events and subscriptions

Subscription requests:

Each subscriber endpoint instance will at start send a subscription message to each configured publisher instance. Each publisher instance receives a subscription requests and stores this. In most cases the subscription storage is shared.

Events:

When an event is published the event will be send to only one of the endpoint instances. Which instance depends on the distribution strategy

Limitations

Sender-side distribution does not use message processing confirmations. Therefore the sender has no feedback on the availability of workers and, by default, sends the messages in a round-robin behavior. Should one of the nodes stop processing, the messages will pile up in its input queue. As such, nodes running in sender-side distribution mode require more careful monitoring.

Decommissioning endpoint instances

For the reasons outlined above, when scaling down (removing a "target" endpoint instance from service), it is important to properly decommission the instance:

1. Change the instance mapping file to remove the target endpoint instance.
2. Ensure that the updated instance mapping information is distributed to all endpoint instances that might send a message to the target endpoint.
3. Allow time (30 seconds by default) for all endpoints to reread the instance mapping file, and ensure no new messages are arriving in the target instance's queue.
4. Allow the target endpoint instance to complete processing all messages in its queue.
5. Disable the target endpoint instance.
6. Check the input queue of the decommissioned instance for leftover messages and move them to other instances if necessary.