Scaling out with sender-side distribution

Component: MSMQ Transport
NuGet Package NServiceBus (6.x)

Endpoints using the MSMQ transport are unable to use the competing consumers pattern to scale out by adding additional worker instances. Sender-side distribution is a method of scaling out an endpoint using the MSMQ transport, without relying on a centralized distributor assigning messages to available workers.

When using sender-side distribution:

  • Multiple endpoint instances (deployed to different servers) are capable of processing a message that requires scaled-out processing.
  • A client sending a message is aware of all the endpoint instances that can process the message.
  • The client sends the message to a worker endpoint instance based on round-robin distribution, or a custom distribution strategy.

Using sender-side distribution requires two parts. The first part maps message types to logical endpoints, and occurs in code. The second part maps logical endpoints to physical endpoint instances running on a specific machine.

Mapping logical endpoints

To map message types to logical endpoints, use the following config:

var transport = endpointConfiguration.UseTransport<MsmqTransport>();
var routing = transport.Routing();

routing.RouteToEndpoint(
    messageType: typeof(AcceptOrder),
    destination: "Sales");
routing.RouteToEndpoint(
    messageType: typeof(SendOrder),
    destination: "Shipping");

This creates mappings specifying that the AcceptOrder command is handled by the Sales endpoint, while the SendOrder command is handled by the Shipping endpoint.

Meanwhile, the logical-to-physical mappings will be configured in the instance-mapping.xml file, as this information is an operational concern that must be changed for deployment to multiple machines.

If a message is mapped in an App.config file via the UnicastBusConfig/MessageEndpointMappings configuration section, then that message cannot participate in sender-side distribution. The endpoint address specified by a message endpoint mapping is a physical address (QueueName@MachineName, where machine name is assumed to be localhost if omitted) which combines the message-to-owner-endpoint and endpoint-to-physical-address concerns in a way that can't be separated.

Mapping physical endpoint instances

The routing configuration file specifies how logical endpoint names are mapped to physical queues on specific machines:

<endpoints>
  <endpoint name="Sales">
    <instance machine="VM-S-1"/>
    <instance machine="VM-S-2"/>
    <instance machine="VM-S-3"/>
  </endpoint>
</endpoints>

To read more about the instance mapping, refer to the MSMQ routing page.

Message distribution

Every message is always delivered to a single physical instance of the logical endpoint. When scaling out there are multiple instances of a single logical endpoint registered in the routing system. Each outgoing message has to undergo the distribution process to determine which instance is going to receive this particular message. By default a round-robin algorithm is used to determine the destination. Routing extensions can override this behavior by registering a custom DistributionStrategy for a given destination endpoint.

var transport = endpointConfiguration.UseTransport<MsmqTransport>();
var routing = transport.Routing();
routing.SetMessageDistributionStrategy(new RandomStrategy("Sales", DistributionStrategyScope.Send));
6.x NServiceBus
class RandomStrategy :
    DistributionStrategy
{
    static Random random = new Random();

    public RandomStrategy(string endpoint, DistributionStrategyScope scope) : base(endpoint, scope)
    {
    }

    public override string SelectReceiver(string[] receiverAddresses)
    {
        int index;
        var length = receiverAddresses.Length;
        lock(random) index = random.Next(length);
        return receiverAddresses[random.Next(index)];
    }
}
6.2 - N NServiceBus
class RandomStrategy :
    DistributionStrategy
{
    static Random random = new Random();

    public RandomStrategy(string endpoint, DistributionStrategyScope scope) : base(endpoint, scope)
    {
    }

    // Method will not be called since SelectDestination doesn't call base.SelectDestination
    public override string SelectReceiver(string[] receiverAddresses)
    {
        throw new NotImplementedException();
    }

    public override string SelectDestination(DistributionContext context)
    {
        int index;
        var length = context.ReceiverAddresses.Length;
        lock(random) index = random.Next(length);
        // access to headers, payload...
        return context.ReceiverAddresses[index];
    }
}

In Versions 6.2 and above it is possible to override the virtual method SelectDestination. The method provides access to the DistributionStrategyContext that enables implementing more advanced distribution scenarios, such as distributing based on the headers of the message. When SelectDestination is overridden, do not call base.SelectDestination since the base method calls SelectReceiver for backward compatibility reasons.

To learn more about creating custom distribution strategies see the fair distribution sample.

Limitations

Sender-side distribution does not use message processing confirmations (the Distributor approach). Therefore the sender has no feedback on the availability of workers and, by default, sends the messages in a round-robin behavior. Should one of the nodes stop processing, the messages will start piling up in its input queue. As such nodes running in sender-side distribution mode require more careful monitoring compared to distributor workers.

Using Sender-Side distribution feature in combination with the Distributor will affect the routing of delayed retries. These messages will be routed directly to the Distributor instead of the endpoint instance even though these messages were originally sent to the endpoint instance.

Decommissioning endpoint instances

For the reasons outlined above, when scaling down (removing a "target" endpoint instance from service), it is important to properly decommission the instance:

  1. Change the instance mapping file to remove the target endpoint instance.
  2. Ensure that the updated instance mapping information is distributed to all endpoint instances that might send a message to the target endpoint.
  3. Allow time (30 seconds by default) for all endpoints to reread the instance mapping file, and ensure no new messages are arriving in the target instance's queue.
  4. Allow the target endpoint instance to complete processing all messages in its queue.
  5. Disable the target endpoint instance.
  6. Check the input queue of the decommissioned instance for leftover messages and move them to other instances if necessary.

Related Articles


Last modified