Upgrading a Distributor endpoint from Version 5 to 6

Initial state

This sample uses the following setup:

• Sender sends commands to the scaled out endpoint.
• Worker1 runs both distributor and a worker node.
• Worker2 runs only a worker node.

Upgrade

NServiceBus 6 does not have a distributor role. The distributor remains on NServiceBus 5.x, the workers can upgrade to 6.x.

Adding standalone distributor

Versions 6 and above no longer support running an embedded distributor so a separate project has to be added to the solution.

This new project references NServiceBus Version 5.X because the distributor does not have a Version 6 compatible release. The project also needs to reference the NServiceBus.Distributor.MSMQ package.

Writing the distributor code

The following code is required to set up the distributor

var busConfiguration = new BusConfiguration();
busConfiguration.EndpointName("Samples.Scaleout.Distributor");
busConfiguration.RunMSMQDistributor(withWorker: false);
busConfiguration.UsePersistence<InMemoryPersistence>();
busConfiguration.EnableInstallers();
using (var bus = Bus.Create(busConfiguration).Start())
{
    Console.WriteLine("Press any key to exit");
    Console.ReadKey();
}

Notice that false is passed as a value for withWorker argument of the RunMSMQDistributor API call.

Reconfiguring sender's routing

In the original sample the sender was sending messages to the master (Worker1). Now it needs to send messages to the new standalone distributor

bus.Send("Samples.Scaleout.Distributor", placeOrder);

Upgrading a worker

The NServiceBus package needs to be upgraded to 6.x. Workers no longer need the NServiceBus.Distributor.MSMQ package as the worker logic is embedded in the Version 6 package.

Sample specific changes

Emulating multiple workers

In Versions 6 and above each endpoint instance is identified by name of the endpoint and an ID of the instance. Both workers are going to be named Samples.Scaleout.Worker and the instance ID is going to be loaded from the app.config file. If the workers are deployed to separate machines the instance ID can be omitted.

var endpointConfiguration = new EndpointConfiguration("Samples.Scaleout.Worker");
endpointConfiguration.MakeInstanceUniquelyAddressable(
    discriminator: ConfigurationManager.AppSettings["InstanceId"]);

Enlisting with the distributor

In Versions 6 and above there is a new API for enlisting with the distributor. The API requires passing addresses of distributor data and control queues which were previously read from the configuration section.

endpointConfiguration.EnlistWithLegacyMSMQDistributor(
    masterNodeAddress: ConfigurationManager.AppSettings["DistributorAddress"],
    masterNodeControlAddress: ConfigurationManager.AppSettings["DistributorControlAddress"],
    capacity: PushRuntimeSettings.Default.MaxConcurrency);

The last parameter, capacity is the maximum number of messages in-flight between the distributor and the worker. The worker sends an acknowledgement (ACK) message (also called ready message) for each message processed. The distributor keeps track of these ACKs to ensure no worker is flooded with work that it can't process

In Versions 5 and below the capacity was always set to the same value as the maximum concurrency setting, which means there were never more messages in-flight than threads ready to process them on the worker side. If the worker was processing it meant that messages never waited in the queue. This approach limits the potential consequences of worker failure at the expense of throughput.

A new setting was introduced in Version 6, which allows to explicitly control the capacity value. The snippet above shows how to set it to the default concurrency value of Version 6. If the endpoint configuration overrides this parameter with a custom value, an equal or larger value should be provided as capacity when enlisting with distributor.

Upgrading the handlers

The handlers need to be upgraded to conform to new Version 6 API.

Removing the shared code

Because the sender still uses Version 5 and the workers are on Version 6, the shared elements need to be moved to their target projects:

• Copy the ConfigErrorQueue class to the Sender and Distributor projects.
• Configure the Sender and Worker projects to use unobtrusive mode.

Final state

After the upgrade is done both worker projects are identical (apart from the configuration file). The sender needed only a minor routing correction.