Initial state
This sample uses the following setup:
- Sender sends commands to the scaled out endpoint.
- Worker1 runs both distributor and a worker node.
- Worker2 runs only a worker node.
Upgrade
NServiceBus 6 does not have a distributor role. The distributor remains on NServiceBus 5.x, the workers can upgrade to 6.x.
Adding standalone distributor
Versions 6 and above no longer support running an embedded distributor so a separate project has to be added to the solution.
This new project references NServiceBus
Version 5.X because the distributor does not have a Version 6 compatible release. The project also needs to reference the NServiceBus.
package.
Writing the distributor code
The following code is required to set up the distributor
var busConfiguration = new BusConfiguration();
busConfiguration.EndpointName("Samples.Scaleout.Distributor");
busConfiguration.RunMSMQDistributor(withWorker: false);
busConfiguration.UsePersistence<InMemoryPersistence>();
busConfiguration.EnableInstallers();
using (var bus = Bus.Create(busConfiguration).Start())
{
Console.WriteLine("Press any key to exit");
Console.ReadKey();
}
Notice that false
is passed as a value for withWorker
argument of the RunMSMQDistributor
API call.
Reconfiguring sender's routing
In the original sample the sender was sending messages to the master (Worker1). Now it needs to send messages to the new standalone distributor
bus.Send("Samples.Scaleout.Distributor", placeOrder);
Upgrading a worker
The NServiceBus
package needs to be upgraded to 6.x. Workers no longer need the NServiceBus.
package as the worker logic is embedded in the Version 6 package.
Sample specific changes
Emulating multiple workers
In Versions 6 and above each endpoint instance is identified by name of the endpoint and an ID of the instance. Both workers are going to be named Samples.
and the instance ID is going to be loaded from the app.config file. If the workers are deployed to separate machines the instance ID can be omitted.
var endpointConfiguration = new EndpointConfiguration("Samples.Scaleout.Worker");
endpointConfiguration.MakeInstanceUniquelyAddressable(
discriminator: ConfigurationManager.AppSettings["InstanceId"]);
Enlisting with the distributor
In Versions 6 and above there is a new API for enlisting with the distributor. The API requires passing addresses of distributor data and control queues which were previously read from the configuration section.
endpointConfiguration.EnlistWithLegacyMSMQDistributor(
masterNodeAddress: ConfigurationManager.AppSettings["DistributorAddress"],
masterNodeControlAddress: ConfigurationManager.AppSettings["DistributorControlAddress"],
capacity: PushRuntimeSettings.Default.MaxConcurrency);
The last parameter, capacity
is the maximum number of messages in-flight between the distributor and the worker. The worker sends an acknowledgement (ACK) message (also called ready
message) for each message processed. The distributor keeps track of these ACKs to ensure no worker is flooded with work that it can't process
In Versions 5 and below the capacity was always set to the same value as the maximum concurrency setting, which means there were never more messages in-flight than threads ready to process them on the worker side. If the worker was processing it meant that messages never waited in the queue. This approach limits the potential consequences of worker failure at the expense of throughput.
A new setting was introduced in Version 6, which allows to explicitly control the capacity value. The snippet above shows how to set it to the default concurrency value of Version 6. If the endpoint configuration overrides this parameter with a custom value, an equal or larger value should be provided as capacity
when enlisting with distributor.
Upgrading the handlers
The handlers need to be upgraded to conform to new Version 6 API.
Removing the shared code
Because the sender still uses Version 5 and the workers are on Version 6, the shared elements need to be moved to their target projects:
- Copy the
ConfigErrorQueue
class to theSender
andDistributor
projects. - Configure the
Sender
andWorker
projects to use unobtrusive mode.
Final state
After the upgrade is done both worker projects are identical (apart from the configuration file). The sender needed only a minor routing correction.