NServiceBus is designed for scalability and reliability, but to take advantage of these features, it is necessary to deploy it in a Windows Failover Cluster. Unfortunately, information on how to do this effectively is, as yet, incomplete and scattered. This article describes the process for deploying NServiceBus in a failover cluster. This article does not cover the generic setup of a failover cluster. There are other, better resources for that, such as Creating a Cluster in Windows Server 2008 or Server 2012. The focus here is the setup related to NServiceBus.
A simple setup for scalability and reliability includes at least two servers in a failover cluster. The failover cluster servers run a distributor process with a timeout manager for each logical message queue.
In addition there must exist one or more additional servers called worker nodes. These contain endpoints containing message handlers and they are the servers that can be scaled out. The endpoints on worker nodes request work from the clustered distributors, do the work, and then ask for more.
While technically it shouldn't matter from which clustered server is set up, generally it is more reliable to set up everything from whichever server currently holds the Quorum disk. Find the server that has it (it moves around when the server holding it restarts), and open up Failover Cluster Management under Administrative Tools.
Set up a clustered DTC access point:
Configure DTC for NServiceBus:
Administrative Tools - Component Services, expand
Component Services - Computers - My Computer - Distributed Transaction Coordinator.
Set up a MSMQ Cluster Group. Cluster group is a group of resources that have a unique DNS name and can be addressed externally like a computer.
For more information, see https://technet.microsoft.com/en-us/library/cc753575.aspx
For NServiceBus endpoint destination, queues are addressed by the MSMQ cluster group's name, clustered resources be added will later. In non-cluster terms, typically the machine name is added to the address of the queue, i.e.
queue@MachineName. In cluster terms it is addressed by queue@MSMQ Network name.
In Failover Cluster Management, from the server with Quorum:
This will result in a clustered MSMQ instance. Click the instance under Services and Applications to see the summary, which contains the Server Name, Storage, and MSMQ instance. Right click the MSMQ instance, select Properties, and the Dependencies tab. Make sure that it contains dependencies for the MSMQ Network Name AND IP Address AND Storage.
View the MSMQ MMC snap-in by right clicking the MSMQ cluster in the left pane and selecting Manage MSMQ, which opens the Computer Management tool geared toward the clustered instance.
Go to MSMQ by expanding Services and Applications - Message Queuing.
Keep in mind that this only to work if viewing Failover Cluster Management from the server where the MSMQ Network Name currently resides. If on Server A and an attempt is made to manage MSMQ on a MSMQ Network residing on Server B, Message Queuing will not show up in the Computer Management window.
Try swapping the MSMQ Network Name back and forth between nodes a few times. It's best to make sure that everything is working properly now before continuing.
The "cluster name" is a Network Name created for the cluster as part of the core Cluster Group - a group created by default for each cluster. The core cluster group is different than the MSMQ cluster group and it has a Different network name. One of the most common confusions while using MSMQ on a cluster is using the Cluster Name in the client instead of the MSMQ Network Name.
In this picture:
NServiceBus.Host.exe processes can be clustered, they need to be installed services on all clustered nodes.
Copy the Distributor binary as many times as there are have logical queues, and then configure each one as described in the NServiceBus Distributor page. To keep everything straight, the queues are named according to the following convention:
A review of how the distributor works: Other endpoints send messages to the queue specified by the Distributor Data Bus, where they accumulate if no worker is running. When a worker comes online, it sends a ReadyMessage to the queue specified by the Distributor Control Bus. If there is work to be done, the distributor sends an item from the Data Bus to the endpoint's local input queue, otherwise, it files it in the Distributor Storage Queue so that when work does come in, the distributor knows who is ready to process it.
Using this naming convention, all the queues of an endpoint end up grouped together.
Install each distributor from the command line:
NServiceBus.Host.exe /install /serviceName:EndpointName /displayName:Distributor.EndpointName /description:Distributor.EndpointName /userName:DOMAIN\us /password:thepassword NServiceBus.Production NServiceBus.MSMQDistributor
It's easier to set the service name, display name, and description to be the same. It helps when trying to start and stop things from a NET START/STOP command and when viewing them in the multiple graphical tools. Starting each one with Distributor puts them all together alphabetically in the Services MMC snap-in.
Don't forget the
NServiceBus.Production at the end, which sets the profile for the NServiceBus generic host, as described in the Generic Host page and the
NServiceBus.MSMQDistributor which sets up the host in distributor mode.
Do not try starting the services as they will run in the scope of the local server node, and will attempt to create their queues there.
Now, add each distributor to the cluster:
Environment.MachineNameshould return the cluster name, not the cluster node's computer name. Click Apply.
Again, try swapping the cluster back and forth, to ensure it can move freely between the cluster nodes.
MasterNodeConfigare obsoleted and
endpointConfiguraiton.EnlistWithLegacyMSMQDistributorshould be used instead. For more information refer to Upgrading a Distributor-based scaled out endpoint to V6 documentation.
Ensure that the worker servers have unique QMIds.
Set up the worker processes on all worker servers (not the cluster nodes!) as windows services, but instead of using
NServiceBus.MSMQWorker profile instead.
Configure the workers'
MasterNodeConfig section to point to the machine running the distributor as described on the Distributor Page under Routing with the Distributor.
With the distributors running in the cluster and the worker processes coming online, note the Storage queues for each process start to fill up. The more worker threads configured, the more messages will appear in each Storage queue.
While in development, the endpoint configurations probably don't have any
@ symbols in them, in production change to point to the Data Bus queue on the cluster, i.e., for application MyApp and logical queue MyQueue, the worker config looks like this:
<?xml version="1.0" encoding="utf-8" ?> <configuration> <configSections> <!-- Other sections go here --> <section name="MasterNodeConfig" type="NServiceBus.Config.MasterNodeConfig, NServiceBus.Core" /> <section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/> </configSections> <!-- Other config options go here --> <MasterNodeConfig Node="MachineWhereDistributorRuns"/> <UnicastBusConfig"> <MessageEndpointMappings> <!-- regular entries --> </MessageEndpointMappings> </UnicastBusConfig> </configuration>
This article shows how to set up a Windows Failover Cluster and one or more worker node servers to run a scalable, maintainable, and reliable NServiceBus application infrastructure.
Last modified 2016-05-04 09:20:26Z