Configuring RavenDB DTC

Component: RavenDB Persistence
NuGet Package NServiceBus.RavenDB (3.x)
Target NServiceBus Version: 5.x

In order to provide reliable support for distributed transactions in RavenDB, a custom DocumentStore must be provided and configured to uniquely identify the endpoint to the Distributed Transaction Coordinator (DTC) and provide a storage location for uncommitted transaction recovery.

Definitions

Support for the Distributed Transaction Coordinator (DTC) in RavenDB is dependent upon the ResourceManagerId and TransactionRecoveryStorage settings.

When using the Outbox feature instead of distributed transactions, these settings are not required and should not be used. See Outbox with RavenDB persistence for more information.

ResourceManagerId

The ResourceManagerId is a Guid that uniquely identifies a transactional resource on a machine, and must be both unique system wide, and stable (deterministic) between restarts of the process utilizing the DTC. If more than one RavenDB document store attempts to use the same ResourceManagerId, it can result in the following error during a commit operation:

"A resource manager with the same identifier is already registered with the specified transaction coordinator"

It is possible to configure the ResourceManagerId from a RavenDB connection string, however this is not recommended as this method does not allow for the configuration of a suitable TransactionRecoveryStorage.

TransactionRecoveryStorage

To guard against the loss of a committed transaction, RavenDB requires a storage location to persist data in the event of a process crash immediately following a transaction commit.

RavenDB provides transaction recovery storage options for volatile (in-memory) storage, IsolatedStorage, and local directory storage. LocalDirectoryTransactionRecoveryStorage is recommended as the only stable and reliable option.

Configuring safe settings

In order to configure safe settings for production use of RavenDB, construct a DocumentStore instance and configure the ResourceManagerId and TransactionRecoveryStorage properties as shown in the following code:

// Value must uniquely identify endpoint on the machine and remain stable on process restarts
var resourceManagerId = new Guid("xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx");

var dtcRecoveryBasePath = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData);
var recoveryPath = Path.Combine(dtcRecoveryBasePath, "NServiceBus.RavenDB", resourceManagerId.ToString());

var store = new DocumentStore
{
    Url = UrlToRavenDB,
    ResourceManagerId = resourceManagerId,
    TransactionRecoveryStorage = new LocalDirectoryTransactionRecoveryStorage(recoveryPath)
};
store.Initialize();

var busConfiguration = new BusConfiguration();
var persistence = busConfiguration.UsePersistence<RavenDBPersistence>();
persistence.SetDefaultDocumentStore(store);

In order to provide transaction safety, the following must be observed:

  • documentStore.ResourceManagerId must be constant across process restarts, and uniquely identify the process running on the machine. Do not use Guid.NewGuid(). Otherwise, the transaction recovery process will fail when the process restarts.
  • documentStore.TransactionRecoveryStorage must be set to an instance of LocalDirectoryTransactionRecoveryStorage, configured to a directory that is constant across process restarts, and writable by the process.

Configuration by convention

It can be cumbersome to manage these settings for multiple endpoints, so it is preferable to create a convention that will calculate a unique ResourceManagerId, and then use this value to create a storage location for TransactionRecoveryStorage as well.

public class EndpointConfig :
    IConfigureThisEndpoint
{
    public void Customize(BusConfiguration configuration)
    {
        var persistence = configuration.UsePersistence<RavenDBPersistence>();
        persistence.SetRavenDtcSettings("MyEndpointName");
    }
}

static class RavenDtcExtensions
{
    public static void SetRavenDtcSettings(this PersistenceExtentions<RavenDBPersistence> persistenceConfig, string endpointName)
    {
        // Calculate a ResourceManagerId unique to this endpoint using just endpoint name
        // Not suitable for side-by-side installations!
        var resourceManagerId = DeterministicGuidBuilder(endpointName);

        // Calculate a DTC transaction recovery storage path including the ResourceManagerId
        var programDataPath = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData);
        var txRecoveryPath = Path.Combine(programDataPath, "NServiceBus.RavenDB", resourceManagerId.ToString());
        var store = new DocumentStore
        {
            // RavenServerUrl
            Url = "http://localhost:8083",
            DefaultDatabase = endpointName,
            ResourceManagerId = resourceManagerId,
            TransactionRecoveryStorage = new LocalDirectoryTransactionRecoveryStorage(txRecoveryPath)
        };
        persistenceConfig.SetDefaultDocumentStore(store);
    }

    static Guid DeterministicGuidBuilder(string input)
    {
        // use MD5 hash to get a 16-byte hash of the string
        using (var provider = new MD5CryptoServiceProvider())
        {
            var inputBytes = Encoding.UTF8.GetBytes(input);
            var hashBytes = provider.ComputeHash(inputBytes);
            // generate a guid from the hash:
            return new Guid(hashBytes);
        }
    }
}

It is important to keep a few things in mind when determining a convention.

The string provided to DeterministicGuidBuilder will define the ResourceManagerId, and thus the identity of the endpoint. This string value must then be unique within the scope of the server. The EndpointName or LocalAddress provide attractive options as they are normally unique per server.

An exception is side-by-side deployment, where an old version and new version of the same endpoint run concurrently, processing messages from the same queue, in order to verify the new version and enable rollback to the previous version. In this case using EndpointName or LocalAddress would result in duplicate ResourceManagerId values on the same server, which would lead to DTC exceptions. In this case, each release of the endpoint must be versioned (for example, with the AssemblyFileVersion attribute), and the endpoint's version must be included in the string provided to DeterministicGuidBuilder.

The exact convention used must be appropriately defined to match the deployment strategy in use for the endpoint. If a new endpoint version is deployed by taking the old one offline, replacing the binaries, and then starting it back up, fixed values should be used at all times so that the new version can recover transactions for the old version. If endpoint versions are run side-by-side, then independent values should be used for each version, and old versions should be safely decommissioned when they are shut down.

Safely decommissioning endpoints

If an endpoint terminates unexpectedly for any reason, data can be left behind in the transaction recovery storage which represents transactions not yet committed to the RavenDB database, but which may be recovered when the endpoint restarts.

In order to avoid losing data, it is important to ensure that endpoints that are decommissioned are taken offline gracefully, i.e. stopped at the request of the Service Control Manager, and not terminated from the Task Manager. Then, the transaction recovery storage directory should be inspected to ensure it is empty. If any files remain, the endpoint should be started again briefly so that RavenDB can perform a recovery.

Related Articles


Last modified