Migration from Azure Table Persistence

For existing system running in Azure and using Azure Table Persistence, where a migration to Azure Cosmos DB SQL API is desired, a multi-step migration process is recommended, using Particular and Azure Cosmos DB tools.

For each Azure Table Persistence table containing saga data, the following four major steps must be performed:

1. Export saga data from Table Storage
2. Inspect exported saga data for quality
3. Import data into Cosmos DB
4. Inspect imported saga data for quality

Prior to starting the endpoint, configure the endpoint to use migration mode.

Export data

To export data from Table Storage, a .NET tool provided by Particular is required. Install the tool using the following command:

dotnet tool install Particular.AzureTable.Export --tool-path <installation-path> --version 1.*

Once installed, the particular-azuretable-export command line tool will be available for use at the installation path used earlier. For example:

particular-azuretable-export -c "UseDevelopmentStorage=true" -s OrderSagaData

<exported-path> is the destination path where the particular-azuretable-export.exe will be found.

Once the tool is executed, saga data for the selected saga data table will be stored in the current working directory as a sub-folder named after the saga data class with each saga data record as individual JSON files. These files can be inspected and imported into Cosmos DB using the instructions below.

Export tool options

-c | --connectionstring: Set the connection string to the Table Storage
-s | --sagadataname: The saga data class name without the namespace (e.g. OrderSagaData) of the saga data to export. This will be used to derive the table storage name.
-i | --ignore-updates: Allow use of the tool even if a newer version is available.
-v | --verbose: Enable verbose output.
--version: Show the current version of the tool.

Updating the tool

The tool can be updated with the following command:

dotnet tool update --tool-path <installation-path> Particular.AzureTable.Export --version 1.*

Exported saga ID

As part of the export process a new saga ID is generated for each saga that is compliant with the NServiceBus.Persistence.CosmosDB package. The original saga ID is stored in the _NServiceBus-Persistence-Metadata.SagaDataContainer-MigratedSagaId metadata property embedded in the exported saga data.

Export tool customizations

For customers that require a certain degree of customization, the .NET tool code is publicly available at https://github.com/Particular/Particular.AzureTable.Export and can be forked.

Import data

The exported saga data JSON files can be imported into Cosmos DB using the Data migration tool provided by Microsoft. The import tool provides both a UI and a command line option.

For example, to import a single saga data table called OrderSagaData originally exported to the location C:\path\to\OrderSagaData, the following command is required:

dt.exe /s:JsonFile /s.Files:C:\\path\\to\\OrderSagaData\\*.* /t:DocumentDB /t.IdField:id /t.DisableIdGeneration /t.Collection:OrderSagaData /t.PartitionKey:/id /t.ConnectionString:AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==;database=CosmosDBPersistence

where the following parameters must be adjusted:

s.Files: The file path to the folder containing specific saga data JSON files exported from Table Storage.
t.Collection: The Cosmos DB collection to be used for the imported data.
t.ConnectionString: The Cosmos DB connection string including Cosmos DB database name (e.g;database=MyCosmosDb).

Data inspection

Due to the limited types supported by Azure Storage Tables, some types are stored in the table by the Azure Storage persister as serialized JSON strings. The export tool makes a best effort to re-serialize these values for import into Cosmos DB. As a result, the data can and should be inspected for quality both before and after the import. The migrated endpoint and all saga types migrated should be thoroughly tested before moving into production to ensure the migration is correct.

Using migration mode

Auto-correlated messages include the saga ID in the message headers. For unprocessed auto-correlated messages sent prior to migration, this may result in a saga not found error, since the saga ID contained in the message headers is not the new saga ID expected by the Cosmos DB persister.

By enabling migration mode only for auto-correlated messages, the saga persister will attempt to query the collection using the original saga ID in the saga metadata when the saga is not found. Messages explicitly mapped using the ConfigureHowToFindSaga method do not require the additional query.

To enable migration mode use the configuration option:

var persistence = endpointConfiguration.UsePersistence<CosmosPersistence>();
var sagas = persistence.Sagas();
sagas.EnableMigrationMode();

Migration mode can be disabled when all auto-correlated pre-migration messages have been processed, or when all migrated sagas have ended.

Counting migrated sagas

Perform the following query on each collection containing migrated saga data:

SELECT COUNT(c["_NServiceBus-Persistence-Metadata"]["SagaDataContainer-MigratedSagaId"]) AS SagasInMigration FROM c

This will produce a result that returns the number of migrated sagas still open in the collection:

[
    {
        "SagasInMigration": 2
    }
]

Migration mode can be disabled when the query result returns 0 for all collections that contained migrated saga data.