ServiceControl version 6.20 changes the default RavenDB search engine from Corax to Lucene to address performance and stability issues on large database instances. Databases created by version 6.20 use Lucene. Existing error and audit databases are not modified by the upgrade and continue to use Corax. Migrating existing databases to Lucene is recommended, see Should existing databases be migrated?. Monitoring instances are not affected.
New databases use the Lucene search engine
Databases created by version 6.20 or later use the Lucene search engine for their indexes instead of Corax. Lucene indexes are smaller, use less memory, and perform better for the ServiceControl workload of continuous message ingestion and deletion combined with occasional queries.
Databases created by earlier versions keep the search engine they were created with, typically Corax. The upgrade does not switch existing databases to Lucene automatically because doing so triggers a full rebuild of all indexes. On very large databases the rebuild can take days depending on the available compute, and ingestion and indexing rates are degraded while it runs. The migration must therefore be planned separately for each environment.
Indexes still using Corax are reported
After upgrading, error and audit instances whose indexes still use Corax:
- log a warning at every start-up listing the affected indexes
- report a failing custom check,
Error Database Search EngineorAudit Database Search Engine, in ServicePulse
The instance continues to operate on Corax, but the check keeps failing until all indexes have been migrated to Lucene. To migrate existing indexes and clear the check, follow the migration procedure during a maintenance window.
Should existing databases be migrated?
Yes. Migration is recommended to avoid performance and stability issues as the database grows. Migrate as soon as possible when an instance shows one or more of the following symptoms:
- Frequent or persistent index lag, reported by the stale indexes custom check
- High RAM utilization or RavenDB dirty memory warnings
- High CPU utilization caused by indexing
- Corrupted indexes or a lengthy database recovery after a service shutdown, see Audit instances: Corrupted indexes or corrupted database after a service shutdown
Instances without these symptoms should be migrated in the next planned maintenance window. Migrate development and test instances first to estimate the rebuild duration for production. See Should existing indexes be migrated? for guidance on planning the migration and the migration procedure for the steps.
Previously migrated and locked indexes
Indexes that were already migrated to Lucene and locked, as described in the troubleshooting guide, remain locked and on Lucene after the upgrade. No action is required for them; the custom check only reports indexes that still use Corax.