用于公开共享支持工具___下载.zip

# MongoDB Validation Support Tools * [Pre-requisites](#pre-requisites) * [Remediation Steps](#remediation-steps) * [Checking for Local Inconsistencies](#checking-for-local-inconsistencies) * [Checking for Replica Set Inconsistencies](#checking-for-replica-set-inconsistencies) ## Warning The scripts provided here are best run by those experienced with MongoDB, ideally with guidance from MongoDB Technical Support. They are provided without warranty or guarantee of any kind (see disclaimer below). If you do not have access to MongoDB Technical Support, consider reaching out to our community at [community.mongodb.com](community.mongodb.com) to ask questions. Some adaptation of these scripts may be required for your use-case. If you are using these scripts on your own, we strongly recommend: * reading all instructions before attempting any action, and ensuring you are prepared for all steps. * taking regular backups of the [dbpaths](https://docs.mongodb.com/manual/core/backups/#back-up-by-copying-underlying-data-files) of each node, and working off of backups as much as possible. * reviewing the scripts themselves to understand their behavior. * testing this process and all scripts on a copy of the environment it is to be run. * for sharded clusters, disabling the balancer. # Summary This process provides for identifying and resolving data inconsistencies between MongoDB replica set members, above and beyond what is currently supported by existing and documented `validate` and `dbHash` commands. The primary use-cases for this process are when: * a MongoDB replica set running on system architectures with weak memory ordering (ARM64 and POWER) is affected by [WT-10461](https://jira.mongodb.org/browse/WT-10461) which can cause affected nodes to store documents or index entries out of order, leading to inconsistencies and improperly sorted or incomplete query results on versions 4.2.0-4.2.23, 4.4.0-4.4.18, 5.0.0-5.0.14, 6.0.0-6.0.4 or rapid release versions 6.1.0-6.2.0. * a MongoDB replica set has undergone an unsafe upgrade path identified by [WT-8395](https://jira.mongodb.org/browse/WT-8395), a defect which can introduce data inconsistencies when upgrading from versions 4.4.3-4.4.4 directly to 4.4.8-4.4.10 or 5.0.2-5.0.5. * all nodes of a MongoDB replica set have been separately impacted by [WT-7995](https://jira.mongodb.org/browse/WT-7995), [WT-7984](https://jira.mongodb.org/browse/WT-7984) on versions 4.4.2-4.4.8 and 5.0.0-5.0.2, and validate() output alone is not sufficient to rule out document-level inconsistencies between nodes. **Warning**: Use this process with care and perform backups before proceeding. Be certain about what bug you are responding to, and be familiar with the details of that bug. This process involves scripts which cause load on and perform writes to your cluster. In particular, `repair_checked_documents.py` issues delete operations against user collections, depending on options provided when it is run. ## Remediation components The overall remediation process relies on a combination of `mongod` database commands and scripts in this directory. * The [validate database command](https://docs.mongodb.com/manual/reference/method/db.collection.validate/), optionally scripted for all databases/collections in `validate.js`. * The [dbCheck database command](https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/dbcheck.idl), optionally scripted for all databases/collections in `dbcheck.js`: * a new server command, run per collection. * on each replica set node, generates hashes at points in time for ranges of documents. * outputs results to a non-replicated collection on each node (in each node's `local` database). * secondaries report inconsistencies with the primary. * The `scan_checked_replset.js` javascript script for the legacy mongo shell ("mongo"): * consumes dbCheck output on each node. * uses specific ranges of inconsistencies reported by dbCheck on secondaries to identify specific differing documents. * writes results to new databases/collections in the cluster. * each collection on each node includes that node's version of the differing document (or no document, if that node is missing a document). * The `repair_checked_documents.py` python3 script: * consumes scan_checked_replset.js script output. * resolves inconsistencies by overwriting documents according to a consensus between replica set nodes, depending on the strategy specified. * **Importantly**: `repair_checked_documents.py` restores the state of inconsistent documents as of the time they were backed up by `scan_checked_replset.js`. If the system is receiving writes, between the start of `scan_checked_replset.js` and `repair_checked_documents.py` those writes could be undone when `repair_checked_documents.py` remediates based on the state of the document at the time it was archived during `scan_checked_replset.js`. ## Pre-requisites To perform this process, you will need to: * Upgrade your cluster to MongoDB 4.4.11+ or 5.0.6+. * Create on your cluster a database user that can: * read and write to any database * read the 'local.system.healthlog' collection * read and write to the '__corruption_repair.unhealthyRanges' collection * issue the following commands: * applyOps * listDatabases * listCollections * validate * dbCheck * Note that additional explicit privileges to read and write to system collections may be needed * Prepare a machine for script execution with: * The (legacy) mongo shell * python3 * the (latest) pymongo MongoDB driver * Network access to all members of the replica set being remediated ``` db.adminCommand({ createRole: "remediationAdmin", roles: [ "clusterManager", "clusterMonitor", "readWriteAnyDatabase"], privileges: [ { resource: {cluster: true}, actions: ["applyOps", "listDatabases", "serverStatus"] }, { resource: {db: "local", collection: "system.healthlog"}, actions: ["find"] }, { resource: {db: "__corruption_repair", collection: "unhealthyRanges"}, actions: ["find", "insert", "update", "remove", "createCollection", "dropCollection", "createIndex", "dropIndex"] }, { resource: { anyResource: true }, actions: ["listCollections", "validate"] }, ] }); db.getSiblingDB("admin").createUser({user: "remediate", pwd: "<password>", roles: [{role: "remediationAdmin", db: "admin"}]}) ``` ## Important notes: False positives (inconsistencies that are not a concern) can occur on collections with TTL indexes, because MongoDB does not synchronize the removal of TTL documents between nodes. # Remediation Steps The overall high level process is: 1. Use a combination of the `validate` command and the `reIndex` command to ensure the `_id` index is consistent in all collections on all nodes. 1. For each collection in the cluster: 1. Run `dbCheck` to generate initial inconsistency information. 1. Stop writes to the collection. 1. Run `scan_checked_replset.js` to generate specific inconsistency information. 1. Run `repair_checked_documents.py` to resolve the identified inconsistencies, or resolve them manually. 1. Optionally, re-run `dbCheck` to confirm consistency 1. Resume writes to the collection. ## Checking for Local Inconsistencies Note: The [db.collection.validate()](https://www.mongodb.com/docs/manual/reference/method/db.collection.validate/) method is potentially resource intensive and may impact the performance of your MongoDB instance, particularly on larger data sets. Please review performance considerations [here](https://www.mongodb.com/docs/manual/reference/method/db.collection.validate/#performance) before proceeding. For each node, run `validate` on all collections on all nodes. [`validate.js`](https://github.com/mongodb/support-tools/blob/master/replset-consistency/validate.js) runs validate on every collection of a given node. On each node with validation issues, any “m