The MongoDB Database

# Replication Internals Replication is the set of systems used to continuously copy data from a primary server to secondary servers so if the primary server fails a secondary server can take over soon. This process is intended to be mostly transparent to the user, with drivers taking care of routing queries to the requested replica. Replication in MongoDB is facilitated through [**replica sets**](https://docs.mongodb.com/manual/replication/). Replica sets are a group of nodes with one primary and multiple secondaries. The primary is responsible for all writes. Users may specify that reads from secondaries are acceptable via [`setSecondaryOk`](https://docs.mongodb.com/manual/reference/method/Mongo.setSecondaryOk/) or through [**read preference**](#read-preference), but they are not by default. # Steady State Replication The normal running of a replica set is referred to as steady state replication. This is when there is one primary and multiple secondaries. Each secondary is replicating data from the primary, or another secondary off of which it is **chaining**. ## Life as a Primary ### Doing a Write When a user does a write, all a primary node does is apply the write to the database like a standalone would. The one difference from a standalone write is that replica set nodes have an `OpObserver` that inserts a document to the **oplog** whenever a write to the database happens, describing the write. The oplog is a capped collection called `oplog.rs` in the `local` database. There are a few optimizations made for it in WiredTiger, and it is the only collection that doesn't include an \_id field. If a write does multiple operations, each will have its own oplog entry; for example, inserts with implicit collection creation create two oplog entries, one for the `create` and one for the `insert`. These entries are rewritten from the initial operation to make them idempotent; for example, updates with `$inc` are changed to use `$set`. Secondaries drive oplog replication via a pull process. Writes can also specify a [**write concern**](https://docs.mongodb.com/manual/reference/write-concern/). If a command includes a write concern, the command will just block in its own thread until the oplog entries it generates have been replicated to the requested number of nodes. The primary keeps track of how up-to-date the secondaries are to know when to return. A write concern can specify a number of nodes to wait for, or **majority**. If **majority** is specified, the write waits for that write to be in the **committed snapshot** as well, so that it can be read with `readConcern: { level: majority }` reads. (If this last sentence made no sense, come back to it at the end). ### Default Write Concern If a write operation does not explicitly specify a write concern, the server will use a default write concern. This default write concern will be defined by either the **cluster-wide write concern**, explicitly set by the user, or the **implicit default write concern**, implicitly set by the server based on replica set configuration. #### Cluster-Wide Write Concern Users can set the cluster-wide write concern (CWWC) using the [`setDefaultRWConcern`](https://docs.mongodb.com/manual/reference/command/setDefaultRWConcern/) command. Setting the CWWC will cause the implicit default write concern to no longer take effect. Once a user sets a CWWC, we disallow unsetting it. The reasoning behind this is explored in the section [Implicit Default Write Concern and Sharded Clusters](#implicit-default-write-concern-and-sharded-clusters). On sharded clusters, the CWWC will be stored on config servers. Shard servers themselves do not store the CWWC. Instead, mongos polls the config server and applies the default write concern to requests it forwards to shards. #### Implicit Default Write Concern If there is no cluster-wide default write concern set, the server will set the default. This is known as the implicit default write concern (IDWC). For most cases, the IDWC will default to `{w: "majority")`. The IDWC is calculated on startup using the **Default Write Concern Formula (DWCF)**: `implicitDefaultWriteConcern = if ((#arbiters > 0) AND (#non-arbiters <= majority(#voting nodes)) then {w:1} else {w:majority}` This formula specifies that for replica sets with arbiters, we want to ensure that we set the implicit default to a value that the set can satisfy in the event of one data-bearing node going down. That is, the number of data-bearing nodes must be strictly greater than the majority of voting nodes for the set to set `{w: "majority"}`. For example, if we have a PSA replica set, and the secondary goes down, the primary cannot successfully acknowledge a majority write as the majority for the set is two nodes. However, the primary will remain primary with the arbiter's vote. In this case, the DWCF will have preemptively set the IDWC to `{w: 1}` so the user can still perform writes to the replica set. #### Implicit Default Write Concern and Sharded Clusters For sharded clusters, the implicit default write concern will always be `{w: "majority"}`. As mentioned above, mongos will send the default write concern with all requests that it forwards to shards, which means the default write concern on shards will always be consistent in the cluster. We don't want to specify `{w: "majority"}` for shard replica sets that can keep a primary due to an arbiter's vote, but lose the ability to acknowledge majority writes if a majority of data-bearing nodes goes down. So if the result of the DWCF for any replica set in the cluster is `{w: 1}`, we require the cluster to set a CWWC. Once set, we disallow unsetting it so we can prevent PSA shards from implicitly defaulting to `{w: "majority"}` for reasons mentioned above. However, if a user decides to set the CWWC to `{w: "majority"}` for a PSA set, they may do so. We assume that in this case the user understands the tradeoffs they are making. We will fassert shard servers on startup if no CWWC is set and the result of the default write concern formula is `{w: 1}`. Similarly, we will also fail any `addShard` command that attempts to add a shard replica set with a default write concern of `{w: 1}` when CWWC is unset. This is because we want to maintain a consistent implicit default of `{w: "majority"}` across the cluster, but we do not want to specify that for PSA sets for reasons listed above. #### Replica Set Reconfigs and Default Write Concern A replica set reconfig will recalculate the default write concern using the Default Write Concern Formula if CWWC is not set. If the new value of the implicit default write concern is different from the old value, we will fail the reconfig. Users must set a CWWC before issuing a reconfig that would change the IDWC. #### Force Reconfigs As an important note, we will also fail force reconfigs that may change the IDWC. In cases where a replica set is facing degraded performance and cannot satisfy a majority write concern needed to set the CWWC, users can run `setDefaultRWConcern` with write concern `{w: 1}` instead of making it a majority write so that setting CWWC does not get in the way of being able to do a force reconfig. #### Code References - [The definition of an Oplog Entry](https://github.com/mongodb/mongo/blob/r6.2.0/src/mongo/db/repl/oplog_entry.idl) - [Upper layer uses OpObserver class to write Oplog](https://github.com/mongodb/mongo/blob/r6.2.0/src/mongo/db/op_observer/op_observer.h#L112), for example, [it is helpful to take a look at ObObserverImpl::logOperation()](https://github.com/mongodb/mongo/blob/r6.2.0/src/mongo/db/op_observer/op_observer_impl.cpp#L114) - [repl::logOplogRecords() is a common function to write Oplogs into Oplog Collection](https://github.com/mongodb/mongo/blob/r7.1.0/src/mongo/db/repl/oplog.cpp#L440) - [WriteConcernOptions is filled in extractWriteConcern()](https://github.com/mongodb/mongo/blob/r6.2.0/src/mongo/db/write_concern.cpp#L71) - [Upper leve