drbd-8.4.4.tar.gz

# vim: set foldenable foldmethod=indent sw=4 ts=8 : # Copyright 2013 Linbit HA Solutions GmbH # Lars Ellenberg @ linbit.com TODO: someone convert this into proper ascii doc please ;-) ... and draw some pictures ... How crm-fence-peer.sh, pacemaker, and the OCF Linbit DRBD resource agent are supposed to work together. Two node cluster is the trickier one, because it has not real quorum. Relative Timeouts --dc-timeout > dead-time resp. stonith-timeout if stonith enabled, --timeout >= --dc-timeout if no stonith, then timeout may be small. Pacemaker operations timeouts monitor and promote action timeout > max(dc_timeout, timeout) Node reboot, possibly because of crash or stonith due to communication loss no peer reachable [no delay] crm may decide to elect itself, shoot the peer, and start services. If DRBD peer disk state is known Outdated or worse, DRBD will switch itself to UpToDate, allowing it to be promoted, without further fencing actions. If DRBD peer disk state is DUnknown, DRBD will be only Consistent. In case crm decides to promote this instance, the fence-peer callback runs, finds the peer "unreachable", finds itself Consistent only, does NOT set any constraint, and DRBD refuses to be promoted. CRM will now try in an endless loop to promote this instance. Avoid this by adding param adjust_master_score="0 10 1000 10000" to the DRBD resource definition. no replication link CRM can see both nodes. [delay: crmadmin -S $peer] If currently both nodes are Secondary Consistent, CRM will decide to promote one instance. The fence-peer callback will find the other node still reachable after timeout, and set the constraint. If there is already one Primary, and this is a node rejoining the cluster, there should already be a constraint preventing this node from being promoted. Only Replication link breaks during normal operation Single Primary [delay: crmadmin -S $peer] fence-peer callback finds DC, crmadmin -S confirms peer still "reachable", and sets contraint. Dual Primary both fence-peer callbacks find DC, both see node_state "reachable", optionaly delay for --network-hickup timeout, and if DRBD is still disconnected, both try to set the constraint. Only one succeeds. The loser should probably commit suicide, to reduce the overall recovery time. --suicide-on-failure-if-primary Node crash surviving node is Secondary, [no delay] If not DC, triggers DC election, elects itself. Is DC now. If stonith enabled, shoots the peer. Promotes this node. During promotion, fenc-peer callback finds a DC, and a node_state "unreachable", so sets the constraint "immediately". surviving node is Primary (DC) [delay up to timeout] If stonith enabled, shoots the peer. fence-peer callback finds DC, after some time sees node_state "unreachable", or times out while node_state is still "reachable". Either way still sets the constraint. surviving node is Primary (not DC) [delay up to mac(dc_timeout,timeout)] fence-peer callback loops trying to contact DC. eventually this node is elected DC. If stonith enabled, shoots the peer. Fence-peer callback either times out while no DC is available, thus fails. Make sure you chose a suitable --dc-timeout. Or it finds the other node "unreachable", and sets the constraint. Total communication loss To the single node, this looks like node crash, so see above. The difference is the potential of data divergence. If DRBD was configured for "fencing resource-and-stonith", IO on any Primary is frozen while the fence-peer callback runs. If stonith is enabled, timeouts should be selected so that we are shot while waiting for the DC to confirm node_state "unreachable" of the peer, thus combined with freezing IO, no harmful data diversion can happen at this time. If there is no stonith enabled, data divergence is unavoidable. ==> Multi-Primary *requires* both node level fencing (stonith) AND drbd resource level fencing Again: Multi-Primary REQUIRES stonith enabled and working.