# Example 621 — The quorum was three and two shared a rack `the_quorum_was_three_and_two_shared_a_rack.eml` - Every shard has three replicas on three distinct nodes and needs two to serve. The placement check passes on all of them. What a single rack failure removes is computed below. ## EML ```eml # Self-authored for the EML case corpus (no external origin). Every shard has # three replicas on three distinct nodes and needs two to serve. The placement # check passes on all of them. What a single rack failure removes is computed # below. # # Three replicas with a quorum of two is the right configuration and the # placement rule is enforced, not aspirational: the scheduler refuses to put # two replicas of a shard on one node, and a background auditor re-checks every # shard continuously. Distinct nodes is a real property and it is real here. # # A quorum tolerates failures that are independent. Nodes in one rack share a # power feed and a switch, so they do not fail independently — they fail # together, and "together" is exactly the case a quorum is sized against. # # The placement rule counts nodes. Failures arrive by rack. 4096 => shards 3 => replicas_per_shard 2 => quorum 48 => racks 1180 => shards_with_two_in_one_rack "shards : " + str(shards) ^0 "replicas per shard : " + str(replicas_per_shard) ^0 "quorum : " + str(quorum) ^0 "racks : " + str(racks) ^0 "" ^0 # ---- what the placement rule enforces ---- "the placement rule, checked on every shard" ^0 " shards with " + str(replicas_per_shard) + " replicas : " + str(shards) + " of " + str(shards) ^0 " shards with two on one node : 0" ^0 " scheduler violations : 0" ^0 " auditor findings : 0" ^0 " defects in the placement rule : 0" ^0 "" ^0 " distinct NODES is true of every shard in the cluster" ^0 "" ^0 # ---- the property nothing checks ---- shards - shards_with_two_in_one_rack => shards_across_three_racks int(shards_with_two_in_one_rack * 10000 / shards) => exposed_per_myriad "the same shards, counted by rack" ^0 " spread across three racks : " + str(shards_across_three_racks) ^0 " two replicas in one rack : " + str(shards_with_two_in_one_rack) ^0 " share : " + str(exposed_per_myriad) + " per ten thousand" ^0 "" ^0 " no rule was broken to produce that column" ^0 " it was never a rule" ^0 "" ^0 # ---- one rack goes ---- int(shards_with_two_in_one_rack / racks) => shards_losing_quorum int(shards_across_three_racks / racks) => shards_losing_one_replica "one rack fails" ^0 " shards losing one replica : " + str(shards_losing_one_replica) + ", still have " + str(quorum) + ", still serving" ^0 " shards losing two replicas : " + str(shards_losing_quorum) + ", below quorum, unavailable" ^0 "" ^0 " the second row is the one the configuration was chosen to" ^0 " make empty, and it is the one the placement rule cannot see" ^0 "" ^0 # ---- what each layer reports before the failure ---- "instrument reads" ^0 " replica count " + str(replicas_per_shard) + " on every shard" ^0 " distinct nodes enforced, 0 violations" ^0 " under-replicated shards 0" ^0 " shards below quorum 0" ^0 " shards that WOULD be, per rack not measured" ^0 "" ^0 " the first four are green and each of them is true" ^0 "" ^0 # ---- rack by rack ---- "rack shards losing 1 shards losing 2 unavailable" ^0 for r in [1:4]: " " + str(r) + " " + str(shards_losing_one_replica) + " " + str(shards_losing_quorum) + " yes" ^0 "" ^0 " every rack has the same exposure, because the placement" ^0 " was uniform with respect to the property it optimised" ^0 "" ^0 # ---- the control ---- # # The rule that IS enforced, against what it prevents. Two replicas on one node # is a single process losing a shard's quorum, and that has never happened. "control - is the node rule earning its place" ^0 " shards a single NODE could take below quorum : 0" ^0 " scheduler placements refused for this reason : enforced continuously" ^0 " auditor disagreements with the scheduler : 0" ^0 " defects in the rule : 0" ^0 "" ^0 " the rule is correct, enforced, and audited" ^0 " it is about the wrong unit of failure" ^0 "" ^0 # ---- the null control ---- # # The same three replicas, same quorum, same auditor, with rack as the # placement domain instead of node. Nothing about the replication factor or the # quorum changes. 0 => nc_shards_with_two_in_one_rack "null control - the same configuration placed by rack" ^0 " replicas per shard : " + str(replicas_per_shard) + ", unchanged" ^0 " quorum : " + str(quorum) + ", unchanged" ^0 " two replicas in one rack : " + str(nc_shards_with_two_in_one_rack) ^0 " shards below quorum after one rack fails : " + str(nc_shards_with_two_in_one_rack) ^0 " the redundancy did not increase" ^0 " the domain it is measured over did" ^0 "" ^0 # ---- the rule ---- "what distinct replicas guarantees" ^0 " they are separate copies : yes" ^0 " they fail separately : only in the domain you named" ^0 " and the domain that matters is the one failures arrive in," ^0 " which is decided by power, network and cooling rather than" ^0 " by the scheduler's data model" ^0 "" ^0 "the number to publish is not the replication factor, it is" ^0 "the count of shards that lose quorum to one failure of each" ^0 "kind - node, rack, zone - because those are three different" ^0 "numbers and only the first one is being checked" ^0 "" ^0 "The scheduler enforces three replicas on three distinct nodes on all " + str(shards) ^0 "shards, with 0 violations and 0 auditor findings, and no single node can take" ^0 "any shard below quorum. Counted by rack, " + str(shards_with_two_in_one_rack) + " shards - " + str(exposed_per_myriad) + " per ten" ^0 "thousand - hold two replicas in one rack, so one rack failure takes " + str(shards_losing_quorum) ^0 "shards below quorum while every green indicator above stays green." ^0 ``` ## Python (deterministic transpilation) ```python shards = 4096 replicas_per_shard = 3 quorum = 2 racks = 48 shards_with_two_in_one_rack = 1180 print("shards : " + str(shards)) print("replicas per shard : " + str(replicas_per_shard)) print("quorum : " + str(quorum)) print("racks : " + str(racks)) print("") print("the placement rule, checked on every shard") print(" shards with " + str(replicas_per_shard) + " replicas : " + str(shards) + " of " + str(shards)) print(" shards with two on one node : 0") print(" scheduler violations : 0") print(" auditor findings : 0") print(" defects in the placement rule : 0") print("") print(" distinct NODES is true of every shard in the cluster") print("") shards_across_three_racks = shards - shards_with_two_in_one_rack exposed_per_myriad = int(shards_with_two_in_one_rack * 10000 / shards) print("the same shards, counted by rack") print(" spread across three racks : " + str(shards_across_three_racks)) print(" two replicas in one rack : " + str(shards_with_two_in_one_rack)) print(" share : " + str(exposed_per_myriad) + " per ten thousand") print("") print(" no rule was broken to produce that column") print(" it was never a rule") print("") shards_losing_quorum = int(shards_with_two_in_one_rack / racks) shards_losing_one_replica = int(shards_across_three_racks / racks) print("one rack fails") print(" shards losing one replica : " + str(shards_losing_one_replica) + ", still have " + str(quorum) + ", still serving") print(" shards losing two replicas : " + str(shards_losing_quorum) + ", below quorum, unavailable") print("") print(" the second row is the one the configuration was chosen to") print(" make empty, and it is the one the placement rule cannot see") print("") print("instrument reads") print(" replica count " + str(replicas_per_shard) + " on every shard") print(" distinct nodes enforced, 0 violations") print(" under-replicated shards 0") print(" shards below quorum 0") print(" shards that WOULD be, per rack not measured") print("") print(" the first four are green and each of them is true") print("") print("rack shards losing 1 shards losing 2 unavailable") for r in range(1, 5): print(" " + str(r) + " " + str(shards_losing_one_replica) + " " + str(shards_losing_quorum) + " yes") print("") print(" every rack has the same exposure, because the placement") print(" was uniform with respect to the property it optimised") print("") print("control - is the node rule earning its place") print(" shards a single NODE could take below quorum : 0") print(" scheduler placements refused for this reason : enforced continuously") print(" auditor disagreements with the scheduler : 0") print(" defects in the rule : 0") print("") print(" the rule is correct, enforced, and audited") print(" it is about the wrong unit of failure") print("") nc_shards_with_two_in_one_rack = 0 print("null control - the same configuration placed by rack") print(" replicas per shard : " + str(replicas_per_shard) + ", unchanged") print(" quorum : " + str(quorum) + ", unchanged") print(" two replicas in one rack : " + str(nc_shards_with_two_in_one_rack)) print(" shards below quorum after one rack fails : " + str(nc_shards_with_two_in_one_rack)) print(" the redundancy did not increase") print(" the domain it is measured over did") print("") print("what distinct replicas guarantees") print(" they are separate copies : yes") print(" they fail separately : only in the domain you named") print(" and the domain that matters is the one failures arrive in,") print(" which is decided by power, network and cooling rather than") print(" by the scheduler's data model") print("") print("the number to publish is not the replication factor, it is") print("the count of shards that lose quorum to one failure of each") print("kind - node, rack, zone - because those are three different") print("numbers and only the first one is being checked") print("") print("The scheduler enforces three replicas on three distinct nodes on all " + str(shards)) print("shards, with 0 violations and 0 auditor findings, and no single node can take") print("any shard below quorum. Counted by rack, " + str(shards_with_two_in_one_rack) + " shards - " + str(exposed_per_myriad) + " per ten") print("thousand - hold two replicas in one rack, so one rack failure takes " + str(shards_losing_quorum)) print("shards below quorum while every green indicator above stays green.") ``` ## stdout (executed) ```text shards : 4096 replicas per shard : 3 quorum : 2 racks : 48 the placement rule, checked on every shard shards with 3 replicas : 4096 of 4096 shards with two on one node : 0 scheduler violations : 0 auditor findings : 0 defects in the placement rule : 0 distinct NODES is true of every shard in the cluster the same shards, counted by rack spread across three racks : 2916 two replicas in one rack : 1180 share : 2880 per ten thousand no rule was broken to produce that column it was never a rule one rack fails shards losing one replica : 60, still have 2, still serving shards losing two replicas : 24, below quorum, unavailable the second row is the one the configuration was chosen to make empty, and it is the one the placement rule cannot see instrument reads replica count 3 on every shard distinct nodes enforced, 0 violations under-replicated shards 0 shards below quorum 0 shards that WOULD be, per rack not measured the first four are green and each of them is true rack shards losing 1 shards losing 2 unavailable 1 60 24 yes 2 60 24 yes 3 60 24 yes 4 60 24 yes every rack has the same exposure, because the placement was uniform with respect to the property it optimised control - is the node rule earning its place shards a single NODE could take below quorum : 0 scheduler placements refused for this reason : enforced continuously auditor disagreements with the scheduler : 0 defects in the rule : 0 the rule is correct, enforced, and audited it is about the wrong unit of failure null control - the same configuration placed by rack replicas per shard : 3, unchanged quorum : 2, unchanged two replicas in one rack : 0 shards below quorum after one rack fails : 0 the redundancy did not increase the domain it is measured over did what distinct replicas guarantees they are separate copies : yes they fail separately : only in the domain you named and the domain that matters is the one failures arrive in, which is decided by power, network and cooling rather than by the scheduler's data model the number to publish is not the replication factor, it is the count of shards that lose quorum to one failure of each kind - node, rack, zone - because those are three different numbers and only the first one is being checked The scheduler enforces three replicas on three distinct nodes on all 4096 shards, with 0 violations and 0 auditor findings, and no single node can take any shard below quorum. Counted by rack, 1180 shards - 2880 per ten thousand - hold two replicas in one rack, so one rack failure takes 24 shards below quorum while every green indicator above stays green. ``` ## Round-trip `ok: true` — round-trip fixpoint reached (python1 == python2) ## Trace event types eml:run:start · eml:assign · eml:output · eml:run:done