# Example 924 — The sixteenth node made the cluster slower `the_sixteenth_node_made_the_cluster_slower.eml` - A cluster of eight nodes is doubled to sixteen to raise throughput, every node is identical and fully working, and the sixteen-node cluster does less work per second than the eight. What each added node costs the others is computed below. ## EML ```eml # Self-authored for the EML case corpus (no external origin). A cluster of # eight nodes is doubled to sixteen to raise throughput, every node is # identical and fully working, and the sixteen-node cluster does less work per # second than the eight. What each added node costs the others is computed # below. # # The expansion is careful. Every node is the same hardware; every node is # healthy and busy; the load balancer spreads work evenly; and the intent is # exactly 'twice the nodes, more throughput'. # # Each node must keep its shared state coherent with every other, and that # coordination grows with the number of pairs of nodes, so past a point the # nodes spend more of each second agreeing than working - sixteen agree more # than eight and do less. 8 => nodes_before 16 => nodes_after 500 => contention_per_extra_node_per_myriad 100 => coherence_per_pair_per_myriad nodes_before - 1 => extra_nodes_before nodes_after - 1 => extra_nodes_after 10000 + contention_per_extra_node_per_myriad * extra_nodes_before + coherence_per_pair_per_myriad * nodes_before * extra_nodes_before => time_per_unit_before_per_myriad 10000 + contention_per_extra_node_per_myriad * extra_nodes_after + coherence_per_pair_per_myriad * nodes_after * extra_nodes_after => time_per_unit_after_per_myriad int(nodes_before * 100 * 10000 / time_per_unit_before_per_myriad) => throughput_before_hundredths int(nodes_after * 100 * 10000 / time_per_unit_after_per_myriad) => throughput_after_hundredths throughput_before_hundredths - throughput_after_hundredths => throughput_lost_by_doubling_hundredths coherence_per_pair_per_myriad * nodes_after * extra_nodes_after => coherence_cost_at_sixteen_per_myriad coherence_per_pair_per_myriad * nodes_before * extra_nodes_before => coherence_cost_at_eight_per_myriad "nodes before / after : " + str(nodes_before) + " / " + str(nodes_after) + ", all identical and healthy" ^0 "contention per extra node : " + str(contention_per_extra_node_per_myriad) + " per ten thousand of a unit's time" ^0 "coherence per pair of nodes : " + str(coherence_per_pair_per_myriad) + " per ten thousand" ^0 "" ^0 "coherence cost at " + str(nodes_before) + " : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand" ^0 "coherence cost at " + str(nodes_after) + " : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand" ^0 "throughput at " + str(nodes_before) + " : " + str(throughput_before_hundredths) + " hundredths of one node's" ^0 "throughput at " + str(nodes_after) + " : " + str(throughput_after_hundredths) + " hundredths of one node's" ^0 "throughput lost by doubling : " + str(throughput_lost_by_doubling_hundredths) + " hundredths" ^0 "" ^0 # ---- what the expansion verified ---- "the doubled cluster" ^0 " hardware : identical on every node" ^0 " health : every node up and busy" ^0 " balancing : even" ^0 " intent : twice the nodes, more throughput" ^0 " idle or failed nodes : 0" ^0 " verdict : SIXTEEN HEALTHY NODES, ALL BUSY" ^0 "" ^0 " sixteen identical, healthy, evenly loaded nodes is the" ^0 " part done right here, and it is why the drop is not a" ^0 " bad node or an uneven split" ^0 "" ^0 # ---- what each node costs the others ---- "coordination" ^0 " what shared state requires : every node agreeing with" ^0 " every other" ^0 " how that grows : with the pairs, " + str(nodes_before) + " times " + str(extra_nodes_before) + " then " + str(nodes_after) + " times " + str(extra_nodes_after) ^0 " at " + str(nodes_before) + " nodes : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand of every unit spent agreeing" ^0 " at " + str(nodes_after) + " nodes : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand" ^0 " so each node at " + str(nodes_after) + " : does much less than a node at " + str(nodes_before) + ", and" ^0 " twice as many of them do less in total" ^0 "" ^0 # ---- what the operators got ---- "the throughput" ^0 " eight nodes : " + str(throughput_before_hundredths) + " hundredths" ^0 " sixteen nodes : " + str(throughput_after_hundredths) + " hundredths" ^0 " is any node broken : no" ^0 " is more nodes more work : up to the point where agreeing" ^0 " costs more than the node adds, and sixteen is past it" ^0 "" ^0 # ---- null control ---- # The same cluster with the shared state partitioned so nodes coordinate only # within small groups, removing the pairwise term. 385 => nc_throughput_at_sixteen_with_pairwise_coherence_hundredths 914 => nc_throughput_at_sixteen_with_coherence_partitioned_hundredths 1 => nc_sixteen_beats_eight_once_partitioned "null control - partition the shared state" ^0 " throughput at sixteen, pairwise coherence : " + str(nc_throughput_at_sixteen_with_pairwise_coherence_hundredths) + " hundredths" ^0 " throughput at sixteen, coherence partitioned : " + str(nc_throughput_at_sixteen_with_coherence_partitioned_hundredths) + " hundredths" ^0 " sixteen beats eight once partitioned : " + str(nc_sixteen_beats_eight_once_partitioned) ^0 " no node changed; the nodes stopped having to agree with" ^0 " every other" ^0 "" ^0 # ---- the rule ---- "what sixteen healthy, evenly loaded nodes guarantee" ^0 " twice the machines are working : exactly, identical," ^0 " healthy, balanced" ^0 " twice the machines do more : not addressed; coherence" ^0 " grows with the pairs, so sixteen nodes spend " + str(coherence_cost_at_sixteen_per_myriad) + " per ten" ^0 " thousand of each unit agreeing and deliver " + str(throughput_after_hundredths) + " hundredths" ^0 " where eight delivered " + str(throughput_before_hundredths) ^0 "" ^0 "a machine added to a group that must agree adds a worker and a voice, and" ^0 "the voices grow faster than the workers; past the point where a new node's" ^0 "talk costs more than its work, adding capacity subtracts throughput" ^0 "" ^0 "Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the" ^0 "nodes must keep shared state coherent pairwise, and sixteen have " + str(coherence_cost_at_sixteen_per_myriad) + " per ten" ^0 "thousand of every unit tied up agreeing against " + str(coherence_cost_at_eight_per_myriad) + " for eight, so throughput" ^0 "falls from " + str(throughput_before_hundredths) + " to " + str(throughput_after_hundredths) + " hundredths, until the shared state is partitioned." ^0 ``` ## Python (deterministic transpilation) ```python nodes_before = 8 nodes_after = 16 contention_per_extra_node_per_myriad = 500 coherence_per_pair_per_myriad = 100 extra_nodes_before = nodes_before - 1 extra_nodes_after = nodes_after - 1 time_per_unit_before_per_myriad = 10000 + contention_per_extra_node_per_myriad * extra_nodes_before + coherence_per_pair_per_myriad * nodes_before * extra_nodes_before time_per_unit_after_per_myriad = 10000 + contention_per_extra_node_per_myriad * extra_nodes_after + coherence_per_pair_per_myriad * nodes_after * extra_nodes_after throughput_before_hundredths = int(nodes_before * 100 * 10000 / time_per_unit_before_per_myriad) throughput_after_hundredths = int(nodes_after * 100 * 10000 / time_per_unit_after_per_myriad) throughput_lost_by_doubling_hundredths = throughput_before_hundredths - throughput_after_hundredths coherence_cost_at_sixteen_per_myriad = coherence_per_pair_per_myriad * nodes_after * extra_nodes_after coherence_cost_at_eight_per_myriad = coherence_per_pair_per_myriad * nodes_before * extra_nodes_before print("nodes before / after : " + str(nodes_before) + " / " + str(nodes_after) + ", all identical and healthy") print("contention per extra node : " + str(contention_per_extra_node_per_myriad) + " per ten thousand of a unit's time") print("coherence per pair of nodes : " + str(coherence_per_pair_per_myriad) + " per ten thousand") print("") print("coherence cost at " + str(nodes_before) + " : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand") print("coherence cost at " + str(nodes_after) + " : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand") print("throughput at " + str(nodes_before) + " : " + str(throughput_before_hundredths) + " hundredths of one node's") print("throughput at " + str(nodes_after) + " : " + str(throughput_after_hundredths) + " hundredths of one node's") print("throughput lost by doubling : " + str(throughput_lost_by_doubling_hundredths) + " hundredths") print("") print("the doubled cluster") print(" hardware : identical on every node") print(" health : every node up and busy") print(" balancing : even") print(" intent : twice the nodes, more throughput") print(" idle or failed nodes : 0") print(" verdict : SIXTEEN HEALTHY NODES, ALL BUSY") print("") print(" sixteen identical, healthy, evenly loaded nodes is the") print(" part done right here, and it is why the drop is not a") print(" bad node or an uneven split") print("") print("coordination") print(" what shared state requires : every node agreeing with") print(" every other") print(" how that grows : with the pairs, " + str(nodes_before) + " times " + str(extra_nodes_before) + " then " + str(nodes_after) + " times " + str(extra_nodes_after)) print(" at " + str(nodes_before) + " nodes : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand of every unit spent agreeing") print(" at " + str(nodes_after) + " nodes : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand") print(" so each node at " + str(nodes_after) + " : does much less than a node at " + str(nodes_before) + ", and") print(" twice as many of them do less in total") print("") print("the throughput") print(" eight nodes : " + str(throughput_before_hundredths) + " hundredths") print(" sixteen nodes : " + str(throughput_after_hundredths) + " hundredths") print(" is any node broken : no") print(" is more nodes more work : up to the point where agreeing") print(" costs more than the node adds, and sixteen is past it") print("") nc_throughput_at_sixteen_with_pairwise_coherence_hundredths = 385 nc_throughput_at_sixteen_with_coherence_partitioned_hundredths = 914 nc_sixteen_beats_eight_once_partitioned = 1 print("null control - partition the shared state") print(" throughput at sixteen, pairwise coherence : " + str(nc_throughput_at_sixteen_with_pairwise_coherence_hundredths) + " hundredths") print(" throughput at sixteen, coherence partitioned : " + str(nc_throughput_at_sixteen_with_coherence_partitioned_hundredths) + " hundredths") print(" sixteen beats eight once partitioned : " + str(nc_sixteen_beats_eight_once_partitioned)) print(" no node changed; the nodes stopped having to agree with") print(" every other") print("") print("what sixteen healthy, evenly loaded nodes guarantee") print(" twice the machines are working : exactly, identical,") print(" healthy, balanced") print(" twice the machines do more : not addressed; coherence") print(" grows with the pairs, so sixteen nodes spend " + str(coherence_cost_at_sixteen_per_myriad) + " per ten") print(" thousand of each unit agreeing and deliver " + str(throughput_after_hundredths) + " hundredths") print(" where eight delivered " + str(throughput_before_hundredths)) print("") print("a machine added to a group that must agree adds a worker and a voice, and") print("the voices grow faster than the workers; past the point where a new node's") print("talk costs more than its work, adding capacity subtracts throughput") print("") print("Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the") print("nodes must keep shared state coherent pairwise, and sixteen have " + str(coherence_cost_at_sixteen_per_myriad) + " per ten") print("thousand of every unit tied up agreeing against " + str(coherence_cost_at_eight_per_myriad) + " for eight, so throughput") print("falls from " + str(throughput_before_hundredths) + " to " + str(throughput_after_hundredths) + " hundredths, until the shared state is partitioned.") ``` ## stdout (executed) ```text nodes before / after : 8 / 16, all identical and healthy contention per extra node : 500 per ten thousand of a unit's time coherence per pair of nodes : 100 per ten thousand coherence cost at 8 : 5600 per ten thousand coherence cost at 16 : 24000 per ten thousand throughput at 8 : 418 hundredths of one node's throughput at 16 : 385 hundredths of one node's throughput lost by doubling : 33 hundredths the doubled cluster hardware : identical on every node health : every node up and busy balancing : even intent : twice the nodes, more throughput idle or failed nodes : 0 verdict : SIXTEEN HEALTHY NODES, ALL BUSY sixteen identical, healthy, evenly loaded nodes is the part done right here, and it is why the drop is not a bad node or an uneven split coordination what shared state requires : every node agreeing with every other how that grows : with the pairs, 8 times 7 then 16 times 15 at 8 nodes : 5600 per ten thousand of every unit spent agreeing at 16 nodes : 24000 per ten thousand so each node at 16 : does much less than a node at 8, and twice as many of them do less in total the throughput eight nodes : 418 hundredths sixteen nodes : 385 hundredths is any node broken : no is more nodes more work : up to the point where agreeing costs more than the node adds, and sixteen is past it null control - partition the shared state throughput at sixteen, pairwise coherence : 385 hundredths throughput at sixteen, coherence partitioned : 914 hundredths sixteen beats eight once partitioned : 1 no node changed; the nodes stopped having to agree with every other what sixteen healthy, evenly loaded nodes guarantee twice the machines are working : exactly, identical, healthy, balanced twice the machines do more : not addressed; coherence grows with the pairs, so sixteen nodes spend 24000 per ten thousand of each unit agreeing and deliver 385 hundredths where eight delivered 418 a machine added to a group that must agree adds a worker and a voice, and the voices grow faster than the workers; past the point where a new node's talk costs more than its work, adding capacity subtracts throughput Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the nodes must keep shared state coherent pairwise, and sixteen have 24000 per ten thousand of every unit tied up agreeing against 5600 for eight, so throughput falls from 418 to 385 hundredths, until the shared state is partitioned. ``` ## Round-trip `ok: true` — round-trip fixpoint reached (python1 == python2) ## Trace event types eml:run:start · eml:assign · eml:output · eml:run:done