The sixteenth node made the cluster slower

# Self-authored for the EML case corpus (no external origin). A cluster of
# eight nodes is doubled to sixteen to raise throughput, every node is
# identical and fully working, and the sixteen-node cluster does less work per
# second than the eight. What each added node costs the others is computed
# below.
#
# The expansion is careful. Every node is the same hardware; every node is
# healthy and busy; the load balancer spreads work evenly; and the intent is
# exactly 'twice the nodes, more throughput'.
#
# Each node must keep its shared state coherent with every other, and that
# coordination grows with the number of pairs of nodes, so past a point the
# nodes spend more of each second agreeing than working - sixteen agree more
# than eight and do less.

8 => nodes_before
16 => nodes_after
500 => contention_per_extra_node_per_myriad
100 => coherence_per_pair_per_myriad

nodes_before - 1 => extra_nodes_before
nodes_after - 1 => extra_nodes_after
10000 + contention_per_extra_node_per_myriad * extra_nodes_before + coherence_per_pair_per_myriad * nodes_before * extra_nodes_before => time_per_unit_before_per_myriad
10000 + contention_per_extra_node_per_myriad * extra_nodes_after + coherence_per_pair_per_myriad * nodes_after * extra_nodes_after => time_per_unit_after_per_myriad
int(nodes_before * 100 * 10000 / time_per_unit_before_per_myriad) => throughput_before_hundredths
int(nodes_after * 100 * 10000 / time_per_unit_after_per_myriad) => throughput_after_hundredths
throughput_before_hundredths - throughput_after_hundredths => throughput_lost_by_doubling_hundredths
coherence_per_pair_per_myriad * nodes_after * extra_nodes_after => coherence_cost_at_sixteen_per_myriad
coherence_per_pair_per_myriad * nodes_before * extra_nodes_before => coherence_cost_at_eight_per_myriad

"nodes before / after            : " + str(nodes_before) + " / " + str(nodes_after) + ", all identical and healthy" ^0
"contention per extra node       : " + str(contention_per_extra_node_per_myriad) + " per ten thousand of a unit's time" ^0
"coherence per pair of nodes     : " + str(coherence_per_pair_per_myriad) + " per ten thousand" ^0
"" ^0
"coherence cost at " + str(nodes_before) + "              : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand" ^0
"coherence cost at " + str(nodes_after) + "             : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand" ^0
"throughput at " + str(nodes_before) + "                 : " + str(throughput_before_hundredths) + " hundredths of one node's" ^0
"throughput at " + str(nodes_after) + "                : " + str(throughput_after_hundredths) + " hundredths of one node's" ^0
"throughput lost by doubling     : " + str(throughput_lost_by_doubling_hundredths) + " hundredths" ^0
"" ^0

# ---- what the expansion verified ----

"the doubled cluster" ^0
"  hardware : identical on every node" ^0
"  health : every node up and busy" ^0
"  balancing : even" ^0
"  intent : twice the nodes, more throughput" ^0
"  idle or failed nodes : 0" ^0
"  verdict : SIXTEEN HEALTHY NODES, ALL BUSY" ^0
"" ^0
"  sixteen identical, healthy, evenly loaded nodes is the" ^0
"  part done right here, and it is why the drop is not a" ^0
"  bad node or an uneven split" ^0
"" ^0

# ---- what each node costs the others ----

"coordination" ^0
"  what shared state requires : every node agreeing with" ^0
"    every other" ^0
"  how that grows : with the pairs, " + str(nodes_before) + " times " + str(extra_nodes_before) + " then " + str(nodes_after) + " times " + str(extra_nodes_after) ^0
"  at " + str(nodes_before) + " nodes : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand of every unit spent agreeing" ^0
"  at " + str(nodes_after) + " nodes : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand" ^0
"  so each node at " + str(nodes_after) + " : does much less than a node at " + str(nodes_before) + ", and" ^0
"    twice as many of them do less in total" ^0
"" ^0

# ---- what the operators got ----

"the throughput" ^0
"  eight nodes : " + str(throughput_before_hundredths) + " hundredths" ^0
"  sixteen nodes : " + str(throughput_after_hundredths) + " hundredths" ^0
"  is any node broken : no" ^0
"  is more nodes more work : up to the point where agreeing" ^0
"    costs more than the node adds, and sixteen is past it" ^0
"" ^0

# ---- null control ----

# The same cluster with the shared state partitioned so nodes coordinate only
# within small groups, removing the pairwise term.
385 => nc_throughput_at_sixteen_with_pairwise_coherence_hundredths
914 => nc_throughput_at_sixteen_with_coherence_partitioned_hundredths
1 => nc_sixteen_beats_eight_once_partitioned

"null control - partition the shared state" ^0
"  throughput at sixteen, pairwise coherence : " + str(nc_throughput_at_sixteen_with_pairwise_coherence_hundredths) + " hundredths" ^0
"  throughput at sixteen, coherence partitioned : " + str(nc_throughput_at_sixteen_with_coherence_partitioned_hundredths) + " hundredths" ^0
"  sixteen beats eight once partitioned : " + str(nc_sixteen_beats_eight_once_partitioned) ^0
"  no node changed; the nodes stopped having to agree with" ^0
"  every other" ^0
"" ^0

# ---- the rule ----

"what sixteen healthy, evenly loaded nodes guarantee" ^0
"  twice the machines are working : exactly, identical," ^0
"    healthy, balanced" ^0
"  twice the machines do more : not addressed; coherence" ^0
"    grows with the pairs, so sixteen nodes spend " + str(coherence_cost_at_sixteen_per_myriad) + " per ten" ^0
"    thousand of each unit agreeing and deliver " + str(throughput_after_hundredths) + " hundredths" ^0
"    where eight delivered " + str(throughput_before_hundredths) ^0
"" ^0

"a machine added to a group that must agree adds a worker and a voice, and" ^0
"the voices grow faster than the workers; past the point where a new node's" ^0
"talk costs more than its work, adding capacity subtracts throughput" ^0
"" ^0

"Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the" ^0
"nodes must keep shared state coherent pairwise, and sixteen have " + str(coherence_cost_at_sixteen_per_myriad) + " per ten" ^0
"thousand of every unit tied up agreeing against " + str(coherence_cost_at_eight_per_myriad) + " for eight, so throughput" ^0
"falls from " + str(throughput_before_hundredths) + " to " + str(throughput_after_hundredths) + " hundredths, until the shared state is partitioned." ^0

nodes_before = 8
nodes_after = 16
contention_per_extra_node_per_myriad = 500
coherence_per_pair_per_myriad = 100
extra_nodes_before = nodes_before - 1
extra_nodes_after = nodes_after - 1
time_per_unit_before_per_myriad = 10000 + contention_per_extra_node_per_myriad * extra_nodes_before + coherence_per_pair_per_myriad * nodes_before * extra_nodes_before
time_per_unit_after_per_myriad = 10000 + contention_per_extra_node_per_myriad * extra_nodes_after + coherence_per_pair_per_myriad * nodes_after * extra_nodes_after
throughput_before_hundredths = int(nodes_before * 100 * 10000 / time_per_unit_before_per_myriad)
throughput_after_hundredths = int(nodes_after * 100 * 10000 / time_per_unit_after_per_myriad)
throughput_lost_by_doubling_hundredths = throughput_before_hundredths - throughput_after_hundredths
coherence_cost_at_sixteen_per_myriad = coherence_per_pair_per_myriad * nodes_after * extra_nodes_after
coherence_cost_at_eight_per_myriad = coherence_per_pair_per_myriad * nodes_before * extra_nodes_before
print("nodes before / after            : " + str(nodes_before) + " / " + str(nodes_after) + ", all identical and healthy")
print("contention per extra node       : " + str(contention_per_extra_node_per_myriad) + " per ten thousand of a unit's time")
print("coherence per pair of nodes     : " + str(coherence_per_pair_per_myriad) + " per ten thousand")
print("")
print("coherence cost at " + str(nodes_before) + "              : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand")
print("coherence cost at " + str(nodes_after) + "             : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand")
print("throughput at " + str(nodes_before) + "                 : " + str(throughput_before_hundredths) + " hundredths of one node's")
print("throughput at " + str(nodes_after) + "                : " + str(throughput_after_hundredths) + " hundredths of one node's")
print("throughput lost by doubling     : " + str(throughput_lost_by_doubling_hundredths) + " hundredths")
print("")
print("the doubled cluster")
print("  hardware : identical on every node")
print("  health : every node up and busy")
print("  balancing : even")
print("  intent : twice the nodes, more throughput")
print("  idle or failed nodes : 0")
print("  verdict : SIXTEEN HEALTHY NODES, ALL BUSY")
print("")
print("  sixteen identical, healthy, evenly loaded nodes is the")
print("  part done right here, and it is why the drop is not a")
print("  bad node or an uneven split")
print("")
print("coordination")
print("  what shared state requires : every node agreeing with")
print("    every other")
print("  how that grows : with the pairs, " + str(nodes_before) + " times " + str(extra_nodes_before) + " then " + str(nodes_after) + " times " + str(extra_nodes_after))
print("  at " + str(nodes_before) + " nodes : " + str(coherence_cost_at_eight_per_myriad) + " per ten thousand of every unit spent agreeing")
print("  at " + str(nodes_after) + " nodes : " + str(coherence_cost_at_sixteen_per_myriad) + " per ten thousand")
print("  so each node at " + str(nodes_after) + " : does much less than a node at " + str(nodes_before) + ", and")
print("    twice as many of them do less in total")
print("")
print("the throughput")
print("  eight nodes : " + str(throughput_before_hundredths) + " hundredths")
print("  sixteen nodes : " + str(throughput_after_hundredths) + " hundredths")
print("  is any node broken : no")
print("  is more nodes more work : up to the point where agreeing")
print("    costs more than the node adds, and sixteen is past it")
print("")
nc_throughput_at_sixteen_with_pairwise_coherence_hundredths = 385
nc_throughput_at_sixteen_with_coherence_partitioned_hundredths = 914
nc_sixteen_beats_eight_once_partitioned = 1
print("null control - partition the shared state")
print("  throughput at sixteen, pairwise coherence : " + str(nc_throughput_at_sixteen_with_pairwise_coherence_hundredths) + " hundredths")
print("  throughput at sixteen, coherence partitioned : " + str(nc_throughput_at_sixteen_with_coherence_partitioned_hundredths) + " hundredths")
print("  sixteen beats eight once partitioned : " + str(nc_sixteen_beats_eight_once_partitioned))
print("  no node changed; the nodes stopped having to agree with")
print("  every other")
print("")
print("what sixteen healthy, evenly loaded nodes guarantee")
print("  twice the machines are working : exactly, identical,")
print("    healthy, balanced")
print("  twice the machines do more : not addressed; coherence")
print("    grows with the pairs, so sixteen nodes spend " + str(coherence_cost_at_sixteen_per_myriad) + " per ten")
print("    thousand of each unit agreeing and deliver " + str(throughput_after_hundredths) + " hundredths")
print("    where eight delivered " + str(throughput_before_hundredths))
print("")
print("a machine added to a group that must agree adds a worker and a voice, and")
print("the voices grow faster than the workers; past the point where a new node's")
print("talk costs more than its work, adding capacity subtracts throughput")
print("")
print("Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the")
print("nodes must keep shared state coherent pairwise, and sixteen have " + str(coherence_cost_at_sixteen_per_myriad) + " per ten")
print("thousand of every unit tied up agreeing against " + str(coherence_cost_at_eight_per_myriad) + " for eight, so throughput")
print("falls from " + str(throughput_before_hundredths) + " to " + str(throughput_after_hundredths) + " hundredths, until the shared state is partitioned.")

nodes before / after            : 8 / 16, all identical and healthy
contention per extra node       : 500 per ten thousand of a unit's time
coherence per pair of nodes     : 100 per ten thousand

coherence cost at 8              : 5600 per ten thousand
coherence cost at 16             : 24000 per ten thousand
throughput at 8                 : 418 hundredths of one node's
throughput at 16                : 385 hundredths of one node's
throughput lost by doubling     : 33 hundredths

the doubled cluster
  hardware : identical on every node
  health : every node up and busy
  balancing : even
  intent : twice the nodes, more throughput
  idle or failed nodes : 0
  verdict : SIXTEEN HEALTHY NODES, ALL BUSY

  sixteen identical, healthy, evenly loaded nodes is the
  part done right here, and it is why the drop is not a
  bad node or an uneven split

coordination
  what shared state requires : every node agreeing with
    every other
  how that grows : with the pairs, 8 times 7 then 16 times 15
  at 8 nodes : 5600 per ten thousand of every unit spent agreeing
  at 16 nodes : 24000 per ten thousand
  so each node at 16 : does much less than a node at 8, and
    twice as many of them do less in total

the throughput
  eight nodes : 418 hundredths
  sixteen nodes : 385 hundredths
  is any node broken : no
  is more nodes more work : up to the point where agreeing
    costs more than the node adds, and sixteen is past it

null control - partition the shared state
  throughput at sixteen, pairwise coherence : 385 hundredths
  throughput at sixteen, coherence partitioned : 914 hundredths
  sixteen beats eight once partitioned : 1
  no node changed; the nodes stopped having to agree with
  every other

what sixteen healthy, evenly loaded nodes guarantee
  twice the machines are working : exactly, identical,
    healthy, balanced
  twice the machines do more : not addressed; coherence
    grows with the pairs, so sixteen nodes spend 24000 per ten
    thousand of each unit agreeing and deliver 385 hundredths
    where eight delivered 418

a machine added to a group that must agree adds a worker and a voice, and
the voices grow faster than the workers; past the point where a new node's
talk costs more than its work, adding capacity subtracts throughput

Sixteen identical, healthy, evenly loaded nodes - nothing is broken. But the
nodes must keep shared state coherent pairwise, and sixteen have 24000 per ten
thousand of every unit tied up agreeing against 5600 for eight, so throughput
falls from 418 to 385 hundredths, until the shared state is partitioned.

Trace event types