The two nodes agreed and both were behind

# Self-authored for the EML case corpus (no external origin). Every read is
# confirmed by two replicas that agree, and they do agree. How old the answer is
# is computed below.
#
# The quorum is correctly implemented. A read is not returned until two of the
# three replicas have answered with the same value; a disagreement is detected,
# logged and retried against the third; and a deliberate corruption drill on one
# replica was caught on the first read. Two out of three is a real check and it
# is doing real work.
#
# Agreement is a statement about two ANSWERS. It says the two replicas hold the
# same bytes. It does not say when either of them last heard from the leader,
# and two replicas behind by the same amount agree perfectly.
#
# The client asks the two fastest. The current replica is the slow one, because
# applying the write stream is what makes it slow.

3 => replicas
2 => quorum
41 => replica_a_lag_seconds
43 => replica_b_lag_seconds
0 => replica_c_lag_seconds
12 => fast_replica_p99_ms
180 => current_replica_p99_ms
3400 => writes_per_second
26000 => reads_per_second
0 => quorum_disagreements

# The client's policy is "the two fastest", and the two fastest are the two that
# are not applying the backlog.
replica_a_lag_seconds => lag_of_the_answer_seconds
lag_of_the_answer_seconds * writes_per_second => writes_the_quorum_cannot_see

"replicas                     : " + str(replicas) ^0
"quorum                       : " + str(quorum) ^0
"replica A lag, seconds       : " + str(replica_a_lag_seconds) ^0
"replica B lag, seconds       : " + str(replica_b_lag_seconds) ^0
"replica C lag, seconds       : " + str(replica_c_lag_seconds) ^0
"" ^0
"p99 of A and B, ms           : " + str(fast_replica_p99_ms) ^0
"p99 of C, ms                 : " + str(current_replica_p99_ms) ^0
"writes the quorum cannot see : " + str(writes_the_quorum_cannot_see) ^0
"" ^0

# ---- what the quorum verified ----

"the quorum check" ^0
"  replicas that must agree : " + str(quorum) ^0
"  disagreements this week  : " + str(quorum_disagreements) ^0
"  corruption drill on one replica : caught on the first read" ^0
"  reads served per second  : " + str(reads_per_second) ^0
"  verdict                  : CONSISTENT" ^0
"" ^0
"  the drill is why this is trusted, and it found the" ^0
"  corruption immediately" ^0
"" ^0

# ---- which two answered ----

"choosing the two" ^0
"  policy            : the two fastest to respond" ^0
"  A and B, p99 ms   : " + str(fast_replica_p99_ms) ^0
"  C, p99 ms         : " + str(current_replica_p99_ms) ^0
"  why C is slow     : it is applying the write stream" ^0
"  how often C is in the quorum : never" ^0
"" ^0
"  the policy selects for the property that is anti-" ^0
"  correlated with freshness, and it does so every time" ^0
"" ^0

int(current_replica_p99_ms / fast_replica_p99_ms) => how_much_slower_current_is
"the current replica is " + str(how_much_slower_current_is) + " times slower, which is why it is excluded" ^0
"" ^0

# ---- what a reader concludes ----

# Two independent sources agreeing is the strongest evidence most systems can
# offer, and here the independence is in the storage and not in the clock.
"the evidence the reader has" ^0
"  independent replicas agreeing : " + str(quorum) ^0
"  what that rules out           : one replica corrupted," ^0
"    one replica rolled back, one disk lying" ^0
"  what it does not rule out     : both replicas being" ^0
"    " + str(lag_of_the_answer_seconds) + " seconds old, which is exactly what they are" ^0
"" ^0

# ---- null control ----

# The same quorum, with the replica's lag returned alongside its answer and the
# read refused above a threshold.
5 => nc_max_lag_seconds
nc_max_lag_seconds * writes_per_second => nc_writes_the_quorum_cannot_see

"null control - each answer carries its lag, bounded at 5 s" ^0
"  quorum disagreements    : " + str(quorum_disagreements) + ", unchanged" ^0
"  max lag accepted, seconds : " + str(nc_max_lag_seconds) ^0
"  writes the quorum cannot see : " + str(nc_writes_the_quorum_cannot_see) ^0
"  the replicas did not become more consistent; the answer" ^0
"  started carrying the one fact the comparison omits" ^0
"" ^0

# ---- the rule ----

"what an agreeing quorum guarantees" ^0
"  these replicas hold the same value : exactly" ^0
"  the value is current                : not addressed;" ^0
"    agreement is a comparison between replicas and" ^0
"    currency is a comparison with the leader" ^0
"" ^0
"two sources agreeing is strong evidence against corruption" ^0
"and no evidence at all about age; when the selection policy" ^0
"prefers the idle replicas, it is evidence against currency" ^0
"" ^0

"The quorum agrees on every read and the check is real: " + str(quorum_disagreements) + " disagreements this" ^0
"week and a corruption drill caught on the first read. The two that answer are" ^0
"the two fastest, which are the two not applying the write stream - " + str(how_much_slower_current_is) + " times" ^0
"faster than the current replica - so the agreed answer is " + str(lag_of_the_answer_seconds) + " seconds old and" ^0
str(writes_the_quorum_cannot_see) + " writes are invisible to a read that two independent replicas confirmed." ^0

replicas = 3
quorum = 2
replica_a_lag_seconds = 41
replica_b_lag_seconds = 43
replica_c_lag_seconds = 0
fast_replica_p99_ms = 12
current_replica_p99_ms = 180
writes_per_second = 3400
reads_per_second = 26000
quorum_disagreements = 0
lag_of_the_answer_seconds = replica_a_lag_seconds
writes_the_quorum_cannot_see = lag_of_the_answer_seconds * writes_per_second
print("replicas                     : " + str(replicas))
print("quorum                       : " + str(quorum))
print("replica A lag, seconds       : " + str(replica_a_lag_seconds))
print("replica B lag, seconds       : " + str(replica_b_lag_seconds))
print("replica C lag, seconds       : " + str(replica_c_lag_seconds))
print("")
print("p99 of A and B, ms           : " + str(fast_replica_p99_ms))
print("p99 of C, ms                 : " + str(current_replica_p99_ms))
print("writes the quorum cannot see : " + str(writes_the_quorum_cannot_see))
print("")
print("the quorum check")
print("  replicas that must agree : " + str(quorum))
print("  disagreements this week  : " + str(quorum_disagreements))
print("  corruption drill on one replica : caught on the first read")
print("  reads served per second  : " + str(reads_per_second))
print("  verdict                  : CONSISTENT")
print("")
print("  the drill is why this is trusted, and it found the")
print("  corruption immediately")
print("")
print("choosing the two")
print("  policy            : the two fastest to respond")
print("  A and B, p99 ms   : " + str(fast_replica_p99_ms))
print("  C, p99 ms         : " + str(current_replica_p99_ms))
print("  why C is slow     : it is applying the write stream")
print("  how often C is in the quorum : never")
print("")
print("  the policy selects for the property that is anti-")
print("  correlated with freshness, and it does so every time")
print("")
how_much_slower_current_is = int(current_replica_p99_ms / fast_replica_p99_ms)
print("the current replica is " + str(how_much_slower_current_is) + " times slower, which is why it is excluded")
print("")
print("the evidence the reader has")
print("  independent replicas agreeing : " + str(quorum))
print("  what that rules out           : one replica corrupted,")
print("    one replica rolled back, one disk lying")
print("  what it does not rule out     : both replicas being")
print("    " + str(lag_of_the_answer_seconds) + " seconds old, which is exactly what they are")
print("")
nc_max_lag_seconds = 5
nc_writes_the_quorum_cannot_see = nc_max_lag_seconds * writes_per_second
print("null control - each answer carries its lag, bounded at 5 s")
print("  quorum disagreements    : " + str(quorum_disagreements) + ", unchanged")
print("  max lag accepted, seconds : " + str(nc_max_lag_seconds))
print("  writes the quorum cannot see : " + str(nc_writes_the_quorum_cannot_see))
print("  the replicas did not become more consistent; the answer")
print("  started carrying the one fact the comparison omits")
print("")
print("what an agreeing quorum guarantees")
print("  these replicas hold the same value : exactly")
print("  the value is current                : not addressed;")
print("    agreement is a comparison between replicas and")
print("    currency is a comparison with the leader")
print("")
print("two sources agreeing is strong evidence against corruption")
print("and no evidence at all about age; when the selection policy")
print("prefers the idle replicas, it is evidence against currency")
print("")
print("The quorum agrees on every read and the check is real: " + str(quorum_disagreements) + " disagreements this")
print("week and a corruption drill caught on the first read. The two that answer are")
print("the two fastest, which are the two not applying the write stream - " + str(how_much_slower_current_is) + " times")
print("faster than the current replica - so the agreed answer is " + str(lag_of_the_answer_seconds) + " seconds old and")
print(str(writes_the_quorum_cannot_see) + " writes are invisible to a read that two independent replicas confirmed.")

replicas                     : 3
quorum                       : 2
replica A lag, seconds       : 41
replica B lag, seconds       : 43
replica C lag, seconds       : 0

p99 of A and B, ms           : 12
p99 of C, ms                 : 180
writes the quorum cannot see : 139400

the quorum check
  replicas that must agree : 2
  disagreements this week  : 0
  corruption drill on one replica : caught on the first read
  reads served per second  : 26000
  verdict                  : CONSISTENT

  the drill is why this is trusted, and it found the
  corruption immediately

choosing the two
  policy            : the two fastest to respond
  A and B, p99 ms   : 12
  C, p99 ms         : 180
  why C is slow     : it is applying the write stream
  how often C is in the quorum : never

  the policy selects for the property that is anti-
  correlated with freshness, and it does so every time

the current replica is 15 times slower, which is why it is excluded

the evidence the reader has
  independent replicas agreeing : 2
  what that rules out           : one replica corrupted,
    one replica rolled back, one disk lying
  what it does not rule out     : both replicas being
    41 seconds old, which is exactly what they are

null control - each answer carries its lag, bounded at 5 s
  quorum disagreements    : 0, unchanged
  max lag accepted, seconds : 5
  writes the quorum cannot see : 17000
  the replicas did not become more consistent; the answer
  started carrying the one fact the comparison omits

what an agreeing quorum guarantees
  these replicas hold the same value : exactly
  the value is current                : not addressed;
    agreement is a comparison between replicas and
    currency is a comparison with the leader

two sources agreeing is strong evidence against corruption
and no evidence at all about age; when the selection policy
prefers the idle replicas, it is evidence against currency

The quorum agrees on every read and the check is real: 0 disagreements this
week and a corruption drill caught on the first read. The two that answer are
the two fastest, which are the two not applying the write stream - 15 times
faster than the current replica - so the agreed answer is 41 seconds old and
139400 writes are invisible to a read that two independent replicas confirmed.

Trace event types