The counter was emptied by whoever read it first

# Self-authored for the EML case corpus (no external origin). A counter that
# reports events since the last read, and a second collector added during a
# region migration. What each collector sees afterwards is computed below.
#
# A read-and-reset counter is a real design and a defensible one. The endpoint
# holds a small integer instead of a monotonic total that has to survive
# restarts, the collector does no subtraction and needs no memory of the
# previous value, and a missed scrape is visible as a larger number rather than
# invisible as a gap. Plenty of metrics protocols work exactly this way.
#
# Adding a second collector is the safest operation there is. It is a read. It
# takes no lock anyone can see, it changes no configuration on the instance,
# and it is how every other metric on the fleet is collected twice for
# redundancy. The migration runbook lists it under low risk and it belongs
# there for every metric except this one.
#
# A read that mutates is not a read. Two collectors on a normal counter both
# see the truth; two collectors on this one divide it, because the interval
# each of them measures is bounded by the other one's last visit rather than by
# its own.

100 => rate_per_s
30 => period_s
80 => alert_threshold

"true event rate    : " + str(rate_per_s) + " per second" ^0
"collector interval : " + str(period_s) + " seconds" ^0
"alert threshold    : " + str(alert_threshold) + " per second" ^0
"" ^0

# ---- the timeline with two collectors, offset by half a period ----

"the timeline, collectors A and B fifteen seconds apart" ^0
0 => last_read
for t in [15, 30, 45, 60, 75, 90]:
    "B" => who
    if t % period_s == 0:
        "A" => who
    (t - last_read) * rate_per_s => drained
    ("  t=%-4s %s reads, drains %s seconds of events, counter returns %s" % (str(t), who, str(t - last_read), str(drained)))^0
    t => last_read
"" ^0

# ---- what each collector reports, for one to four collectors ----

"collectors   seconds each read covers   events per read   reported rate   alert" ^0
for n in [1, 2, 3, 4]:
    int(period_s / n) => window
    window * rate_per_s => per_read
    int(per_read / period_s) => reported
    "silent" => fires
    if reported > alert_threshold:
        "fires" => fires
    ("  %-12s %-26s %-17s %-15s %s" % (str(n), str(window), str(per_read), str(reported), fires))^0
"" ^0

int(int(period_s / 2) * rate_per_s / period_s) => two_reported
("  with two collectors each reports %s per second against a true %s" % (str(two_reported), str(rate_per_s)))^0
("  the alert needs %s and the highest number anywhere in the system is %s" % (str(alert_threshold), str(two_reported)))^0
"" ^0

# ---- the control ----
#
# The same endpoint, the same two collectors, the same network. A gauge is
# read without being reset.

"control - a gauge on the same endpoint, read by the same two collectors" ^0
"  queue_depth reported by A : 412" ^0
"  queue_depth reported by B : 412" ^0
"  they agree, and they are right" ^0
"  so the endpoint, both collectors and the transport are all correct" ^0
"" ^0

"control - the same counter before the migration, one collector" ^0
("  seconds each read covers : %s" % str(period_s))^0
("  reported rate            : %s" % str(int(period_s * rate_per_s / period_s)))^0
("  true rate                : %s" % str(rate_per_s))^0
"  correct, for eleven months" ^0
"  so the counter is not wrong either, and nothing was edited" ^0
"" ^0

# ---- the aggregation that would fix it ----

"how to combine two collectors' readings" ^0
("  take either one    : %s, which is the rule for every other metric" % str(two_reported))^0
("  take the maximum   : %s, which is the rule for a counter that is not drained" % str(two_reported))^0
("  add them together  : %s, which is correct here and catastrophic anywhere else" % str(two_reported * 2))^0
"  the right aggregation for this metric is the one that is a bug for the" ^0
"  rest of them, and the collector cannot tell which kind it is holding" ^0
"" ^0

# ---- what the shape of the defect is ----

"why nothing looked broken" ^0
"  errors raised          : 0" ^0
"  scrapes failed         : 0" ^0
"  series missing         : 0" ^0
"  collectors disagreeing : 0, they both report the same wrong number" ^0
"  a disagreement would have been noticed within a day" ^0
"  agreement is what made it survive" ^0
"" ^0

"A read-and-reset counter is a real design and adding a second collector is" ^0
"a read. A read that resets is not one: each collector measures the interval" ^0
("since the other one's visit, so both report %s per second against a true %s," % (str(two_reported), str(rate_per_s)))^0
("they agree with each other, and the %s threshold is never crossed." % str(alert_threshold))^0

rate_per_s = 100
period_s = 30
alert_threshold = 80
print("true event rate    : " + str(rate_per_s) + " per second")
print("collector interval : " + str(period_s) + " seconds")
print("alert threshold    : " + str(alert_threshold) + " per second")
print("")
print("the timeline, collectors A and B fifteen seconds apart")
last_read = 0
for t in [15, 30, 45, 60, 75, 90]:
    who = "B"
    if t % period_s == 0:
        who = "A"
    drained = (t - last_read) * rate_per_s
    print("  t=%-4s %s reads, drains %s seconds of events, counter returns %s" % (str(t), who, str(t - last_read), str(drained)))
    last_read = t
print("")
print("collectors   seconds each read covers   events per read   reported rate   alert")
for n in [1, 2, 3, 4]:
    window = int(period_s / n)
    per_read = window * rate_per_s
    reported = int(per_read / period_s)
    fires = "silent"
    if reported > alert_threshold:
        fires = "fires"
    print("  %-12s %-26s %-17s %-15s %s" % (str(n), str(window), str(per_read), str(reported), fires))
print("")
two_reported = int(int(period_s / 2) * rate_per_s / period_s)
print("  with two collectors each reports %s per second against a true %s" % (str(two_reported), str(rate_per_s)))
print("  the alert needs %s and the highest number anywhere in the system is %s" % (str(alert_threshold), str(two_reported)))
print("")
print("control - a gauge on the same endpoint, read by the same two collectors")
print("  queue_depth reported by A : 412")
print("  queue_depth reported by B : 412")
print("  they agree, and they are right")
print("  so the endpoint, both collectors and the transport are all correct")
print("")
print("control - the same counter before the migration, one collector")
print("  seconds each read covers : %s" % str(period_s))
print("  reported rate            : %s" % str(int(period_s * rate_per_s / period_s)))
print("  true rate                : %s" % str(rate_per_s))
print("  correct, for eleven months")
print("  so the counter is not wrong either, and nothing was edited")
print("")
print("how to combine two collectors' readings")
print("  take either one    : %s, which is the rule for every other metric" % str(two_reported))
print("  take the maximum   : %s, which is the rule for a counter that is not drained" % str(two_reported))
print("  add them together  : %s, which is correct here and catastrophic anywhere else" % str(two_reported * 2))
print("  the right aggregation for this metric is the one that is a bug for the")
print("  rest of them, and the collector cannot tell which kind it is holding")
print("")
print("why nothing looked broken")
print("  errors raised          : 0")
print("  scrapes failed         : 0")
print("  series missing         : 0")
print("  collectors disagreeing : 0, they both report the same wrong number")
print("  a disagreement would have been noticed within a day")
print("  agreement is what made it survive")
print("")
print("A read-and-reset counter is a real design and adding a second collector is")
print("a read. A read that resets is not one: each collector measures the interval")
print("since the other one's visit, so both report %s per second against a true %s," % (str(two_reported), str(rate_per_s)))
print("they agree with each other, and the %s threshold is never crossed." % str(alert_threshold))

true event rate    : 100 per second
collector interval : 30 seconds
alert threshold    : 80 per second

the timeline, collectors A and B fifteen seconds apart
  t=15   B reads, drains 15 seconds of events, counter returns 1500
  t=30   A reads, drains 15 seconds of events, counter returns 1500
  t=45   B reads, drains 15 seconds of events, counter returns 1500
  t=60   A reads, drains 15 seconds of events, counter returns 1500
  t=75   B reads, drains 15 seconds of events, counter returns 1500
  t=90   A reads, drains 15 seconds of events, counter returns 1500

collectors   seconds each read covers   events per read   reported rate   alert
  1            30                         3000              100             fires
  2            15                         1500              50              silent
  3            10                         1000              33              silent
  4            7                          700               23              silent

  with two collectors each reports 50 per second against a true 100
  the alert needs 80 and the highest number anywhere in the system is 50

control - a gauge on the same endpoint, read by the same two collectors
  queue_depth reported by A : 412
  queue_depth reported by B : 412
  they agree, and they are right
  so the endpoint, both collectors and the transport are all correct

control - the same counter before the migration, one collector
  seconds each read covers : 30
  reported rate            : 100
  true rate                : 100
  correct, for eleven months
  so the counter is not wrong either, and nothing was edited

how to combine two collectors' readings
  take either one    : 50, which is the rule for every other metric
  take the maximum   : 50, which is the rule for a counter that is not drained
  add them together  : 100, which is correct here and catastrophic anywhere else
  the right aggregation for this metric is the one that is a bug for the
  rest of them, and the collector cannot tell which kind it is holding

why nothing looked broken
  errors raised          : 0
  scrapes failed         : 0
  series missing         : 0
  collectors disagreeing : 0, they both report the same wrong number
  a disagreement would have been noticed within a day
  agreement is what made it survive

A read-and-reset counter is a real design and adding a second collector is
a read. A read that resets is not one: each collector measures the interval
since the other one's visit, so both report 50 per second against a true 100,
they agree with each other, and the 80 threshold is never crossed.

Trace event types