The alert threshold was set from the week it was written

# Self-authored for the EML case corpus (no external origin). An alert fires
# when requests per second cross 2400. That number was chosen in month 0 as
# twice the busiest second of the week the alert was written. What it means in
# each later month is computed below.
#
# Twice the observed peak is a good rule and was chosen with care. It is not a
# guess: it comes from a real week of real traffic. It leaves genuine headroom
# rather than tripping on ordinary variation. It is a single number an on-call
# engineer can hold in their head. And it was written down with the reasoning
# attached, which is more than most thresholds get.
#
# The reasoning is a ratio. The threshold is a constant. Traffic compounds at
# about 4 percent a month, which nobody would call growth worth mentioning, and
# a constant does not compound. The gap closes on its own, with no decision,
# no deploy, and no line in any changelog.
#
# What makes this expensive is not the crossing. It is that the alert keeps
# working perfectly the whole way: it fires exactly when the rule says, and the
# rule stopped meaning what it was written to mean.

1200 => peak_at_writing
2400 => threshold
104 => growth_per_month_pct
30 => days_per_month

"month 0 peak: " + str(peak_at_writing) + " rps" ^0
"threshold set to twice that: " + str(threshold) + " rps" ^0
"traffic compounds at " + str(growth_per_month_pct - 100) + " percent per month" ^0
"" ^0

"month   daily peak   threshold   headroom   alerts that month" ^0
peak_at_writing => peak
0 => crossed_at
0 => false_alerts_total
for m in [1:36]:
    int(peak * growth_per_month_pct / 100) => peak
    if peak > threshold:
        days_per_month => fires
        false_alerts_total + days_per_month => false_alerts_total
        if crossed_at == 0:
            m => crossed_at
    else:
        0 => fires
    if m % 6 == 0:
        "  " + str(m) + "        " + str(peak) + "         " + str(threshold) + "        " + str(threshold - peak) + "        " + str(fires) ^0
"" ^0

"  the threshold was first exceeded by ordinary traffic in month " + str(crossed_at) ^0
"  false alerts since then: " + str(false_alerts_total) ^0
"  nothing was deployed in month " + str(crossed_at) + " and nothing broke" ^0
"" ^0

# ---- what the alert now distinguishes ----
#
# An alert is only useful if the world where it fires differs from the world
# where it does not. By month 36 it fires in both.

peak => peak_36
peak_36 * 3 => incident_rps

"month 36" ^0
"  ordinary daily peak : " + str(peak_36) + " rps -> alert fires" ^0
"  a real incident     : " + str(incident_rps) + " rps -> alert fires" ^0
"  the two are " + str(incident_rps - peak_36) + " rps apart and produce the same page" ^0
"" ^0
"  the alert is not broken; it fires exactly when the rule says" ^0
"  the rule is 'above 2400', and above 2400 is now where the service lives" ^0
"" ^0

# ---- the number that decayed ----
#
# Expressed as a multiple of current peak, the threshold falls every month
# without ever being edited.

"the threshold, expressed as the multiple it was written to be" ^0
peak_at_writing => p
for m in [0:36]:
    if m % 6 == 0:
        "  month " + str(m) + ": threshold is " + str(int(threshold * 100 / p)) + " hundredths of the current peak" ^0
    int(p * growth_per_month_pct / 100) => p
"  it was written to be 200 hundredths and was never changed" ^0
"" ^0

# ---- the control ----
#
# The same rule expressed as a ratio rather than a constant. It is the same
# intent, the same week of data, the same engineer. It tracks, because a ratio
# recomputed against current traffic cannot go stale by sitting still.

"control - the identical rule written as a ratio against LAST month's peak" ^0
"  it must do two separate things: stay quiet on growth, and still fire on a spike" ^0
peak_at_writing => rp
0 => ratio_false_alerts
0 => incidents_caught
0 => incidents_tried
for m in [1:36]:
    rp => last_month_peak
    last_month_peak * 2 => relative_threshold
    int(rp * growth_per_month_pct / 100) => rp
    if rp > relative_threshold:
        ratio_false_alerts + days_per_month => ratio_false_alerts
    rp * 3 => spike
    incidents_tried + 1 => incidents_tried
    if spike > relative_threshold:
        incidents_caught + 1 => incidents_caught
    if m % 12 == 0:
        "  month " + str(m) + ": peak " + str(rp) + ", threshold " + str(relative_threshold) + ", a 3x spike would be " + str(spike) ^0
"  rejection side: false alerts from ordinary growth over 36 months: " + str(ratio_false_alerts) ^0
"  acceptance side: 3x spikes detected: " + str(incidents_caught) + " of " + str(incidents_tried) ^0
"  constant threshold, false alerts over the same 36 months: " + str(false_alerts_total) ^0
"  a threshold that only ever stayed quiet would pass the first test and fail the second" ^0
"" ^0
"  same intent, same data, same week; one of them decays and one does not" ^0
"" ^0

# ---- the null control ----
#
# Flat traffic. The constant threshold never decays, because what decays it is
# growth and not the passage of time. A threshold is not stale because it is
# old.

"null control - the same constant threshold against flat traffic" ^0
peak_at_writing => fp
0 => flat_false_alerts
for m in [1:36]:
    if fp > threshold:
        flat_false_alerts + days_per_month => flat_false_alerts
"  peak after 36 months : " + str(fp) + " rps" ^0
"  false alerts         : " + str(flat_false_alerts) ^0
"  the threshold is the same age and is still correct" ^0
"  so the defect is not 'old threshold', it is 'constant compared against a" ^0
"  quantity that compounds'" ^0
"" ^0

# ---- the rule ----

"how a written-down number ages" ^0
"  an absolute number       decays at the rate the world grows" ^0
"  a ratio to a live value  does not decay" ^0
"  a ratio to a frozen value decays exactly like an absolute number" ^0
"  the reasoning was a ratio and only the result was stored" ^0
"" ^0

"Twice the observed peak was a defensible rule, taken from real traffic, with" ^0
"the reasoning written down beside it. The reasoning was a ratio and what got" ^0
"stored was " + str(threshold) + ". Ordinary traffic passed it in month " + str(crossed_at) + ", and by month 36" ^0
"the page for a normal Tuesday and the page for a " + str(incident_rps) + " rps incident are the" ^0
"same page. " + str(false_alerts_total) + " alerts fired, every one of them obeying the rule exactly." ^0

peak_at_writing = 1200
threshold = 2400
growth_per_month_pct = 104
days_per_month = 30
print("month 0 peak: " + str(peak_at_writing) + " rps")
print("threshold set to twice that: " + str(threshold) + " rps")
print("traffic compounds at " + str(growth_per_month_pct - 100) + " percent per month")
print("")
print("month   daily peak   threshold   headroom   alerts that month")
peak = peak_at_writing
crossed_at = 0
false_alerts_total = 0
for m in range(1, 37):
    peak = int(peak * growth_per_month_pct / 100)
    if peak > threshold:
        fires = days_per_month
        false_alerts_total = false_alerts_total + days_per_month
        if crossed_at == 0:
            crossed_at = m
    else:
        fires = 0
    if m % 6 == 0:
        print("  " + str(m) + "        " + str(peak) + "         " + str(threshold) + "        " + str(threshold - peak) + "        " + str(fires))
print("")
print("  the threshold was first exceeded by ordinary traffic in month " + str(crossed_at))
print("  false alerts since then: " + str(false_alerts_total))
print("  nothing was deployed in month " + str(crossed_at) + " and nothing broke")
print("")
peak_36 = peak
incident_rps = peak_36 * 3
print("month 36")
print("  ordinary daily peak : " + str(peak_36) + " rps -> alert fires")
print("  a real incident     : " + str(incident_rps) + " rps -> alert fires")
print("  the two are " + str(incident_rps - peak_36) + " rps apart and produce the same page")
print("")
print("  the alert is not broken; it fires exactly when the rule says")
print("  the rule is 'above 2400', and above 2400 is now where the service lives")
print("")
print("the threshold, expressed as the multiple it was written to be")
p = peak_at_writing
for m in range(0, 37):
    if m % 6 == 0:
        print("  month " + str(m) + ": threshold is " + str(int(threshold * 100 / p)) + " hundredths of the current peak")
    p = int(p * growth_per_month_pct / 100)
print("  it was written to be 200 hundredths and was never changed")
print("")
print("control - the identical rule written as a ratio against LAST month's peak")
print("  it must do two separate things: stay quiet on growth, and still fire on a spike")
rp = peak_at_writing
ratio_false_alerts = 0
incidents_caught = 0
incidents_tried = 0
for m in range(1, 37):
    last_month_peak = rp
    relative_threshold = last_month_peak * 2
    rp = int(rp * growth_per_month_pct / 100)
    if rp > relative_threshold:
        ratio_false_alerts = ratio_false_alerts + days_per_month
    spike = rp * 3
    incidents_tried = incidents_tried + 1
    if spike > relative_threshold:
        incidents_caught = incidents_caught + 1
    if m % 12 == 0:
        print("  month " + str(m) + ": peak " + str(rp) + ", threshold " + str(relative_threshold) + ", a 3x spike would be " + str(spike))
print("  rejection side: false alerts from ordinary growth over 36 months: " + str(ratio_false_alerts))
print("  acceptance side: 3x spikes detected: " + str(incidents_caught) + " of " + str(incidents_tried))
print("  constant threshold, false alerts over the same 36 months: " + str(false_alerts_total))
print("  a threshold that only ever stayed quiet would pass the first test and fail the second")
print("")
print("  same intent, same data, same week; one of them decays and one does not")
print("")
print("null control - the same constant threshold against flat traffic")
fp = peak_at_writing
flat_false_alerts = 0
for m in range(1, 37):
    if fp > threshold:
        flat_false_alerts = flat_false_alerts + days_per_month
print("  peak after 36 months : " + str(fp) + " rps")
print("  false alerts         : " + str(flat_false_alerts))
print("  the threshold is the same age and is still correct")
print("  so the defect is not 'old threshold', it is 'constant compared against a")
print("  quantity that compounds'")
print("")
print("how a written-down number ages")
print("  an absolute number       decays at the rate the world grows")
print("  a ratio to a live value  does not decay")
print("  a ratio to a frozen value decays exactly like an absolute number")
print("  the reasoning was a ratio and only the result was stored")
print("")
print("Twice the observed peak was a defensible rule, taken from real traffic, with")
print("the reasoning written down beside it. The reasoning was a ratio and what got")
print("stored was " + str(threshold) + ". Ordinary traffic passed it in month " + str(crossed_at) + ", and by month 36")
print("the page for a normal Tuesday and the page for a " + str(incident_rps) + " rps incident are the")
print("same page. " + str(false_alerts_total) + " alerts fired, every one of them obeying the rule exactly.")

month 0 peak: 1200 rps
threshold set to twice that: 2400 rps
traffic compounds at 4 percent per month

month   daily peak   threshold   headroom   alerts that month
  6        1515         2400        885        0
  12        1914         2400        486        0
  18        2419         2400        -19        30
  24        3057         2400        -657        30
  30        3866         2400        -1466        30
  36        4888         2400        -2488        30

  the threshold was first exceeded by ordinary traffic in month 18
  false alerts since then: 570
  nothing was deployed in month 18 and nothing broke

month 36
  ordinary daily peak : 4888 rps -> alert fires
  a real incident     : 14664 rps -> alert fires
  the two are 9776 rps apart and produce the same page

  the alert is not broken; it fires exactly when the rule says
  the rule is 'above 2400', and above 2400 is now where the service lives

the threshold, expressed as the multiple it was written to be
  month 0: threshold is 200 hundredths of the current peak
  month 6: threshold is 158 hundredths of the current peak
  month 12: threshold is 125 hundredths of the current peak
  month 18: threshold is 99 hundredths of the current peak
  month 24: threshold is 78 hundredths of the current peak
  month 30: threshold is 62 hundredths of the current peak
  month 36: threshold is 49 hundredths of the current peak
  it was written to be 200 hundredths and was never changed

control - the identical rule written as a ratio against LAST month's peak
  it must do two separate things: stay quiet on growth, and still fire on a spike
  month 12: peak 1914, threshold 3682, a 3x spike would be 5742
  month 24: peak 3057, threshold 5880, a 3x spike would be 9171
  month 36: peak 4888, threshold 9400, a 3x spike would be 14664
  rejection side: false alerts from ordinary growth over 36 months: 0
  acceptance side: 3x spikes detected: 36 of 36
  constant threshold, false alerts over the same 36 months: 570
  a threshold that only ever stayed quiet would pass the first test and fail the second

  same intent, same data, same week; one of them decays and one does not

null control - the same constant threshold against flat traffic
  peak after 36 months : 1200 rps
  false alerts         : 0
  the threshold is the same age and is still correct
  so the defect is not 'old threshold', it is 'constant compared against a
  quantity that compounds'

how a written-down number ages
  an absolute number       decays at the rate the world grows
  a ratio to a live value  does not decay
  a ratio to a frozen value decays exactly like an absolute number
  the reasoning was a ratio and only the result was stored

Twice the observed peak was a defensible rule, taken from real traffic, with
the reasoning written down beside it. The reasoning was a ratio and what got
stored was 2400. Ordinary traffic passed it in month 18, and by month 36
the page for a normal Tuesday and the page for a 14664 rps incident are the
same page. 570 alerts fired, every one of them obeying the rule exactly.

Trace event types