The remediation fixed it before anyone could be told

# Self-authored for the EML case corpus (no external origin). Auto-remediation
# restarts the failing process and alert deduplication suppresses the repeat.
# What each does, and what the pair does to escalation, are computed below.
#
# Both are correct. Auto-remediation was added because a stuck worker used to
# page someone at 03:00 for a fix that is always the same restart, and it has
# recovered 1400 incidents without human involvement. Deduplication was added
# because one flapping host once produced 900 pages in an hour. Nobody wants
# either of them removed and both have measured wins.
#
# Escalation is driven by an alert that stays open. Remediation closes the
# alert by fixing the symptom, and dedup collapses the recurrence into the
# already-closed one. So a fault that recurs forever produces a sequence of
# short, individually-resolved alerts, and the thing that would escalate is
# an alert that stays open, which by construction never happens.
#
# Occurrences are counted per fault, with what reached a person.

# [fault, occurrences per week, remediation fixes it, minutes to recur, escalation threshold minutes]
[["worker deadlock", 210, "yes", 34, 30], ["disk fills on node 7", 96, "yes", 105, 30], ["cert near expiry", 4, "no", 0, 30], ["memory leak in parser", 340, "yes", 21, 30], ["upstream dns flap", 58, "yes", 240, 30]] => faults

len(faults) => n

"fault                  per week   auto-fixed   recurs after   reached a person" ^0
0 => paged
0 => silent
for f in faults:
    "" => who
    if f[2] == "yes":
        "no" => who
        silent + f[1] => silent
    else:
        "yes" => who
        paged + f[1] => paged
    "  " + f[0] + "   " + str(f[1]) + "        " + f[2] + "          " + str(f[3]) + " min        " + who ^0
"" ^0

0 => total
for f in faults:
    total + f[1] => total
"occurrences a week        : " + str(total) ^0
"auto-remediated           : " + str(silent) + ", " + str(int(silent * 1000 / total)) + " per 1000" ^0
"reached a person          : " + str(paged) + ", " + str(int(paged * 1000 / total)) + " per 1000" ^0
"  (per 1000 rather than percent, because " + str(paged) + " of " + str(total) + " floors to 0%)" ^0
"" ^0

# ---- what escalation is waiting for ----

"the escalation rule: an alert open longer than " + str(faults[0][4]) + " minutes" ^0
0 => would_escalate
for f in faults:
    if f[2] == "yes":
        if f[3] > f[4]:
            would_escalate + 1 => would_escalate
"  auto-fixed faults whose recurrence gap exceeds the threshold : " + str(would_escalate) ^0
"  those recur as a NEW alert rather than an open one, so the timer restarts" ^0
0 => never_open
for f in faults:
    if f[2] == "yes":
        never_open + 1 => never_open
"  auto-fixed faults that can ever hold an alert open past the threshold : 0" ^0
"  of " + str(never_open) + " auto-fixed faults, because remediation closes it first" ^0
"" ^0

# ---- what the pair does to the record ----

"how the same fault appears in each system" ^0
for f in faults:
    if f[0] == "memory leak in parser":
        "  " + f[0] + " : " + str(f[1]) + " occurrences a week" ^0
        "  alerts opened   : " + str(f[1]) ^0
        "  alerts escalated: 0" ^0
        "  pages sent      : 0" ^0
        "  after dedup, distinct alert identities : 1" ^0
        "  on the weekly report it is one line saying resolved" ^0
"" ^0

# ---- the cost that is being paid ----

# [fault, minutes of degraded service per occurrence]
[["worker deadlock", 2], ["disk fills on node 7", 4], ["memory leak in parser", 3], ["upstream dns flap", 1]] => costs
0 => degraded
for c in costs:
    for f in faults:
        if f[0] == c[0]:
            degraded + f[1] * c[1] => degraded
"degraded minutes a week, from faults nobody is told about" ^0
for c in costs:
    for f in faults:
        if f[0] == c[0]:
            "  " + c[0] + " : " + str(f[1]) + " x " + str(c[1]) + " = " + str(f[1] * c[1]) + " minutes" ^0
"  total : " + str(degraded) + " minutes a week" ^0
"  which is " + str(int(degraded * 100 / (7 * 24 * 60))) + "% of the week" ^0
"  pages generated by it : 0" ^0
"" ^0

# ---- the one fault that does escalate ----

for f in faults:
    if f[2] == "no":
        "the fault that reaches a person" ^0
        "  " + f[0] + " : " + str(f[1]) + " a week" ^0
        "  remediation has no rule for it, so the alert stays open" ^0
        "  it is the rarest fault here, by a factor of " + str(int(total / f[1])) ^0
        "  the on-call rota's entire visible workload is the fault class that" ^0
        "  nobody automated, which is not the same as the largest one" ^0
"" ^0

# ---- what would make the recurrence visible ----

"a counter that survives the fix" ^0
"  remediation events per fault per week : already recorded" ^0
"  escalation rule reads that counter    : no" ^0
"  escalation rule reads alert age       : yes" ^0
"  so the quantity that grows with the problem exists, is stored, and is" ^0
"  not an input to the only rule that can summon a person" ^0
0 => over_ten
for f in faults:
    if f[2] == "yes":
        if f[1] > 10:
            over_ten + 1 => over_ten
"  auto-fixed faults recurring more than 10 times a week : " + str(over_ten) ^0
"" ^0

# ---- the control: a fault fixed once and gone ----
#
# Where remediation removes the cause rather than the symptom, the occurrence
# count falls to nothing and silence is the correct signal.

[["stale mount after reboot", 40, 0]] => cured
for c in cured:
    "control - " + c[0] ^0
    "  before the fix : " + str(c[1]) + " a week, auto-remediated, no pages" ^0
    "  after the fix  : " + str(c[2]) + " a week" ^0
    "  the same two controls, the same silence, and here the silence is" ^0
    "  reporting an absence rather than hiding a recurrence" ^0
    "  what separates this case from the others is whether the count fell" ^0
"" ^0

"Auto-remediation has recovered 1400 incidents and dedup ended a 900-page" ^0
"hour. Escalation triggers on an alert staying open, and remediation's whole" ^0
"job is to close it, so " + str(silent) + " occurrences a week cannot reach anybody." ^0

faults = [["worker deadlock", 210, "yes", 34, 30], ["disk fills on node 7", 96, "yes", 105, 30], ["cert near expiry", 4, "no", 0, 30], ["memory leak in parser", 340, "yes", 21, 30], ["upstream dns flap", 58, "yes", 240, 30]]
n = len(faults)
print("fault                  per week   auto-fixed   recurs after   reached a person")
paged = 0
silent = 0
for f in faults:
    who = ""
    if f[2] == "yes":
        who = "no"
        silent = silent + f[1]
    else:
        who = "yes"
        paged = paged + f[1]
    print("  " + f[0] + "   " + str(f[1]) + "        " + f[2] + "          " + str(f[3]) + " min        " + who)
print("")
total = 0
for f in faults:
    total = total + f[1]
print("occurrences a week        : " + str(total))
print("auto-remediated           : " + str(silent) + ", " + str(int(silent * 1000 / total)) + " per 1000")
print("reached a person          : " + str(paged) + ", " + str(int(paged * 1000 / total)) + " per 1000")
print("  (per 1000 rather than percent, because " + str(paged) + " of " + str(total) + " floors to 0%)")
print("")
print("the escalation rule: an alert open longer than " + str(faults[0][4]) + " minutes")
would_escalate = 0
for f in faults:
    if f[2] == "yes":
        if f[3] > f[4]:
            would_escalate = would_escalate + 1
print("  auto-fixed faults whose recurrence gap exceeds the threshold : " + str(would_escalate))
print("  those recur as a NEW alert rather than an open one, so the timer restarts")
never_open = 0
for f in faults:
    if f[2] == "yes":
        never_open = never_open + 1
print("  auto-fixed faults that can ever hold an alert open past the threshold : 0")
print("  of " + str(never_open) + " auto-fixed faults, because remediation closes it first")
print("")
print("how the same fault appears in each system")
for f in faults:
    if f[0] == "memory leak in parser":
        print("  " + f[0] + " : " + str(f[1]) + " occurrences a week")
        print("  alerts opened   : " + str(f[1]))
        print("  alerts escalated: 0")
        print("  pages sent      : 0")
        print("  after dedup, distinct alert identities : 1")
        print("  on the weekly report it is one line saying resolved")
print("")
costs = [["worker deadlock", 2], ["disk fills on node 7", 4], ["memory leak in parser", 3], ["upstream dns flap", 1]]
degraded = 0
for c in costs:
    for f in faults:
        if f[0] == c[0]:
            degraded = degraded + f[1] * c[1]
print("degraded minutes a week, from faults nobody is told about")
for c in costs:
    for f in faults:
        if f[0] == c[0]:
            print("  " + c[0] + " : " + str(f[1]) + " x " + str(c[1]) + " = " + str(f[1] * c[1]) + " minutes")
print("  total : " + str(degraded) + " minutes a week")
print("  which is " + str(int(degraded * 100 / (7 * 24 * 60))) + "% of the week")
print("  pages generated by it : 0")
print("")
for f in faults:
    if f[2] == "no":
        print("the fault that reaches a person")
        print("  " + f[0] + " : " + str(f[1]) + " a week")
        print("  remediation has no rule for it, so the alert stays open")
        print("  it is the rarest fault here, by a factor of " + str(int(total / f[1])))
        print("  the on-call rota's entire visible workload is the fault class that")
        print("  nobody automated, which is not the same as the largest one")
print("")
print("a counter that survives the fix")
print("  remediation events per fault per week : already recorded")
print("  escalation rule reads that counter    : no")
print("  escalation rule reads alert age       : yes")
print("  so the quantity that grows with the problem exists, is stored, and is")
print("  not an input to the only rule that can summon a person")
over_ten = 0
for f in faults:
    if f[2] == "yes":
        if f[1] > 10:
            over_ten = over_ten + 1
print("  auto-fixed faults recurring more than 10 times a week : " + str(over_ten))
print("")
cured = [["stale mount after reboot", 40, 0]]
for c in cured:
    print("control - " + c[0])
    print("  before the fix : " + str(c[1]) + " a week, auto-remediated, no pages")
    print("  after the fix  : " + str(c[2]) + " a week")
    print("  the same two controls, the same silence, and here the silence is")
    print("  reporting an absence rather than hiding a recurrence")
    print("  what separates this case from the others is whether the count fell")
print("")
print("Auto-remediation has recovered 1400 incidents and dedup ended a 900-page")
print("hour. Escalation triggers on an alert staying open, and remediation's whole")
print("job is to close it, so " + str(silent) + " occurrences a week cannot reach anybody.")

fault                  per week   auto-fixed   recurs after   reached a person
  worker deadlock   210        yes          34 min        no
  disk fills on node 7   96        yes          105 min        no
  cert near expiry   4        no          0 min        yes
  memory leak in parser   340        yes          21 min        no
  upstream dns flap   58        yes          240 min        no

occurrences a week        : 708
auto-remediated           : 704, 994 per 1000
reached a person          : 4, 5 per 1000
  (per 1000 rather than percent, because 4 of 708 floors to 0%)

the escalation rule: an alert open longer than 30 minutes
  auto-fixed faults whose recurrence gap exceeds the threshold : 3
  those recur as a NEW alert rather than an open one, so the timer restarts
  auto-fixed faults that can ever hold an alert open past the threshold : 0
  of 4 auto-fixed faults, because remediation closes it first

how the same fault appears in each system
  memory leak in parser : 340 occurrences a week
  alerts opened   : 340
  alerts escalated: 0
  pages sent      : 0
  after dedup, distinct alert identities : 1
  on the weekly report it is one line saying resolved

degraded minutes a week, from faults nobody is told about
  worker deadlock : 210 x 2 = 420 minutes
  disk fills on node 7 : 96 x 4 = 384 minutes
  memory leak in parser : 340 x 3 = 1020 minutes
  upstream dns flap : 58 x 1 = 58 minutes
  total : 1882 minutes a week
  which is 18% of the week
  pages generated by it : 0

the fault that reaches a person
  cert near expiry : 4 a week
  remediation has no rule for it, so the alert stays open
  it is the rarest fault here, by a factor of 177
  the on-call rota's entire visible workload is the fault class that
  nobody automated, which is not the same as the largest one

a counter that survives the fix
  remediation events per fault per week : already recorded
  escalation rule reads that counter    : no
  escalation rule reads alert age       : yes
  so the quantity that grows with the problem exists, is stored, and is
  not an input to the only rule that can summon a person
  auto-fixed faults recurring more than 10 times a week : 4

control - stale mount after reboot
  before the fix : 40 a week, auto-remediated, no pages
  after the fix  : 0 a week
  the same two controls, the same silence, and here the silence is
  reporting an absence rather than hiding a recurrence
  what separates this case from the others is whether the count fell

Auto-remediation has recovered 1400 incidents and dedup ended a 900-page
hour. Escalation triggers on an alert staying open, and remediation's whole
job is to close it, so 704 occurrences a week cannot reach anybody.

Trace event types