The cost appeared only when it stopped being paid

# Self-authored for the EML case corpus (no external origin). A weekly
# maintenance job was cancelled after 140 runs that each reported nothing
# found. When the bill arrived, and who was blamed for it, are computed below.
#
# Cancelling it was the correct reading of the evidence. It had run every week
# for nearly three years, it cost six engineer-hours each time, and its own log
# line said "0 problems found" on all 140 occasions. Every measure available to
# the person deciding said the job produced nothing. Keeping work that has
# never once produced an output is how a team ends up with no time for the work
# that does.
#
# The difficulty is that "0 problems found" is what a job prints when it is
# working, and also what it prints when it is pointless. The two worlds emit
# the same log, so the log cannot tell them apart, and the log was the whole
# case for cancellation.
#
# What separates them is what happens afterwards, and afterwards is far away.
# The consequence accumulates a little each week and crosses the threshold long
# after the change that caused it has left everyone's memory and, more
# importantly, left the window the team searches when something breaks.

120 => base_ms
250 => sla_ms
12 => frag_points_per_week
7 => attribution_window_days

140 => runs
6 => hours_per_run

"the job as the ledger recorded it" ^0
"  runs            : " + str(runs) ^0
"  hours per run   : " + str(hours_per_run) ^0
"  hours spent     : " + str(runs * hours_per_run) ^0
"  problems found  : 0" ^0
"  problems found per run : 0" ^0
"  on this evidence the job returns nothing for " + str(runs * hours_per_run) + " hours" ^0
"" ^0

# ---- what the two worlds print ----

"world A - the job was unnecessary" ^0
"  weekly log line : 0 problems found" ^0
"world B - the job was preventing the problem" ^0
"  weekly log line : 0 problems found" ^0
"  the observation does not distinguish them, and it is the only one taken" ^0
"" ^0

# ---- after it stopped ----

0 => week
0 => frag
0 => breach_week
0 => breach_ms
"week   fragmentation   query ms   sla" ^0
for w in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
    w * frag_points_per_week => frag
    int(base_ms * (100 + frag) / 100) => q
    if breach_week == 0:
        if q > sla_ms:
            w => breach_week
            q => breach_ms
    "  w" + str(w) + "     " + str(frag) + "              " + str(q) + "        " + str(sla_ms) ^0
"" ^0

breach_week * 7 => lag_days
"  breach at week  : " + str(breach_week) + ", query time " + str(breach_ms) + " ms against a " + str(sla_ms) + " ms limit" ^0
"  days since the cancellation : " + str(lag_days) ^0
"  attribution window          : " + str(attribution_window_days) + " days" ^0
"  the cause is outside the window by a factor of " + str(int(lag_days / attribution_window_days)) ^0
"" ^0

# ---- what was inside the window ----

# [change, days before the incident, related to the cause]
[["checkout copy edit", 1, "no"], ["dependency bump", 2, "no"], ["new dashboard panel", 4, "no"], ["log format change", 6, "no"]] => candidates

"changes inside the " + str(attribution_window_days) + "-day window" ^0
0 => related
for c in candidates:
    if c[2] == "yes":
        related + 1 => related
    "  " + c[0] + ", " + str(c[1]) + " days before, related: " + c[2] ^0
"  candidates examined : " + str(len(candidates)) ^0
"  candidates related to the cause : " + str(related) ^0
"  the window was searched correctly and completely, and contained nothing" ^0
"" ^0

# ---- the control ----
#
# A second job was cancelled in the same week. Its effect is immediate rather
# than cumulative, and the same team using the same process diagnosed it in a
# day. The process is not the weakness.

1 => control_lag_days
"control - the other job cancelled that week" ^0
"  job              : cache warmer, also cancelled, also 0 problems found" ^0
"  effect appears after : " + str(control_lag_days) + " day" ^0
"  inside the " + str(attribution_window_days) + "-day window : yes" ^0
"  correctly diagnosed  : yes, next morning, restored the same day" ^0
"  the difference between the two cases is not the team and not the" ^0
"  reasoning, it is " + str(lag_days) + " days against " + str(control_lag_days) ^0
"" ^0

# ---- the two bills ----

40 => incident_hours
"what each choice cost" ^0
"  keeping the job    : " + str(runs * hours_per_run) + " hours, visible, on a line item, every week" ^0
"  cancelling the job : " + str(incident_hours) + " hours of incident plus " + str(lag_days) + " days of degradation" ^0
"  cancelling looks cheaper on any report that covers " + str(attribution_window_days) + " days" ^0
"  and on any report that covers less than " + str(lag_days) ^0
"" ^0

"Cancelling was the right reading of the evidence: " + str(runs) + " runs, 0 findings," ^0
str(runs * hours_per_run) + " hours. A preventive job prints the same line whether it is working" ^0
"or useless, and the bill arrived " + str(lag_days) + " days later, " + str(int(lag_days / attribution_window_days)) + " windows outside the" ^0
"search. The " + str(len(candidates)) + " changes that were examined were all innocent." ^0

base_ms = 120
sla_ms = 250
frag_points_per_week = 12
attribution_window_days = 7
runs = 140
hours_per_run = 6
print("the job as the ledger recorded it")
print("  runs            : " + str(runs))
print("  hours per run   : " + str(hours_per_run))
print("  hours spent     : " + str(runs * hours_per_run))
print("  problems found  : 0")
print("  problems found per run : 0")
print("  on this evidence the job returns nothing for " + str(runs * hours_per_run) + " hours")
print("")
print("world A - the job was unnecessary")
print("  weekly log line : 0 problems found")
print("world B - the job was preventing the problem")
print("  weekly log line : 0 problems found")
print("  the observation does not distinguish them, and it is the only one taken")
print("")
week = 0
frag = 0
breach_week = 0
breach_ms = 0
print("week   fragmentation   query ms   sla")
for w in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]:
    frag = w * frag_points_per_week
    q = int(base_ms * (100 + frag) / 100)
    if breach_week == 0:
        if q > sla_ms:
            breach_week = w
            breach_ms = q
    print("  w" + str(w) + "     " + str(frag) + "              " + str(q) + "        " + str(sla_ms))
print("")
lag_days = breach_week * 7
print("  breach at week  : " + str(breach_week) + ", query time " + str(breach_ms) + " ms against a " + str(sla_ms) + " ms limit")
print("  days since the cancellation : " + str(lag_days))
print("  attribution window          : " + str(attribution_window_days) + " days")
print("  the cause is outside the window by a factor of " + str(int(lag_days / attribution_window_days)))
print("")
candidates = [["checkout copy edit", 1, "no"], ["dependency bump", 2, "no"], ["new dashboard panel", 4, "no"], ["log format change", 6, "no"]]
print("changes inside the " + str(attribution_window_days) + "-day window")
related = 0
for c in candidates:
    if c[2] == "yes":
        related = related + 1
    print("  " + c[0] + ", " + str(c[1]) + " days before, related: " + c[2])
print("  candidates examined : " + str(len(candidates)))
print("  candidates related to the cause : " + str(related))
print("  the window was searched correctly and completely, and contained nothing")
print("")
control_lag_days = 1
print("control - the other job cancelled that week")
print("  job              : cache warmer, also cancelled, also 0 problems found")
print("  effect appears after : " + str(control_lag_days) + " day")
print("  inside the " + str(attribution_window_days) + "-day window : yes")
print("  correctly diagnosed  : yes, next morning, restored the same day")
print("  the difference between the two cases is not the team and not the")
print("  reasoning, it is " + str(lag_days) + " days against " + str(control_lag_days))
print("")
incident_hours = 40
print("what each choice cost")
print("  keeping the job    : " + str(runs * hours_per_run) + " hours, visible, on a line item, every week")
print("  cancelling the job : " + str(incident_hours) + " hours of incident plus " + str(lag_days) + " days of degradation")
print("  cancelling looks cheaper on any report that covers " + str(attribution_window_days) + " days")
print("  and on any report that covers less than " + str(lag_days))
print("")
print("Cancelling was the right reading of the evidence: " + str(runs) + " runs, 0 findings,")
print(str(runs * hours_per_run) + " hours. A preventive job prints the same line whether it is working")
print("or useless, and the bill arrived " + str(lag_days) + " days later, " + str(int(lag_days / attribution_window_days)) + " windows outside the")
print("search. The " + str(len(candidates)) + " changes that were examined were all innocent.")

the job as the ledger recorded it
  runs            : 140
  hours per run   : 6
  hours spent     : 840
  problems found  : 0
  problems found per run : 0
  on this evidence the job returns nothing for 840 hours

world A - the job was unnecessary
  weekly log line : 0 problems found
world B - the job was preventing the problem
  weekly log line : 0 problems found
  the observation does not distinguish them, and it is the only one taken

week   fragmentation   query ms   sla
  w1     12              134        250
  w2     24              148        250
  w3     36              163        250
  w4     48              177        250
  w5     60              192        250
  w6     72              206        250
  w7     84              220        250
  w8     96              235        250
  w9     108              249        250
  w10     120              264        250
  w11     132              278        250

  breach at week  : 10, query time 264 ms against a 250 ms limit
  days since the cancellation : 70
  attribution window          : 7 days
  the cause is outside the window by a factor of 10

changes inside the 7-day window
  checkout copy edit, 1 days before, related: no
  dependency bump, 2 days before, related: no
  new dashboard panel, 4 days before, related: no
  log format change, 6 days before, related: no
  candidates examined : 4
  candidates related to the cause : 0
  the window was searched correctly and completely, and contained nothing

control - the other job cancelled that week
  job              : cache warmer, also cancelled, also 0 problems found
  effect appears after : 1 day
  inside the 7-day window : yes
  correctly diagnosed  : yes, next morning, restored the same day
  the difference between the two cases is not the team and not the
  reasoning, it is 70 days against 1

what each choice cost
  keeping the job    : 840 hours, visible, on a line item, every week
  cancelling the job : 40 hours of incident plus 70 days of degradation
  cancelling looks cheaper on any report that covers 7 days
  and on any report that covers less than 70

Cancelling was the right reading of the evidence: 140 runs, 0 findings,
840 hours. A preventive job prints the same line whether it is working
or useless, and the bill arrived 70 days later, 10 windows outside the
search. The 4 changes that were examined were all innocent.

Trace event types