Every rollback made the next deploy bigger

# Self-authored for the EML case corpus (no external origin). A bad deploy is
# followed by more review before the next one. What that does to the next
# deploy is simulated rather than argued.
#
# Slowing down after a failure is the correct instinct and it is what every
# incident review recommends. More eyes on a change finds more in it, and the
# team that just broke production has evidence that its current rate is too
# fast for its current process.
#
# Waiting does not stop the changes arriving. It stores them, so the next
# deploy carries everything written during the wait, and a deploy's risk is
# mostly its size. The response to a failure is an input to the next failure.
#
# Both policies are run over the same stream of changes.

3 => changes_per_day
60 => days
6 => risk_per_failure

# Returns [failures, changes shipped, largest batch, interval at the end]
def run(after_failure, after_clean, start_interval):
    start_interval => interval
    0 => day
    0 => pending
    0 => failures
    0 => shipped
    0 => biggest
    while day < days:
        pending + changes_per_day => pending
        day + 1 => day
        if day % interval == 0:
            pending => batch
            if batch > biggest:
                batch => biggest
            shipped + batch => shipped
            int(batch / risk_per_failure) => broke
            0 => pending
            if broke > 0:
                failures + broke => failures
                interval + after_failure => interval
            else:
                interval - after_clean => interval
            if interval < 1:
                1 => interval
            if interval > 30:
                30 => interval
    return [failures, shipped, biggest, interval]

"changes written per day : " + str(changes_per_day) ^0
"days simulated          : " + str(days) ^0
"a deploy breaks once per " + str(risk_per_failure) + " changes it carries" ^0
"both runs start the day after an incident, at a 4-day interval" ^0
"" ^0

run(1, 1, 4) => cautious
run(0 - 1, 1, 4) => forward

"policy                          failures   shipped   largest deploy   interval at the end" ^0
"  wait longer after a failure   " + str(cautious[0]) + "         " + str(cautious[1]) + "       " + str(cautious[2]) + "               " + str(cautious[3]) ^0
"  deploy sooner after a failure " + str(forward[0]) + "          " + str(forward[1]) + "       " + str(forward[2]) + "                " + str(forward[3]) ^0
"" ^0

if cautious[0] > forward[0]:
    "waiting longer produced " + str(cautious[0] - forward[0]) + " more failures than deploying sooner" ^0
if cautious[2] > forward[2]:
    "and its largest deploy was " + str(cautious[2] - forward[2]) + " changes bigger" ^0
"" ^0

# ---- the interval, step by step, under the cautious rule ----

"the cautious rule, deploy by deploy" ^0
4 => interval
0 => day
0 => pending
0 => k
0 => broke_deploys
while day < days:
    pending + changes_per_day => pending
    day + 1 => day
    if day % interval == 0:
        pending => batch
        int(batch / risk_per_failure) => broke
        k + 1 => k
        if broke > 0:
            broke_deploys + 1 => broke_deploys
        if k <= 6:
            "" => verdict
            if broke > 0:
                "broke, wait " + str(interval + 1) + " days next" => verdict
            else:
                "clean" => verdict
            "  deploy " + str(k) + " : after " + str(interval) + " days, " + str(batch) + " changes, " + verdict ^0
        0 => pending
        if broke > 0:
            interval + 1 => interval
        else:
            interval - 1 => interval
        if interval < 1:
            1 => interval
        if interval > 30:
            30 => interval
"  deploys in the whole run : " + str(k) + ", of which broke : " + str(broke_deploys) ^0
"  breakages counted : " + str(cautious[0]) + ", because a big deploy breaks more than once" ^0
"  the interval goes 4 to " + str(cautious[3]) + "; the batches do not rise monotonically," ^0
"  because a deploy only happens on a day divisible by the current interval," ^0
"  so lengthening it sometimes lands on a shorter gap first" ^0
"" ^0

# ---- how long a change waits before it is live ----
#
# The interval is also latency. A change written on day one of a wait sits
# until the wait ends, whatever happens to it after that.

"the interval each run ends on, which a change waits half of on average" ^0
"  cautious : " + str(cautious[3]) + " days" ^0
"  forward  : " + str(forward[3]) + " days" ^0
if cautious[3] > forward[3]:
    "  the cautious rule ends " + str(cautious[3] - forward[3]) + " days slower per deploy as well as more broken" ^0
"" ^0

# ---- where the caution would have been right ----
#
# If risk came from the number of deploys rather than their size, waiting
# would reduce it. That is the world the instinct is calibrated for.

30 => per_deploy_risk
int(days / cautious[3]) => cautious_deploys
int(days / forward[3]) => forward_deploys
"if each deploy carried a fixed risk regardless of size" ^0
"  deploys under the cautious rule : about " + str(cautious_deploys) ^0
"  deploys under the forward rule  : about " + str(forward_deploys) ^0
if forward_deploys > cautious_deploys:
    "  the cautious rule would win, with " + str(forward_deploys - cautious_deploys) + " fewer exposures" ^0
"  which risk model holds is a fact about the change, not about the policy" ^0
"" ^0

# ---- the control: changes that do not accumulate ----
#
# Where nothing arrives during the wait, a longer interval produces the same
# deploy later, and the loop has nothing to feed on.

0 => still_pending
4 => c_interval
0 => c_day
0 => c_failures
while c_day < days:
    c_day + 1 => c_day
    if c_day % c_interval == 0:
        int(still_pending / risk_per_failure) => broke
        if broke > 0:
            c_failures + broke => c_failures
            c_interval + 1 => c_interval
"control - a frozen codebase, no changes arriving" ^0
"  failures : " + str(c_failures) + ", interval at the end : " + str(c_interval) ^0
if c_failures == 0:
    "  waiting costs nothing here, because the batch does not grow while you wait" ^0
"" ^0

"Reviewing harder after a failure finds more in each change, and the wait it" ^0
"buys is stored as batch size. The size is what the next failure is drawn" ^0
"from, so the response is upstream of the thing it responds to." ^0

changes_per_day = 3
days = 60
risk_per_failure = 6

def run(after_failure, after_clean, start_interval):
    interval = start_interval
    day = 0
    pending = 0
    failures = 0
    shipped = 0
    biggest = 0
    while day < days:
        pending = pending + changes_per_day
        day = day + 1
        if day % interval == 0:
            batch = pending
            if batch > biggest:
                biggest = batch
            shipped = shipped + batch
            broke = int(batch / risk_per_failure)
            pending = 0
            if broke > 0:
                failures = failures + broke
                interval = interval + after_failure
            else:
                interval = interval - after_clean
            if interval < 1:
                interval = 1
            if interval > 30:
                interval = 30
    return [failures, shipped, biggest, interval]

print("changes written per day : " + str(changes_per_day))
print("days simulated          : " + str(days))
print("a deploy breaks once per " + str(risk_per_failure) + " changes it carries")
print("both runs start the day after an incident, at a 4-day interval")
print("")
cautious = run(1, 1, 4)
forward = run(0 - 1, 1, 4)
print("policy                          failures   shipped   largest deploy   interval at the end")
print("  wait longer after a failure   " + str(cautious[0]) + "         " + str(cautious[1]) + "       " + str(cautious[2]) + "               " + str(cautious[3]))
print("  deploy sooner after a failure " + str(forward[0]) + "          " + str(forward[1]) + "       " + str(forward[2]) + "                " + str(forward[3]))
print("")
if cautious[0] > forward[0]:
    print("waiting longer produced " + str(cautious[0] - forward[0]) + " more failures than deploying sooner")
if cautious[2] > forward[2]:
    print("and its largest deploy was " + str(cautious[2] - forward[2]) + " changes bigger")
print("")
print("the cautious rule, deploy by deploy")
interval = 4
day = 0
pending = 0
k = 0
broke_deploys = 0
while day < days:
    pending = pending + changes_per_day
    day = day + 1
    if day % interval == 0:
        batch = pending
        broke = int(batch / risk_per_failure)
        k = k + 1
        if broke > 0:
            broke_deploys = broke_deploys + 1
        if k <= 6:
            verdict = ""
            if broke > 0:
                verdict = "broke, wait " + str(interval + 1) + " days next"
            else:
                verdict = "clean"
            print("  deploy " + str(k) + " : after " + str(interval) + " days, " + str(batch) + " changes, " + verdict)
        pending = 0
        if broke > 0:
            interval = interval + 1
        else:
            interval = interval - 1
        if interval < 1:
            interval = 1
        if interval > 30:
            interval = 30
print("  deploys in the whole run : " + str(k) + ", of which broke : " + str(broke_deploys))
print("  breakages counted : " + str(cautious[0]) + ", because a big deploy breaks more than once")
print("  the interval goes 4 to " + str(cautious[3]) + "; the batches do not rise monotonically,")
print("  because a deploy only happens on a day divisible by the current interval,")
print("  so lengthening it sometimes lands on a shorter gap first")
print("")
print("the interval each run ends on, which a change waits half of on average")
print("  cautious : " + str(cautious[3]) + " days")
print("  forward  : " + str(forward[3]) + " days")
if cautious[3] > forward[3]:
    print("  the cautious rule ends " + str(cautious[3] - forward[3]) + " days slower per deploy as well as more broken")
print("")
per_deploy_risk = 30
cautious_deploys = int(days / cautious[3])
forward_deploys = int(days / forward[3])
print("if each deploy carried a fixed risk regardless of size")
print("  deploys under the cautious rule : about " + str(cautious_deploys))
print("  deploys under the forward rule  : about " + str(forward_deploys))
if forward_deploys > cautious_deploys:
    print("  the cautious rule would win, with " + str(forward_deploys - cautious_deploys) + " fewer exposures")
print("  which risk model holds is a fact about the change, not about the policy")
print("")
still_pending = 0
c_interval = 4
c_day = 0
c_failures = 0
while c_day < days:
    c_day = c_day + 1
    if c_day % c_interval == 0:
        broke = int(still_pending / risk_per_failure)
        if broke > 0:
            c_failures = c_failures + broke
            c_interval = c_interval + 1
print("control - a frozen codebase, no changes arriving")
print("  failures : " + str(c_failures) + ", interval at the end : " + str(c_interval))
if c_failures == 0:
    print("  waiting costs nothing here, because the batch does not grow while you wait")
print("")
print("Reviewing harder after a failure finds more in each change, and the wait it")
print("buys is stored as batch size. The size is what the next failure is drawn")
print("from, so the response is upstream of the thing it responds to.")

changes written per day : 3
days simulated          : 60
a deploy breaks once per 6 changes it carries
both runs start the day after an incident, at a 4-day interval

policy                          failures   shipped   largest deploy   interval at the end
  wait longer after a failure   29         180       12               30
  deploy sooner after a failure 4          180       12                1

waiting longer produced 25 more failures than deploying sooner

the cautious rule, deploy by deploy
  deploy 1 : after 4 days, 12 changes, broke, wait 5 days next
  deploy 2 : after 5 days, 3 changes, clean
  deploy 3 : after 4 days, 9 changes, broke, wait 5 days next
  deploy 4 : after 5 days, 6 changes, broke, wait 6 days next
  deploy 5 : after 6 days, 6 changes, broke, wait 7 days next
  deploy 6 : after 7 days, 6 changes, broke, wait 8 days next
  deploys in the whole run : 29, of which broke : 28
  breakages counted : 29, because a big deploy breaks more than once
  the interval goes 4 to 30; the batches do not rise monotonically,
  because a deploy only happens on a day divisible by the current interval,
  so lengthening it sometimes lands on a shorter gap first

the interval each run ends on, which a change waits half of on average
  cautious : 30 days
  forward  : 1 days
  the cautious rule ends 29 days slower per deploy as well as more broken

if each deploy carried a fixed risk regardless of size
  deploys under the cautious rule : about 2
  deploys under the forward rule  : about 60
  the cautious rule would win, with 58 fewer exposures
  which risk model holds is a fact about the change, not about the policy

control - a frozen codebase, no changes arriving
  failures : 0, interval at the end : 4
  waiting costs nothing here, because the batch does not grow while you wait

Reviewing harder after a failure finds more in each change, and the wait it
buys is stored as batch size. The size is what the next failure is drawn
from, so the response is upstream of the thing it responds to.

Trace event types