The total was capped and the parts were not

# Self-authored for the EML case corpus (no external origin). The cloud account
# has a hard spending cap of 10000 a month. Eight services share it. On day 7
# the cap fired. Which services stopped is computed below.
#
# A cap on the account total is the right control and it was argued for at the
# top level, correctly. It is the number the finance team is accountable for.
# It is the only figure that cannot be gamed by moving spend between line items.
# It needs no per-service forecast, so it does not go stale when a service is
# added, and adding a service does not require anyone to re-derive a budget. One
# number, one owner, one alarm.
#
# A cap is a shared resource, and a shared resource with no per-holder limit is
# consumed by whoever consumes fastest. The service that spends the budget is
# not the service that the cap stops - the cap stops everyone, and it stops them
# in the order their next request happens to arrive, which is unrelated to who
# spent the money.
#
# The runaway service is also the one that reaches the cap first and therefore
# gets the most work done before it fires.

10000 => budget
8 => services
40 => normal_per_day
1200 => runaway_per_day
30 => days_in_month

"budget            : " + str(budget) + " per month" ^0
"services sharing it: " + str(services) ^0
"normal spend      : " + str(normal_per_day) + " per service per day" ^0
"normal month total: " + str(services * normal_per_day * days_in_month) + ", which fits" ^0
"" ^0

# ---- one service starts spending 30 times its normal rate ----

((services - 1) * normal_per_day) + runaway_per_day => daily_total

"service X develops a retry loop and spends " + str(runaway_per_day) + " a day" ^0
"account now spends " + str(daily_total) + " a day" ^0
"" ^0

"day   spend that day   cumulative   cap" ^0
0 => spent
0 => cap_day
for d in [1:10]:
    if spent + daily_total <= budget:
        spent + daily_total => spent
        "  " + str(d) + "      " + str(daily_total) + "            " + str(spent) + "        " + str(budget) ^0
    else:
        if cap_day == 0:
            d => cap_day
"" ^0
"  the cap fired on day " + str(cap_day) ^0
"" ^0

# ---- who spent it and who paid for it ----

cap_day - 1 => full_days
runaway_per_day * full_days => x_spent
(services - 1) * normal_per_day * full_days => others_spent

"through day " + str(full_days) ^0
"  service X spent   : " + str(x_spent) + ", which is " + str(int(x_spent * 100 / spent)) + " percent of the budget" ^0
"  the other " + str(services - 1) + " spent : " + str(others_spent) + ", which is " + str(int(others_spent * 100 / spent)) + " percent" ^0
"" ^0
"after the cap fires" ^0
days_in_month - full_days => days_stopped
"  services stopped  : " + str(services) ^0
"  days stopped      : " + str(days_stopped) ^0
"  service-days lost : " + str(services * days_stopped) ^0
"  service-days lost by the service that caused it : " + str(days_stopped) ^0
"  service-days lost by services that did nothing  : " + str((services - 1) * days_stopped) ^0
"" ^0

"  " + str(int((services - 1) * 100 / services)) + " percent of the harm landed on services with no defect" ^0
"" ^0

# ---- the same budget, divided ----
#
# Give each service its own cap of budget/services. Nothing else changes: same
# total, same alarm, same finance number.

int(budget / services) => per_service_cap
int(per_service_cap / runaway_per_day) => x_days_until_own_cap
normal_per_day * days_in_month => normal_month_spend

"per-service cap of " + str(per_service_cap) ^0
"  service X hits its own cap on day  : " + str(x_days_until_own_cap + 1) ^0
"  X spends at most                   : " + str(per_service_cap) ^0
"  a healthy service spends all month : " + str(normal_month_spend) ^0
"  is that under its cap              : " + str(per_service_cap - normal_month_spend) + " to spare" ^0
"  services stopped                   : 1" ^0
"" ^0
"  affected services: " + str(services) + " under one shared cap, 1 under divided caps" ^0
"  the total, the alarm and the finance number are unchanged" ^0
"" ^0

# ---- the shape of the problem ----

"a cap on a sum, with no cap on the terms" ^0
"  bounds the total                        yes, exactly" ^0
"  bounds any single term                  no" ^0
"  stops the term that consumed the budget  no, it stops all of them" ^0
"  fires earlier the worse the runaway is   yes, which shortens the warning" ^0
"  the alarm and the outage are the same event, so there is no interval in" ^0
"  which anyone could have acted" ^0
"" ^0

# ---- the control ----
#
# The cap did its job. The account did not overspend by one unit. Every promise
# the control made was kept, which is why no one reviewed it.

"control - what the cap was built to guarantee" ^0
"  budget          : " + str(budget) ^0
"  actually spent  : " + str(spent) ^0
"  overspend       : " + str(spent - budget) ^0
"  the cap held, and it would have held against any runaway of any size" ^0
"  a control can be perfectly effective and still be aimed at the wrong risk" ^0
"" ^0

# ---- the null control ----
#
# The same single shared cap, with no runaway. Eight well-behaved services
# spend 9600 of 10000 and the cap never fires. The shared cap is not a defect
# on its own; it is a defect exactly when one term can grow without bound.

services * normal_per_day * days_in_month => quiet_month

"null control - the same shared cap, no runaway service" ^0
"  spend for the month : " + str(quiet_month) ^0
"  budget              : " + str(budget) ^0
"  cap fires           : no, " + str(budget - quiet_month) + " to spare" ^0
"  services stopped    : 0" ^0
"  the identical control, and for eleven months it was the right one" ^0
"" ^0

"A cap on the account total is the figure finance is accountable for, it cannot" ^0
"be gamed by moving spend between line items, and it does not need a per-service" ^0
"forecast that goes stale. It held: " + str(spent) + " against a budget of " + str(budget) + ", overspend" ^0
str(spent - budget) + ". It fired on day " + str(cap_day) + ", after one service had taken " + str(int(x_spent * 100 / spent)) + " percent of the" ^0
"month, and it stopped all " + str(services) + " of them for " + str(days_stopped) + " days." ^0

budget = 10000
services = 8
normal_per_day = 40
runaway_per_day = 1200
days_in_month = 30
print("budget            : " + str(budget) + " per month")
print("services sharing it: " + str(services))
print("normal spend      : " + str(normal_per_day) + " per service per day")
print("normal month total: " + str(services * normal_per_day * days_in_month) + ", which fits")
print("")
daily_total = (services - 1) * normal_per_day + runaway_per_day
print("service X develops a retry loop and spends " + str(runaway_per_day) + " a day")
print("account now spends " + str(daily_total) + " a day")
print("")
print("day   spend that day   cumulative   cap")
spent = 0
cap_day = 0
for d in range(1, 11):
    if spent + daily_total <= budget:
        spent = spent + daily_total
        print("  " + str(d) + "      " + str(daily_total) + "            " + str(spent) + "        " + str(budget))
    elif cap_day == 0:
        cap_day = d
print("")
print("  the cap fired on day " + str(cap_day))
print("")
full_days = cap_day - 1
x_spent = runaway_per_day * full_days
others_spent = (services - 1) * normal_per_day * full_days
print("through day " + str(full_days))
print("  service X spent   : " + str(x_spent) + ", which is " + str(int(x_spent * 100 / spent)) + " percent of the budget")
print("  the other " + str(services - 1) + " spent : " + str(others_spent) + ", which is " + str(int(others_spent * 100 / spent)) + " percent")
print("")
print("after the cap fires")
days_stopped = days_in_month - full_days
print("  services stopped  : " + str(services))
print("  days stopped      : " + str(days_stopped))
print("  service-days lost : " + str(services * days_stopped))
print("  service-days lost by the service that caused it : " + str(days_stopped))
print("  service-days lost by services that did nothing  : " + str((services - 1) * days_stopped))
print("")
print("  " + str(int((services - 1) * 100 / services)) + " percent of the harm landed on services with no defect")
print("")
per_service_cap = int(budget / services)
x_days_until_own_cap = int(per_service_cap / runaway_per_day)
normal_month_spend = normal_per_day * days_in_month
print("per-service cap of " + str(per_service_cap))
print("  service X hits its own cap on day  : " + str(x_days_until_own_cap + 1))
print("  X spends at most                   : " + str(per_service_cap))
print("  a healthy service spends all month : " + str(normal_month_spend))
print("  is that under its cap              : " + str(per_service_cap - normal_month_spend) + " to spare")
print("  services stopped                   : 1")
print("")
print("  affected services: " + str(services) + " under one shared cap, 1 under divided caps")
print("  the total, the alarm and the finance number are unchanged")
print("")
print("a cap on a sum, with no cap on the terms")
print("  bounds the total                        yes, exactly")
print("  bounds any single term                  no")
print("  stops the term that consumed the budget  no, it stops all of them")
print("  fires earlier the worse the runaway is   yes, which shortens the warning")
print("  the alarm and the outage are the same event, so there is no interval in")
print("  which anyone could have acted")
print("")
print("control - what the cap was built to guarantee")
print("  budget          : " + str(budget))
print("  actually spent  : " + str(spent))
print("  overspend       : " + str(spent - budget))
print("  the cap held, and it would have held against any runaway of any size")
print("  a control can be perfectly effective and still be aimed at the wrong risk")
print("")
quiet_month = services * normal_per_day * days_in_month
print("null control - the same shared cap, no runaway service")
print("  spend for the month : " + str(quiet_month))
print("  budget              : " + str(budget))
print("  cap fires           : no, " + str(budget - quiet_month) + " to spare")
print("  services stopped    : 0")
print("  the identical control, and for eleven months it was the right one")
print("")
print("A cap on the account total is the figure finance is accountable for, it cannot")
print("be gamed by moving spend between line items, and it does not need a per-service")
print("forecast that goes stale. It held: " + str(spent) + " against a budget of " + str(budget) + ", overspend")
print(str(spent - budget) + ". It fired on day " + str(cap_day) + ", after one service had taken " + str(int(x_spent * 100 / spent)) + " percent of the")
print("month, and it stopped all " + str(services) + " of them for " + str(days_stopped) + " days.")

budget            : 10000 per month
services sharing it: 8
normal spend      : 40 per service per day
normal month total: 9600, which fits

service X develops a retry loop and spends 1200 a day
account now spends 1480 a day

day   spend that day   cumulative   cap
  1      1480            1480        10000
  2      1480            2960        10000
  3      1480            4440        10000
  4      1480            5920        10000
  5      1480            7400        10000
  6      1480            8880        10000

  the cap fired on day 7

through day 6
  service X spent   : 7200, which is 81 percent of the budget
  the other 7 spent : 1680, which is 18 percent

after the cap fires
  services stopped  : 8
  days stopped      : 24
  service-days lost : 192
  service-days lost by the service that caused it : 24
  service-days lost by services that did nothing  : 168

  87 percent of the harm landed on services with no defect

per-service cap of 1250
  service X hits its own cap on day  : 2
  X spends at most                   : 1250
  a healthy service spends all month : 1200
  is that under its cap              : 50 to spare
  services stopped                   : 1

  affected services: 8 under one shared cap, 1 under divided caps
  the total, the alarm and the finance number are unchanged

a cap on a sum, with no cap on the terms
  bounds the total                        yes, exactly
  bounds any single term                  no
  stops the term that consumed the budget  no, it stops all of them
  fires earlier the worse the runaway is   yes, which shortens the warning
  the alarm and the outage are the same event, so there is no interval in
  which anyone could have acted

control - what the cap was built to guarantee
  budget          : 10000
  actually spent  : 8880
  overspend       : -1120
  the cap held, and it would have held against any runaway of any size
  a control can be perfectly effective and still be aimed at the wrong risk

null control - the same shared cap, no runaway service
  spend for the month : 9600
  budget              : 10000
  cap fires           : no, 400 to spare
  services stopped    : 0
  the identical control, and for eleven months it was the right one

A cap on the account total is the figure finance is accountable for, it cannot
be gamed by moving spend between line items, and it does not need a per-service
forecast that goes stale. It held: 8880 against a budget of 10000, overspend
-1120. It fired on day 7, after one service had taken 81 percent of the
month, and it stopped all 8 of them for 24 days.

Trace event types