The redundancy shared a dependency

# Self-authored for the EML case corpus (no external origin). The service runs
# in three availability zones. Each zone is measured at 99.9 percent. The
# availability the three of them actually provide is computed below.
#
# Three zones is the right architecture and the reasoning behind it is sound.
# Zones fail independently by design: separate power, separate cooling,
# separate network, and the provider contracts on exactly that independence.
# Three is the smallest number that survives one loss and still holds a
# quorum. The 99.9 per zone is not a marketing figure either - it was measured
# over a year of real operation.
#
# Independent failures multiply. Every zone fetches its TLS certificate from
# one internal certificate authority at startup and on renewal. That authority
# was never part of the availability calculation because it is not a zone; it
# is a small internal service that had never gone down.
#
# A product of independent terms is only as independent as its least
# independent term. Adding a fourth zone, or a tenth, moves the product term
# and leaves the shared term exactly where it is.

3 => zones
1 => per_zone_unavail_permille
5 => shared_unavail_permille
525600 => minutes_per_year

"zones                    : " + str(zones) ^0
"per-zone unavailability  : " + str(per_zone_unavail_permille) + " per mille, i.e. 99.9 percent" ^0
"shared CA unavailability : " + str(shared_unavail_permille) + " per ten thousand, i.e. 99.95 percent" ^0
"" ^0

# ---- the calculation that was done ----
#
# All three zones down at once, treating the zones as independent. Working in
# parts per thousand cubed keeps this exact in integers: 1/1000 cubed is
# 1 in 1000000000.

1000 => permille_base
permille_base * permille_base * permille_base => independent_denominator

"all three zones down at once, zones treated as independent" ^0
"  probability : 1 in " + str(independent_denominator) ^0
"  expected downtime from that cause : " + str(int(minutes_per_year * 60 / independent_denominator)) + " seconds per year" ^0
"  which rounds to zero on any dashboard" ^0
"" ^0

# ---- the term that was not in the calculation ----

int(minutes_per_year * shared_unavail_permille / 10000) => shared_minutes

"the certificate authority, which all three zones call" ^0
"  unavailability : " + str(shared_unavail_permille) + " per ten thousand" ^0
"  expected downtime : " + str(shared_minutes) + " minutes per year" ^0
"  during which all three zones are up and none of them can serve" ^0
"" ^0

"downtime attributable to" ^0
"  three zones failing together : less than 1 second per year" ^0
"  the one shared dependency    : " + str(shared_minutes) + " minutes per year" ^0
"  the second is larger than the first by a factor no dashboard shows," ^0
"  because the first was the only one that was ever computed" ^0
"" ^0

# ---- what another zone buys ----
#
# The independent term shrinks by a factor of 1000 per zone. The shared term
# does not move at all, and it is the whole of the answer.

"zones   independent term          shared term       total downtime/yr" ^0
for z in [1:6]:
    if z == 1:
        "  1     " + str(int(minutes_per_year * per_zone_unavail_permille / 1000)) + " min                 " + str(shared_minutes) + " min           " + str(int(minutes_per_year * per_zone_unavail_permille / 1000) + shared_minutes) + " min" ^0
    elif z == 2:
        "  2     " + str(int(minutes_per_year * 60 / (permille_base * permille_base))) + " sec                  " + str(shared_minutes) + " min           " + str(shared_minutes) + " min" ^0
    else:
        "  " + str(z) + "     under a second         " + str(shared_minutes) + " min           " + str(shared_minutes) + " min" ^0
"" ^0
"  from two zones onward the answer stops changing" ^0
"  every zone after the second costs money and buys nothing measurable" ^0
"" ^0

# ---- where the shared term hides ----
#
# It is not in the zone list, so it is not in the zone calculation. Every
# question the review asked was about zones.

"questions the design review asked" ^0
"  are the zones independent            yes, contractually" ^0
"  does the service survive one zone    yes, demonstrated" ^0
"  does it survive two zones            yes, demonstrated" ^0
"  is there anything all three call     not asked" ^0
"" ^0
"  the first three questions are about the term that was already negligible" ^0
"  the fourth is about the term that is the entire answer" ^0
"" ^0

# ---- the control ----
#
# The zones really are independent, and the measurement proving it is real: a
# zone loss was exercised and the service stayed up. That drill passes under
# both architectures, which is why it could not distinguish them.

"control - the zone-failure drill, which passed" ^0
"  zones killed in the drill : 1" ^0
"  service stayed up         : yes" ^0
"  zones killed             : 2" ^0
"  service stayed up         : yes" ^0
"  what the drill would show if the CA were also killed : not run" ^0
"  the drill exercises the independent term and is silent on the other" ^0
"  a drill that kills a zone cannot find a dependency shared BY zones" ^0
"" ^0

# ---- the null control ----
#
# The same three zones with a per-zone certificate cache that survives a CA
# outage. The shared term leaves the calculation and the independent term is
# once again the whole answer - which is what the design review believed it
# was measuring all along.

"null control - the same three zones, certificate cached per zone" ^0
"  shared dependency at request time : none" ^0
"  downtime from three zones failing : under a second per year" ^0
"  downtime from the CA              : 0 minutes, requests do not reach it" ^0
"  total                             : under a second per year" ^0
"  the architecture is unchanged; one call moved off the request path" ^0
"  so the finding is not 'three zones is wrong'" ^0
"  it is 'a product of independent terms is worth exactly as much as its" ^0
"  least independent term'" ^0
"" ^0

# ---- the rule ----

"how to find the term that is missing from an availability product" ^0
"  list the components               done, three zones" ^0
"  multiply their failure rates      done, and it is negligible" ^0
"  list what EVERY component calls   this is the missing step" ^0
"  a dependency shared by all N does not appear in any per-component review" ^0
"  and it is not reduced by raising N" ^0
"" ^0

"Zones fail independently by contract, three is the smallest number that holds" ^0
"a quorum after one loss, and the 99.9 was measured over a real year. The" ^0
"three-zone term contributes under a second of downtime a year. The one" ^0
"service all three call contributes " + str(shared_minutes) + " minutes, it was never in the" ^0
"calculation because it is not a zone, and a fourth zone would not have" ^0
"changed it by a single second." ^0

zones = 3
per_zone_unavail_permille = 1
shared_unavail_permille = 5
minutes_per_year = 525600
print("zones                    : " + str(zones))
print("per-zone unavailability  : " + str(per_zone_unavail_permille) + " per mille, i.e. 99.9 percent")
print("shared CA unavailability : " + str(shared_unavail_permille) + " per ten thousand, i.e. 99.95 percent")
print("")
permille_base = 1000
independent_denominator = permille_base * permille_base * permille_base
print("all three zones down at once, zones treated as independent")
print("  probability : 1 in " + str(independent_denominator))
print("  expected downtime from that cause : " + str(int(minutes_per_year * 60 / independent_denominator)) + " seconds per year")
print("  which rounds to zero on any dashboard")
print("")
shared_minutes = int(minutes_per_year * shared_unavail_permille / 10000)
print("the certificate authority, which all three zones call")
print("  unavailability : " + str(shared_unavail_permille) + " per ten thousand")
print("  expected downtime : " + str(shared_minutes) + " minutes per year")
print("  during which all three zones are up and none of them can serve")
print("")
print("downtime attributable to")
print("  three zones failing together : less than 1 second per year")
print("  the one shared dependency    : " + str(shared_minutes) + " minutes per year")
print("  the second is larger than the first by a factor no dashboard shows,")
print("  because the first was the only one that was ever computed")
print("")
print("zones   independent term          shared term       total downtime/yr")
for z in range(1, 7):
    if z == 1:
        print("  1     " + str(int(minutes_per_year * per_zone_unavail_permille / 1000)) + " min                 " + str(shared_minutes) + " min           " + str(int(minutes_per_year * per_zone_unavail_permille / 1000) + shared_minutes) + " min")
    elif z == 2:
        print("  2     " + str(int(minutes_per_year * 60 / (permille_base * permille_base))) + " sec                  " + str(shared_minutes) + " min           " + str(shared_minutes) + " min")
    else:
        print("  " + str(z) + "     under a second         " + str(shared_minutes) + " min           " + str(shared_minutes) + " min")
print("")
print("  from two zones onward the answer stops changing")
print("  every zone after the second costs money and buys nothing measurable")
print("")
print("questions the design review asked")
print("  are the zones independent            yes, contractually")
print("  does the service survive one zone    yes, demonstrated")
print("  does it survive two zones            yes, demonstrated")
print("  is there anything all three call     not asked")
print("")
print("  the first three questions are about the term that was already negligible")
print("  the fourth is about the term that is the entire answer")
print("")
print("control - the zone-failure drill, which passed")
print("  zones killed in the drill : 1")
print("  service stayed up         : yes")
print("  zones killed             : 2")
print("  service stayed up         : yes")
print("  what the drill would show if the CA were also killed : not run")
print("  the drill exercises the independent term and is silent on the other")
print("  a drill that kills a zone cannot find a dependency shared BY zones")
print("")
print("null control - the same three zones, certificate cached per zone")
print("  shared dependency at request time : none")
print("  downtime from three zones failing : under a second per year")
print("  downtime from the CA              : 0 minutes, requests do not reach it")
print("  total                             : under a second per year")
print("  the architecture is unchanged; one call moved off the request path")
print("  so the finding is not 'three zones is wrong'")
print("  it is 'a product of independent terms is worth exactly as much as its")
print("  least independent term'")
print("")
print("how to find the term that is missing from an availability product")
print("  list the components               done, three zones")
print("  multiply their failure rates      done, and it is negligible")
print("  list what EVERY component calls   this is the missing step")
print("  a dependency shared by all N does not appear in any per-component review")
print("  and it is not reduced by raising N")
print("")
print("Zones fail independently by contract, three is the smallest number that holds")
print("a quorum after one loss, and the 99.9 was measured over a real year. The")
print("three-zone term contributes under a second of downtime a year. The one")
print("service all three call contributes " + str(shared_minutes) + " minutes, it was never in the")
print("calculation because it is not a zone, and a fourth zone would not have")
print("changed it by a single second.")

zones                    : 3
per-zone unavailability  : 1 per mille, i.e. 99.9 percent
shared CA unavailability : 5 per ten thousand, i.e. 99.95 percent

all three zones down at once, zones treated as independent
  probability : 1 in 1000000000
  expected downtime from that cause : 0 seconds per year
  which rounds to zero on any dashboard

the certificate authority, which all three zones call
  unavailability : 5 per ten thousand
  expected downtime : 262 minutes per year
  during which all three zones are up and none of them can serve

downtime attributable to
  three zones failing together : less than 1 second per year
  the one shared dependency    : 262 minutes per year
  the second is larger than the first by a factor no dashboard shows,
  because the first was the only one that was ever computed

zones   independent term          shared term       total downtime/yr
  1     525 min                 262 min           787 min
  2     31 sec                  262 min           262 min
  3     under a second         262 min           262 min
  4     under a second         262 min           262 min
  5     under a second         262 min           262 min
  6     under a second         262 min           262 min

  from two zones onward the answer stops changing
  every zone after the second costs money and buys nothing measurable

questions the design review asked
  are the zones independent            yes, contractually
  does the service survive one zone    yes, demonstrated
  does it survive two zones            yes, demonstrated
  is there anything all three call     not asked

  the first three questions are about the term that was already negligible
  the fourth is about the term that is the entire answer

control - the zone-failure drill, which passed
  zones killed in the drill : 1
  service stayed up         : yes
  zones killed             : 2
  service stayed up         : yes
  what the drill would show if the CA were also killed : not run
  the drill exercises the independent term and is silent on the other
  a drill that kills a zone cannot find a dependency shared BY zones

null control - the same three zones, certificate cached per zone
  shared dependency at request time : none
  downtime from three zones failing : under a second per year
  downtime from the CA              : 0 minutes, requests do not reach it
  total                             : under a second per year
  the architecture is unchanged; one call moved off the request path
  so the finding is not 'three zones is wrong'
  it is 'a product of independent terms is worth exactly as much as its
  least independent term'

how to find the term that is missing from an availability product
  list the components               done, three zones
  multiply their failure rates      done, and it is negligible
  list what EVERY component calls   this is the missing step
  a dependency shared by all N does not appear in any per-component review
  and it is not reduced by raising N

Zones fail independently by contract, three is the smallest number that holds
a quorum after one loss, and the 99.9 was measured over a real year. The
three-zone term contributes under a second of downtime a year. The one
service all three call contributes 262 minutes, it was never in the
calculation because it is not a zone, and a fourth zone would not have
changed it by a single second.

Trace event types