The breaker was per instance and the dependency was shared

# Self-authored for the EML case corpus (no external origin). The circuit
# breaker is per dependency, trips on consecutive failures, probes with a single
# half-open call, and it demonstrably shed load during a real outage. Where it
# lives is computed below.
#
# The breaker is properly implemented. It is per dependency rather than global,
# so one sick backend does not open the path to a healthy one; it trips on
# consecutive failures rather than a rate, so a single blip does not open it;
# recovery is a single half-open probe rather than a flood; and during the
# March outage it cut the error path from a full timeout to an immediate
# rejection, which is what kept the front end responsive.
#
# The breaker is an object in a process. Two hundred forty processes hold two
# hundred forty of them, and the dependency they protect is one.
#
# Nothing is shared between them.

240 => instances
1 => breakers_per_instance_per_dependency
20 => consecutive_failures_to_trip
1 => half_open_probes_per_instance_per_interval
30 => probe_interval_seconds
0 => breaker_state_shared_between_instances
1 => dependencies_being_protected
60 => seconds_in_a_minute

instances * breakers_per_instance_per_dependency => breakers_protecting_this_dependency
instances * consecutive_failures_to_trip => failed_calls_before_the_whole_fleet_is_open
instances * half_open_probes_per_instance_per_interval => probes_per_interval_against_a_dead_dependency
int(seconds_in_a_minute / probe_interval_seconds) => probe_intervals_per_minute
probes_per_interval_against_a_dead_dependency * probe_intervals_per_minute => probes_per_minute_against_a_dead_dependency

"instances                       : " + str(instances) ^0
"dependencies being protected    : " + str(dependencies_being_protected) ^0
"breakers protecting it          : " + str(breakers_protecting_this_dependency) ^0
"breaker state shared            : " + str(breaker_state_shared_between_instances) ^0
"" ^0
"consecutive failures to trip one: " + str(consecutive_failures_to_trip) ^0
"failed calls before the fleet is open : " + str(failed_calls_before_the_whole_fleet_is_open) ^0
"" ^0
"half-open probes per instance   : " + str(half_open_probes_per_instance_per_interval) ^0
"probe interval, seconds         : " + str(probe_interval_seconds) ^0
"probes per interval, fleet-wide : " + str(probes_per_interval_against_a_dead_dependency) ^0
"probes per minute against a dead dependency : " + str(probes_per_minute_against_a_dead_dependency) ^0
"" ^0

# ---- what the breaker verified ----

"the circuit breaker" ^0
"  scope of one breaker : per dependency, not global" ^0
"  trips on : consecutive failures, not a rate, so a blip" ^0
"    does not open it" ^0
"  recovery : a single half-open probe, not a flood" ^0
"  measured in the March outage : the error path went from" ^0
"    a full timeout to an immediate rejection" ^0
"  verdict : PROTECTS THE CALLER" ^0
"" ^0
"  per dependency rather than global is the design decision" ^0
"  that makes a breaker useful instead of dangerous" ^0
"" ^0

# ---- which side it protects ----

"the two things a breaker can do" ^0
"  stop this process waiting on a dead dependency : yes," ^0
"    completely, and that is what it is for" ^0
"  stop the dependency being called while it is dead : that" ^0
"    is a property of the fleet, not of a process" ^0
"  processes holding one : " + str(instances) ^0
"  state shared between them : " + str(breaker_state_shared_between_instances) ^0
"  so each one learns the outage : independently" ^0
"" ^0
"  the guarantee is local and correct, and the quantity the" ^0
"  dependency experiences is a sum over all of them" ^0
"" ^0
# ---- what the dependency experiences ----

# From the failing service's side, the fleet does not have a breaker. It has two
# hundred and forty independent learners, each of which must be taught the same
# lesson at its own cost, and each of which then knocks once every interval to
# ask whether the lesson is over.
"from the dependency's side" ^0
"  calls it must fail to open one breaker : " + str(consecutive_failures_to_trip) ^0
"  calls it must fail to open all of them : " + str(failed_calls_before_the_whole_fleet_is_open) ^0
"  probes it receives per minute while dead : " + str(probes_per_minute_against_a_dead_dependency) ^0
"  what those probes are : correct, minimal, one per" ^0
"    instance per interval" ^0
"  what they are in aggregate : steady traffic to" ^0
"    something that is trying to restart" ^0
"" ^0

# ---- the recovery is the part that compounds ----

# A single half-open probe per instance is the gentlest possible recovery
# policy, chosen exactly to avoid a thundering herd. Multiplied by the fleet it
# is a herd, and it arrives at the moment the dependency first answers.
"the moment it comes back" ^0
"  probes in flight when it first answers : " + str(probes_per_interval_against_a_dead_dependency) ^0
"  each of them succeeding : yes" ^0
"  what each instance does then : closes its breaker and" ^0
"    resumes full traffic" ^0
"  how many do that : all of them, at once" ^0
"  what the policy was designed to prevent : exactly this" ^0
"  is the policy wrong : no; it is per instance" ^0
"" ^0

# ---- what the outage graph showed ----

"why March looked like a success" ^0
"  front end responsive : yes, measured" ^0
"  error path latency   : cut, measured" ^0
"  the metric watched   : this service's own latency" ^0
"  the dependency's inbound rate during its outage : not on" ^0
"    that graph, and it is on the dependency's" ^0
"  who read both together : nobody" ^0
"" ^0

# ---- null control ----

# The same breaker, with its state in a shared store: the fleet opens once and
# one instance holds the probe token.
1 => nc_breakers_protecting_this_dependency
consecutive_failures_to_trip => nc_failed_calls_before_the_fleet_is_open
half_open_probes_per_instance_per_interval => nc_probes_per_interval

"null control - one breaker for the fleet" ^0
"  scope per dependency : unchanged" ^0
"  breakers protecting it : " + str(nc_breakers_protecting_this_dependency) ^0
"  failed calls before the fleet is open : " + str(nc_failed_calls_before_the_fleet_is_open) ^0
"  probes per interval : " + str(nc_probes_per_interval) ^0
"  the policy did not change; the object holding it stopped" ^0
"  being one per process" ^0
"" ^0

# ---- the rule ----

"what a per-instance breaker guarantees" ^0
"  this process stops waiting on a dead dependency :" ^0
"    exactly, and it is what kept the front end up" ^0
"  the dead dependency stops being called : not addressed;" ^0
"    that is a claim about " + str(instances) + " processes and the breaker" ^0
"    is a claim about one" ^0
"" ^0
"a protection installed per instance is multiplied by the" ^0
"instance count wherever it touches something shared; the" ^0
"gentlest per-process policy is still a fleet-sized policy" ^0
"when the thing it is gentle towards is singular" ^0
"" ^0

"The breaker is per dependency rather than global, trips on " + str(consecutive_failures_to_trip) + " consecutive" ^0
"failures rather than a rate, recovers with " + str(half_open_probes_per_instance_per_interval) + " half-open probe, and in March it" ^0
"cut the error path from a full timeout to an immediate rejection. It is an" ^0
"object in a process and there are " + str(instances) + " of them with " + str(breaker_state_shared_between_instances) + " shared state, so the" ^0
"dependency must fail " + str(failed_calls_before_the_whole_fleet_is_open) + " calls to be shut out and receives " + str(probes_per_minute_against_a_dead_dependency) + " probes a minute" ^0
"while it is down." ^0

instances = 240
breakers_per_instance_per_dependency = 1
consecutive_failures_to_trip = 20
half_open_probes_per_instance_per_interval = 1
probe_interval_seconds = 30
breaker_state_shared_between_instances = 0
dependencies_being_protected = 1
seconds_in_a_minute = 60
breakers_protecting_this_dependency = instances * breakers_per_instance_per_dependency
failed_calls_before_the_whole_fleet_is_open = instances * consecutive_failures_to_trip
probes_per_interval_against_a_dead_dependency = instances * half_open_probes_per_instance_per_interval
probe_intervals_per_minute = int(seconds_in_a_minute / probe_interval_seconds)
probes_per_minute_against_a_dead_dependency = probes_per_interval_against_a_dead_dependency * probe_intervals_per_minute
print("instances                       : " + str(instances))
print("dependencies being protected    : " + str(dependencies_being_protected))
print("breakers protecting it          : " + str(breakers_protecting_this_dependency))
print("breaker state shared            : " + str(breaker_state_shared_between_instances))
print("")
print("consecutive failures to trip one: " + str(consecutive_failures_to_trip))
print("failed calls before the fleet is open : " + str(failed_calls_before_the_whole_fleet_is_open))
print("")
print("half-open probes per instance   : " + str(half_open_probes_per_instance_per_interval))
print("probe interval, seconds         : " + str(probe_interval_seconds))
print("probes per interval, fleet-wide : " + str(probes_per_interval_against_a_dead_dependency))
print("probes per minute against a dead dependency : " + str(probes_per_minute_against_a_dead_dependency))
print("")
print("the circuit breaker")
print("  scope of one breaker : per dependency, not global")
print("  trips on : consecutive failures, not a rate, so a blip")
print("    does not open it")
print("  recovery : a single half-open probe, not a flood")
print("  measured in the March outage : the error path went from")
print("    a full timeout to an immediate rejection")
print("  verdict : PROTECTS THE CALLER")
print("")
print("  per dependency rather than global is the design decision")
print("  that makes a breaker useful instead of dangerous")
print("")
print("the two things a breaker can do")
print("  stop this process waiting on a dead dependency : yes,")
print("    completely, and that is what it is for")
print("  stop the dependency being called while it is dead : that")
print("    is a property of the fleet, not of a process")
print("  processes holding one : " + str(instances))
print("  state shared between them : " + str(breaker_state_shared_between_instances))
print("  so each one learns the outage : independently")
print("")
print("  the guarantee is local and correct, and the quantity the")
print("  dependency experiences is a sum over all of them")
print("")
print("from the dependency's side")
print("  calls it must fail to open one breaker : " + str(consecutive_failures_to_trip))
print("  calls it must fail to open all of them : " + str(failed_calls_before_the_whole_fleet_is_open))
print("  probes it receives per minute while dead : " + str(probes_per_minute_against_a_dead_dependency))
print("  what those probes are : correct, minimal, one per")
print("    instance per interval")
print("  what they are in aggregate : steady traffic to")
print("    something that is trying to restart")
print("")
print("the moment it comes back")
print("  probes in flight when it first answers : " + str(probes_per_interval_against_a_dead_dependency))
print("  each of them succeeding : yes")
print("  what each instance does then : closes its breaker and")
print("    resumes full traffic")
print("  how many do that : all of them, at once")
print("  what the policy was designed to prevent : exactly this")
print("  is the policy wrong : no; it is per instance")
print("")
print("why March looked like a success")
print("  front end responsive : yes, measured")
print("  error path latency   : cut, measured")
print("  the metric watched   : this service's own latency")
print("  the dependency's inbound rate during its outage : not on")
print("    that graph, and it is on the dependency's")
print("  who read both together : nobody")
print("")
nc_breakers_protecting_this_dependency = 1
nc_failed_calls_before_the_fleet_is_open = consecutive_failures_to_trip
nc_probes_per_interval = half_open_probes_per_instance_per_interval
print("null control - one breaker for the fleet")
print("  scope per dependency : unchanged")
print("  breakers protecting it : " + str(nc_breakers_protecting_this_dependency))
print("  failed calls before the fleet is open : " + str(nc_failed_calls_before_the_fleet_is_open))
print("  probes per interval : " + str(nc_probes_per_interval))
print("  the policy did not change; the object holding it stopped")
print("  being one per process")
print("")
print("what a per-instance breaker guarantees")
print("  this process stops waiting on a dead dependency :")
print("    exactly, and it is what kept the front end up")
print("  the dead dependency stops being called : not addressed;")
print("    that is a claim about " + str(instances) + " processes and the breaker")
print("    is a claim about one")
print("")
print("a protection installed per instance is multiplied by the")
print("instance count wherever it touches something shared; the")
print("gentlest per-process policy is still a fleet-sized policy")
print("when the thing it is gentle towards is singular")
print("")
print("The breaker is per dependency rather than global, trips on " + str(consecutive_failures_to_trip) + " consecutive")
print("failures rather than a rate, recovers with " + str(half_open_probes_per_instance_per_interval) + " half-open probe, and in March it")
print("cut the error path from a full timeout to an immediate rejection. It is an")
print("object in a process and there are " + str(instances) + " of them with " + str(breaker_state_shared_between_instances) + " shared state, so the")
print("dependency must fail " + str(failed_calls_before_the_whole_fleet_is_open) + " calls to be shut out and receives " + str(probes_per_minute_against_a_dead_dependency) + " probes a minute")
print("while it is down.")

instances                       : 240
dependencies being protected    : 1
breakers protecting it          : 240
breaker state shared            : 0

consecutive failures to trip one: 20
failed calls before the fleet is open : 4800

half-open probes per instance   : 1
probe interval, seconds         : 30
probes per interval, fleet-wide : 240
probes per minute against a dead dependency : 480

the circuit breaker
  scope of one breaker : per dependency, not global
  trips on : consecutive failures, not a rate, so a blip
    does not open it
  recovery : a single half-open probe, not a flood
  measured in the March outage : the error path went from
    a full timeout to an immediate rejection
  verdict : PROTECTS THE CALLER

  per dependency rather than global is the design decision
  that makes a breaker useful instead of dangerous

the two things a breaker can do
  stop this process waiting on a dead dependency : yes,
    completely, and that is what it is for
  stop the dependency being called while it is dead : that
    is a property of the fleet, not of a process
  processes holding one : 240
  state shared between them : 0
  so each one learns the outage : independently

  the guarantee is local and correct, and the quantity the
  dependency experiences is a sum over all of them

from the dependency's side
  calls it must fail to open one breaker : 20
  calls it must fail to open all of them : 4800
  probes it receives per minute while dead : 480
  what those probes are : correct, minimal, one per
    instance per interval
  what they are in aggregate : steady traffic to
    something that is trying to restart

the moment it comes back
  probes in flight when it first answers : 240
  each of them succeeding : yes
  what each instance does then : closes its breaker and
    resumes full traffic
  how many do that : all of them, at once
  what the policy was designed to prevent : exactly this
  is the policy wrong : no; it is per instance

why March looked like a success
  front end responsive : yes, measured
  error path latency   : cut, measured
  the metric watched   : this service's own latency
  the dependency's inbound rate during its outage : not on
    that graph, and it is on the dependency's
  who read both together : nobody

null control - one breaker for the fleet
  scope per dependency : unchanged
  breakers protecting it : 1
  failed calls before the fleet is open : 20
  probes per interval : 1
  the policy did not change; the object holding it stopped
  being one per process

what a per-instance breaker guarantees
  this process stops waiting on a dead dependency :
    exactly, and it is what kept the front end up
  the dead dependency stops being called : not addressed;
    that is a claim about 240 processes and the breaker
    is a claim about one

a protection installed per instance is multiplied by the
instance count wherever it touches something shared; the
gentlest per-process policy is still a fleet-sized policy
when the thing it is gentle towards is singular

The breaker is per dependency rather than global, trips on 20 consecutive
failures rather than a rate, recovers with 1 half-open probe, and in March it
cut the error path from a full timeout to an immediate rejection. It is an
object in a process and there are 240 of them with 0 shared state, so the
dependency must fail 4800 calls to be shut out and receives 480 probes a minute
while it is down.

Trace event types