The throttle protected the service and moved the failure out

# Self-authored for the EML case corpus (no external origin). A rate limiter
# holds the service at its measured capacity. The service's error rate is zero
# and its latency is flat. What the system as a whole delivers is computed
# below.
#
# Shedding load at the edge is correct and it is the standard answer, for good
# reasons. A service that accepts more than it can serve does not serve more; it
# serves everything slowly, then times out, and a timeout costs the work already
# done plus the retry. Refusing early returns a clear signal to the caller while
# it can still act on it, and it keeps the accepted requests fast. The limit was
# set from a measured capacity, not a guess.
#
# A rejection is a failure that has been moved rather than removed. The service
# stops recording it because the service is no longer where it happens. The
# caller records it, and the caller's response to it is a retry, which arrives
# as new offered load.
#
# The limiter has no term for what the caller does next, so the loop it closes
# is not visible from either side alone.

2000 => capacity_rps
2600 => offered_rps
3 => retries_per_rejection

offered_rps - capacity_rps => rejected_rps

"capacity      : " + str(capacity_rps) + " rps, measured" ^0
"offered       : " + str(offered_rps) + " rps" ^0
"rejected      : " + str(rejected_rps) + " rps" ^0
"" ^0

"the service's own numbers" ^0
"  requests served     : " + str(capacity_rps) + " rps" ^0
"  latency             : flat, it is never overloaded" ^0
"  server error rate   : 0, a 429 is not a server error" ^0
"  SLO                 : met" ^0
"" ^0
"the caller's numbers" ^0
"  requests refused    : " + str(rejected_rps) + " rps" ^0
"  share refused       : " + str(int(rejected_rps * 100 / offered_rps)) + " percent" ^0
"" ^0

# ---- what the caller does with a 429 ----

rejected_rps * retries_per_rejection => retry_rps
offered_rps + retry_rps => offered_round_two
offered_round_two - capacity_rps => rejected_round_two

"each rejection is retried " + str(retries_per_rejection) + " times" ^0
"  retry traffic added : " + str(retry_rps) + " rps" ^0
"  offered, round two  : " + str(offered_round_two) + " rps" ^0
"  rejected, round two : " + str(rejected_round_two) + " rps" ^0
"  rejections grew by  : " + str(int((rejected_round_two - rejected_rps) * 100 / rejected_rps)) + " percent" ^0
"" ^0

"  served, round one : " + str(capacity_rps) ^0
"  served, round two : " + str(capacity_rps) ^0
"  the served column does not move, which is the limiter working" ^0
"" ^0

# ---- the loop ----

"round   offered   served   rejected" ^0
offered_rps => current_offered
for r in [1:4]:
    current_offered - capacity_rps => rej
    "  " + str(r) + "       " + str(current_offered) + "      " + str(capacity_rps) + "     " + str(rej) ^0
    offered_rps + (rej * retries_per_rejection) => current_offered
"" ^0
"  the limiter is doing exactly what it promised at every row" ^0
"  and the rejected column is the input to the next row" ^0
"" ^0

# ---- where the work goes ----

int(offered_round_two * 100 / offered_rps) => load_multiplier_pct

"  original demand        : " + str(offered_rps) + " rps of real work" ^0
"  load reaching the limiter : " + str(offered_round_two) + " rps" ^0
"  of that, retries       : " + str(retry_rps) + " rps, none of it new work" ^0
"  load multiplier        : " + str(load_multiplier_pct) + " percent of the original" ^0
"  every retry consumes a connection, a TLS handshake and a limiter" ^0
"  decision, and none of them reach a worker" ^0
"" ^0

# ---- what a caller-side answer would do ----
#
# The retry is the caller's correct response to a transient failure. It is the
# wrong response to a capacity signal, and the two are not distinguishable from
# the status code alone unless the response says how long to wait.

"the same rejection, with a retry-after the caller honours" ^0
"  retries within the second : 0" ^0
"  offered                   : " + str(offered_rps) + " rps, unchanged" ^0
"  rejected                  : " + str(rejected_rps) + " rps" ^0
"  the demand is the same and the amplification is gone" ^0
"  the limiter's job was to tell the caller something, and the number it" ^0
"  needed to say is not in the status code" ^0
"" ^0

# ---- the control ----
#
# The limiter, against everything it promised. It holds the service at capacity,
# it never lets latency rise, it never returns a server error, and it was tuned
# from a real measurement. Every claim it makes is true.

"control - is the limiter doing its job" ^0
"  requests admitted beyond capacity : 0" ^0
"  latency excursions                : 0" ^0
"  server errors                     : 0" ^0
"  the service protected              : yes, completely" ^0
"  defects in the limiter             : 0" ^0
"" ^0
"  it protects the service, which is the thing it was asked to protect" ^0
"  and nothing measures the pair" ^0
"" ^0

# ---- the null control ----
#
# The same limiter when offered load is under capacity. Nothing is rejected,
# nothing is retried, no loop forms, and the limiter is pure insurance. It is
# not the limiter that closes the loop; it is a rejection meeting a retry.

1800 => nc_offered_rps

"null control - the same limiter under capacity" ^0
"  offered   : " + str(nc_offered_rps) + " rps" ^0
"  capacity  : " + str(capacity_rps) + " rps" ^0
"  rejected  : 0" ^0
"  retries   : 0" ^0
"  amplification : none" ^0
"  same limiter, same threshold, same callers" ^0
"  the loop needs a rejection to exist before it can begin" ^0
"" ^0

# ---- the rule ----

"a failure that is moved rather than removed" ^0
"  disappears from the mover's metrics    yes, immediately" ^0
"  appears in the receiver's metrics      yes, and the receiver is elsewhere" ^0
"  changes the receiver's behaviour       yes, and that behaviour returns" ^0
"  the return path is what nobody owns" ^0
"" ^0
"the measurement that closes it is not on either side" ^0
"it is offered load against original demand, and neither party can" ^0
"compute it alone" ^0
"" ^0

"Shedding at the edge is the right answer and this limiter implements it" ^0
"correctly: the service never exceeds its measured capacity, latency never" ^0
"moves, and its error rate is zero because a 429 is not a server error. Each" ^0
"rejection is retried " + str(retries_per_rejection) + " times, so offered load reaches " + str(offered_round_two) + " rps and" ^0
"rejections reach " + str(rejected_round_two) + " - " + str(int((rejected_round_two - rejected_rps) * 100 / rejected_rps)) + " percent more than before - while the served" ^0
"column holds at " + str(capacity_rps) + " and every SLO stays green." ^0

capacity_rps = 2000
offered_rps = 2600
retries_per_rejection = 3
rejected_rps = offered_rps - capacity_rps
print("capacity      : " + str(capacity_rps) + " rps, measured")
print("offered       : " + str(offered_rps) + " rps")
print("rejected      : " + str(rejected_rps) + " rps")
print("")
print("the service's own numbers")
print("  requests served     : " + str(capacity_rps) + " rps")
print("  latency             : flat, it is never overloaded")
print("  server error rate   : 0, a 429 is not a server error")
print("  SLO                 : met")
print("")
print("the caller's numbers")
print("  requests refused    : " + str(rejected_rps) + " rps")
print("  share refused       : " + str(int(rejected_rps * 100 / offered_rps)) + " percent")
print("")
retry_rps = rejected_rps * retries_per_rejection
offered_round_two = offered_rps + retry_rps
rejected_round_two = offered_round_two - capacity_rps
print("each rejection is retried " + str(retries_per_rejection) + " times")
print("  retry traffic added : " + str(retry_rps) + " rps")
print("  offered, round two  : " + str(offered_round_two) + " rps")
print("  rejected, round two : " + str(rejected_round_two) + " rps")
print("  rejections grew by  : " + str(int((rejected_round_two - rejected_rps) * 100 / rejected_rps)) + " percent")
print("")
print("  served, round one : " + str(capacity_rps))
print("  served, round two : " + str(capacity_rps))
print("  the served column does not move, which is the limiter working")
print("")
print("round   offered   served   rejected")
current_offered = offered_rps
for r in range(1, 5):
    rej = current_offered - capacity_rps
    print("  " + str(r) + "       " + str(current_offered) + "      " + str(capacity_rps) + "     " + str(rej))
    current_offered = offered_rps + rej * retries_per_rejection
print("")
print("  the limiter is doing exactly what it promised at every row")
print("  and the rejected column is the input to the next row")
print("")
load_multiplier_pct = int(offered_round_two * 100 / offered_rps)
print("  original demand        : " + str(offered_rps) + " rps of real work")
print("  load reaching the limiter : " + str(offered_round_two) + " rps")
print("  of that, retries       : " + str(retry_rps) + " rps, none of it new work")
print("  load multiplier        : " + str(load_multiplier_pct) + " percent of the original")
print("  every retry consumes a connection, a TLS handshake and a limiter")
print("  decision, and none of them reach a worker")
print("")
print("the same rejection, with a retry-after the caller honours")
print("  retries within the second : 0")
print("  offered                   : " + str(offered_rps) + " rps, unchanged")
print("  rejected                  : " + str(rejected_rps) + " rps")
print("  the demand is the same and the amplification is gone")
print("  the limiter's job was to tell the caller something, and the number it")
print("  needed to say is not in the status code")
print("")
print("control - is the limiter doing its job")
print("  requests admitted beyond capacity : 0")
print("  latency excursions                : 0")
print("  server errors                     : 0")
print("  the service protected              : yes, completely")
print("  defects in the limiter             : 0")
print("")
print("  it protects the service, which is the thing it was asked to protect")
print("  and nothing measures the pair")
print("")
nc_offered_rps = 1800
print("null control - the same limiter under capacity")
print("  offered   : " + str(nc_offered_rps) + " rps")
print("  capacity  : " + str(capacity_rps) + " rps")
print("  rejected  : 0")
print("  retries   : 0")
print("  amplification : none")
print("  same limiter, same threshold, same callers")
print("  the loop needs a rejection to exist before it can begin")
print("")
print("a failure that is moved rather than removed")
print("  disappears from the mover's metrics    yes, immediately")
print("  appears in the receiver's metrics      yes, and the receiver is elsewhere")
print("  changes the receiver's behaviour       yes, and that behaviour returns")
print("  the return path is what nobody owns")
print("")
print("the measurement that closes it is not on either side")
print("it is offered load against original demand, and neither party can")
print("compute it alone")
print("")
print("Shedding at the edge is the right answer and this limiter implements it")
print("correctly: the service never exceeds its measured capacity, latency never")
print("moves, and its error rate is zero because a 429 is not a server error. Each")
print("rejection is retried " + str(retries_per_rejection) + " times, so offered load reaches " + str(offered_round_two) + " rps and")
print("rejections reach " + str(rejected_round_two) + " - " + str(int((rejected_round_two - rejected_rps) * 100 / rejected_rps)) + " percent more than before - while the served")
print("column holds at " + str(capacity_rps) + " and every SLO stays green.")

capacity      : 2000 rps, measured
offered       : 2600 rps
rejected      : 600 rps

the service's own numbers
  requests served     : 2000 rps
  latency             : flat, it is never overloaded
  server error rate   : 0, a 429 is not a server error
  SLO                 : met

the caller's numbers
  requests refused    : 600 rps
  share refused       : 23 percent

each rejection is retried 3 times
  retry traffic added : 1800 rps
  offered, round two  : 4400 rps
  rejected, round two : 2400 rps
  rejections grew by  : 300 percent

  served, round one : 2000
  served, round two : 2000
  the served column does not move, which is the limiter working

round   offered   served   rejected
  1       2600      2000     600
  2       4400      2000     2400
  3       9800      2000     7800
  4       26000      2000     24000

  the limiter is doing exactly what it promised at every row
  and the rejected column is the input to the next row

  original demand        : 2600 rps of real work
  load reaching the limiter : 4400 rps
  of that, retries       : 1800 rps, none of it new work
  load multiplier        : 169 percent of the original
  every retry consumes a connection, a TLS handshake and a limiter
  decision, and none of them reach a worker

the same rejection, with a retry-after the caller honours
  retries within the second : 0
  offered                   : 2600 rps, unchanged
  rejected                  : 600 rps
  the demand is the same and the amplification is gone
  the limiter's job was to tell the caller something, and the number it
  needed to say is not in the status code

control - is the limiter doing its job
  requests admitted beyond capacity : 0
  latency excursions                : 0
  server errors                     : 0
  the service protected              : yes, completely
  defects in the limiter             : 0

  it protects the service, which is the thing it was asked to protect
  and nothing measures the pair

null control - the same limiter under capacity
  offered   : 1800 rps
  capacity  : 2000 rps
  rejected  : 0
  retries   : 0
  amplification : none
  same limiter, same threshold, same callers
  the loop needs a rejection to exist before it can begin

a failure that is moved rather than removed
  disappears from the mover's metrics    yes, immediately
  appears in the receiver's metrics      yes, and the receiver is elsewhere
  changes the receiver's behaviour       yes, and that behaviour returns
  the return path is what nobody owns

the measurement that closes it is not on either side
it is offered load against original demand, and neither party can
compute it alone

Shedding at the edge is the right answer and this limiter implements it
correctly: the service never exceeds its measured capacity, latency never
moves, and its error rate is zero because a 429 is not a server error. Each
rejection is retried 3 times, so offered load reaches 4400 rps and
rejections reach 2400 - 300 percent more than before - while the served
column holds at 2000 and every SLO stays green.

Trace event types