The limit was on concurrency and the work was queued

# Self-authored for the EML case corpus (no external origin). The downstream
# never sees more than thirty-two concurrent requests and it never has. What a
# caller waits is computed below.
#
# The concurrency limit is correct and it saved the downstream. Thirty-two was
# measured, not guessed: above it the downstream's latency rose superlinearly
# and at forty it fell over. The semaphore is acquired and released in a finally,
# so a panic cannot leak a permit, and a soak test confirmed the count never
# exceeds the limit over eight hours.
#
# A request over the limit does not fail. It waits, in an unbounded queue, with
# no deadline — which is what makes the protection perfect and is the whole of
# the problem.
#
# Arrivals are four hundred and eighty a second and the limit serves two
# hundred and sixty-six.

32 => concurrency_limit
120 => downstream_service_ms
480 => arrivals_per_second
60 => burst_seconds
0 => times_the_limit_was_exceeded
0 => queue_bound

int(concurrency_limit * 1000 / downstream_service_ms) => served_per_second
arrivals_per_second - served_per_second => overflow_per_second
overflow_per_second * burst_seconds => queued_after_the_burst
int(queued_after_the_burst / served_per_second) => wait_seconds_at_the_back

"concurrency limit          : " + str(concurrency_limit) ^0
"downstream service time, ms: " + str(downstream_service_ms) ^0
"served per second          : " + str(served_per_second) ^0
"arrivals per second        : " + str(arrivals_per_second) ^0
"overflow per second        : " + str(overflow_per_second) ^0
"" ^0
"after a " + str(burst_seconds) + " second burst" ^0
"  queued                   : " + str(queued_after_the_burst) ^0
"  wait at the back, seconds: " + str(wait_seconds_at_the_back) ^0
"  queue bound              : " + str(queue_bound) + ", meaning none" ^0
"  times the limit was exceeded : " + str(times_the_limit_was_exceeded) ^0
"" ^0

# ---- what the limit verified ----

"the semaphore" ^0
"  limit measured, not guessed : yes, the downstream falls" ^0
"    over at 40 and degrades above 32" ^0
"  acquired and released in a finally : yes, a panic cannot" ^0
"    leak a permit" ^0
"  soak test over eight hours  : never exceeded" ^0
"  times exceeded in production: " + str(times_the_limit_was_exceeded) ^0
"  verdict                     : LIMITED" ^0
"" ^0
"  the downstream is genuinely protected and removing this" ^0
"  would take it down" ^0
"" ^0

# ---- what happens to the rest ----

"a request that cannot get a permit" ^0
"  rejected            : no" ^0
"  given a deadline    : no" ^0
"  counted             : no, the queue has no metric" ^0
"  what it does        : waits" ^0
"  what the caller sees: a request in flight" ^0
"" ^0
"  the protection is perfect because nothing is refused," ^0
"  and nothing is refused because the queue is unbounded" ^0
"" ^0

int(overflow_per_second * 10000 / arrivals_per_second) => overflow_per_myriad
"share of arrivals that must wait : " + str(overflow_per_myriad) + " per ten thousand" ^0
"" ^0

# ---- what the dashboards show ----

# The downstream's dashboard is the healthiest it has ever been: latency flat,
# error rate zero, concurrency pinned at exactly the limit. The caller's
# dashboard is the one on fire, and they are different teams.
"the two dashboards" ^0
"  downstream : latency flat, errors zero, concurrency at" ^0
"    exactly " + str(concurrency_limit) ^0
"  caller     : latency climbing without bound" ^0
"  a panel showing the queue : none exists" ^0
"  which team owns which : different teams" ^0
"" ^0

# ---- null control ----

# The same limit, with a bounded queue and a deadline, so overflow is refused
# quickly instead of accepted slowly.
2 => nc_queue_seconds
nc_queue_seconds * served_per_second => nc_queue_bound
overflow_per_second => nc_refused_per_second

"null control - a bounded queue with a deadline" ^0
"  times the limit was exceeded : " + str(times_the_limit_was_exceeded) + ", unchanged" ^0
"  queue bound                  : " + str(nc_queue_bound) ^0
"  refused per second           : " + str(nc_refused_per_second) ^0
"  wait at the back, seconds    : " + str(nc_queue_seconds) ^0
"  the downstream is no better protected; the overflow" ^0
"  became visible and answerable instead of pending" ^0
"" ^0

# ---- the rule ----

"what a concurrency limit guarantees" ^0
"  the downstream sees at most N at once : exactly" ^0
"  the system degrades gracefully        : not addressed;" ^0
"    where the excess goes is a separate decision, and an" ^0
"    unbounded queue is the default nobody chose" ^0
"" ^0
"limiting is half a policy; the other half is what happens to" ^0
"what is over the limit, and a queue with no bound and no" ^0
"deadline converts a rejection into a latency nobody has a" ^0
"metric for" ^0
"" ^0

"The limit holds and the downstream is protected: " + str(concurrency_limit) + " measured rather than guessed," ^0
"released in a finally, " + str(times_the_limit_was_exceeded) + " exceedances in production or in an eight-hour soak." ^0
"It serves " + str(served_per_second) + " a second against " + str(arrivals_per_second) + " arriving, so " + str(overflow_per_myriad) + " per ten thousand" ^0
"must wait, and after " + str(burst_seconds) + " seconds " + str(queued_after_the_burst) + " requests are queued with nothing bounding" ^0
"them and " + str(wait_seconds_at_the_back) + " seconds of wait at the back, on a queue that has no panel." ^0

concurrency_limit = 32
downstream_service_ms = 120
arrivals_per_second = 480
burst_seconds = 60
times_the_limit_was_exceeded = 0
queue_bound = 0
served_per_second = int(concurrency_limit * 1000 / downstream_service_ms)
overflow_per_second = arrivals_per_second - served_per_second
queued_after_the_burst = overflow_per_second * burst_seconds
wait_seconds_at_the_back = int(queued_after_the_burst / served_per_second)
print("concurrency limit          : " + str(concurrency_limit))
print("downstream service time, ms: " + str(downstream_service_ms))
print("served per second          : " + str(served_per_second))
print("arrivals per second        : " + str(arrivals_per_second))
print("overflow per second        : " + str(overflow_per_second))
print("")
print("after a " + str(burst_seconds) + " second burst")
print("  queued                   : " + str(queued_after_the_burst))
print("  wait at the back, seconds: " + str(wait_seconds_at_the_back))
print("  queue bound              : " + str(queue_bound) + ", meaning none")
print("  times the limit was exceeded : " + str(times_the_limit_was_exceeded))
print("")
print("the semaphore")
print("  limit measured, not guessed : yes, the downstream falls")
print("    over at 40 and degrades above 32")
print("  acquired and released in a finally : yes, a panic cannot")
print("    leak a permit")
print("  soak test over eight hours  : never exceeded")
print("  times exceeded in production: " + str(times_the_limit_was_exceeded))
print("  verdict                     : LIMITED")
print("")
print("  the downstream is genuinely protected and removing this")
print("  would take it down")
print("")
print("a request that cannot get a permit")
print("  rejected            : no")
print("  given a deadline    : no")
print("  counted             : no, the queue has no metric")
print("  what it does        : waits")
print("  what the caller sees: a request in flight")
print("")
print("  the protection is perfect because nothing is refused,")
print("  and nothing is refused because the queue is unbounded")
print("")
overflow_per_myriad = int(overflow_per_second * 10000 / arrivals_per_second)
print("share of arrivals that must wait : " + str(overflow_per_myriad) + " per ten thousand")
print("")
print("the two dashboards")
print("  downstream : latency flat, errors zero, concurrency at")
print("    exactly " + str(concurrency_limit))
print("  caller     : latency climbing without bound")
print("  a panel showing the queue : none exists")
print("  which team owns which : different teams")
print("")
nc_queue_seconds = 2
nc_queue_bound = nc_queue_seconds * served_per_second
nc_refused_per_second = overflow_per_second
print("null control - a bounded queue with a deadline")
print("  times the limit was exceeded : " + str(times_the_limit_was_exceeded) + ", unchanged")
print("  queue bound                  : " + str(nc_queue_bound))
print("  refused per second           : " + str(nc_refused_per_second))
print("  wait at the back, seconds    : " + str(nc_queue_seconds))
print("  the downstream is no better protected; the overflow")
print("  became visible and answerable instead of pending")
print("")
print("what a concurrency limit guarantees")
print("  the downstream sees at most N at once : exactly")
print("  the system degrades gracefully        : not addressed;")
print("    where the excess goes is a separate decision, and an")
print("    unbounded queue is the default nobody chose")
print("")
print("limiting is half a policy; the other half is what happens to")
print("what is over the limit, and a queue with no bound and no")
print("deadline converts a rejection into a latency nobody has a")
print("metric for")
print("")
print("The limit holds and the downstream is protected: " + str(concurrency_limit) + " measured rather than guessed,")
print("released in a finally, " + str(times_the_limit_was_exceeded) + " exceedances in production or in an eight-hour soak.")
print("It serves " + str(served_per_second) + " a second against " + str(arrivals_per_second) + " arriving, so " + str(overflow_per_myriad) + " per ten thousand")
print("must wait, and after " + str(burst_seconds) + " seconds " + str(queued_after_the_burst) + " requests are queued with nothing bounding")
print("them and " + str(wait_seconds_at_the_back) + " seconds of wait at the back, on a queue that has no panel.")

concurrency limit          : 32
downstream service time, ms: 120
served per second          : 266
arrivals per second        : 480
overflow per second        : 214

after a 60 second burst
  queued                   : 12840
  wait at the back, seconds: 48
  queue bound              : 0, meaning none
  times the limit was exceeded : 0

the semaphore
  limit measured, not guessed : yes, the downstream falls
    over at 40 and degrades above 32
  acquired and released in a finally : yes, a panic cannot
    leak a permit
  soak test over eight hours  : never exceeded
  times exceeded in production: 0
  verdict                     : LIMITED

  the downstream is genuinely protected and removing this
  would take it down

a request that cannot get a permit
  rejected            : no
  given a deadline    : no
  counted             : no, the queue has no metric
  what it does        : waits
  what the caller sees: a request in flight

  the protection is perfect because nothing is refused,
  and nothing is refused because the queue is unbounded

share of arrivals that must wait : 4458 per ten thousand

the two dashboards
  downstream : latency flat, errors zero, concurrency at
    exactly 32
  caller     : latency climbing without bound
  a panel showing the queue : none exists
  which team owns which : different teams

null control - a bounded queue with a deadline
  times the limit was exceeded : 0, unchanged
  queue bound                  : 532
  refused per second           : 214
  wait at the back, seconds    : 2
  the downstream is no better protected; the overflow
  became visible and answerable instead of pending

what a concurrency limit guarantees
  the downstream sees at most N at once : exactly
  the system degrades gracefully        : not addressed;
    where the excess goes is a separate decision, and an
    unbounded queue is the default nobody chose

limiting is half a policy; the other half is what happens to
what is over the limit, and a queue with no bound and no
deadline converts a rejection into a latency nobody has a
metric for

The limit holds and the downstream is protected: 32 measured rather than guessed,
released in a finally, 0 exceedances in production or in an eight-hour soak.
It serves 266 a second against 480 arriving, so 4458 per ten thousand
must wait, and after 60 seconds 12840 requests are queued with nothing bounding
them and 48 seconds of wait at the back, on a queue that has no panel.

Trace event types