The worker was stateless and the filesystem was not

# Self-authored for the EML case corpus (no external origin). The workers hold
# no session state, any request can go to any pod, and a chaos test kills one
# every five minutes without a failed request. What persists is computed below.
#
# The statelessness is real and it was earned. There is no in-memory session,
# no sticky routing, no affinity in the load balancer; scaling out works
# linearly; and the chaos test is not a drill on paper - it kills a random pod
# every five minutes, all day, and the failed-request count has been zero for
# eleven months.
#
# Statelessness was established about memory. The container has a writable
# filesystem, a template compiler caches to it, and that cache survives every
# request boundary the memory claim was proved across.
#
# A pod lives about thirty-four days before disk pressure evicts it.

240 => worker_pods
0 => in_memory_session_state
0 => failed_requests_under_the_chaos_test
288 => chaos_kills_per_day
41000 => files_written_per_pod_per_day
380 => megabytes_written_per_pod_per_day
34 => days_until_disk_pressure_evicts_a_pod
30 => days_in_the_month
0 => alerts_separating_the_two_causes
120 => p99_ms_on_a_warm_pod
890 => p99_ms_on_a_fresh_pod

# Both restart counts are derived, so neither can drift from the fleet size,
# the pod lifetime, or the kill rate that produced them.
chaos_kills_per_day * days_in_the_month => chaos_kills_per_month
int(worker_pods * days_in_the_month / days_until_disk_pressure_evicts_a_pod) => restarts_caused_by_disk_pressure
chaos_kills_per_month + restarts_caused_by_disk_pressure => pod_restarts_per_month
pod_restarts_per_month - restarts_caused_by_disk_pressure => restarts_from_chaos_or_churn
int(restarts_caused_by_disk_pressure * 10000 / pod_restarts_per_month) => disk_pressure_per_myriad
p99_ms_on_a_fresh_pod - p99_ms_on_a_warm_pod => p99_gap_ms

"worker pods                     : " + str(worker_pods) ^0
"in-memory session state         : " + str(in_memory_session_state) ^0
"chaos kills per day             : " + str(chaos_kills_per_day) ^0
"failed requests under the chaos test : " + str(failed_requests_under_the_chaos_test) ^0
"" ^0
"files written per pod per day   : " + str(files_written_per_pod_per_day) ^0
"megabytes per pod per day       : " + str(megabytes_written_per_pod_per_day) ^0
"days until disk pressure evicts : " + str(days_until_disk_pressure_evicts_a_pod) ^0
"" ^0
"pod restarts per month          : " + str(pod_restarts_per_month) ^0
"  from chaos or normal churn    : " + str(restarts_from_chaos_or_churn) ^0
"  from disk pressure            : " + str(restarts_caused_by_disk_pressure) ^0
"  share                         : " + str(disk_pressure_per_myriad) + " per ten thousand" ^0
"alerts separating the two causes : " + str(alerts_separating_the_two_causes) ^0
"" ^0
"p99 on a warm pod, ms           : " + str(p99_ms_on_a_warm_pod) ^0
"p99 on a fresh pod, ms          : " + str(p99_ms_on_a_fresh_pod) ^0
"  gap                           : " + str(p99_gap_ms) + " ms" ^0
"" ^0

# ---- what the chaos test verified ----

"the statelessness claim" ^0
"  in-memory session state : " + str(in_memory_session_state) ^0
"  sticky routing or affinity : none" ^0
"  pods killed per day     : " + str(chaos_kills_per_day) ^0
"  failed requests         : " + str(failed_requests_under_the_chaos_test) ^0
"  months at zero          : eleven" ^0
"  verdict : STATELESS" ^0
"" ^0
"  killing a pod every five minutes in production is a real" ^0
"  test and almost nobody runs it" ^0
"" ^0

# ---- what the test ranges over ----

"what the chaos test can observe" ^0
"  a request in flight when a pod dies : retried elsewhere," ^0
"    and it succeeds" ^0
"  what that proves : no request depends on memory in one" ^0
"    process" ^0
"  what it does not observe : the pods it did not kill, and" ^0
"    what they have been accumulating" ^0
"  the filesystem : writable, and never asserted about" ^0
"" ^0
"  correctness under pod loss and independence from pod" ^0
"  identity are different claims, and only the first was run" ^0
"" ^0
# ---- what the cache makes observable ----

# The template cache is a correctness-neutral optimisation and it works. It also
# makes a pod's age an input to what a caller experiences, which is the thing
# statelessness was supposed to have removed.
"pod age as an input" ^0
"  p99 on a pod minutes old : " + str(p99_ms_on_a_fresh_pod) + " ms" ^0
"  p99 on a pod days old    : " + str(p99_ms_on_a_warm_pod) + " ms" ^0
"  difference               : " + str(p99_gap_ms) + " ms" ^0
"  is the response different : no, the same bytes" ^0
"  is the experience different : yes, and it depends on" ^0
"    which pod answered" ^0
"  what routing guarantees about that : nothing, by design" ^0
"" ^0

# ---- the restart count ----

# Two causes produce the same event. The chaos test kills pods on purpose, disk
# pressure kills them because a directory grew, and the restart counter sums
# them into one number that is expected to be large.
"why the eviction is invisible" ^0
"  restarts a month : " + str(pod_restarts_per_month) ^0
"  expected, from the chaos test and churn : " + str(restarts_from_chaos_or_churn) ^0
"  from disk pressure : " + str(restarts_caused_by_disk_pressure) ^0
"  alerts separating them : " + str(alerts_separating_the_two_causes) ^0
"  why a high restart count reads as healthy : because the" ^0
"    team deliberately causes " + str(chaos_kills_per_day) + " of them a day" ^0
"" ^0

# ---- null control ----

# The same workers, with the cache moved to a shared store under an explicit
# key and the container filesystem read-only.
0 => nc_restarts_caused_by_disk_pressure
p99_ms_on_a_warm_pod => nc_p99_ms_on_a_fresh_pod
0 => nc_p99_gap_ms

"null control - a shared cache and a read-only container" ^0
"  in-memory session state : " + str(in_memory_session_state) + ", unchanged" ^0
"  restarts from disk pressure : " + str(nc_restarts_caused_by_disk_pressure) ^0
"  p99 on a fresh pod : " + str(nc_p99_ms_on_a_fresh_pod) + " ms" ^0
"  gap between fresh and warm : " + str(nc_p99_gap_ms) + " ms" ^0
"  the workers did not become more stateless; the state they" ^0
"  had stopped being per pod and started having a name" ^0
"" ^0

# ---- the rule ----

"what a passing chaos test guarantees" ^0
"  no request depends on state in one process : exactly," ^0
"    proved " + str(chaos_kills_per_day) + " times a day for eleven months" ^0
"  the workers are interchangeable             : not" ^0
"    addressed; a process is not the only thing a pod has," ^0
"    and the test only ever destroys pods" ^0
"" ^0
"statelessness is a claim about what a request reads, so it" ^0
"has to name every store a request can read; a claim proved" ^0
"by killing processes covers memory exactly and leaves the" ^0
"disk under it untouched and accumulating" ^0
"" ^0

"The workers hold " + str(in_memory_session_state) + " session state and a chaos test kills " + str(chaos_kills_per_day) + " pods a day with" ^0
str(failed_requests_under_the_chaos_test) + " failed requests, which almost nobody proves. A template cache writes " + str(megabytes_written_per_pod_per_day) ^0
"megabytes a pod a day to the container filesystem, so p99 depends on pod age -" ^0
str(p99_ms_on_a_fresh_pod) + " ms against " + str(p99_ms_on_a_warm_pod) + ", a gap of " + str(p99_gap_ms) + " - and " + str(restarts_caused_by_disk_pressure) + " of " + str(pod_restarts_per_month) + " restarts a month," ^0
str(disk_pressure_per_myriad) + " per ten thousand, are evictions hidden inside a number the test inflates." ^0

worker_pods = 240
in_memory_session_state = 0
failed_requests_under_the_chaos_test = 0
chaos_kills_per_day = 288
files_written_per_pod_per_day = 41000
megabytes_written_per_pod_per_day = 380
days_until_disk_pressure_evicts_a_pod = 34
days_in_the_month = 30
alerts_separating_the_two_causes = 0
p99_ms_on_a_warm_pod = 120
p99_ms_on_a_fresh_pod = 890
chaos_kills_per_month = chaos_kills_per_day * days_in_the_month
restarts_caused_by_disk_pressure = int(worker_pods * days_in_the_month / days_until_disk_pressure_evicts_a_pod)
pod_restarts_per_month = chaos_kills_per_month + restarts_caused_by_disk_pressure
restarts_from_chaos_or_churn = pod_restarts_per_month - restarts_caused_by_disk_pressure
disk_pressure_per_myriad = int(restarts_caused_by_disk_pressure * 10000 / pod_restarts_per_month)
p99_gap_ms = p99_ms_on_a_fresh_pod - p99_ms_on_a_warm_pod
print("worker pods                     : " + str(worker_pods))
print("in-memory session state         : " + str(in_memory_session_state))
print("chaos kills per day             : " + str(chaos_kills_per_day))
print("failed requests under the chaos test : " + str(failed_requests_under_the_chaos_test))
print("")
print("files written per pod per day   : " + str(files_written_per_pod_per_day))
print("megabytes per pod per day       : " + str(megabytes_written_per_pod_per_day))
print("days until disk pressure evicts : " + str(days_until_disk_pressure_evicts_a_pod))
print("")
print("pod restarts per month          : " + str(pod_restarts_per_month))
print("  from chaos or normal churn    : " + str(restarts_from_chaos_or_churn))
print("  from disk pressure            : " + str(restarts_caused_by_disk_pressure))
print("  share                         : " + str(disk_pressure_per_myriad) + " per ten thousand")
print("alerts separating the two causes : " + str(alerts_separating_the_two_causes))
print("")
print("p99 on a warm pod, ms           : " + str(p99_ms_on_a_warm_pod))
print("p99 on a fresh pod, ms          : " + str(p99_ms_on_a_fresh_pod))
print("  gap                           : " + str(p99_gap_ms) + " ms")
print("")
print("the statelessness claim")
print("  in-memory session state : " + str(in_memory_session_state))
print("  sticky routing or affinity : none")
print("  pods killed per day     : " + str(chaos_kills_per_day))
print("  failed requests         : " + str(failed_requests_under_the_chaos_test))
print("  months at zero          : eleven")
print("  verdict : STATELESS")
print("")
print("  killing a pod every five minutes in production is a real")
print("  test and almost nobody runs it")
print("")
print("what the chaos test can observe")
print("  a request in flight when a pod dies : retried elsewhere,")
print("    and it succeeds")
print("  what that proves : no request depends on memory in one")
print("    process")
print("  what it does not observe : the pods it did not kill, and")
print("    what they have been accumulating")
print("  the filesystem : writable, and never asserted about")
print("")
print("  correctness under pod loss and independence from pod")
print("  identity are different claims, and only the first was run")
print("")
print("pod age as an input")
print("  p99 on a pod minutes old : " + str(p99_ms_on_a_fresh_pod) + " ms")
print("  p99 on a pod days old    : " + str(p99_ms_on_a_warm_pod) + " ms")
print("  difference               : " + str(p99_gap_ms) + " ms")
print("  is the response different : no, the same bytes")
print("  is the experience different : yes, and it depends on")
print("    which pod answered")
print("  what routing guarantees about that : nothing, by design")
print("")
print("why the eviction is invisible")
print("  restarts a month : " + str(pod_restarts_per_month))
print("  expected, from the chaos test and churn : " + str(restarts_from_chaos_or_churn))
print("  from disk pressure : " + str(restarts_caused_by_disk_pressure))
print("  alerts separating them : " + str(alerts_separating_the_two_causes))
print("  why a high restart count reads as healthy : because the")
print("    team deliberately causes " + str(chaos_kills_per_day) + " of them a day")
print("")
nc_restarts_caused_by_disk_pressure = 0
nc_p99_ms_on_a_fresh_pod = p99_ms_on_a_warm_pod
nc_p99_gap_ms = 0
print("null control - a shared cache and a read-only container")
print("  in-memory session state : " + str(in_memory_session_state) + ", unchanged")
print("  restarts from disk pressure : " + str(nc_restarts_caused_by_disk_pressure))
print("  p99 on a fresh pod : " + str(nc_p99_ms_on_a_fresh_pod) + " ms")
print("  gap between fresh and warm : " + str(nc_p99_gap_ms) + " ms")
print("  the workers did not become more stateless; the state they")
print("  had stopped being per pod and started having a name")
print("")
print("what a passing chaos test guarantees")
print("  no request depends on state in one process : exactly,")
print("    proved " + str(chaos_kills_per_day) + " times a day for eleven months")
print("  the workers are interchangeable             : not")
print("    addressed; a process is not the only thing a pod has,")
print("    and the test only ever destroys pods")
print("")
print("statelessness is a claim about what a request reads, so it")
print("has to name every store a request can read; a claim proved")
print("by killing processes covers memory exactly and leaves the")
print("disk under it untouched and accumulating")
print("")
print("The workers hold " + str(in_memory_session_state) + " session state and a chaos test kills " + str(chaos_kills_per_day) + " pods a day with")
print(str(failed_requests_under_the_chaos_test) + " failed requests, which almost nobody proves. A template cache writes " + str(megabytes_written_per_pod_per_day))
print("megabytes a pod a day to the container filesystem, so p99 depends on pod age -")
print(str(p99_ms_on_a_fresh_pod) + " ms against " + str(p99_ms_on_a_warm_pod) + ", a gap of " + str(p99_gap_ms) + " - and " + str(restarts_caused_by_disk_pressure) + " of " + str(pod_restarts_per_month) + " restarts a month,")
print(str(disk_pressure_per_myriad) + " per ten thousand, are evictions hidden inside a number the test inflates.")

worker pods                     : 240
in-memory session state         : 0
chaos kills per day             : 288
failed requests under the chaos test : 0

files written per pod per day   : 41000
megabytes per pod per day       : 380
days until disk pressure evicts : 34

pod restarts per month          : 8851
  from chaos or normal churn    : 8640
  from disk pressure            : 211
  share                         : 238 per ten thousand
alerts separating the two causes : 0

p99 on a warm pod, ms           : 120
p99 on a fresh pod, ms          : 890
  gap                           : 770 ms

the statelessness claim
  in-memory session state : 0
  sticky routing or affinity : none
  pods killed per day     : 288
  failed requests         : 0
  months at zero          : eleven
  verdict : STATELESS

  killing a pod every five minutes in production is a real
  test and almost nobody runs it

what the chaos test can observe
  a request in flight when a pod dies : retried elsewhere,
    and it succeeds
  what that proves : no request depends on memory in one
    process
  what it does not observe : the pods it did not kill, and
    what they have been accumulating
  the filesystem : writable, and never asserted about

  correctness under pod loss and independence from pod
  identity are different claims, and only the first was run

pod age as an input
  p99 on a pod minutes old : 890 ms
  p99 on a pod days old    : 120 ms
  difference               : 770 ms
  is the response different : no, the same bytes
  is the experience different : yes, and it depends on
    which pod answered
  what routing guarantees about that : nothing, by design

why the eviction is invisible
  restarts a month : 8851
  expected, from the chaos test and churn : 8640
  from disk pressure : 211
  alerts separating them : 0
  why a high restart count reads as healthy : because the
    team deliberately causes 288 of them a day

null control - a shared cache and a read-only container
  in-memory session state : 0, unchanged
  restarts from disk pressure : 0
  p99 on a fresh pod : 120 ms
  gap between fresh and warm : 0 ms
  the workers did not become more stateless; the state they
  had stopped being per pod and started having a name

what a passing chaos test guarantees
  no request depends on state in one process : exactly,
    proved 288 times a day for eleven months
  the workers are interchangeable             : not
    addressed; a process is not the only thing a pod has,
    and the test only ever destroys pods

statelessness is a claim about what a request reads, so it
has to name every store a request can read; a claim proved
by killing processes covers memory exactly and leaves the
disk under it untouched and accumulating

The workers hold 0 session state and a chaos test kills 288 pods a day with
0 failed requests, which almost nobody proves. A template cache writes 380
megabytes a pod a day to the container filesystem, so p99 depends on pod age -
890 ms against 120, a gap of 770 - and 211 of 8851 restarts a month,
238 per ten thousand, are evictions hidden inside a number the test inflates.

Trace event types