The canary was healthy and it served the easy requests

# Self-authored for the EML case corpus (no external origin). The canary took
# one percent of traffic for an hour and beat the baseline on every metric. What
# it was asked to do is computed below.
#
# The canary process is well built. One percent of traffic, an hour of soak,
# error rate and latency compared against the same window on the baseline pool,
# an automatic rollback if either regresses, and it has stopped two bad releases
# this quarter. The comparison is like-for-like on time, version and hardware.
#
# Which requests reach it is decided by routing. Sessions are sticky, so an
# established session keeps the pool it was assigned; only NEW sessions can land
# on the canary, and a new session is a person who has just arrived.
#
# An established session carries sixty-one items. A new one carries three.

100 => canary_share_per_myriad
60 => soak_minutes
240 => baseline_mean_ms
96 => canary_mean_ms
3 => items_in_a_new_session
61 => items_in_an_established_session
2 => bad_releases_stopped_this_quarter

int(items_in_an_established_session / items_in_a_new_session) => work_ratio
baseline_mean_ms - canary_mean_ms => apparent_improvement_ms

"canary share                  : " + str(canary_share_per_myriad) + " per ten thousand" ^0
"soak, minutes                 : " + str(soak_minutes) ^0
"baseline mean, ms             : " + str(baseline_mean_ms) ^0
"canary mean, ms               : " + str(canary_mean_ms) ^0
"apparent improvement, ms      : " + str(apparent_improvement_ms) ^0
"" ^0
"items in a new session        : " + str(items_in_a_new_session) ^0
"items in an established session : " + str(items_in_an_established_session) ^0
"work ratio                    : " + str(work_ratio) + " times" ^0
"" ^0

# ---- what the canary process verified ----

"the canary comparison" ^0
"  traffic share       : " + str(canary_share_per_myriad) + " per ten thousand" ^0
"  soak                : " + str(soak_minutes) + " minutes" ^0
"  compared against    : the same window, same hardware" ^0
"  automatic rollback on regression : yes" ^0
"  bad releases stopped this quarter : " + str(bad_releases_stopped_this_quarter) ^0
"  verdict             : HEALTHY, BETTER THAN BASELINE" ^0
"" ^0
"  it is a real gate and it has caught real regressions" ^0
"" ^0

# ---- what routing decided ----

"which requests can reach it" ^0
"  routing            : sticky by session" ^0
"  established sessions : stay on their existing pool" ^0
"  new sessions       : may land on the canary" ^0
"  so the canary's population is : arrivals" ^0
"" ^0
"  stickiness is there for a good reason and it is also" ^0
"  the sampling rule" ^0
"" ^0

int(items_in_a_new_session * 10000 / items_in_an_established_session) => canary_work_per_myriad
"work per request on the canary, against production : " + str(canary_work_per_myriad) + " per ten thousand" ^0
"" ^0

# ---- what the improvement was ----

# The canary is not faster. It is doing a twentieth of the work per request,
# and the metric is per request.
int(canary_mean_ms * items_in_an_established_session / items_in_a_new_session) => canary_ms_at_production_mix

"the same canary, on the production mix" ^0
"  measured mean, ms          : " + str(canary_mean_ms) ^0
"  work ratio                 : " + str(work_ratio) ^0
"  implied mean at that mix, ms : " + str(canary_ms_at_production_mix) ^0
"  baseline mean, ms          : " + str(baseline_mean_ms) ^0
"" ^0
canary_ms_at_production_mix - baseline_mean_ms => implied_regression_ms
"  implied change, ms         : " + str(implied_regression_ms) ^0
"  which direction the gate reported : better" ^0
"" ^0

# ---- null control ----

# The same canary, with routing that assigns a percentage of ESTABLISHED
# sessions as well, accepting the reassignment cost.
items_in_an_established_session => nc_items_per_canary_request
canary_ms_at_production_mix => nc_canary_mean_ms

"null control - established sessions also sampled" ^0
"  soak, minutes         : " + str(soak_minutes) + ", unchanged" ^0
"  items per canary request : " + str(nc_items_per_canary_request) ^0
"  canary mean, ms       : " + str(nc_canary_mean_ms) ^0
"  the canary did not get slower; it was asked the" ^0
"  question the baseline is being asked" ^0
"" ^0

# ---- the rule ----

"what a healthy canary guarantees" ^0
"  this version is healthy on the traffic it received : exactly" ^0
"  this version is healthy in production               : not" ^0
"    addressed; the routing that selects the sample is" ^0
"    chosen for other reasons and is not a sampling design" ^0
"" ^0
"a canary is an experiment and its assignment rule is its" ^0
"randomisation; when the rule correlates with the workload," ^0
"the result is about the sample and reads as being about the" ^0
"population" ^0
"" ^0

"The canary is healthy and the gate is real: " + str(canary_share_per_myriad) + " per ten thousand of traffic," ^0
"a " + str(soak_minutes) + " minute soak, like-for-like comparison, automatic rollback, and " + str(bad_releases_stopped_this_quarter) ^0
"bad releases stopped this quarter. Sticky routing sends it only new sessions," ^0
"which carry " + str(items_in_a_new_session) + " items against " + str(items_in_an_established_session) + " - " + str(canary_work_per_myriad) + " per ten thousand of the work -" ^0
"so its " + str(canary_mean_ms) + " ms implies " + str(canary_ms_at_production_mix) + " ms at the production mix, against " + str(baseline_mean_ms) + "." ^0

canary_share_per_myriad = 100
soak_minutes = 60
baseline_mean_ms = 240
canary_mean_ms = 96
items_in_a_new_session = 3
items_in_an_established_session = 61
bad_releases_stopped_this_quarter = 2
work_ratio = int(items_in_an_established_session / items_in_a_new_session)
apparent_improvement_ms = baseline_mean_ms - canary_mean_ms
print("canary share                  : " + str(canary_share_per_myriad) + " per ten thousand")
print("soak, minutes                 : " + str(soak_minutes))
print("baseline mean, ms             : " + str(baseline_mean_ms))
print("canary mean, ms               : " + str(canary_mean_ms))
print("apparent improvement, ms      : " + str(apparent_improvement_ms))
print("")
print("items in a new session        : " + str(items_in_a_new_session))
print("items in an established session : " + str(items_in_an_established_session))
print("work ratio                    : " + str(work_ratio) + " times")
print("")
print("the canary comparison")
print("  traffic share       : " + str(canary_share_per_myriad) + " per ten thousand")
print("  soak                : " + str(soak_minutes) + " minutes")
print("  compared against    : the same window, same hardware")
print("  automatic rollback on regression : yes")
print("  bad releases stopped this quarter : " + str(bad_releases_stopped_this_quarter))
print("  verdict             : HEALTHY, BETTER THAN BASELINE")
print("")
print("  it is a real gate and it has caught real regressions")
print("")
print("which requests can reach it")
print("  routing            : sticky by session")
print("  established sessions : stay on their existing pool")
print("  new sessions       : may land on the canary")
print("  so the canary's population is : arrivals")
print("")
print("  stickiness is there for a good reason and it is also")
print("  the sampling rule")
print("")
canary_work_per_myriad = int(items_in_a_new_session * 10000 / items_in_an_established_session)
print("work per request on the canary, against production : " + str(canary_work_per_myriad) + " per ten thousand")
print("")
canary_ms_at_production_mix = int(canary_mean_ms * items_in_an_established_session / items_in_a_new_session)
print("the same canary, on the production mix")
print("  measured mean, ms          : " + str(canary_mean_ms))
print("  work ratio                 : " + str(work_ratio))
print("  implied mean at that mix, ms : " + str(canary_ms_at_production_mix))
print("  baseline mean, ms          : " + str(baseline_mean_ms))
print("")
implied_regression_ms = canary_ms_at_production_mix - baseline_mean_ms
print("  implied change, ms         : " + str(implied_regression_ms))
print("  which direction the gate reported : better")
print("")
nc_items_per_canary_request = items_in_an_established_session
nc_canary_mean_ms = canary_ms_at_production_mix
print("null control - established sessions also sampled")
print("  soak, minutes         : " + str(soak_minutes) + ", unchanged")
print("  items per canary request : " + str(nc_items_per_canary_request))
print("  canary mean, ms       : " + str(nc_canary_mean_ms))
print("  the canary did not get slower; it was asked the")
print("  question the baseline is being asked")
print("")
print("what a healthy canary guarantees")
print("  this version is healthy on the traffic it received : exactly")
print("  this version is healthy in production               : not")
print("    addressed; the routing that selects the sample is")
print("    chosen for other reasons and is not a sampling design")
print("")
print("a canary is an experiment and its assignment rule is its")
print("randomisation; when the rule correlates with the workload,")
print("the result is about the sample and reads as being about the")
print("population")
print("")
print("The canary is healthy and the gate is real: " + str(canary_share_per_myriad) + " per ten thousand of traffic,")
print("a " + str(soak_minutes) + " minute soak, like-for-like comparison, automatic rollback, and " + str(bad_releases_stopped_this_quarter))
print("bad releases stopped this quarter. Sticky routing sends it only new sessions,")
print("which carry " + str(items_in_a_new_session) + " items against " + str(items_in_an_established_session) + " - " + str(canary_work_per_myriad) + " per ten thousand of the work -")
print("so its " + str(canary_mean_ms) + " ms implies " + str(canary_ms_at_production_mix) + " ms at the production mix, against " + str(baseline_mean_ms) + ".")

canary share                  : 100 per ten thousand
soak, minutes                 : 60
baseline mean, ms             : 240
canary mean, ms               : 96
apparent improvement, ms      : 144

items in a new session        : 3
items in an established session : 61
work ratio                    : 20 times

the canary comparison
  traffic share       : 100 per ten thousand
  soak                : 60 minutes
  compared against    : the same window, same hardware
  automatic rollback on regression : yes
  bad releases stopped this quarter : 2
  verdict             : HEALTHY, BETTER THAN BASELINE

  it is a real gate and it has caught real regressions

which requests can reach it
  routing            : sticky by session
  established sessions : stay on their existing pool
  new sessions       : may land on the canary
  so the canary's population is : arrivals

  stickiness is there for a good reason and it is also
  the sampling rule

work per request on the canary, against production : 491 per ten thousand

the same canary, on the production mix
  measured mean, ms          : 96
  work ratio                 : 20
  implied mean at that mix, ms : 1952
  baseline mean, ms          : 240

  implied change, ms         : 1712
  which direction the gate reported : better

null control - established sessions also sampled
  soak, minutes         : 60, unchanged
  items per canary request : 61
  canary mean, ms       : 1952
  the canary did not get slower; it was asked the
  question the baseline is being asked

what a healthy canary guarantees
  this version is healthy on the traffic it received : exactly
  this version is healthy in production               : not
    addressed; the routing that selects the sample is
    chosen for other reasons and is not a sampling design

a canary is an experiment and its assignment rule is its
randomisation; when the rule correlates with the workload,
the result is about the sample and reads as being about the
population

The canary is healthy and the gate is real: 100 per ten thousand of traffic,
a 60 minute soak, like-for-like comparison, automatic rollback, and 2
bad releases stopped this quarter. Sticky routing sends it only new sessions,
which carry 3 items against 61 - 491 per ten thousand of the work -
so its 96 ms implies 1952 ms at the production mix, against 240.

Trace event types