The alert was tested by firing it and the delivery was not

# Self-authored for the EML case corpus (no external origin). Alert rules are
# tested rather than trusted: a synthetic metric is injected, the rule is
# expected to fire, and a rule that does not fire fails the build. Where the
# test stops is computed below.
#
# The practice is a real one. The test injects a series through the same
# evaluation engine production uses rather than a mock of it; it asserts the
# firing rather than the absence of an error; it runs on every change to a rule
# and nightly besides; and seventy-four rules have been fixed because it went
# red on them.
#
# It asserts that the rule fires. Between a rule firing and a person waking up
# there are three more hops, and the test does not cross them.

640 => alert_rules
612 => rules_with_an_automated_firing_test
1224 => test_firings_a_month
18 => months_the_practice_has_run
74 => rules_fixed_because_the_test_failed
5 => steps_between_the_condition_and_a_person
2 => steps_the_test_exercises
318 => pages_raised_last_quarter
296 => pages_that_reached_a_phone
9 => pages_where_nobody_was_rostered
5 => pages_lost_to_an_expired_integration_token
8 => pages_silenced_by_do_not_disturb
0 => tests_that_page_a_real_device

alert_rules - rules_with_an_automated_firing_test => rules_without_a_firing_test
pages_raised_last_quarter - pages_that_reached_a_phone => pages_that_reached_nobody
pages_where_nobody_was_rostered + pages_lost_to_an_expired_integration_token => pages_lost_before_the_device
pages_lost_before_the_device + pages_silenced_by_do_not_disturb => pages_lost_in_total
steps_between_the_condition_and_a_person - steps_the_test_exercises => steps_nothing_exercises
int(rules_with_an_automated_firing_test * 10000 / alert_rules) => rules_tested_per_myriad
int(pages_that_reached_a_phone * 10000 / pages_raised_last_quarter) => pages_delivered_per_myriad
int(steps_the_test_exercises * 10000 / steps_between_the_condition_and_a_person) => path_covered_per_myriad

"alert rules                     : " + str(alert_rules) ^0
"  with an automated firing test : " + str(rules_with_an_automated_firing_test) ^0
"  without one                   : " + str(rules_without_a_firing_test) ^0
"  tested share                  : " + str(rules_tested_per_myriad) + " per ten thousand" ^0
"test firings a month            : " + str(test_firings_a_month) ^0
"months the practice has run     : " + str(months_the_practice_has_run) ^0
"rules fixed because it went red : " + str(rules_fixed_because_the_test_failed) ^0
"" ^0
"steps from condition to a person: " + str(steps_between_the_condition_and_a_person) ^0
"  the test exercises            : " + str(steps_the_test_exercises) ^0
"  nothing exercises             : " + str(steps_nothing_exercises) ^0
"  path covered                  : " + str(path_covered_per_myriad) + " per ten thousand" ^0
"tests that page a real device   : " + str(tests_that_page_a_real_device) ^0
"" ^0
"pages raised last quarter       : " + str(pages_raised_last_quarter) ^0
"  that reached a phone          : " + str(pages_that_reached_a_phone) ^0
"  that reached nobody           : " + str(pages_that_reached_nobody) ^0
"  delivered                     : " + str(pages_delivered_per_myriad) + " per ten thousand" ^0
"    nobody rostered             : " + str(pages_where_nobody_was_rostered) ^0
"    integration token expired   : " + str(pages_lost_to_an_expired_integration_token) ^0
"    silenced by do not disturb  : " + str(pages_silenced_by_do_not_disturb) ^0
"" ^0

# ---- what the test verified ----

"the alert-rule test" ^0
"  engine : the same evaluation engine production uses," ^0
"    not a mock of it" ^0
"  assertion : that the rule fires, not that nothing" ^0
"    errored" ^0
"  cadence : on every change to a rule, and nightly" ^0
"  rules fixed because it went red : " + str(rules_fixed_because_the_test_failed) ^0
"  months in place : " + str(months_the_practice_has_run) ^0
"  verdict : FIRES" ^0
"" ^0
"  asserting the firing rather than the absence of an" ^0
"  error is the part almost nobody does, and it is why" ^0
"  the " + str(rules_fixed_because_the_test_failed) + " were found before an incident" ^0
"" ^0

# ---- where the test stops ----

"the path from a condition to a person" ^0
"  1 the condition holds : exercised" ^0
"  2 the rule fires : exercised" ^0
"  3 the notifier accepts it : not exercised" ^0
"  4 the schedule says who : not exercised" ^0
"  5 the device rings : not exercised" ^0
"  steps covered : " + str(steps_the_test_exercises) + " of " + str(steps_between_the_condition_and_a_person) + ", " + str(path_covered_per_myriad) + " per ten" ^0
"    thousand" ^0
"" ^0
"  the boundary of the test is the boundary of the system" ^0
"  it was written for, and the question crosses it" ^0
"" ^0

# ---- what a lost page looks like ----

"a page that woke nobody" ^0
"  did the condition hold : yes" ^0
"  did the rule fire : yes, and the test says it always" ^0
"    will" ^0
"  did anything record a failure : no; delivery is" ^0
"    accepted asynchronously and nothing reads the result" ^0
"  where it stopped : " + str(pages_where_nobody_was_rostered) + " at an empty rota, " ^0
"    " + str(pages_lost_to_an_expired_integration_token) + " at an expired token, " + str(pages_silenced_by_do_not_disturb) + " at a silent phone" ^0
"  pages in that state : " + str(pages_lost_in_total) + " of " + str(pages_raised_last_quarter) ^0
"" ^0

# ---- null control ----

# The same practice, with one rule per hour paging a device that is expected to
# acknowledge, and the acknowledgement asserted.
5 => nc_steps_the_test_exercises
22 => nc_delivery_failures_found_before_an_incident
74 => nc_rules_fixed_because_the_test_failed

"null control - page a real device and require the ack" ^0
"  rules fixed by the firing test : " + str(nc_rules_fixed_because_the_test_failed) + ", unchanged" ^0
"  steps the test exercises : " + str(nc_steps_the_test_exercises) ^0
"  delivery failures found before an incident : " + str(nc_delivery_failures_found_before_an_incident) ^0
"  the rules did not get better; the assertion was moved" ^0
"  to the end of the path instead of the end of the" ^0
"  component" ^0
"" ^0

# ---- the rule ----

"what a fully tested alert set guarantees" ^0
"  every rule fires when its condition holds : exactly," ^0
"    " + str(rules_with_an_automated_firing_test) + " of " + str(alert_rules) + " rules, " + str(test_firings_a_month) + " firings a month, " + str(months_the_practice_has_run) + " months" ^0
"  somebody is woken : not addressed; the test ends where" ^0
"    the alerting system does, " + str(steps_nothing_exercises) + " hops short of a phone" ^0
"" ^0
"a test that ends at a component boundary is evidence about" ^0
"that component; the promise is about a person, and the" ^0
"part between them is exercised only by real incidents" ^0
"" ^0

"The firing test runs the production evaluation engine, asserts the firing" ^0
"rather than the absence of an error, and has fixed " + str(rules_fixed_because_the_test_failed) + " rules in " + str(months_the_practice_has_run) + " months." ^0
"It covers " + str(steps_the_test_exercises) + " of the " + str(steps_between_the_condition_and_a_person) + " steps to a person - " + str(path_covered_per_myriad) + " per ten thousand - so " + str(pages_lost_in_total) ^0
"of " + str(pages_raised_last_quarter) + " pages last quarter reached nobody, across " + str(tests_that_page_a_real_device) + " tests that page a" ^0
"device." ^0

alert_rules = 640
rules_with_an_automated_firing_test = 612
test_firings_a_month = 1224
months_the_practice_has_run = 18
rules_fixed_because_the_test_failed = 74
steps_between_the_condition_and_a_person = 5
steps_the_test_exercises = 2
pages_raised_last_quarter = 318
pages_that_reached_a_phone = 296
pages_where_nobody_was_rostered = 9
pages_lost_to_an_expired_integration_token = 5
pages_silenced_by_do_not_disturb = 8
tests_that_page_a_real_device = 0
rules_without_a_firing_test = alert_rules - rules_with_an_automated_firing_test
pages_that_reached_nobody = pages_raised_last_quarter - pages_that_reached_a_phone
pages_lost_before_the_device = pages_where_nobody_was_rostered + pages_lost_to_an_expired_integration_token
pages_lost_in_total = pages_lost_before_the_device + pages_silenced_by_do_not_disturb
steps_nothing_exercises = steps_between_the_condition_and_a_person - steps_the_test_exercises
rules_tested_per_myriad = int(rules_with_an_automated_firing_test * 10000 / alert_rules)
pages_delivered_per_myriad = int(pages_that_reached_a_phone * 10000 / pages_raised_last_quarter)
path_covered_per_myriad = int(steps_the_test_exercises * 10000 / steps_between_the_condition_and_a_person)
print("alert rules                     : " + str(alert_rules))
print("  with an automated firing test : " + str(rules_with_an_automated_firing_test))
print("  without one                   : " + str(rules_without_a_firing_test))
print("  tested share                  : " + str(rules_tested_per_myriad) + " per ten thousand")
print("test firings a month            : " + str(test_firings_a_month))
print("months the practice has run     : " + str(months_the_practice_has_run))
print("rules fixed because it went red : " + str(rules_fixed_because_the_test_failed))
print("")
print("steps from condition to a person: " + str(steps_between_the_condition_and_a_person))
print("  the test exercises            : " + str(steps_the_test_exercises))
print("  nothing exercises             : " + str(steps_nothing_exercises))
print("  path covered                  : " + str(path_covered_per_myriad) + " per ten thousand")
print("tests that page a real device   : " + str(tests_that_page_a_real_device))
print("")
print("pages raised last quarter       : " + str(pages_raised_last_quarter))
print("  that reached a phone          : " + str(pages_that_reached_a_phone))
print("  that reached nobody           : " + str(pages_that_reached_nobody))
print("  delivered                     : " + str(pages_delivered_per_myriad) + " per ten thousand")
print("    nobody rostered             : " + str(pages_where_nobody_was_rostered))
print("    integration token expired   : " + str(pages_lost_to_an_expired_integration_token))
print("    silenced by do not disturb  : " + str(pages_silenced_by_do_not_disturb))
print("")
print("the alert-rule test")
print("  engine : the same evaluation engine production uses,")
print("    not a mock of it")
print("  assertion : that the rule fires, not that nothing")
print("    errored")
print("  cadence : on every change to a rule, and nightly")
print("  rules fixed because it went red : " + str(rules_fixed_because_the_test_failed))
print("  months in place : " + str(months_the_practice_has_run))
print("  verdict : FIRES")
print("")
print("  asserting the firing rather than the absence of an")
print("  error is the part almost nobody does, and it is why")
print("  the " + str(rules_fixed_because_the_test_failed) + " were found before an incident")
print("")
print("the path from a condition to a person")
print("  1 the condition holds : exercised")
print("  2 the rule fires : exercised")
print("  3 the notifier accepts it : not exercised")
print("  4 the schedule says who : not exercised")
print("  5 the device rings : not exercised")
print("  steps covered : " + str(steps_the_test_exercises) + " of " + str(steps_between_the_condition_and_a_person) + ", " + str(path_covered_per_myriad) + " per ten")
print("    thousand")
print("")
print("  the boundary of the test is the boundary of the system")
print("  it was written for, and the question crosses it")
print("")
print("a page that woke nobody")
print("  did the condition hold : yes")
print("  did the rule fire : yes, and the test says it always")
print("    will")
print("  did anything record a failure : no; delivery is")
print("    accepted asynchronously and nothing reads the result")
print("  where it stopped : " + str(pages_where_nobody_was_rostered) + " at an empty rota, ")
print("    " + str(pages_lost_to_an_expired_integration_token) + " at an expired token, " + str(pages_silenced_by_do_not_disturb) + " at a silent phone")
print("  pages in that state : " + str(pages_lost_in_total) + " of " + str(pages_raised_last_quarter))
print("")
nc_steps_the_test_exercises = 5
nc_delivery_failures_found_before_an_incident = 22
nc_rules_fixed_because_the_test_failed = 74
print("null control - page a real device and require the ack")
print("  rules fixed by the firing test : " + str(nc_rules_fixed_because_the_test_failed) + ", unchanged")
print("  steps the test exercises : " + str(nc_steps_the_test_exercises))
print("  delivery failures found before an incident : " + str(nc_delivery_failures_found_before_an_incident))
print("  the rules did not get better; the assertion was moved")
print("  to the end of the path instead of the end of the")
print("  component")
print("")
print("what a fully tested alert set guarantees")
print("  every rule fires when its condition holds : exactly,")
print("    " + str(rules_with_an_automated_firing_test) + " of " + str(alert_rules) + " rules, " + str(test_firings_a_month) + " firings a month, " + str(months_the_practice_has_run) + " months")
print("  somebody is woken : not addressed; the test ends where")
print("    the alerting system does, " + str(steps_nothing_exercises) + " hops short of a phone")
print("")
print("a test that ends at a component boundary is evidence about")
print("that component; the promise is about a person, and the")
print("part between them is exercised only by real incidents")
print("")
print("The firing test runs the production evaluation engine, asserts the firing")
print("rather than the absence of an error, and has fixed " + str(rules_fixed_because_the_test_failed) + " rules in " + str(months_the_practice_has_run) + " months.")
print("It covers " + str(steps_the_test_exercises) + " of the " + str(steps_between_the_condition_and_a_person) + " steps to a person - " + str(path_covered_per_myriad) + " per ten thousand - so " + str(pages_lost_in_total))
print("of " + str(pages_raised_last_quarter) + " pages last quarter reached nobody, across " + str(tests_that_page_a_real_device) + " tests that page a")
print("device.")

alert rules                     : 640
  with an automated firing test : 612
  without one                   : 28
  tested share                  : 9562 per ten thousand
test firings a month            : 1224
months the practice has run     : 18
rules fixed because it went red : 74

steps from condition to a person: 5
  the test exercises            : 2
  nothing exercises             : 3
  path covered                  : 4000 per ten thousand
tests that page a real device   : 0

pages raised last quarter       : 318
  that reached a phone          : 296
  that reached nobody           : 22
  delivered                     : 9308 per ten thousand
    nobody rostered             : 9
    integration token expired   : 5
    silenced by do not disturb  : 8

the alert-rule test
  engine : the same evaluation engine production uses,
    not a mock of it
  assertion : that the rule fires, not that nothing
    errored
  cadence : on every change to a rule, and nightly
  rules fixed because it went red : 74
  months in place : 18
  verdict : FIRES

  asserting the firing rather than the absence of an
  error is the part almost nobody does, and it is why
  the 74 were found before an incident

the path from a condition to a person
  1 the condition holds : exercised
  2 the rule fires : exercised
  3 the notifier accepts it : not exercised
  4 the schedule says who : not exercised
  5 the device rings : not exercised
  steps covered : 2 of 5, 4000 per ten
    thousand

  the boundary of the test is the boundary of the system
  it was written for, and the question crosses it

a page that woke nobody
  did the condition hold : yes
  did the rule fire : yes, and the test says it always
    will
  did anything record a failure : no; delivery is
    accepted asynchronously and nothing reads the result
  where it stopped : 9 at an empty rota, 
    5 at an expired token, 8 at a silent phone
  pages in that state : 22 of 318

null control - page a real device and require the ack
  rules fixed by the firing test : 74, unchanged
  steps the test exercises : 5
  delivery failures found before an incident : 22
  the rules did not get better; the assertion was moved
  to the end of the path instead of the end of the
  component

what a fully tested alert set guarantees
  every rule fires when its condition holds : exactly,
    612 of 640 rules, 1224 firings a month, 18 months
  somebody is woken : not addressed; the test ends where
    the alerting system does, 3 hops short of a phone

a test that ends at a component boundary is evidence about
that component; the promise is about a person, and the
part between them is exercised only by real incidents

The firing test runs the production evaluation engine, asserts the firing
rather than the absence of an error, and has fixed 74 rules in 18 months.
It covers 2 of the 5 steps to a person - 4000 per ten thousand - so 22
of 318 pages last quarter reached nobody, across 0 tests that page a
device.

Trace event types