The run stopped before it had seen them all

# Self-authored for the EML case corpus (no external origin). A fuzzing run draws
# random inputs that each trigger one of ten equally likely error categories, and
# after ten draws it reports which categories can occur. Every draw was genuine
# and every category seen was logged. How many draws it takes to see all ten is
# computed below.
#
# The run is careful. Every input is a genuine uniform draw from the engine's
# generator; every triggered category is logged exactly; the run makes as many
# draws as there are categories; and the intent is exactly 'which error
# categories can this code produce'.
#
# Seeing all of n equally likely categories takes about n times the n-th harmonic
# number of draws on average - twenty-nine for ten - so a run of ten draws is
# expected to leave about a third of them unseen.

10 => categories
10 => draws_run
6 => distinct_categories_seen

1000 + 500 + 333 + 250 + 200 + 166 + 142 + 125 + 111 + 100 => harmonic_ten_per_mille
int(categories * harmonic_ten_per_mille / 1000) => expected_draws_to_see_all
categories - distinct_categories_seen => categories_declared_unreachable
int(draws_run * 10000 / expected_draws_to_see_all) => share_of_needed_draws_made_per_myriad

"error categories, equally likely : " + str(categories) ^0
"draws the run made              : " + str(draws_run) ^0
"distinct categories seen        : " + str(distinct_categories_seen) ^0
"categories declared unreachable : " + str(categories_declared_unreachable) ^0
"" ^0
"harmonic number H(10), per mille : " + str(harmonic_ten_per_mille) ^0
"expected draws to see all ten   : " + str(expected_draws_to_see_all) ^0
"share of needed draws made      : " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand" ^0
"" ^0

# ---- what the run verified ----

"the fuzzing run" ^0
"  each input : a genuine uniform draw from the generator" ^0
"  each category : logged exactly when triggered" ^0
"  draws made : " + str(draws_run) + ", one per category" ^0
"  intent : which error categories can this code produce" ^0
"  draws that were not random : 0" ^0
"  verdict : SIX CATEGORIES OBSERVED IN TEN DRAWS" ^0
"" ^0
"  logging every triggered category from genuine random" ^0
"  inputs is the part done right here, and it is why the six" ^0
"  seen are certainly reachable" ^0
"" ^0

# ---- how many draws it takes to see them all ----

"the coupon collector" ^0
"  the first draw : a new category, certainly" ^0
"  the last new category : found at one chance in ten per" ^0
"    draw, about ten draws of waiting" ^0
"  the total, summed over all ten : ten times H(10), about" ^0
"    " + str(expected_draws_to_see_all) + " draws" ^0
"  what ten draws are expected to reach : about six or" ^0
"    seven distinct categories" ^0
"  so four unseen after ten : the expected outcome, not a" ^0
"    finding" ^0
"" ^0

# ---- what the caller got ----

"the report" ^0
"  categories declared unreachable : " + str(categories_declared_unreachable) ^0
"  categories that actually exist : " + str(categories) ^0
"  draws made vs draws needed : " + str(draws_run) + " of about " + str(expected_draws_to_see_all) ^0
"  is any seen category wrong : no; all six are real" ^0
"  is 'not seen in ten draws' the same as 'cannot occur' :" ^0
"    no; the run made " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand of the" ^0
"    draws needed to expect a full set" ^0
"" ^0

# ---- null control ----

# The same run, continued to the coupon-collector expectation and beyond (or
# until the last new category is followed by a long silence).
6 => nc_distinct_after_ten_draws
10 => nc_distinct_after_the_expected_draws
4 => nc_categories_the_longer_run_recovers

"null control - run to the expected number of draws" ^0
"  distinct after ten draws : " + str(nc_distinct_after_ten_draws) ^0
"  distinct after about " + str(expected_draws_to_see_all) + " draws : " + str(nc_distinct_after_the_expected_draws) ^0
"  categories the longer run recovers : " + str(nc_categories_the_longer_run_recovers) ^0
"  no code and no generator changed; the run stopped being" ^0
"  cut off at the count of categories" ^0
"" ^0

# ---- the rule ----

"what a ten-draw fuzzing run guarantees" ^0
"  the categories it saw are reachable : exactly, genuine" ^0
"    draws, exact logging" ^0
"  the categories it did not see are unreachable : not" ^0
"    addressed; collecting all ten takes about " + str(expected_draws_to_see_all) + " draws on" ^0
"    average, so ten draws expect to miss about a third -" ^0
"    the " + str(categories_declared_unreachable) + " declared unreachable are simply not yet drawn" ^0
"" ^0

"each new kind is easy to find while most are still unfound and hard once few" ^0
"remain; collecting them all costs n times the harmonic sum, not n, and a run" ^0
"sized to the count of kinds stops while the rare tail is still expected" ^0
"" ^0

"It logs every category from genuine random draws - the six seen are real. But" ^0
"collecting all ten equally likely kinds takes about " + str(expected_draws_to_see_all) + " draws on average, and" ^0
"the run made " + str(draws_run) + ", " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand of that; the " + str(categories_declared_unreachable) + " categories it declared" ^0
"unreachable were the expected leftover, until the run was continued." ^0

categories = 10
draws_run = 10
distinct_categories_seen = 6
harmonic_ten_per_mille = 1000 + 500 + 333 + 250 + 200 + 166 + 142 + 125 + 111 + 100
expected_draws_to_see_all = int(categories * harmonic_ten_per_mille / 1000)
categories_declared_unreachable = categories - distinct_categories_seen
share_of_needed_draws_made_per_myriad = int(draws_run * 10000 / expected_draws_to_see_all)
print("error categories, equally likely : " + str(categories))
print("draws the run made              : " + str(draws_run))
print("distinct categories seen        : " + str(distinct_categories_seen))
print("categories declared unreachable : " + str(categories_declared_unreachable))
print("")
print("harmonic number H(10), per mille : " + str(harmonic_ten_per_mille))
print("expected draws to see all ten   : " + str(expected_draws_to_see_all))
print("share of needed draws made      : " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand")
print("")
print("the fuzzing run")
print("  each input : a genuine uniform draw from the generator")
print("  each category : logged exactly when triggered")
print("  draws made : " + str(draws_run) + ", one per category")
print("  intent : which error categories can this code produce")
print("  draws that were not random : 0")
print("  verdict : SIX CATEGORIES OBSERVED IN TEN DRAWS")
print("")
print("  logging every triggered category from genuine random")
print("  inputs is the part done right here, and it is why the six")
print("  seen are certainly reachable")
print("")
print("the coupon collector")
print("  the first draw : a new category, certainly")
print("  the last new category : found at one chance in ten per")
print("    draw, about ten draws of waiting")
print("  the total, summed over all ten : ten times H(10), about")
print("    " + str(expected_draws_to_see_all) + " draws")
print("  what ten draws are expected to reach : about six or")
print("    seven distinct categories")
print("  so four unseen after ten : the expected outcome, not a")
print("    finding")
print("")
print("the report")
print("  categories declared unreachable : " + str(categories_declared_unreachable))
print("  categories that actually exist : " + str(categories))
print("  draws made vs draws needed : " + str(draws_run) + " of about " + str(expected_draws_to_see_all))
print("  is any seen category wrong : no; all six are real")
print("  is 'not seen in ten draws' the same as 'cannot occur' :")
print("    no; the run made " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand of the")
print("    draws needed to expect a full set")
print("")
nc_distinct_after_ten_draws = 6
nc_distinct_after_the_expected_draws = 10
nc_categories_the_longer_run_recovers = 4
print("null control - run to the expected number of draws")
print("  distinct after ten draws : " + str(nc_distinct_after_ten_draws))
print("  distinct after about " + str(expected_draws_to_see_all) + " draws : " + str(nc_distinct_after_the_expected_draws))
print("  categories the longer run recovers : " + str(nc_categories_the_longer_run_recovers))
print("  no code and no generator changed; the run stopped being")
print("  cut off at the count of categories")
print("")
print("what a ten-draw fuzzing run guarantees")
print("  the categories it saw are reachable : exactly, genuine")
print("    draws, exact logging")
print("  the categories it did not see are unreachable : not")
print("    addressed; collecting all ten takes about " + str(expected_draws_to_see_all) + " draws on")
print("    average, so ten draws expect to miss about a third -")
print("    the " + str(categories_declared_unreachable) + " declared unreachable are simply not yet drawn")
print("")
print("each new kind is easy to find while most are still unfound and hard once few")
print("remain; collecting them all costs n times the harmonic sum, not n, and a run")
print("sized to the count of kinds stops while the rare tail is still expected")
print("")
print("It logs every category from genuine random draws - the six seen are real. But")
print("collecting all ten equally likely kinds takes about " + str(expected_draws_to_see_all) + " draws on average, and")
print("the run made " + str(draws_run) + ", " + str(share_of_needed_draws_made_per_myriad) + " per ten thousand of that; the " + str(categories_declared_unreachable) + " categories it declared")
print("unreachable were the expected leftover, until the run was continued.")

error categories, equally likely : 10
draws the run made              : 10
distinct categories seen        : 6
categories declared unreachable : 4

harmonic number H(10), per mille : 2927
expected draws to see all ten   : 29
share of needed draws made      : 3448 per ten thousand

the fuzzing run
  each input : a genuine uniform draw from the generator
  each category : logged exactly when triggered
  draws made : 10, one per category
  intent : which error categories can this code produce
  draws that were not random : 0
  verdict : SIX CATEGORIES OBSERVED IN TEN DRAWS

  logging every triggered category from genuine random
  inputs is the part done right here, and it is why the six
  seen are certainly reachable

the coupon collector
  the first draw : a new category, certainly
  the last new category : found at one chance in ten per
    draw, about ten draws of waiting
  the total, summed over all ten : ten times H(10), about
    29 draws
  what ten draws are expected to reach : about six or
    seven distinct categories
  so four unseen after ten : the expected outcome, not a
    finding

the report
  categories declared unreachable : 4
  categories that actually exist : 10
  draws made vs draws needed : 10 of about 29
  is any seen category wrong : no; all six are real
  is 'not seen in ten draws' the same as 'cannot occur' :
    no; the run made 3448 per ten thousand of the
    draws needed to expect a full set

null control - run to the expected number of draws
  distinct after ten draws : 6
  distinct after about 29 draws : 10
  categories the longer run recovers : 4
  no code and no generator changed; the run stopped being
  cut off at the count of categories

what a ten-draw fuzzing run guarantees
  the categories it saw are reachable : exactly, genuine
    draws, exact logging
  the categories it did not see are unreachable : not
    addressed; collecting all ten takes about 29 draws on
    average, so ten draws expect to miss about a third -
    the 4 declared unreachable are simply not yet drawn

each new kind is easy to find while most are still unfound and hard once few
remain; collecting them all costs n times the harmonic sum, not n, and a run
sized to the count of kinds stops while the rare tail is still expected

It logs every category from genuine random draws - the six seen are real. But
collecting all ten equally likely kinds takes about 29 draws on average, and
the run made 10, 3448 per ten thousand of that; the 4 categories it declared
unreachable were the expected leftover, until the run was continued.

Trace event types