The sample was taken after the filter

# Self-authored for the EML case corpus (no external origin). A quality audit
# draws five hundred records at random from the processed table and finds a
# defect rate of two per thousand. What that table contains is computed below.
#
# The sampling is correct and it was done properly. The draw is uniform, the
# sample size was chosen from the width of the interval the audit needed, the
# records were pulled without replacement, and nobody chose which ones. Within
# its frame the estimate is unbiased and the confidence interval is real.
#
# A sample estimates the population it is DRAWN FROM. The processed table is
# not the population the audit is reporting on; it is what survived ingestion,
# and ingestion drops what it cannot parse.
#
# The records that failed are the ones the question is about, and they were
# removed before the sampling frame existed.

240000 => submitted
800 => rejected_per_myriad
500 => sample_size
20 => defects_per_myriad_in_sample

int(submitted * rejected_per_myriad / 10000) => rejected_at_ingest
submitted - rejected_at_ingest => reached_the_table

"records submitted        : " + str(submitted) ^0
"rejected at ingestion    : " + str(rejected_at_ingest) ^0
"reached the table        : " + str(reached_the_table) ^0
"sample drawn from        : the table" ^0
"sample size              : " + str(sample_size) ^0
"" ^0

# ---- the audit's finding ----

"what the audit reports" ^0
"  defect rate in the sample : " + str(defects_per_myriad_in_sample) + " per ten thousand" ^0
"  drawn uniformly           : yes" ^0
"  chosen by a person        : no" ^0
"  replacement               : none" ^0
"  the estimate is unbiased for the table" ^0
"" ^0

# ---- what the table is ----

int(defects_per_myriad_in_sample * reached_the_table / 10000) => defects_in_table

"the frame the estimate describes" ^0
"  records in the table      : " + str(reached_the_table) ^0
"  estimated defects there   : " + str(defects_in_table) ^0
"  records NOT in the table  : " + str(rejected_at_ingest) ^0
"  defects among those       : " + str(rejected_at_ingest) + ", by definition" ^0
"" ^0
"  a record is in the second group BECAUSE it was malformed," ^0
"  so that group's defect rate is not estimated, it is known," ^0
"  and it is total" ^0
"" ^0

# ---- the rate over what was submitted ----

defects_in_table + rejected_at_ingest => defects_over_submitted
int(defects_over_submitted * 10000 / submitted) => true_rate_per_myriad

"defect rate, two frames" ^0
"  over the table     : " + str(defects_per_myriad_in_sample) + " per ten thousand" ^0
"  over what was sent : " + str(true_rate_per_myriad) + " per ten thousand" ^0
"  ratio              : " + str(int(true_rate_per_myriad / defects_per_myriad_in_sample)) + " times" ^0
"" ^0
"  both numbers are correct about their own denominator" ^0
"  the report names neither denominator" ^0
"" ^0

# ---- where the filter sits ----

"the pipeline, in order" ^0
"  1  records are submitted            " + str(submitted) ^0
"  2  ingestion parses and drops       " + str(rejected_at_ingest) + " removed" ^0
"  3  the table receives               " + str(reached_the_table) ^0
"  4  the audit samples                " + str(sample_size) + " from step 3" ^0
"" ^0
"  the audit is asking about step 1 and drawing from step 3" ^0
"  and steps 1 and 3 differ by exactly the records that failed" ^0
"" ^0

# ---- what a larger sample would do ----
#
# The obvious response to a surprising rate is more data. Every extra record
# comes from the same table.

5000 => bigger_sample

"a sample ten times larger" ^0
"  sample size           : " + str(bigger_sample) ^0
"  drawn from            : the table, still" ^0
"  interval width        : narrower" ^0
"  estimate              : " + str(defects_per_myriad_in_sample) + " per ten thousand, more precisely" ^0
"  rejected records included : 0" ^0
"" ^0
"  precision improves and the frame does not move" ^0
"" ^0

# ---- the control ----
#
# The sampling method, on its own terms. It is unbiased within its frame and it
# is the reason the number is trusted at all.

"control - is the sampling correct" ^0
"  selection bias within the table : none" ^0
"  sample size adequate            : yes, for the stated interval" ^0
"  records excluded by the sampler : 0" ^0
"  defects in the method           : 0" ^0
"" ^0
"  nothing about the draw needs fixing; what needs stating is" ^0
"  the sentence 'of the records that reached the table'" ^0
"" ^0

# ---- the null control ----
#
# The same method, same size, same auditor, drawing from the submission log
# instead of the table. The technique did not change; the frame became the one
# the question was about.

"null control - the same sample drawn from the submission log" ^0
"  frame              : " + str(submitted) + " submitted records" ^0
"  sample size        : " + str(sample_size) + ", unchanged" ^0
"  method             : unchanged" ^0
"  estimated rate     : " + str(true_rate_per_myriad) + " per ten thousand" ^0
"  the sampler did not improve; it was pointed at the population" ^0
"  the report was already describing" ^0
"" ^0

# ---- the rule ----

"what a sample is evidence about" ^0
"  the frame it was drawn from : exactly, and with an interval" ^0
"  the population upstream of a filter : nothing at all" ^0
"  and a filter that removes failures makes the two differ by" ^0
"  precisely the thing being measured" ^0
"" ^0
"the question to ask of any rate is not how many were sampled," ^0
"it is what had to happen to a record for it to be eligible" ^0
"" ^0

"The draw is uniform, unbiased and adequately sized, and " + str(defects_per_myriad_in_sample) + " per ten thousand is" ^0
"a correct estimate for the " + str(reached_the_table) + " records in the table. Ingestion dropped " + str(rejected_at_ingest) ^0
"records for being malformed, which is the defect the audit exists to count, so" ^0
"over the " + str(submitted) + " actually submitted the rate is " + str(true_rate_per_myriad) + " per ten thousand - " + str(int(true_rate_per_myriad / defects_per_myriad_in_sample)) + " times" ^0
"the reported one - and a larger sample moves the interval, not the frame." ^0

submitted = 240000
rejected_per_myriad = 800
sample_size = 500
defects_per_myriad_in_sample = 20
rejected_at_ingest = int(submitted * rejected_per_myriad / 10000)
reached_the_table = submitted - rejected_at_ingest
print("records submitted        : " + str(submitted))
print("rejected at ingestion    : " + str(rejected_at_ingest))
print("reached the table        : " + str(reached_the_table))
print("sample drawn from        : the table")
print("sample size              : " + str(sample_size))
print("")
print("what the audit reports")
print("  defect rate in the sample : " + str(defects_per_myriad_in_sample) + " per ten thousand")
print("  drawn uniformly           : yes")
print("  chosen by a person        : no")
print("  replacement               : none")
print("  the estimate is unbiased for the table")
print("")
defects_in_table = int(defects_per_myriad_in_sample * reached_the_table / 10000)
print("the frame the estimate describes")
print("  records in the table      : " + str(reached_the_table))
print("  estimated defects there   : " + str(defects_in_table))
print("  records NOT in the table  : " + str(rejected_at_ingest))
print("  defects among those       : " + str(rejected_at_ingest) + ", by definition")
print("")
print("  a record is in the second group BECAUSE it was malformed,")
print("  so that group's defect rate is not estimated, it is known,")
print("  and it is total")
print("")
defects_over_submitted = defects_in_table + rejected_at_ingest
true_rate_per_myriad = int(defects_over_submitted * 10000 / submitted)
print("defect rate, two frames")
print("  over the table     : " + str(defects_per_myriad_in_sample) + " per ten thousand")
print("  over what was sent : " + str(true_rate_per_myriad) + " per ten thousand")
print("  ratio              : " + str(int(true_rate_per_myriad / defects_per_myriad_in_sample)) + " times")
print("")
print("  both numbers are correct about their own denominator")
print("  the report names neither denominator")
print("")
print("the pipeline, in order")
print("  1  records are submitted            " + str(submitted))
print("  2  ingestion parses and drops       " + str(rejected_at_ingest) + " removed")
print("  3  the table receives               " + str(reached_the_table))
print("  4  the audit samples                " + str(sample_size) + " from step 3")
print("")
print("  the audit is asking about step 1 and drawing from step 3")
print("  and steps 1 and 3 differ by exactly the records that failed")
print("")
bigger_sample = 5000
print("a sample ten times larger")
print("  sample size           : " + str(bigger_sample))
print("  drawn from            : the table, still")
print("  interval width        : narrower")
print("  estimate              : " + str(defects_per_myriad_in_sample) + " per ten thousand, more precisely")
print("  rejected records included : 0")
print("")
print("  precision improves and the frame does not move")
print("")
print("control - is the sampling correct")
print("  selection bias within the table : none")
print("  sample size adequate            : yes, for the stated interval")
print("  records excluded by the sampler : 0")
print("  defects in the method           : 0")
print("")
print("  nothing about the draw needs fixing; what needs stating is")
print("  the sentence 'of the records that reached the table'")
print("")
print("null control - the same sample drawn from the submission log")
print("  frame              : " + str(submitted) + " submitted records")
print("  sample size        : " + str(sample_size) + ", unchanged")
print("  method             : unchanged")
print("  estimated rate     : " + str(true_rate_per_myriad) + " per ten thousand")
print("  the sampler did not improve; it was pointed at the population")
print("  the report was already describing")
print("")
print("what a sample is evidence about")
print("  the frame it was drawn from : exactly, and with an interval")
print("  the population upstream of a filter : nothing at all")
print("  and a filter that removes failures makes the two differ by")
print("  precisely the thing being measured")
print("")
print("the question to ask of any rate is not how many were sampled,")
print("it is what had to happen to a record for it to be eligible")
print("")
print("The draw is uniform, unbiased and adequately sized, and " + str(defects_per_myriad_in_sample) + " per ten thousand is")
print("a correct estimate for the " + str(reached_the_table) + " records in the table. Ingestion dropped " + str(rejected_at_ingest))
print("records for being malformed, which is the defect the audit exists to count, so")
print("over the " + str(submitted) + " actually submitted the rate is " + str(true_rate_per_myriad) + " per ten thousand - " + str(int(true_rate_per_myriad / defects_per_myriad_in_sample)) + " times")
print("the reported one - and a larger sample moves the interval, not the frame.")

records submitted        : 240000
rejected at ingestion    : 19200
reached the table        : 220800
sample drawn from        : the table
sample size              : 500

what the audit reports
  defect rate in the sample : 20 per ten thousand
  drawn uniformly           : yes
  chosen by a person        : no
  replacement               : none
  the estimate is unbiased for the table

the frame the estimate describes
  records in the table      : 220800
  estimated defects there   : 441
  records NOT in the table  : 19200
  defects among those       : 19200, by definition

  a record is in the second group BECAUSE it was malformed,
  so that group's defect rate is not estimated, it is known,
  and it is total

defect rate, two frames
  over the table     : 20 per ten thousand
  over what was sent : 818 per ten thousand
  ratio              : 40 times

  both numbers are correct about their own denominator
  the report names neither denominator

the pipeline, in order
  1  records are submitted            240000
  2  ingestion parses and drops       19200 removed
  3  the table receives               220800
  4  the audit samples                500 from step 3

  the audit is asking about step 1 and drawing from step 3
  and steps 1 and 3 differ by exactly the records that failed

a sample ten times larger
  sample size           : 5000
  drawn from            : the table, still
  interval width        : narrower
  estimate              : 20 per ten thousand, more precisely
  rejected records included : 0

  precision improves and the frame does not move

control - is the sampling correct
  selection bias within the table : none
  sample size adequate            : yes, for the stated interval
  records excluded by the sampler : 0
  defects in the method           : 0

  nothing about the draw needs fixing; what needs stating is
  the sentence 'of the records that reached the table'

null control - the same sample drawn from the submission log
  frame              : 240000 submitted records
  sample size        : 500, unchanged
  method             : unchanged
  estimated rate     : 818 per ten thousand
  the sampler did not improve; it was pointed at the population
  the report was already describing

what a sample is evidence about
  the frame it was drawn from : exactly, and with an interval
  the population upstream of a filter : nothing at all
  and a filter that removes failures makes the two differ by
  precisely the thing being measured

the question to ask of any rate is not how many were sampled,
it is what had to happen to a record for it to be eligible

The draw is uniform, unbiased and adequately sized, and 20 per ten thousand is
a correct estimate for the 220800 records in the table. Ingestion dropped 19200
records for being malformed, which is the defect the audit exists to count, so
over the 240000 actually submitted the rate is 818 per ten thousand - 40 times
the reported one - and a larger sample moves the interval, not the frame.

Trace event types