The audit sampled the records that were easiest to fetch

# Self-authored for the EML case corpus (no external origin). An audit sampled
# 500 of 50000 accounts and found a defect rate of 4 per thousand. What that
# rate is a rate of is computed below.
#
# The sample size was chosen properly, from a table, for a stated confidence
# level, and 500 is genuinely enough for a population of 50000. The auditor
# checked that. The sampling query was written to be fast because the audit runs
# nightly and a full scan would lock the table during the batch window - a real
# operational constraint, raised by the DBA, and respected. The query reads from
# the account cache, which is what every other read path in the system uses.
#
# The cache holds accounts touched in the last 90 days. That is 12000 of the
# 50000. The other 38000 are dormant, and dormant is not a neutral property: an
# account is dormant because nobody has touched it, and nobody touching it is
# also why its address, its tax code and its consent flags were never migrated
# when the schema changed.
#
# The sample is unbiased with respect to everything except the one attribute
# that determines whether a record is in the frame at all - and that attribute
# is correlated with the defect.

50000 => accounts
12000 => active
500 => sample_size

accounts - active => dormant

# defects per thousand, by stratum
4 => rate_active
60 => rate_dormant

"accounts            : " + str(accounts) ^0
"  in cache, active  : " + str(active) ^0
"  dormant           : " + str(dormant) ^0
"sample size         : " + str(sample_size) + ", drawn from the cache" ^0
"" ^0

# ---- what the audit saw ----

int(sample_size * rate_active / 1000) => observed_defects
int(observed_defects * 1000 / sample_size) => observed_rate
int(accounts * observed_defects / sample_size) => extrapolated

"the audit" ^0
"  defects found in the sample : " + str(observed_defects) + " of " + str(sample_size) ^0
"  observed rate               : " + str(observed_rate) + " per thousand" ^0
"  extrapolated to " + str(accounts) + "    : " + str(extrapolated) + " defective accounts" ^0
"" ^0

# ---- what is there ----

int(active * rate_active / 1000) => defects_active
int(dormant * rate_dormant / 1000) => defects_dormant
defects_active + defects_dormant => defects_true

"stratum    accounts   rate per thousand   defects   in the sampling frame" ^0
"  active     " + str(active) + "        " + str(rate_active) + "                  " + str(defects_active) + "       yes" ^0
"  dormant    " + str(dormant) + "        " + str(rate_dormant) + "                 " + str(defects_dormant) + "     no" ^0
"  total      " + str(accounts) + "                            " + str(defects_true) ^0
"" ^0

"  audit reported : " + str(extrapolated) ^0
"  actually there : " + str(defects_true) ^0
"  understated by a factor of " + str(int(defects_true / extrapolated)) ^0
"" ^0

# ---- the rate was correct ----
#
# 4 per thousand is the true rate of the population the sample was drawn from.
# The sample is a good estimate of the cache. The report labelled it "accounts".

"the estimate, judged against the population it actually sampled" ^0
"  estimated rate in cache : " + str(observed_rate) + " per thousand" ^0
"  true rate in cache      : " + str(rate_active) + " per thousand" ^0
"  error                   : " + str(observed_rate - rate_active) + " per thousand" ^0
"  the sampling was correct; every step of it was correct" ^0
"" ^0
"  what was wrong is one word in the sentence that reported it" ^0
"  'we sampled accounts' should have read 'we sampled cached accounts'" ^0
"" ^0

# ---- why convenience is not neutral here ----
#
# A convenience sample is harmless when convenience is unrelated to the thing
# being measured. Here the mechanism that puts a record in the cache is the
# same mechanism that would have fixed it.

"the causal chain that makes the frame biased" ^0
"  an account is cached because it was touched recently" ^0
"  an account was migrated because someone touched it" ^0
"  so cached implies touched implies migrated implies not defective" ^0
"  the frame and the defect share a cause, which is the definition of a" ^0
"  biased frame, and no sample size fixes it" ^0
"" ^0
"  a larger sample from the same frame" ^0
5000 => bigger_sample
int(bigger_sample * rate_active / 1000) => bigger_defects
"    sample " + str(bigger_sample) + " instead of " + str(sample_size) + " : " + str(int(accounts * bigger_defects / bigger_sample)) + " extrapolated" ^0
"    ten times the work, the same wrong answer, with tighter confidence bounds" ^0
"" ^0

# ---- the control ----
#
# Draw the sample from the whole table instead of the cache, keeping every
# other choice identical - same size, same confidence level, same checks. This
# is the measurement that separates "the sample was too small" from "the sample
# came from the wrong place".

int(sample_size * active / accounts) => frame_active
sample_size - frame_active => frame_dormant
int(frame_active * rate_active / 1000) + int(frame_dormant * rate_dormant / 1000) => frame_defects

"control - the same sample size drawn from the whole table" ^0
"  sampled from active  : " + str(frame_active) ^0
"  sampled from dormant : " + str(frame_dormant) ^0
"  defects found        : " + str(frame_defects) ^0
"  extrapolated         : " + str(int(accounts * frame_defects / sample_size)) ^0
"  actually there       : " + str(defects_true) ^0
int(accounts * frame_defects / sample_size) => frame_extrapolated
"  cache-only frame was low by  : " + str(int((defects_true - extrapolated) * 100 / defects_true)) + " percent" ^0
"  whole-table frame is low by  : " + str(int((defects_true - frame_extrapolated) * 100 / defects_true)) + " percent" ^0
"  the same 500 records of effort, and the frame is the only thing that changed" ^0
"" ^0

# ---- the null control ----
#
# The same convenient frame, against a table where both strata have the same
# defect rate. The convenience sample is then unbiased, because convenience is
# no longer correlated with the defect. Sampling from the cache is not the
# defect on its own.

"null control - the same cache-only frame, uniform defect rate across strata" ^0
int(active * rate_active / 1000) + int(dormant * rate_active / 1000) => uniform_true
"  rate in both strata : " + str(rate_active) + " per thousand" ^0
"  extrapolated from cache sample : " + str(extrapolated) ^0
"  actually there                 : " + str(uniform_true) ^0
"  difference                     : " + str(uniform_true - extrapolated) ^0
"  the same query, the same frame, and now it is a fine estimate" ^0
"  so the rule is not 'do not sample from a cache'" ^0
"  it is 'a frame is biased exactly as far as belonging to it predicts the" ^0
"  answer'" ^0
"" ^0

# ---- the rule ----

"what a sample size protects against" ^0
"  random variation in the estimate      yes" ^0
"  a frame that excludes part of the population   no" ^0
"  a frame whose membership predicts the answer   no, and it makes it worse" ^0
"  the confidence interval is computed from the sample size" ^0
"  so a biased frame reports a tight interval around the wrong number" ^0
"" ^0

"500 of 50000 is a correct sample size, the confidence level was chosen from a" ^0
"table, and reading from the cache avoided locking the table during the batch" ^0
"window - which the DBA had asked for. The rate that came back, " + str(observed_rate) + " per" ^0
"thousand, is the true rate of cached accounts. " + str(defects_true) + " accounts are defective," ^0
"the audit reported " + str(extrapolated) + ", and " + str(defects_dormant) + " of the defects were in records the query" ^0
"was written never to read." ^0

accounts = 50000
active = 12000
sample_size = 500
dormant = accounts - active
rate_active = 4
rate_dormant = 60
print("accounts            : " + str(accounts))
print("  in cache, active  : " + str(active))
print("  dormant           : " + str(dormant))
print("sample size         : " + str(sample_size) + ", drawn from the cache")
print("")
observed_defects = int(sample_size * rate_active / 1000)
observed_rate = int(observed_defects * 1000 / sample_size)
extrapolated = int(accounts * observed_defects / sample_size)
print("the audit")
print("  defects found in the sample : " + str(observed_defects) + " of " + str(sample_size))
print("  observed rate               : " + str(observed_rate) + " per thousand")
print("  extrapolated to " + str(accounts) + "    : " + str(extrapolated) + " defective accounts")
print("")
defects_active = int(active * rate_active / 1000)
defects_dormant = int(dormant * rate_dormant / 1000)
defects_true = defects_active + defects_dormant
print("stratum    accounts   rate per thousand   defects   in the sampling frame")
print("  active     " + str(active) + "        " + str(rate_active) + "                  " + str(defects_active) + "       yes")
print("  dormant    " + str(dormant) + "        " + str(rate_dormant) + "                 " + str(defects_dormant) + "     no")
print("  total      " + str(accounts) + "                            " + str(defects_true))
print("")
print("  audit reported : " + str(extrapolated))
print("  actually there : " + str(defects_true))
print("  understated by a factor of " + str(int(defects_true / extrapolated)))
print("")
print("the estimate, judged against the population it actually sampled")
print("  estimated rate in cache : " + str(observed_rate) + " per thousand")
print("  true rate in cache      : " + str(rate_active) + " per thousand")
print("  error                   : " + str(observed_rate - rate_active) + " per thousand")
print("  the sampling was correct; every step of it was correct")
print("")
print("  what was wrong is one word in the sentence that reported it")
print("  'we sampled accounts' should have read 'we sampled cached accounts'")
print("")
print("the causal chain that makes the frame biased")
print("  an account is cached because it was touched recently")
print("  an account was migrated because someone touched it")
print("  so cached implies touched implies migrated implies not defective")
print("  the frame and the defect share a cause, which is the definition of a")
print("  biased frame, and no sample size fixes it")
print("")
print("  a larger sample from the same frame")
bigger_sample = 5000
bigger_defects = int(bigger_sample * rate_active / 1000)
print("    sample " + str(bigger_sample) + " instead of " + str(sample_size) + " : " + str(int(accounts * bigger_defects / bigger_sample)) + " extrapolated")
print("    ten times the work, the same wrong answer, with tighter confidence bounds")
print("")
frame_active = int(sample_size * active / accounts)
frame_dormant = sample_size - frame_active
frame_defects = int(frame_active * rate_active / 1000) + int(frame_dormant * rate_dormant / 1000)
print("control - the same sample size drawn from the whole table")
print("  sampled from active  : " + str(frame_active))
print("  sampled from dormant : " + str(frame_dormant))
print("  defects found        : " + str(frame_defects))
print("  extrapolated         : " + str(int(accounts * frame_defects / sample_size)))
print("  actually there       : " + str(defects_true))
frame_extrapolated = int(accounts * frame_defects / sample_size)
print("  cache-only frame was low by  : " + str(int((defects_true - extrapolated) * 100 / defects_true)) + " percent")
print("  whole-table frame is low by  : " + str(int((defects_true - frame_extrapolated) * 100 / defects_true)) + " percent")
print("  the same 500 records of effort, and the frame is the only thing that changed")
print("")
print("null control - the same cache-only frame, uniform defect rate across strata")
uniform_true = int(active * rate_active / 1000) + int(dormant * rate_active / 1000)
print("  rate in both strata : " + str(rate_active) + " per thousand")
print("  extrapolated from cache sample : " + str(extrapolated))
print("  actually there                 : " + str(uniform_true))
print("  difference                     : " + str(uniform_true - extrapolated))
print("  the same query, the same frame, and now it is a fine estimate")
print("  so the rule is not 'do not sample from a cache'")
print("  it is 'a frame is biased exactly as far as belonging to it predicts the")
print("  answer'")
print("")
print("what a sample size protects against")
print("  random variation in the estimate      yes")
print("  a frame that excludes part of the population   no")
print("  a frame whose membership predicts the answer   no, and it makes it worse")
print("  the confidence interval is computed from the sample size")
print("  so a biased frame reports a tight interval around the wrong number")
print("")
print("500 of 50000 is a correct sample size, the confidence level was chosen from a")
print("table, and reading from the cache avoided locking the table during the batch")
print("window - which the DBA had asked for. The rate that came back, " + str(observed_rate) + " per")
print("thousand, is the true rate of cached accounts. " + str(defects_true) + " accounts are defective,")
print("the audit reported " + str(extrapolated) + ", and " + str(defects_dormant) + " of the defects were in records the query")
print("was written never to read.")

accounts            : 50000
  in cache, active  : 12000
  dormant           : 38000
sample size         : 500, drawn from the cache

the audit
  defects found in the sample : 2 of 500
  observed rate               : 4 per thousand
  extrapolated to 50000    : 200 defective accounts

stratum    accounts   rate per thousand   defects   in the sampling frame
  active     12000        4                  48       yes
  dormant    38000        60                 2280     no
  total      50000                            2328

  audit reported : 200
  actually there : 2328
  understated by a factor of 11

the estimate, judged against the population it actually sampled
  estimated rate in cache : 4 per thousand
  true rate in cache      : 4 per thousand
  error                   : 0 per thousand
  the sampling was correct; every step of it was correct

  what was wrong is one word in the sentence that reported it
  'we sampled accounts' should have read 'we sampled cached accounts'

the causal chain that makes the frame biased
  an account is cached because it was touched recently
  an account was migrated because someone touched it
  so cached implies touched implies migrated implies not defective
  the frame and the defect share a cause, which is the definition of a
  biased frame, and no sample size fixes it

  a larger sample from the same frame
    sample 5000 instead of 500 : 200 extrapolated
    ten times the work, the same wrong answer, with tighter confidence bounds

control - the same sample size drawn from the whole table
  sampled from active  : 120
  sampled from dormant : 380
  defects found        : 22
  extrapolated         : 2200
  actually there       : 2328
  cache-only frame was low by  : 91 percent
  whole-table frame is low by  : 5 percent
  the same 500 records of effort, and the frame is the only thing that changed

null control - the same cache-only frame, uniform defect rate across strata
  rate in both strata : 4 per thousand
  extrapolated from cache sample : 200
  actually there                 : 200
  difference                     : 0
  the same query, the same frame, and now it is a fine estimate
  so the rule is not 'do not sample from a cache'
  it is 'a frame is biased exactly as far as belonging to it predicts the
  answer'

what a sample size protects against
  random variation in the estimate      yes
  a frame that excludes part of the population   no
  a frame whose membership predicts the answer   no, and it makes it worse
  the confidence interval is computed from the sample size
  so a biased frame reports a tight interval around the wrong number

500 of 50000 is a correct sample size, the confidence level was chosen from a
table, and reading from the cache avoided locking the table during the batch
window - which the DBA had asked for. The rate that came back, 4 per
thousand, is the true rate of cached accounts. 2328 accounts are defective,
the audit reported 200, and 2280 of the defects were in records the query
was written never to read.

Trace event types