The null was cheaper to store than the reason

# Self-authored for the EML case corpus (no external origin). A column is null
# on nineteen percent of rows. Three different things produced those nulls.
# What can be recovered from them is computed below.
#
# Storing null was correct and it is what the column is for. The alternative
# was a sentinel value, and a sentinel in a numeric column is a number that
# enters averages, comparisons and sums as though it were data — which is the
# defect a nullable column exists to prevent. The schema is right, and it was
# argued about before it was chosen.
#
# Null is one symbol. The situations that produce it are not one situation, and
# the difference between them is what a reader of the column needs.
#
# The reason was known at write time by whichever code path wrote the row. It
# was not written down, because there was no column to write it in.

420000 => rows
1900 => null_per_myriad

int(rows * null_per_myriad / 10000) => null_rows
rows - null_rows => valued_rows

"rows                : " + str(rows) ^0
"rows with a null    : " + str(null_rows) ^0
"rows with a value   : " + str(valued_rows) ^0
"" ^0

# ---- the three writers ----

45 => not_asked_per_hundred
35 => declined_per_hundred
20 => failed_per_hundred

int(null_rows * not_asked_per_hundred / 100) => not_asked
int(null_rows * declined_per_hundred / 100) => declined
int(null_rows * failed_per_hundred / 100) => lookup_failed

"what wrote the null" ^0
"  the question was not asked      : " + str(not_asked) ^0
"  the person declined to answer   : " + str(declined) ^0
"  the lookup failed at write time : " + str(lookup_failed) ^0
"  total                           : " + str(not_asked + declined + lookup_failed) ^0
"" ^0
"  three different facts, one symbol, and the symbol is" ^0
"  identical in all three cases" ^0
"" ^0

# ---- what each consumer needs ----

"consumer                   needs to distinguish" ^0
"  the completeness report    not-asked from declined" ^0
"  the retry job              failed from the other two" ^0
"  the consent audit          declined from everything else" ^0
"  the average                all three, they are all excluded" ^0
"" ^0
"  three of the four cannot be answered from the column," ^0
"  and the fourth is the only one that ever gets computed" ^0
"" ^0

# ---- the retry job ----
#
# The lookups that failed can be retried and would probably succeed. Nothing
# can tell them from the ones that were never asked.

"the retry job, as it must be written" ^0
"  rows it can identify as retryable : 0" ^0
"  rows it would have to retry       : " + str(null_rows) ^0
"  rows that would succeed           : " + str(lookup_failed) ^0
"  rows retried pointlessly          : " + str(not_asked + declined) ^0
"" ^0
int((not_asked + declined) * 10000 / null_rows) => wasted_per_myriad
"  wasted work : " + str(wasted_per_myriad) + " per ten thousand of the retries" ^0
"" ^0
"  and " + str(declined) + " of them re-ask a person who already said no" ^0
"" ^0

# ---- what the completeness report says ----

int(valued_rows * 10000 / rows) => complete_per_myriad

"the completeness report" ^0
"  rows complete        : " + str(complete_per_myriad) + " per ten thousand" ^0
"  rows incomplete      : " + str(null_per_myriad) + " per ten thousand" ^0
"  of the incomplete, how many are answerable : not computable" ^0
"" ^0
"  the report is exactly right and the follow-up question is" ^0
"  the one anybody actually has" ^0
"" ^0

# ---- what recovering it would take now ----

"recovering the reason after the fact" ^0
"  from the column          : impossible, one symbol" ^0
"  from the write path      : the code no longer exists in that form" ^0
"  from the application log : retained 30 days, these rows are older" ^0
"  by re-asking             : possible, and re-asks the " + str(declined) + " who declined" ^0
"" ^0
"  the information existed at write time, cost one small column" ^0
"  to keep, and is now only obtainable by contacting people" ^0
"" ^0

# ---- the control ----
#
# The choice of null over a sentinel, on its own terms. A sentinel would have
# entered every aggregate as a number, and that is not a hypothetical failure.

"control - was null the right symbol" ^0
"  sentinel values in the average : 0" ^0
"  comparisons against a magic number : 0" ^0
"  rows where absent is treated as a value : 0" ^0
"  defects in the schema choice : 0" ^0
"" ^0
"  the column is correct; what is missing was never in it" ^0
"" ^0

# ---- the null control ----
#
# The same nullable column beside a reason column. The null stays null, the
# aggregates behave identically, nothing about the schema decision is revisited.

0 => nc_unattributable

"null control - the same null with a reason column beside it" ^0
"  nulls in the column      : " + str(null_rows) + ", unchanged" ^0
"  sentinel values          : 0, unchanged" ^0
"  nulls whose cause is unknown : " + str(nc_unattributable) ^0
"  retries that are pointless   : " + str(nc_unattributable) ^0
"  the symbol did not change; a second field records what the" ^0
"  first one was never able to say" ^0
"" ^0

# ---- the rule ----

"what a null records" ^0
"  there is no value here : exactly, and unambiguously" ^0
"  why there is no value  : nothing" ^0
"  and 'why' is the part every consumer downstream needs," ^0
"  because it is what decides whether to retry, to re-ask," ^0
"  to exclude, or to leave alone" ^0
"" ^0
"absence is not one state; it is at least three, and they are" ^0
"distinguishable only at the moment of writing, by the code" ^0
"that already knows which one it is" ^0
"" ^0

"The column is nullable rather than sentinel-valued, which is the right choice:" ^0
"0 magic numbers enter the averages and 0 rows treat absent as a value. Of the" ^0
str(null_rows) + " nulls, " + str(not_asked) + " were never asked, " + str(declined) + " were declined and " + str(lookup_failed) + " failed a" ^0
"lookup, and nothing distinguishes them, so a retry job must attempt all " + str(null_rows) ^0
"to reach " + str(lookup_failed) + " - " + str(wasted_per_myriad) + " per ten thousand wasted - while re-asking " + str(declined) + " people" ^0
"who already answered." ^0

rows = 420000
null_per_myriad = 1900
null_rows = int(rows * null_per_myriad / 10000)
valued_rows = rows - null_rows
print("rows                : " + str(rows))
print("rows with a null    : " + str(null_rows))
print("rows with a value   : " + str(valued_rows))
print("")
not_asked_per_hundred = 45
declined_per_hundred = 35
failed_per_hundred = 20
not_asked = int(null_rows * not_asked_per_hundred / 100)
declined = int(null_rows * declined_per_hundred / 100)
lookup_failed = int(null_rows * failed_per_hundred / 100)
print("what wrote the null")
print("  the question was not asked      : " + str(not_asked))
print("  the person declined to answer   : " + str(declined))
print("  the lookup failed at write time : " + str(lookup_failed))
print("  total                           : " + str(not_asked + declined + lookup_failed))
print("")
print("  three different facts, one symbol, and the symbol is")
print("  identical in all three cases")
print("")
print("consumer                   needs to distinguish")
print("  the completeness report    not-asked from declined")
print("  the retry job              failed from the other two")
print("  the consent audit          declined from everything else")
print("  the average                all three, they are all excluded")
print("")
print("  three of the four cannot be answered from the column,")
print("  and the fourth is the only one that ever gets computed")
print("")
print("the retry job, as it must be written")
print("  rows it can identify as retryable : 0")
print("  rows it would have to retry       : " + str(null_rows))
print("  rows that would succeed           : " + str(lookup_failed))
print("  rows retried pointlessly          : " + str(not_asked + declined))
print("")
wasted_per_myriad = int((not_asked + declined) * 10000 / null_rows)
print("  wasted work : " + str(wasted_per_myriad) + " per ten thousand of the retries")
print("")
print("  and " + str(declined) + " of them re-ask a person who already said no")
print("")
complete_per_myriad = int(valued_rows * 10000 / rows)
print("the completeness report")
print("  rows complete        : " + str(complete_per_myriad) + " per ten thousand")
print("  rows incomplete      : " + str(null_per_myriad) + " per ten thousand")
print("  of the incomplete, how many are answerable : not computable")
print("")
print("  the report is exactly right and the follow-up question is")
print("  the one anybody actually has")
print("")
print("recovering the reason after the fact")
print("  from the column          : impossible, one symbol")
print("  from the write path      : the code no longer exists in that form")
print("  from the application log : retained 30 days, these rows are older")
print("  by re-asking             : possible, and re-asks the " + str(declined) + " who declined")
print("")
print("  the information existed at write time, cost one small column")
print("  to keep, and is now only obtainable by contacting people")
print("")
print("control - was null the right symbol")
print("  sentinel values in the average : 0")
print("  comparisons against a magic number : 0")
print("  rows where absent is treated as a value : 0")
print("  defects in the schema choice : 0")
print("")
print("  the column is correct; what is missing was never in it")
print("")
nc_unattributable = 0
print("null control - the same null with a reason column beside it")
print("  nulls in the column      : " + str(null_rows) + ", unchanged")
print("  sentinel values          : 0, unchanged")
print("  nulls whose cause is unknown : " + str(nc_unattributable))
print("  retries that are pointless   : " + str(nc_unattributable))
print("  the symbol did not change; a second field records what the")
print("  first one was never able to say")
print("")
print("what a null records")
print("  there is no value here : exactly, and unambiguously")
print("  why there is no value  : nothing")
print("  and 'why' is the part every consumer downstream needs,")
print("  because it is what decides whether to retry, to re-ask,")
print("  to exclude, or to leave alone")
print("")
print("absence is not one state; it is at least three, and they are")
print("distinguishable only at the moment of writing, by the code")
print("that already knows which one it is")
print("")
print("The column is nullable rather than sentinel-valued, which is the right choice:")
print("0 magic numbers enter the averages and 0 rows treat absent as a value. Of the")
print(str(null_rows) + " nulls, " + str(not_asked) + " were never asked, " + str(declined) + " were declined and " + str(lookup_failed) + " failed a")
print("lookup, and nothing distinguishes them, so a retry job must attempt all " + str(null_rows))
print("to reach " + str(lookup_failed) + " - " + str(wasted_per_myriad) + " per ten thousand wasted - while re-asking " + str(declined) + " people")
print("who already answered.")

rows                : 420000
rows with a null    : 79800
rows with a value   : 340200

what wrote the null
  the question was not asked      : 35910
  the person declined to answer   : 27930
  the lookup failed at write time : 15960
  total                           : 79800

  three different facts, one symbol, and the symbol is
  identical in all three cases

consumer                   needs to distinguish
  the completeness report    not-asked from declined
  the retry job              failed from the other two
  the consent audit          declined from everything else
  the average                all three, they are all excluded

  three of the four cannot be answered from the column,
  and the fourth is the only one that ever gets computed

the retry job, as it must be written
  rows it can identify as retryable : 0
  rows it would have to retry       : 79800
  rows that would succeed           : 15960
  rows retried pointlessly          : 63840

  wasted work : 8000 per ten thousand of the retries

  and 27930 of them re-ask a person who already said no

the completeness report
  rows complete        : 8100 per ten thousand
  rows incomplete      : 1900 per ten thousand
  of the incomplete, how many are answerable : not computable

  the report is exactly right and the follow-up question is
  the one anybody actually has

recovering the reason after the fact
  from the column          : impossible, one symbol
  from the write path      : the code no longer exists in that form
  from the application log : retained 30 days, these rows are older
  by re-asking             : possible, and re-asks the 27930 who declined

  the information existed at write time, cost one small column
  to keep, and is now only obtainable by contacting people

control - was null the right symbol
  sentinel values in the average : 0
  comparisons against a magic number : 0
  rows where absent is treated as a value : 0
  defects in the schema choice : 0

  the column is correct; what is missing was never in it

null control - the same null with a reason column beside it
  nulls in the column      : 79800, unchanged
  sentinel values          : 0, unchanged
  nulls whose cause is unknown : 0
  retries that are pointless   : 0
  the symbol did not change; a second field records what the
  first one was never able to say

what a null records
  there is no value here : exactly, and unambiguously
  why there is no value  : nothing
  and 'why' is the part every consumer downstream needs,
  because it is what decides whether to retry, to re-ask,
  to exclude, or to leave alone

absence is not one state; it is at least three, and they are
distinguishable only at the moment of writing, by the code
that already knows which one it is

The column is nullable rather than sentinel-valued, which is the right choice:
0 magic numbers enter the averages and 0 rows treat absent as a value. Of the
79800 nulls, 35910 were never asked, 27930 were declined and 15960 failed a
lookup, and nothing distinguishes them, so a retry job must attempt all 79800
to reach 15960 - 8000 per ten thousand wasted - while re-asking 27930 people
who already answered.

Trace event types