The column nobody queried

# Self-authored for the EML case corpus (no external origin). The column has
# been in the table for three years and no report has ever selected it. What is
# in it is computed below.
#
# Adding the column was right. It was cheap, it captures something the domain
# really has, and a column that exists from the start is far easier than a
# backfill later. Nobody was wrong to add it.
#
# A column nobody reads is also a column nobody checks. Whether the writer ever
# populated it, whether it means the same thing in rows written by different
# versions, and whether it is null for whole eras are all questions that only a
# reader asks - and there has not been one.
#
# The rows are counted by era rather than in total.

# [era, rows written, rows with the column populated, what the writer meant by it]
[["v1 2023", 410000, 0, "not written at all"], ["v2 2024 H1", 260000, 260000, "seconds"], ["v2 2024 H2", 300000, 300000, "seconds"], ["v3 2025", 520000, 520000, "milliseconds"], ["v3 2026", 180000, 0, "dropped from the writer"]] => eras

len(eras) => n
0 => total_rows
0 => populated
for e in eras:
    total_rows + e[1] => total_rows
    populated + e[2] => populated

"rows in the table : " + str(total_rows) ^0
"rows with the column populated : " + str(populated) + ", which is " + str(int(populated * 100 / total_rows)) + "%" ^0
"reports that select it : 0" ^0
"" ^0

"era            rows      populated   meaning" ^0
for e in eras:
    "  " + e[0] + "   " + str(e[1]) + "    " + str(e[2]) + "     " + e[3] ^0
"" ^0

# ---- what a first query would return ----

"a report written today, averaging the column" ^0
0 => rows_it_sees
for e in eras:
    rows_it_sees + e[2] => rows_it_sees
"  rows it would average over : " + str(rows_it_sees) ^0
"  rows silently excluded     : " + str(total_rows - rows_it_sees) + ", which is " + str(int((total_rows - rows_it_sees) * 100 / total_rows)) + "%" ^0
"  and nothing in the query says so, because a null is not an error" ^0
"" ^0

# ---- the units changed and nobody noticed ----

[] => units
for e in eras:
    if e[2] > 0:
        if not (e[3] in units):
            units + [e[3]] => units
"distinct meanings among the populated rows : " + str(len(units)) ^0
"" => u
for x in units:
    u + x + ", " => u
"  " + u ^0
if len(units) > 1:
    "  the same column holds two units, and the change was invisible because" ^0
    "  no reader was comparing one era to another" ^0
"" ^0

0 => ms_rows
0 => s_rows
for e in eras:
    if e[3] == "milliseconds":
        ms_rows + e[2] => ms_rows
    if e[3] == "seconds":
        s_rows + e[2] => s_rows
"  rows in seconds      : " + str(s_rows) ^0
"  rows in milliseconds : " + str(ms_rows) ^0
if ms_rows > 0:
    if s_rows > 0:
        "  an average over both is a number in neither unit" ^0
"" ^0

# ---- what a reader would have caught, and when ----

"what a single report, written in each era, would have caught" ^0
"  in v1  : the column is empty, caught on the first run" ^0
"  in v2  : nothing to catch, it was populated and consistent" ^0
"  in v3  : the unit change, on the first comparison to a v2 figure" ^0
"  in 2026: the writer dropping it, when the latest rows came back null" ^0
"  three of the four eras had something a reader would have found, and the" ^0
"  reader is the part that was never added" ^0
"" ^0

# ---- what the column is worth now ----

"the column, three years on" ^0
"  rows usable without a unit decision : " + str(s_rows) + " or " + str(ms_rows) + ", not both" ^0
"  rows usable with one               : " + str(populated) ^0
"  rows that will never be recoverable : " + str(total_rows - populated) ^0
if total_rows - populated > 0:
    "  the backfill the column was added to avoid is now required anyway, for" ^0
    "  the eras where nothing was written" ^0
"" ^0

# ---- the control: a column with a reader from day one ----
#
# Where something selects the column in every release, an empty era or a unit
# change is a broken report rather than a discovery three years later.

"control - a column a dashboard has selected since it was added" ^0
"  eras in which it was unpopulated : 0, because the dashboard broke" ^0
"  unit changes that shipped        : 0, for the same reason" ^0
"  the dashboard is not a better check than a reviewer; it is a check that" ^0
"  runs, which is the property the unread column is missing" ^0
"" ^0

"Adding the column was cheap and correct and the domain really has this" ^0
"field. Nothing has ever read it, so what it contains has been decided by" ^0
"three years of writers and checked by nobody." ^0

eras = [["v1 2023", 410000, 0, "not written at all"], ["v2 2024 H1", 260000, 260000, "seconds"], ["v2 2024 H2", 300000, 300000, "seconds"], ["v3 2025", 520000, 520000, "milliseconds"], ["v3 2026", 180000, 0, "dropped from the writer"]]
n = len(eras)
total_rows = 0
populated = 0
for e in eras:
    total_rows = total_rows + e[1]
    populated = populated + e[2]
print("rows in the table : " + str(total_rows))
print("rows with the column populated : " + str(populated) + ", which is " + str(int(populated * 100 / total_rows)) + "%")
print("reports that select it : 0")
print("")
print("era            rows      populated   meaning")
for e in eras:
    print("  " + e[0] + "   " + str(e[1]) + "    " + str(e[2]) + "     " + e[3])
print("")
print("a report written today, averaging the column")
rows_it_sees = 0
for e in eras:
    rows_it_sees = rows_it_sees + e[2]
print("  rows it would average over : " + str(rows_it_sees))
print("  rows silently excluded     : " + str(total_rows - rows_it_sees) + ", which is " + str(int((total_rows - rows_it_sees) * 100 / total_rows)) + "%")
print("  and nothing in the query says so, because a null is not an error")
print("")
units = []
for e in eras:
    if e[2] > 0:
        if not e[3] in units:
            units = units + [e[3]]
print("distinct meanings among the populated rows : " + str(len(units)))
u = ""
for x in units:
    u = u + x + ", "
print("  " + u)
if len(units) > 1:
    print("  the same column holds two units, and the change was invisible because")
    print("  no reader was comparing one era to another")
print("")
ms_rows = 0
s_rows = 0
for e in eras:
    if e[3] == "milliseconds":
        ms_rows = ms_rows + e[2]
    if e[3] == "seconds":
        s_rows = s_rows + e[2]
print("  rows in seconds      : " + str(s_rows))
print("  rows in milliseconds : " + str(ms_rows))
if ms_rows > 0:
    if s_rows > 0:
        print("  an average over both is a number in neither unit")
print("")
print("what a single report, written in each era, would have caught")
print("  in v1  : the column is empty, caught on the first run")
print("  in v2  : nothing to catch, it was populated and consistent")
print("  in v3  : the unit change, on the first comparison to a v2 figure")
print("  in 2026: the writer dropping it, when the latest rows came back null")
print("  three of the four eras had something a reader would have found, and the")
print("  reader is the part that was never added")
print("")
print("the column, three years on")
print("  rows usable without a unit decision : " + str(s_rows) + " or " + str(ms_rows) + ", not both")
print("  rows usable with one               : " + str(populated))
print("  rows that will never be recoverable : " + str(total_rows - populated))
if total_rows - populated > 0:
    print("  the backfill the column was added to avoid is now required anyway, for")
    print("  the eras where nothing was written")
print("")
print("control - a column a dashboard has selected since it was added")
print("  eras in which it was unpopulated : 0, because the dashboard broke")
print("  unit changes that shipped        : 0, for the same reason")
print("  the dashboard is not a better check than a reviewer; it is a check that")
print("  runs, which is the property the unread column is missing")
print("")
print("Adding the column was cheap and correct and the domain really has this")
print("field. Nothing has ever read it, so what it contains has been decided by")
print("three years of writers and checked by nobody.")

rows in the table : 1670000
rows with the column populated : 1080000, which is 64%
reports that select it : 0

era            rows      populated   meaning
  v1 2023   410000    0     not written at all
  v2 2024 H1   260000    260000     seconds
  v2 2024 H2   300000    300000     seconds
  v3 2025   520000    520000     milliseconds
  v3 2026   180000    0     dropped from the writer

a report written today, averaging the column
  rows it would average over : 1080000
  rows silently excluded     : 590000, which is 35%
  and nothing in the query says so, because a null is not an error

distinct meanings among the populated rows : 2
  seconds, milliseconds, 
  the same column holds two units, and the change was invisible because
  no reader was comparing one era to another

  rows in seconds      : 560000
  rows in milliseconds : 520000
  an average over both is a number in neither unit

what a single report, written in each era, would have caught
  in v1  : the column is empty, caught on the first run
  in v2  : nothing to catch, it was populated and consistent
  in v3  : the unit change, on the first comparison to a v2 figure
  in 2026: the writer dropping it, when the latest rows came back null
  three of the four eras had something a reader would have found, and the
  reader is the part that was never added

the column, three years on
  rows usable without a unit decision : 560000 or 520000, not both
  rows usable with one               : 1080000
  rows that will never be recoverable : 590000
  the backfill the column was added to avoid is now required anyway, for
  the eras where nothing was written

control - a column a dashboard has selected since it was added
  eras in which it was unpopulated : 0, because the dashboard broke
  unit changes that shipped        : 0, for the same reason
  the dashboard is not a better check than a reviewer; it is a check that
  runs, which is the property the unread column is missing

Adding the column was cheap and correct and the domain really has this
field. Nothing has ever read it, so what it contains has been decided by
three years of writers and checked by nobody.

Trace event types