# Example 578 — The rule was enforced at write time and the data predated it `the_rule_was_enforced_at_write_time_and_the_data_predated_it.eml` - A validation rule was added to the write path eighteen months ago. Every row written since satisfies it. What fraction of the table satisfies it is computed below. ## EML ```eml # Self-authored for the EML case corpus (no external origin). A validation rule # was added to the write path eighteen months ago. Every row written since # satisfies it. What fraction of the table satisfies it is computed below. # # Enforcing at the write path was the right decision and the alternative was # considered properly. Adding the constraint to the database would have # required a full table rewrite with an exclusive lock, on a table serving # production traffic, and the backfill needed to make the existing rows pass # was estimated at four days of engineer time that nobody had. Enforcing at the # write path costs nothing, takes effect immediately, and guarantees that the # problem stops growing. All of that is true and all of it happened. # # "The problem stops growing" and "the problem goes away" are different # statements. A write-path rule makes the count of violations constant. It does # not make it smaller, because nothing in a write path ever touches a row that # is not being written. # # What falls is the PROPORTION, and it falls only as fast as the table grows. 2400000 => rows_at_the_time 1900000 => rows_predating_the_rule 600 => violation_rate_of_old_rows_per_ten_thousand 3000 => new_rows_per_day 365 => days_per_year int(rows_predating_the_rule * violation_rate_of_old_rows_per_ten_thousand / 10000) => violating_rows "rows in the table : " + str(rows_at_the_time) ^0 "rows written before the rule : " + str(rows_predating_the_rule) ^0 "of those, violating : " + str(violation_rate_of_old_rows_per_ten_thousand) + " per ten thousand = " + str(violating_rows) + " rows" ^0 "rows written since the rule : " + str(rows_at_the_time - rows_predating_the_rule) + ", all compliant" ^0 "" ^0 int(violating_rows * 10000 / rows_at_the_time) => current_rate "violating rows in the whole table : " + str(violating_rows) ^0 "as a share of the table : " + str(current_rate) + " per ten thousand" ^0 "" ^0 # ---- what the rule changed ---- "before the rule violations grew with every non-compliant write" ^0 "after the rule violations are constant at " + str(violating_rows) ^0 " rows removed from the violating set by the rule : 0" ^0 " the rule cannot reach them; it only sees rows being written" ^0 "" ^0 # ---- dilution, which is the only thing that happens now ---- "years rows in table violating share per ten thousand" ^0 for y in [0:8]: rows_at_the_time + (new_rows_per_day * days_per_year * y) => rows_then int(violating_rows * 10000 / rows_then) => share if y % 2 == 0: " " + str(y) + " " + str(rows_then) + " " + str(violating_rows) + " " + str(share) ^0 "" ^0 " the violating column never moves" ^0 " the share falls because the denominator grows, at " + str(new_rows_per_day) + " rows a day" ^0 "" ^0 # ---- how long until the invariant is nearly true ---- int(violating_rows * 10000 / 100) => rows_needed_for_1pct int(violating_rows * 10000 / 10) => rows_needed_for_tenth "to reach 100 per ten thousand (1 percent)" ^0 " rows needed : " + str(rows_needed_for_1pct) ^0 " rows to add : " + str(rows_needed_for_1pct - rows_at_the_time) ^0 " days : " + str(int((rows_needed_for_1pct - rows_at_the_time) / new_rows_per_day)) ^0 " years : " + str(int((rows_needed_for_1pct - rows_at_the_time) / (new_rows_per_day * days_per_year))) ^0 "" ^0 "to reach 10 per ten thousand (0.1 percent)" ^0 " rows needed : " + str(rows_needed_for_tenth) ^0 " years : " + str(int((rows_needed_for_tenth - rows_at_the_time) / (new_rows_per_day * days_per_year))) ^0 "" ^0 " and at no point does it reach zero" ^0 "" ^0 # ---- what reads the table ---- # # Code written after the rule may assume the invariant, because the invariant # is documented and true of everything the author has ever seen written. "a function written today that assumes the field is well-formed" ^0 " rows it handles correctly : " + str(rows_at_the_time - violating_rows) ^0 " rows it fails on : " + str(violating_rows) ^0 " failure rate : " + str(current_rate) + " per ten thousand" ^0 " and the author is right that every row THEY have written is fine" ^0 "" ^0 " a test fixture built by inserting rows will never contain one" ^0 " because inserting rows goes through the write path" ^0 "" ^0 # ---- the control ---- # # The rule works. Every write since it landed is compliant, with no exceptions, # and that is exactly what it promised. Checking the rule finds nothing wrong. "control - does the rule do what it says" ^0 " rows written since it landed : " + str(rows_at_the_time - rows_predating_the_rule) ^0 " of those, violating : 0" ^0 " rows it has rejected : every non-compliant write attempted" ^0 " the rule is correct, complete and has never been bypassed" ^0 "" ^0 " it promised that the problem stops growing, and the problem stopped growing" ^0 " the promise a reader hears is that the field is well-formed" ^0 "" ^0 # ---- the null control ---- # # The same rule on a table that was EMPTY when it landed. Every row in the # table went through the write path, the invariant holds everywhere, and code # may assume it safely. The rule is identical; the history is not. 0 => empty_at_the_time int(empty_at_the_time * violation_rate_of_old_rows_per_ten_thousand / 10000) => empty_violations "null control - the same rule added to an empty table" ^0 " rows predating the rule : " + str(empty_at_the_time) ^0 " violating rows : " + str(empty_violations) ^0 " share of the table : 0 per ten thousand, permanently" ^0 " same rule, same write path, same documentation" ^0 " the invariant holds because there was no history for it to miss" ^0 "" ^0 # ---- the rule ---- "what a write-path rule guarantees, and what it does not" ^0 " new rows comply guaranteed, immediately" ^0 " the violation count stops growing guaranteed" ^0 " the violation count falls no, nothing removes a row" ^0 " the invariant holds for readers no, and this is what gets assumed" ^0 " the gap closes only by dilution, over years" ^0 "" ^0 "a constraint added to a table is a statement about the table" ^0 "a check added to a write path is a statement about future writes" ^0 "they are documented in the same sentence and they are not the same claim" ^0 "" ^0 "Enforcing at the write path avoided an exclusive lock on a production table" ^0 "and a four-day backfill nobody had time for, it took effect immediately, and" ^0 "every one of the " + str(rows_at_the_time - rows_predating_the_rule) + " rows written since is compliant. The " + str(violating_rows) + " rows that" ^0 "predate it are still there, no write path will ever touch them, and their" ^0 "share falls from " + str(current_rate) + " to 100 per ten thousand only after " + str(int((rows_needed_for_1pct - rows_at_the_time) / (new_rows_per_day * days_per_year))) + " years of growth." ^0 ``` ## Python (deterministic transpilation) ```python rows_at_the_time = 2400000 rows_predating_the_rule = 1900000 violation_rate_of_old_rows_per_ten_thousand = 600 new_rows_per_day = 3000 days_per_year = 365 violating_rows = int(rows_predating_the_rule * violation_rate_of_old_rows_per_ten_thousand / 10000) print("rows in the table : " + str(rows_at_the_time)) print("rows written before the rule : " + str(rows_predating_the_rule)) print("of those, violating : " + str(violation_rate_of_old_rows_per_ten_thousand) + " per ten thousand = " + str(violating_rows) + " rows") print("rows written since the rule : " + str(rows_at_the_time - rows_predating_the_rule) + ", all compliant") print("") current_rate = int(violating_rows * 10000 / rows_at_the_time) print("violating rows in the whole table : " + str(violating_rows)) print("as a share of the table : " + str(current_rate) + " per ten thousand") print("") print("before the rule violations grew with every non-compliant write") print("after the rule violations are constant at " + str(violating_rows)) print(" rows removed from the violating set by the rule : 0") print(" the rule cannot reach them; it only sees rows being written") print("") print("years rows in table violating share per ten thousand") for y in range(0, 9): rows_then = rows_at_the_time + new_rows_per_day * days_per_year * y share = int(violating_rows * 10000 / rows_then) if y % 2 == 0: print(" " + str(y) + " " + str(rows_then) + " " + str(violating_rows) + " " + str(share)) print("") print(" the violating column never moves") print(" the share falls because the denominator grows, at " + str(new_rows_per_day) + " rows a day") print("") rows_needed_for_1pct = int(violating_rows * 10000 / 100) rows_needed_for_tenth = int(violating_rows * 10000 / 10) print("to reach 100 per ten thousand (1 percent)") print(" rows needed : " + str(rows_needed_for_1pct)) print(" rows to add : " + str(rows_needed_for_1pct - rows_at_the_time)) print(" days : " + str(int((rows_needed_for_1pct - rows_at_the_time) / new_rows_per_day))) print(" years : " + str(int((rows_needed_for_1pct - rows_at_the_time) / (new_rows_per_day * days_per_year)))) print("") print("to reach 10 per ten thousand (0.1 percent)") print(" rows needed : " + str(rows_needed_for_tenth)) print(" years : " + str(int((rows_needed_for_tenth - rows_at_the_time) / (new_rows_per_day * days_per_year)))) print("") print(" and at no point does it reach zero") print("") print("a function written today that assumes the field is well-formed") print(" rows it handles correctly : " + str(rows_at_the_time - violating_rows)) print(" rows it fails on : " + str(violating_rows)) print(" failure rate : " + str(current_rate) + " per ten thousand") print(" and the author is right that every row THEY have written is fine") print("") print(" a test fixture built by inserting rows will never contain one") print(" because inserting rows goes through the write path") print("") print("control - does the rule do what it says") print(" rows written since it landed : " + str(rows_at_the_time - rows_predating_the_rule)) print(" of those, violating : 0") print(" rows it has rejected : every non-compliant write attempted") print(" the rule is correct, complete and has never been bypassed") print("") print(" it promised that the problem stops growing, and the problem stopped growing") print(" the promise a reader hears is that the field is well-formed") print("") empty_at_the_time = 0 empty_violations = int(empty_at_the_time * violation_rate_of_old_rows_per_ten_thousand / 10000) print("null control - the same rule added to an empty table") print(" rows predating the rule : " + str(empty_at_the_time)) print(" violating rows : " + str(empty_violations)) print(" share of the table : 0 per ten thousand, permanently") print(" same rule, same write path, same documentation") print(" the invariant holds because there was no history for it to miss") print("") print("what a write-path rule guarantees, and what it does not") print(" new rows comply guaranteed, immediately") print(" the violation count stops growing guaranteed") print(" the violation count falls no, nothing removes a row") print(" the invariant holds for readers no, and this is what gets assumed") print(" the gap closes only by dilution, over years") print("") print("a constraint added to a table is a statement about the table") print("a check added to a write path is a statement about future writes") print("they are documented in the same sentence and they are not the same claim") print("") print("Enforcing at the write path avoided an exclusive lock on a production table") print("and a four-day backfill nobody had time for, it took effect immediately, and") print("every one of the " + str(rows_at_the_time - rows_predating_the_rule) + " rows written since is compliant. The " + str(violating_rows) + " rows that") print("predate it are still there, no write path will ever touch them, and their") print("share falls from " + str(current_rate) + " to 100 per ten thousand only after " + str(int((rows_needed_for_1pct - rows_at_the_time) / (new_rows_per_day * days_per_year))) + " years of growth.") ``` ## stdout (executed) ```text rows in the table : 2400000 rows written before the rule : 1900000 of those, violating : 600 per ten thousand = 114000 rows rows written since the rule : 500000, all compliant violating rows in the whole table : 114000 as a share of the table : 475 per ten thousand before the rule violations grew with every non-compliant write after the rule violations are constant at 114000 rows removed from the violating set by the rule : 0 the rule cannot reach them; it only sees rows being written years rows in table violating share per ten thousand 0 2400000 114000 475 2 4590000 114000 248 4 6780000 114000 168 6 8970000 114000 127 8 11160000 114000 102 the violating column never moves the share falls because the denominator grows, at 3000 rows a day to reach 100 per ten thousand (1 percent) rows needed : 11400000 rows to add : 9000000 days : 3000 years : 8 to reach 10 per ten thousand (0.1 percent) rows needed : 114000000 years : 101 and at no point does it reach zero a function written today that assumes the field is well-formed rows it handles correctly : 2286000 rows it fails on : 114000 failure rate : 475 per ten thousand and the author is right that every row THEY have written is fine a test fixture built by inserting rows will never contain one because inserting rows goes through the write path control - does the rule do what it says rows written since it landed : 500000 of those, violating : 0 rows it has rejected : every non-compliant write attempted the rule is correct, complete and has never been bypassed it promised that the problem stops growing, and the problem stopped growing the promise a reader hears is that the field is well-formed null control - the same rule added to an empty table rows predating the rule : 0 violating rows : 0 share of the table : 0 per ten thousand, permanently same rule, same write path, same documentation the invariant holds because there was no history for it to miss what a write-path rule guarantees, and what it does not new rows comply guaranteed, immediately the violation count stops growing guaranteed the violation count falls no, nothing removes a row the invariant holds for readers no, and this is what gets assumed the gap closes only by dilution, over years a constraint added to a table is a statement about the table a check added to a write path is a statement about future writes they are documented in the same sentence and they are not the same claim Enforcing at the write path avoided an exclusive lock on a production table and a four-day backfill nobody had time for, it took effect immediately, and every one of the 500000 rows written since is compliant. The 114000 rows that predate it are still there, no write path will ever touch them, and their share falls from 475 to 100 per ten thousand only after 8 years of growth. ``` ## Round-trip `ok: true` — round-trip fixpoint reached (python1 == python2) ## Trace event types eml:run:start · eml:assign · eml:output · eml:run:done