The error rate fell because the errors moved

# Self-authored for the EML case corpus (no external origin). Service A's error
# rate went from 3 percent to 0.2 percent after one change. Where the errors
# went is computed below.
#
# The change was right and was argued for well. Service A was rejecting records
# on rules it had no business owning: it knew the wire format but not the
# domain, so it was guessing at what "valid" meant and getting it wrong in both
# directions. Records that were fine were being bounced because A's copy of the
# rules was six months behind B's, and support was fielding those. Moving
# validation to B, which owns the rules, removed a duplicate implementation and
# put the decision where the knowledge is. This is textbook, and it is correct.
#
# A rejected record and a failed record are both one bad record. What separates
# them is where in the pipeline the badness is discovered, and therefore how
# much work has already been done that must now be undone. A rejects before
# anything is written. B fails after a partial write, which means a rollback, an
# orphaned row somewhere, and a support ticket.
#
# The error rate on A's dashboard measures A. The change moved the errors out of
# A. The dashboard did exactly what it was built to do.

10000 => records
300 => bad_records

# after the change, A still rejects the genuinely malformed - it owns the wire
# format - and passes everything else through
20 => still_rejected_by_a

bad_records - still_rejected_by_a => passed_to_b

"records per batch : " + str(records) ^0
"bad records       : " + str(bad_records) ^0
"" ^0

# ---- before ----

"before the change" ^0
"  A rejects            : " + str(bad_records) ^0
"  A error rate         : " + str(int(bad_records * 1000 / records)) + " per mille" ^0
"  B receives           : " + str(records - bad_records) ^0
"  B errors             : 0" ^0
"  B error rate         : 0 per mille" ^0
"" ^0

# ---- after ----

records - still_rejected_by_a => b_receives

"after the change" ^0
"  A rejects            : " + str(still_rejected_by_a) ^0
"  A error rate         : " + str(int(still_rejected_by_a * 1000 / records)) + " per mille" ^0
"  B receives           : " + str(b_receives) ^0
"  B errors             : " + str(passed_to_b) ^0
"  B error rate         : " + str(int(passed_to_b * 1000 / b_receives)) + " per mille" ^0
"" ^0

"  A improved from " + str(int(bad_records * 1000 / records)) + " to " + str(int(still_rejected_by_a * 1000 / records)) + " per mille, a factor of " + str(int(bad_records / still_rejected_by_a)) ^0
"  the dashboard tracked A" ^0
"" ^0

# ---- the conserved quantity ----

still_rejected_by_a + passed_to_b => total_after

"bad records before : " + str(bad_records) ^0
"bad records after  : " + str(total_after) ^0
"difference         : " + str(total_after - bad_records) ^0
"nothing was fixed, and nothing was broken; the same records are still bad" ^0
"" ^0

# ---- what changed is where they are found ----
#
# Cost per bad record, in units of support-minutes. A rejection at the edge is
# a 400 response. A failure inside B is a partial write: rollback, an orphaned
# row, a ticket, and a human reading a log.

1 => cost_at_a
12 => cost_at_b

bad_records * cost_at_a => cost_before
(still_rejected_by_a * cost_at_a) + (passed_to_b * cost_at_b) => cost_after

"cost of a bad record" ^0
"  caught at A : " + str(cost_at_a) + " unit  - a 400 response, nothing written" ^0
"  caught at B : " + str(cost_at_b) + " units - partial write, rollback, orphan row, ticket" ^0
"" ^0
"  total cost before : " + str(cost_before) + " units" ^0
"  total cost after  : " + str(cost_after) + " units" ^0
"  change            : " + str(int(cost_after * 100 / cost_before)) + " hundredths of what it was" ^0
"" ^0
"  the tracked error rate fell " + str(int(bad_records * 100 / still_rejected_by_a)) + " hundredths" ^0
"  the total cost rose " + str(int(cost_after * 100 / cost_before)) + " hundredths" ^0
"  both numbers describe the same change and neither is wrong" ^0
"" ^0

# ---- why moving a check downstream costs more ----
#
# Work done before a failure is work that must be undone. The further down the
# pipeline a check sits, the more has been done.

"stage at which a bad record is caught, and what is already done" ^0
"  at the edge        nothing written                    " + str(cost_at_a) + " unit" ^0
"  after parse        buffer allocated                   2 units" ^0
"  after enrichment   two upstream calls made            5 units" ^0
"  inside the write   partial row, rollback, orphan     " + str(cost_at_b) + " units" ^0
"  the rules did belong to B; the check did not have to run where they live" ^0
"" ^0

# ---- the control ----
#
# If the change had genuinely improved data quality, the count of bad records
# would have fallen. It is identical, because a validation rule does not make
# records good - it decides where they stop.

"control - a count no relocation can move" ^0
bad_records => before_count
total_after => after_count
"  bad records, before : " + str(before_count) ^0
"  bad records, after  : " + str(after_count) ^0
"  difference          : " + str(after_count - before_count) ^0
"  this is the number that would move if quality had changed" ^0
"  it did not move, so quality did not change" ^0
"" ^0

# ---- the null control ----
#
# The same relocation, to a stage that costs the SAME as the edge. The tracked
# rate on A falls by exactly as much, and total cost does not move at all. So
# the defect is not "the check moved" - it is "the check moved to a more
# expensive place, and only the cheap place was instrumented".

1 => cost_at_cheap_b
(still_rejected_by_a * cost_at_a) + (passed_to_b * cost_at_cheap_b) => cost_cheap

"null control - the same move, to a stage that costs the same as the edge" ^0
"  A error rate after  : " + str(int(still_rejected_by_a * 1000 / records)) + " per mille, identical to the real case" ^0
"  total cost before   : " + str(cost_before) + " units" ^0
"  total cost after    : " + str(cost_cheap) + " units" ^0
"  difference          : " + str(cost_cheap - cost_before) + " units" ^0
"  the same dashboard improvement, and this time it is free" ^0
"  so the dashboard cannot tell these two cases apart, and they are not the same" ^0
"" ^0

# ---- the rule ----

"what a per-service error rate can and cannot see" ^0
"  errors inside the service        yes" ^0
"  errors this service caused elsewhere   no" ^0
"  errors this service stopped catching   no, they leave the numerator" ^0
"  total errors in the pipeline     no, there is no such dashboard" ^0
"  every service can improve its own rate by declining to look" ^0
"" ^0

"Moving validation to the service that owns the rules removed a duplicate" ^0
"implementation and stopped bouncing good records against a six-month-old copy" ^0
"of the rules. It was the right change. " + str(bad_records) + " records were bad before and" ^0
str(total_after) + " are bad after. A's error rate fell from " + str(int(bad_records * 1000 / records)) + " to " + str(int(still_rejected_by_a * 1000 / records)) + " per mille and the cost of" ^0
"handling those records went from " + str(cost_before) + " to " + str(cost_after) + " units." ^0

records = 10000
bad_records = 300
still_rejected_by_a = 20
passed_to_b = bad_records - still_rejected_by_a
print("records per batch : " + str(records))
print("bad records       : " + str(bad_records))
print("")
print("before the change")
print("  A rejects            : " + str(bad_records))
print("  A error rate         : " + str(int(bad_records * 1000 / records)) + " per mille")
print("  B receives           : " + str(records - bad_records))
print("  B errors             : 0")
print("  B error rate         : 0 per mille")
print("")
b_receives = records - still_rejected_by_a
print("after the change")
print("  A rejects            : " + str(still_rejected_by_a))
print("  A error rate         : " + str(int(still_rejected_by_a * 1000 / records)) + " per mille")
print("  B receives           : " + str(b_receives))
print("  B errors             : " + str(passed_to_b))
print("  B error rate         : " + str(int(passed_to_b * 1000 / b_receives)) + " per mille")
print("")
print("  A improved from " + str(int(bad_records * 1000 / records)) + " to " + str(int(still_rejected_by_a * 1000 / records)) + " per mille, a factor of " + str(int(bad_records / still_rejected_by_a)))
print("  the dashboard tracked A")
print("")
total_after = still_rejected_by_a + passed_to_b
print("bad records before : " + str(bad_records))
print("bad records after  : " + str(total_after))
print("difference         : " + str(total_after - bad_records))
print("nothing was fixed, and nothing was broken; the same records are still bad")
print("")
cost_at_a = 1
cost_at_b = 12
cost_before = bad_records * cost_at_a
cost_after = still_rejected_by_a * cost_at_a + passed_to_b * cost_at_b
print("cost of a bad record")
print("  caught at A : " + str(cost_at_a) + " unit  - a 400 response, nothing written")
print("  caught at B : " + str(cost_at_b) + " units - partial write, rollback, orphan row, ticket")
print("")
print("  total cost before : " + str(cost_before) + " units")
print("  total cost after  : " + str(cost_after) + " units")
print("  change            : " + str(int(cost_after * 100 / cost_before)) + " hundredths of what it was")
print("")
print("  the tracked error rate fell " + str(int(bad_records * 100 / still_rejected_by_a)) + " hundredths")
print("  the total cost rose " + str(int(cost_after * 100 / cost_before)) + " hundredths")
print("  both numbers describe the same change and neither is wrong")
print("")
print("stage at which a bad record is caught, and what is already done")
print("  at the edge        nothing written                    " + str(cost_at_a) + " unit")
print("  after parse        buffer allocated                   2 units")
print("  after enrichment   two upstream calls made            5 units")
print("  inside the write   partial row, rollback, orphan     " + str(cost_at_b) + " units")
print("  the rules did belong to B; the check did not have to run where they live")
print("")
print("control - a count no relocation can move")
before_count = bad_records
after_count = total_after
print("  bad records, before : " + str(before_count))
print("  bad records, after  : " + str(after_count))
print("  difference          : " + str(after_count - before_count))
print("  this is the number that would move if quality had changed")
print("  it did not move, so quality did not change")
print("")
cost_at_cheap_b = 1
cost_cheap = still_rejected_by_a * cost_at_a + passed_to_b * cost_at_cheap_b
print("null control - the same move, to a stage that costs the same as the edge")
print("  A error rate after  : " + str(int(still_rejected_by_a * 1000 / records)) + " per mille, identical to the real case")
print("  total cost before   : " + str(cost_before) + " units")
print("  total cost after    : " + str(cost_cheap) + " units")
print("  difference          : " + str(cost_cheap - cost_before) + " units")
print("  the same dashboard improvement, and this time it is free")
print("  so the dashboard cannot tell these two cases apart, and they are not the same")
print("")
print("what a per-service error rate can and cannot see")
print("  errors inside the service        yes")
print("  errors this service caused elsewhere   no")
print("  errors this service stopped catching   no, they leave the numerator")
print("  total errors in the pipeline     no, there is no such dashboard")
print("  every service can improve its own rate by declining to look")
print("")
print("Moving validation to the service that owns the rules removed a duplicate")
print("implementation and stopped bouncing good records against a six-month-old copy")
print("of the rules. It was the right change. " + str(bad_records) + " records were bad before and")
print(str(total_after) + " are bad after. A's error rate fell from " + str(int(bad_records * 1000 / records)) + " to " + str(int(still_rejected_by_a * 1000 / records)) + " per mille and the cost of")
print("handling those records went from " + str(cost_before) + " to " + str(cost_after) + " units.")

records per batch : 10000
bad records       : 300

before the change
  A rejects            : 300
  A error rate         : 30 per mille
  B receives           : 9700
  B errors             : 0
  B error rate         : 0 per mille

after the change
  A rejects            : 20
  A error rate         : 2 per mille
  B receives           : 9980
  B errors             : 280
  B error rate         : 28 per mille

  A improved from 30 to 2 per mille, a factor of 15
  the dashboard tracked A

bad records before : 300
bad records after  : 300
difference         : 0
nothing was fixed, and nothing was broken; the same records are still bad

cost of a bad record
  caught at A : 1 unit  - a 400 response, nothing written
  caught at B : 12 units - partial write, rollback, orphan row, ticket

  total cost before : 300 units
  total cost after  : 3380 units
  change            : 1126 hundredths of what it was

  the tracked error rate fell 1500 hundredths
  the total cost rose 1126 hundredths
  both numbers describe the same change and neither is wrong

stage at which a bad record is caught, and what is already done
  at the edge        nothing written                    1 unit
  after parse        buffer allocated                   2 units
  after enrichment   two upstream calls made            5 units
  inside the write   partial row, rollback, orphan     12 units
  the rules did belong to B; the check did not have to run where they live

control - a count no relocation can move
  bad records, before : 300
  bad records, after  : 300
  difference          : 0
  this is the number that would move if quality had changed
  it did not move, so quality did not change

null control - the same move, to a stage that costs the same as the edge
  A error rate after  : 2 per mille, identical to the real case
  total cost before   : 300 units
  total cost after    : 300 units
  difference          : 0 units
  the same dashboard improvement, and this time it is free
  so the dashboard cannot tell these two cases apart, and they are not the same

what a per-service error rate can and cannot see
  errors inside the service        yes
  errors this service caused elsewhere   no
  errors this service stopped catching   no, they leave the numerator
  total errors in the pipeline     no, there is no such dashboard
  every service can improve its own rate by declining to look

Moving validation to the service that owns the rules removed a duplicate
implementation and stopped bouncing good records against a six-month-old copy
of the rules. It was the right change. 300 records were bad before and
300 are bad after. A's error rate fell from 30 to 2 per mille and the cost of
handling those records went from 300 to 3380 units.

Trace event types