The batch committed and the notification did not

# Self-authored for the EML case corpus (no external origin). A batch writes
# rows inside a transaction and then tells the downstream service. The
# transaction is atomic. What "and then" costs is computed below.
#
# The transaction is correct and it is doing real work. Either every row of a
# batch lands or none does; a crash mid-write leaves no half-applied batch to
# reconcile by hand, and the constraint that would have been violated in the
# middle is never observable. That guarantee has held on every batch.
#
# The notification is a network call to another service. It cannot join the
# transaction, so it happens after the commit returns — which is the only place
# it CAN happen, because before the commit there is nothing true to announce.
#
# So there is a moment when the rows exist and nobody has been told, and the
# transaction's guarantee does not reach into it because it has already ended.

4200 => batches_per_day
250 => rows_per_batch
2400 => batch_ms
180 => gap_ms

batches_per_day * rows_per_batch => rows_per_day

"batches per day        : " + str(batches_per_day) ^0
"rows per batch         : " + str(rows_per_batch) ^0
"rows per day           : " + str(rows_per_day) ^0
"commit to notify, gap  : " + str(gap_ms) + " ms" ^0
"" ^0

# ---- what the transaction guarantees, checked ----

"the transaction, against what it promises" ^0
"  batches partially applied  : 0" ^0
"  constraint violations seen : 0" ^0
"  rows written twice         : 0" ^0
"  manual reconciliations     : 0" ^0
"  defects in the transaction : 0" ^0
"" ^0
"  atomicity holds on all " + str(batches_per_day) + " batches a day" ^0
"" ^0

# ---- the gap ----

int(gap_ms * 10000 / batch_ms) => gap_per_myriad
int(batches_per_day * gap_per_myriad / 10000) => lost_per_day
lost_per_day * rows_per_batch => rows_unannounced_per_day

"the window between commit and notify" ^0
"  batch duration        : " + str(batch_ms) + " ms" ^0
"  gap                   : " + str(gap_ms) + " ms" ^0
"  share of the batch    : " + str(gap_per_myriad) + " per ten thousand" ^0
"" ^0
"  a crash in that window leaves the rows and loses the message" ^0
"  batches affected per day : " + str(lost_per_day) ^0
"  rows nobody is told about: " + str(rows_unannounced_per_day) + " per day" ^0
"" ^0

# ---- what each side believes ----

rows_per_day - rows_unannounced_per_day => rows_downstream_knows

"at the end of one day" ^0
"  rows in the database      : " + str(rows_per_day) ^0
"  rows downstream knows of  : " + str(rows_downstream_knows) ^0
"  divergence                : " + str(rows_unannounced_per_day) ^0
"" ^0
"  the database is internally consistent" ^0
"  the downstream service is internally consistent" ^0
"  neither is wrong about anything it can check alone" ^0
"" ^0

# ---- the divergence over a week ----

"day   rows written   downstream knows   cumulative gap" ^0
0 => running
for d in [1:5]:
    running + rows_unannounced_per_day => running
    "  " + str(d) + "     " + str(rows_per_day) + "        " + str(rows_downstream_knows) + "            " + str(running) ^0
"" ^0
"  nothing in that table is an error state on either side" ^0
"" ^0

# ---- what a retry of the notification does ----
#
# The obvious repair is to retry the notify. It cannot run, because the process
# that would have retried it is the one that died; the retry lives in the same
# memory as the message.

"retrying the notification" ^0
"  where the pending message lives : in the crashed process" ^0
"  messages recoverable after a crash : 0" ^0
"  a retry helps when the CALL fails : yes" ^0
"  a retry helps when the CALLER dies : no" ^0
"" ^0
"  the two failure modes look identical from the callee" ^0
"" ^0

# ---- the control ----
#
# The transaction, on the thing it was placed to prevent. Without it a crash
# mid-batch leaves part of a batch behind, and that is the failure this design
# was built to remove.

int(batches_per_day * gap_per_myriad / 10000) => partial_without_txn

"control - is the transaction earning its place" ^0
"  partially applied batches without it : " + str(partial_without_txn) + " a day" ^0
"  partially applied batches with it    : 0" ^0
"  rows needing manual repair with it   : 0" ^0
"  defects in the transaction           : 0" ^0
"" ^0
"  the transaction removes the failure inside the batch" ^0
"  and cannot reach the one immediately after it" ^0
"" ^0

# ---- the null control ----
#
# The same batch, same crash rate, where the message is written to a table in
# the SAME transaction and a separate reader ships it. The transaction did not
# get stronger; the notification moved inside its edge.

0 => nc_rows_unannounced

"null control - the message written inside the transaction" ^0
"  batches per day            : " + str(batches_per_day) ^0
"  crash window               : " + str(gap_ms) + " ms, unchanged" ^0
"  rows nobody is told about  : " + str(nc_rows_unannounced) ^0
"  the crash still happens at the same rate" ^0
"  what changed is which side of the commit the message is on" ^0
"" ^0

# ---- the rule ----

"what a transaction's atomicity covers" ^0
"  every write inside it          : completely" ^0
"  the first statement after it   : not at all" ^0
"  and 'after the commit' is the only place a call to another" ^0
"  system can go, so the gap is not an oversight - it is where" ^0
"  the design put the one step that cannot join" ^0
"" ^0
"the repair is not a wider transaction, which cannot span two" ^0
"systems; it is to make the message a row, so that announcing" ^0
"and writing are the same commit" ^0
"" ^0

"The transaction is atomic on all " + str(batches_per_day) + " batches a day: 0 partial applications, 0" ^0
"constraint violations, 0 manual reconciliations. The notification sits " + str(gap_ms) + " ms" ^0
"after it - " + str(gap_per_myriad) + " per ten thousand of the batch - so " + str(lost_per_day) + " batches a day commit" ^0
str(rows_unannounced_per_day) + " rows that nobody downstream is told about, and neither system can" ^0
"detect it, because each one is entirely consistent with itself." ^0

batches_per_day = 4200
rows_per_batch = 250
batch_ms = 2400
gap_ms = 180
rows_per_day = batches_per_day * rows_per_batch
print("batches per day        : " + str(batches_per_day))
print("rows per batch         : " + str(rows_per_batch))
print("rows per day           : " + str(rows_per_day))
print("commit to notify, gap  : " + str(gap_ms) + " ms")
print("")
print("the transaction, against what it promises")
print("  batches partially applied  : 0")
print("  constraint violations seen : 0")
print("  rows written twice         : 0")
print("  manual reconciliations     : 0")
print("  defects in the transaction : 0")
print("")
print("  atomicity holds on all " + str(batches_per_day) + " batches a day")
print("")
gap_per_myriad = int(gap_ms * 10000 / batch_ms)
lost_per_day = int(batches_per_day * gap_per_myriad / 10000)
rows_unannounced_per_day = lost_per_day * rows_per_batch
print("the window between commit and notify")
print("  batch duration        : " + str(batch_ms) + " ms")
print("  gap                   : " + str(gap_ms) + " ms")
print("  share of the batch    : " + str(gap_per_myriad) + " per ten thousand")
print("")
print("  a crash in that window leaves the rows and loses the message")
print("  batches affected per day : " + str(lost_per_day))
print("  rows nobody is told about: " + str(rows_unannounced_per_day) + " per day")
print("")
rows_downstream_knows = rows_per_day - rows_unannounced_per_day
print("at the end of one day")
print("  rows in the database      : " + str(rows_per_day))
print("  rows downstream knows of  : " + str(rows_downstream_knows))
print("  divergence                : " + str(rows_unannounced_per_day))
print("")
print("  the database is internally consistent")
print("  the downstream service is internally consistent")
print("  neither is wrong about anything it can check alone")
print("")
print("day   rows written   downstream knows   cumulative gap")
running = 0
for d in range(1, 6):
    running = running + rows_unannounced_per_day
    print("  " + str(d) + "     " + str(rows_per_day) + "        " + str(rows_downstream_knows) + "            " + str(running))
print("")
print("  nothing in that table is an error state on either side")
print("")
print("retrying the notification")
print("  where the pending message lives : in the crashed process")
print("  messages recoverable after a crash : 0")
print("  a retry helps when the CALL fails : yes")
print("  a retry helps when the CALLER dies : no")
print("")
print("  the two failure modes look identical from the callee")
print("")
partial_without_txn = int(batches_per_day * gap_per_myriad / 10000)
print("control - is the transaction earning its place")
print("  partially applied batches without it : " + str(partial_without_txn) + " a day")
print("  partially applied batches with it    : 0")
print("  rows needing manual repair with it   : 0")
print("  defects in the transaction           : 0")
print("")
print("  the transaction removes the failure inside the batch")
print("  and cannot reach the one immediately after it")
print("")
nc_rows_unannounced = 0
print("null control - the message written inside the transaction")
print("  batches per day            : " + str(batches_per_day))
print("  crash window               : " + str(gap_ms) + " ms, unchanged")
print("  rows nobody is told about  : " + str(nc_rows_unannounced))
print("  the crash still happens at the same rate")
print("  what changed is which side of the commit the message is on")
print("")
print("what a transaction's atomicity covers")
print("  every write inside it          : completely")
print("  the first statement after it   : not at all")
print("  and 'after the commit' is the only place a call to another")
print("  system can go, so the gap is not an oversight - it is where")
print("  the design put the one step that cannot join")
print("")
print("the repair is not a wider transaction, which cannot span two")
print("systems; it is to make the message a row, so that announcing")
print("and writing are the same commit")
print("")
print("The transaction is atomic on all " + str(batches_per_day) + " batches a day: 0 partial applications, 0")
print("constraint violations, 0 manual reconciliations. The notification sits " + str(gap_ms) + " ms")
print("after it - " + str(gap_per_myriad) + " per ten thousand of the batch - so " + str(lost_per_day) + " batches a day commit")
print(str(rows_unannounced_per_day) + " rows that nobody downstream is told about, and neither system can")
print("detect it, because each one is entirely consistent with itself.")

batches per day        : 4200
rows per batch         : 250
rows per day           : 1050000
commit to notify, gap  : 180 ms

the transaction, against what it promises
  batches partially applied  : 0
  constraint violations seen : 0
  rows written twice         : 0
  manual reconciliations     : 0
  defects in the transaction : 0

  atomicity holds on all 4200 batches a day

the window between commit and notify
  batch duration        : 2400 ms
  gap                   : 180 ms
  share of the batch    : 750 per ten thousand

  a crash in that window leaves the rows and loses the message
  batches affected per day : 315
  rows nobody is told about: 78750 per day

at the end of one day
  rows in the database      : 1050000
  rows downstream knows of  : 971250
  divergence                : 78750

  the database is internally consistent
  the downstream service is internally consistent
  neither is wrong about anything it can check alone

day   rows written   downstream knows   cumulative gap
  1     1050000        971250            78750
  2     1050000        971250            157500
  3     1050000        971250            236250
  4     1050000        971250            315000
  5     1050000        971250            393750

  nothing in that table is an error state on either side

retrying the notification
  where the pending message lives : in the crashed process
  messages recoverable after a crash : 0
  a retry helps when the CALL fails : yes
  a retry helps when the CALLER dies : no

  the two failure modes look identical from the callee

control - is the transaction earning its place
  partially applied batches without it : 315 a day
  partially applied batches with it    : 0
  rows needing manual repair with it   : 0
  defects in the transaction           : 0

  the transaction removes the failure inside the batch
  and cannot reach the one immediately after it

null control - the message written inside the transaction
  batches per day            : 4200
  crash window               : 180 ms, unchanged
  rows nobody is told about  : 0
  the crash still happens at the same rate
  what changed is which side of the commit the message is on

what a transaction's atomicity covers
  every write inside it          : completely
  the first statement after it   : not at all
  and 'after the commit' is the only place a call to another
  system can go, so the gap is not an oversight - it is where
  the design put the one step that cannot join

the repair is not a wider transaction, which cannot span two
systems; it is to make the message a row, so that announcing
and writing are the same commit

The transaction is atomic on all 4200 batches a day: 0 partial applications, 0
constraint violations, 0 manual reconciliations. The notification sits 180 ms
after it - 750 per ten thousand of the batch - so 315 batches a day commit
78750 rows that nobody downstream is told about, and neither system can
detect it, because each one is entirely consistent with itself.

Trace event types