# P129 Batch retries A batch with some writes failing at random, retried whole, retried for the failed records only, or rolled back and retried: writes, records written twice, and attempts that need no failure at all. EML-P project `projects/batch-retries` in the EML language repo: 2 module(s), entry `main.eml`, terminal UI. There, `eml project run projects/batch-retries` runs it and `eml project verify projects/batch-retries` replays every session under CPython (two hash seeds) and in the interpreter; the site build replays every session in the interpreter again and publishes a session only if its screen matches. Built on verified corpus cases: partial-batch-failure-retries-everything (https://efficientnewlanguage.org/cases/316-partial-batch-failure-retries-everything/), partial-write-rollback (https://efficientnewlanguage.org/cases/226-partial-write-rollback/). ## Sessions ### bad-input - interpreter: byte-equal to the golden Input: ```text 0 x 1 1 0 1 1000 1 x 3 3 1 3 13 3 5 3 5 61 3 5 0 1 9 2 4 ``` Screen: ```text == Batch retries == A batch of records written one by one, some writes failing at random, and three ways to retry. 1) one batch 2) many batches 3) settings 4) quit choice> 0 Pick a number from 1 to 4. 1) one batch 2) many batches 3) settings 4) quit choice> x Pick a number from 1 to 4. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> Cancelled. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 0 Type a whole number from 1 to 999. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 1000 Type a whole number from 1 to 999. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> x Type a whole number from 1 to 999. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> Cancelled. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 1 Type a whole number from 2 to 12. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 13 Type a whole number from 2 to 12. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 5 percent of writes failing (0 to 60)> Cancelled. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 5 percent of writes failing (0 to 60)> 61 Type a whole number from 0 to 60. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 5 percent of writes failing (0 to 60)> 0 Batches of 5 records, 0% of writes failing. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 9 A batch of 5 records, 0% of writes failing, draw 9: retry everything: attempt 1: wrote 1 2 3 4 5; failed none 5 writes for 5 records, none written twice retry failed only: attempt 1: wrote 1 2 3 4 5; failed none 5 writes for 5 records, none written twice retry all or none: attempt 1: wrote 1 2 3 4 5; failed none 5 writes for 5 records, none written twice 1) one batch 2) many batches 3) settings 4) quit choice> 2 100 batches of 5 records, 0% of writes failing, at most 50 attempts each: retry attempts writes written again not done everything 100 500 0 0 failed only 100 500 0 0 all or none 100 500 0 0 Only retrying everything writes a record twice. It and all or none both need one attempt with no failure at all, rarer with every record added; retrying only the failed ones needs no such attempt. 1) one batch 2) many batches 3) settings 4) quit choice> 4 Bye. ``` ### basic - interpreter: byte-equal to the golden Input: ```text 1 7 2 3 12 30 2 1 3 3 8 10 1 5 4 ``` Screen: ```text == Batch retries == A batch of records written one by one, some writes failing at random, and three ways to retry. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 7 A batch of 8 records, 20% of writes failing, draw 7: retry everything: attempt 1: wrote 1 2 3 4 5 7 8; failed 6 attempt 2: wrote 1 2 3 5 6 8; failed 4 7 attempt 3: wrote 1 2 4 5 6 7 8; failed 3 attempt 4: wrote 1 2 3 4 5 6 7; failed 8 attempt 5: wrote 1 2 3 4 5 6 7 8; failed none 35 writes for 8 records, 27 of them a record written again retry failed only: attempt 1: wrote 1 2 3 4 5 7 8; failed 6 attempt 2: wrote 6; failed none 8 writes for 8 records, none written twice retry all or none: attempt 1: wrote 1 2 3 4 5 7 8; failed 6; rolled back 1 2 3 4 5 7 8 attempt 2: wrote 1 2 3 5 6 8; failed 4 7; rolled back 1 2 3 5 6 8 attempt 3: wrote 1 2 4 5 6 7 8; failed 3; rolled back 1 2 4 5 6 7 8 attempt 4: wrote 1 2 3 4 5 6 7; failed 8; rolled back 1 2 3 4 5 6 7 attempt 5: wrote 1 2 3 4 5 6 7 8; failed none 35 writes for 8 records, none written twice 1) one batch 2) many batches 3) settings 4) quit choice> 2 100 batches of 8 records, 20% of writes failing, at most 50 attempts each: retry attempts writes written again not done everything 503 3253 2453 0 failed only 205 800 0 0 all or none 503 3253 0 0 Only retrying everything writes a record twice. It and all or none both need one attempt with no failure at all, rarer with every record added; retrying only the failed ones needs no such attempt. 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 12 percent of writes failing (0 to 60)> 30 Batches of 12 records, 30% of writes failing. 1) one batch 2) many batches 3) settings 4) quit choice> 2 100 batches of 12 records, 30% of writes failing, at most 50 attempts each: retry attempts writes written again not done everything 3447 28974 27774 48 failed only 301 1200 0 0 all or none 3447 28974 0 48 Only retrying everything writes a record twice. It and all or none both need one attempt with no failure at all, rarer with every record added; retrying only the failed ones needs no such attempt. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 3 A batch of 12 records, 30% of writes failing, draw 3: retry everything: attempt 1: wrote 1 2 4 6 7 8 9 10 11 12; failed 3 5 attempt 2: wrote 1 2 3 4 6 7 8 10 11 12; failed 5 9 attempt 3: wrote 1 2 3 5 6 8 10 11 12; failed 4 7 9 attempt 4: wrote 1 2 3 5 6 8 9 10 11 12; failed 4 7 attempt 5: wrote 2 3 4 5 6 8 9 10 11 12; failed 1 7 attempt 6: wrote 1 4 6 7 8 9 10 11 12; failed 2 3 5 attempt 7: wrote 1 3 4 5 6 10 11 12; failed 2 7 8 9 attempt 8: wrote 1 2 4 5 6 7 8 9 10 11 12; failed 3 attempt 9: wrote 2 4 5 6 8 10 11 12; failed 1 3 7 9 attempt 10: wrote 1 4 5 6 8 9 10 11; failed 2 3 7 12 attempt 11: wrote 1 3 4 6 7 8 11; failed 2 5 9 10 12 attempt 12: wrote 2 5 7 10 11; failed 1 3 4 6 8 9 12 attempt 13: wrote 1 3 4 5 6 7 8 9 10; failed 2 11 12 attempt 14: wrote 1 4 5 6 8 9 10 11 12; failed 2 3 7 attempt 15: wrote 1 3 5 6 9 10 12; failed 2 4 7 8 11 attempt 16: wrote 1 3 4 6 7 8 10 11 12; failed 2 5 9 attempt 17: wrote 1 2 3 4 6 7 9 10 11 12; failed 5 8 attempt 18: wrote 1 2 3 4 6 7 8 9 10 11; failed 5 12 attempt 19: wrote 1 6 8 9 10 11 12; failed 2 3 4 5 7 attempt 20: wrote 1 3 7 8 9 10 12; failed 2 4 5 6 11 attempt 21: wrote 3 5 7 8 9 11 12; failed 1 2 4 6 10 attempt 22: wrote 1 2 3 5 6 9 10 11 12; failed 4 7 8 attempt 23: wrote 3 5 7 8 9 11 12; failed 1 2 4 6 10 attempt 24: wrote 1 2 3 8 9 10 11 12; failed 4 5 6 7 attempt 25: wrote 2 4 5 6 10 12; failed 1 3 7 8 9 11 attempt 26: wrote 2 3 4 7 8 11; failed 1 5 6 9 10 12 attempt 27: wrote 2 4 5 6 7 9 10 11 12; failed 1 3 8 attempt 28: wrote 1 2 3 5 6 8 9 11 12; failed 4 7 10 attempt 29: wrote 2 3 4 7 10 11 12; failed 1 5 6 8 9 attempt 30: wrote 1 3 4 5 6 7 8 9 10 12; failed 2 11 attempt 31: wrote 2 3 4 5 8 9 10 11 12; failed 1 6 7 attempt 32: wrote 1 2 3 6 7 8 10 11 12; failed 4 5 9 attempt 33: wrote 1 2 4 5 7 8 9 10 11; failed 3 6 12 attempt 34: wrote 1 4 5 6 10 12; failed 2 3 7 8 9 11 attempt 35: wrote 1 2 3 4 5 7 9 11 12; failed 6 8 10 attempt 36: wrote 1 2 3 5 6 7 8 9 10 11; failed 4 12 attempt 37: wrote 2 3 4 7 8 9 11 12; failed 1 5 6 10 attempt 38: wrote 3 4 5 8 10 11; failed 1 2 6 7 9 12 attempt 39: wrote 1 2 3 4 5 6 7 8 9 11 12; failed 10 attempt 40: wrote 2 4 5 7 8 9 12; failed 1 3 6 10 11 attempt 41: wrote 1 2 4 5 6 7 9 10 11; failed 3 8 12 attempt 42: wrote 1 3 4 5 7 8 10 11; failed 2 6 9 12 attempt 43: wrote 3 4 5 9 10 11 12; failed 1 2 6 7 8 attempt 44: wrote 1 3 4 7 9 10 11 12; failed 2 5 6 8 attempt 45: wrote 1 2 3 4 5 6 7 8 9 11 12; failed 10 attempt 46: wrote 2 3 4 5 9 11 12; failed 1 6 7 8 10 attempt 47: wrote 2 4 5 6 7 8 9 10 12; failed 1 3 11 attempt 48: wrote 1 2 3 5 6 7 8 10 11 12; failed 4 9 attempt 49: wrote 1 2 3 4 5 7 8 10; failed 6 9 11 12 attempt 50: wrote 1 8 9 11 12; failed 2 3 4 5 6 7 10 not done after 50 attempts; 417 writes so far retry failed only: attempt 1: wrote 1 2 4 6 7 8 9 10 11 12; failed 3 5 attempt 2: wrote 3; failed 5 attempt 3: wrote 5; failed none 12 writes for 12 records, none written twice retry all or none: attempt 1: wrote 1 2 4 6 7 8 9 10 11 12; failed 3 5; rolled back 1 2 4 6 7 8 9 10 11 12 attempt 2: wrote 1 2 3 4 6 7 8 10 11 12; failed 5 9; rolled back 1 2 3 4 6 7 8 10 11 12 attempt 3: wrote 1 2 3 5 6 8 10 11 12; failed 4 7 9; rolled back 1 2 3 5 6 8 10 11 12 attempt 4: wrote 1 2 3 5 6 8 9 10 11 12; failed 4 7; rolled back 1 2 3 5 6 8 9 10 11 12 attempt 5: wrote 2 3 4 5 6 8 9 10 11 12; failed 1 7; rolled back 2 3 4 5 6 8 9 10 11 12 attempt 6: wrote 1 4 6 7 8 9 10 11 12; failed 2 3 5; rolled back 1 4 6 7 8 9 10 11 12 attempt 7: wrote 1 3 4 5 6 10 11 12; failed 2 7 8 9; rolled back 1 3 4 5 6 10 11 12 attempt 8: wrote 1 2 4 5 6 7 8 9 10 11 12; failed 3; rolled back 1 2 4 5 6 7 8 9 10 11 12 attempt 9: wrote 2 4 5 6 8 10 11 12; failed 1 3 7 9; rolled back 2 4 5 6 8 10 11 12 attempt 10: wrote 1 4 5 6 8 9 10 11; failed 2 3 7 12; rolled back 1 4 5 6 8 9 10 11 attempt 11: wrote 1 3 4 6 7 8 11; failed 2 5 9 10 12; rolled back 1 3 4 6 7 8 11 attempt 12: wrote 2 5 7 10 11; failed 1 3 4 6 8 9 12; rolled back 2 5 7 10 11 attempt 13: wrote 1 3 4 5 6 7 8 9 10; failed 2 11 12; rolled back 1 3 4 5 6 7 8 9 10 attempt 14: wrote 1 4 5 6 8 9 10 11 12; failed 2 3 7; rolled back 1 4 5 6 8 9 10 11 12 attempt 15: wrote 1 3 5 6 9 10 12; failed 2 4 7 8 11; rolled back 1 3 5 6 9 10 12 attempt 16: wrote 1 3 4 6 7 8 10 11 12; failed 2 5 9; rolled back 1 3 4 6 7 8 10 11 12 attempt 17: wrote 1 2 3 4 6 7 9 10 11 12; failed 5 8; rolled back 1 2 3 4 6 7 9 10 11 12 attempt 18: wrote 1 2 3 4 6 7 8 9 10 11; failed 5 12; rolled back 1 2 3 4 6 7 8 9 10 11 attempt 19: wrote 1 6 8 9 10 11 12; failed 2 3 4 5 7; rolled back 1 6 8 9 10 11 12 attempt 20: wrote 1 3 7 8 9 10 12; failed 2 4 5 6 11; rolled back 1 3 7 8 9 10 12 attempt 21: wrote 3 5 7 8 9 11 12; failed 1 2 4 6 10; rolled back 3 5 7 8 9 11 12 attempt 22: wrote 1 2 3 5 6 9 10 11 12; failed 4 7 8; rolled back 1 2 3 5 6 9 10 11 12 attempt 23: wrote 3 5 7 8 9 11 12; failed 1 2 4 6 10; rolled back 3 5 7 8 9 11 12 attempt 24: wrote 1 2 3 8 9 10 11 12; failed 4 5 6 7; rolled back 1 2 3 8 9 10 11 12 attempt 25: wrote 2 4 5 6 10 12; failed 1 3 7 8 9 11; rolled back 2 4 5 6 10 12 attempt 26: wrote 2 3 4 7 8 11; failed 1 5 6 9 10 12; rolled back 2 3 4 7 8 11 attempt 27: wrote 2 4 5 6 7 9 10 11 12; failed 1 3 8; rolled back 2 4 5 6 7 9 10 11 12 attempt 28: wrote 1 2 3 5 6 8 9 11 12; failed 4 7 10; rolled back 1 2 3 5 6 8 9 11 12 attempt 29: wrote 2 3 4 7 10 11 12; failed 1 5 6 8 9; rolled back 2 3 4 7 10 11 12 attempt 30: wrote 1 3 4 5 6 7 8 9 10 12; failed 2 11; rolled back 1 3 4 5 6 7 8 9 10 12 attempt 31: wrote 2 3 4 5 8 9 10 11 12; failed 1 6 7; rolled back 2 3 4 5 8 9 10 11 12 attempt 32: wrote 1 2 3 6 7 8 10 11 12; failed 4 5 9; rolled back 1 2 3 6 7 8 10 11 12 attempt 33: wrote 1 2 4 5 7 8 9 10 11; failed 3 6 12; rolled back 1 2 4 5 7 8 9 10 11 attempt 34: wrote 1 4 5 6 10 12; failed 2 3 7 8 9 11; rolled back 1 4 5 6 10 12 attempt 35: wrote 1 2 3 4 5 7 9 11 12; failed 6 8 10; rolled back 1 2 3 4 5 7 9 11 12 attempt 36: wrote 1 2 3 5 6 7 8 9 10 11; failed 4 12; rolled back 1 2 3 5 6 7 8 9 10 11 attempt 37: wrote 2 3 4 7 8 9 11 12; failed 1 5 6 10; rolled back 2 3 4 7 8 9 11 12 attempt 38: wrote 3 4 5 8 10 11; failed 1 2 6 7 9 12; rolled back 3 4 5 8 10 11 attempt 39: wrote 1 2 3 4 5 6 7 8 9 11 12; failed 10; rolled back 1 2 3 4 5 6 7 8 9 11 12 attempt 40: wrote 2 4 5 7 8 9 12; failed 1 3 6 10 11; rolled back 2 4 5 7 8 9 12 attempt 41: wrote 1 2 4 5 6 7 9 10 11; failed 3 8 12; rolled back 1 2 4 5 6 7 9 10 11 attempt 42: wrote 1 3 4 5 7 8 10 11; failed 2 6 9 12; rolled back 1 3 4 5 7 8 10 11 attempt 43: wrote 3 4 5 9 10 11 12; failed 1 2 6 7 8; rolled back 3 4 5 9 10 11 12 attempt 44: wrote 1 3 4 7 9 10 11 12; failed 2 5 6 8; rolled back 1 3 4 7 9 10 11 12 attempt 45: wrote 1 2 3 4 5 6 7 8 9 11 12; failed 10; rolled back 1 2 3 4 5 6 7 8 9 11 12 attempt 46: wrote 2 3 4 5 9 11 12; failed 1 6 7 8 10; rolled back 2 3 4 5 9 11 12 attempt 47: wrote 2 4 5 6 7 8 9 10 12; failed 1 3 11; rolled back 2 4 5 6 7 8 9 10 12 attempt 48: wrote 1 2 3 5 6 7 8 10 11 12; failed 4 9; rolled back 1 2 3 5 6 7 8 10 11 12 attempt 49: wrote 1 2 3 4 5 7 8 10; failed 6 9 11 12; rolled back 1 2 3 4 5 7 8 10 attempt 50: wrote 1 8 9 11 12; failed 2 3 4 5 6 7 10; rolled back 1 8 9 11 12 not done after 50 attempts; 417 writes so far 1) one batch 2) many batches 3) settings 4) quit choice> 3 records in a batch (2 to 12)> 8 percent of writes failing (0 to 60)> 10 Batches of 8 records, 10% of writes failing. 1) one batch 2) many batches 3) settings 4) quit choice> 1 draw (1 to 999)> 5 A batch of 8 records, 10% of writes failing, draw 5: retry everything: attempt 1: wrote 1 2 3 4 5 6 7 8; failed none 8 writes for 8 records, none written twice retry failed only: attempt 1: wrote 1 2 3 4 5 6 7 8; failed none 8 writes for 8 records, none written twice retry all or none: attempt 1: wrote 1 2 3 4 5 6 7 8; failed none 8 writes for 8 records, none written twice 1) one batch 2) many batches 3) settings 4) quit choice> 4 Bye. ``` ## Modules ### main.eml ```eml # P129 batch retries: one batch attempt by attempt under three ways of # retrying, many batches counted, and the settings. import batch def plain(s): 0 => i len(s) => j while i < j and s[i] == " ": i + 1 => i while j > i and s[j - 1] == " ": j - 1 => j return s[i:j] def pad(s, n): s => out while len(out) < n: out + " " => out return out def lpad(s, n): s => out while len(out) < n: " " + out => out return out def ask_int(prompt, low, high): plain(input(prompt)) => t if t == "": "Cancelled." ^0 return 0 - 1 if len(t) > 3: ("Type a whole number from " + str(low) + " to " + str(high) + ".") ^0 return 0 - 1 for c in t: if c < "0" or c > "9": ("Type a whole number from " + str(low) + " to " + str(high) + ".") ^0 return 0 - 1 if int(t) < low or int(t) > high: ("Type a whole number from " + str(low) + " to " + str(high) + ".") ^0 return 0 - 1 return int(t) def nums(xs): if len(xs) == 0: return "none" "" => out for x in xs: if out != "": out + " " => out out + str(x) => out return out def plans(): return ["everything", "failed only", "all or none"] def one(n, rate): ask_int("draw (1 to 999)> ", 1, 999) => d if d < 0: return ("A batch of " + str(n) + " records, " + str(rate) + "% of writes failing, draw " + str(d) + ":") ^0 for plan in plans(): batch.run(n, rate, d, plan, 50) => r ("retry " + plan + ":") ^0 for k in [0:len(r[0]) - 1]: r[0][k] => a " attempt " + lpad(str(k + 1), 2) + ": wrote " + nums(a[0]) + "; failed " + nums(a[1]) => line if len(a[2]) > 0: line + "; rolled back " + nums(a[2]) => line line ^0 if not r[3]: (" not done after 50 attempts; " + str(r[1]) + " writes so far") ^0 elif r[2] > 0: (" " + str(r[1]) + " writes for " + str(n) + " records, " + str(r[2]) + " of them a record written again") ^0 else: (" " + str(r[1]) + " writes for " + str(n) + " records, none written twice") ^0 def over_many(n, rate): ("100 batches of " + str(n) + " records, " + str(rate) + "% of writes failing, at most 50 attempts each:") ^0 " retry attempts writes written again not done" ^0 for row in batch.many(n, rate, 100, 50): (" " + pad(row[0], 12) + lpad(str(row[1]), 10) + lpad(str(row[2]), 8) + lpad(str(row[3]), 15) + lpad(str(row[4]), 10)) ^0 "Only retrying everything writes a record twice. It and all or none both need one attempt with no failure at all, rarer with every record added; retrying only the failed ones needs no such attempt." ^0 def settings(n, rate): ask_int("records in a batch (2 to 12)> ", 2, 12) => a if a < 0: return [n, rate] ask_int("percent of writes failing (0 to 60)> ", 0, 60) => b if b < 0: return [n, rate] ("Batches of " + str(a) + " records, " + str(b) + "% of writes failing.") ^0 return [a, b] 8 => n 20 => rate "== Batch retries ==" ^0 "A batch of records written one by one, some writes failing at random, and three ways to retry." ^0 True => running while running: "" ^0 "1) one batch 2) many batches 3) settings 4) quit" ^0 plain(input("choice> ")) => choice if choice == "1": one(n, rate) elif choice == "2": over_many(n, rate) elif choice == "3": settings(n, rate) => s s[0] => n s[1] => rate elif choice == "4": False => running else: "Pick a number from 1 to 4." ^0 "Bye." ^0 ``` Python projection of main.eml: ```python import batch def plain(s): i = 0 j = len(s) while i < j and s[i] == " ": i = i + 1 while j > i and s[j - 1] == " ": j = j - 1 return s[i:j] def pad(s, n): out = s while len(out) < n: out = out + " " return out def lpad(s, n): out = s while len(out) < n: out = " " + out return out def ask_int(prompt, low, high): t = plain(input(prompt)) if t == "": print("Cancelled.") return 0 - 1 if len(t) > 3: print("Type a whole number from " + str(low) + " to " + str(high) + ".") return 0 - 1 for c in t: if c < "0" or c > "9": print("Type a whole number from " + str(low) + " to " + str(high) + ".") return 0 - 1 if int(t) < low or int(t) > high: print("Type a whole number from " + str(low) + " to " + str(high) + ".") return 0 - 1 return int(t) def nums(xs): if len(xs) == 0: return "none" out = "" for x in xs: if out != "": out = out + " " out = out + str(x) return out def plans(): return ["everything", "failed only", "all or none"] def one(n, rate): d = ask_int("draw (1 to 999)> ", 1, 999) if d < 0: return print("A batch of " + str(n) + " records, " + str(rate) + "% of writes failing, draw " + str(d) + ":") for plan in plans(): r = batch.run(n, rate, d, plan, 50) print("retry " + plan + ":") for k in range(0, len(r[0])): a = r[0][k] line = " attempt " + lpad(str(k + 1), 2) + ": wrote " + nums(a[0]) + "; failed " + nums(a[1]) if len(a[2]) > 0: line = line + "; rolled back " + nums(a[2]) print(line) if not r[3]: print(" not done after 50 attempts; " + str(r[1]) + " writes so far") elif r[2] > 0: print(" " + str(r[1]) + " writes for " + str(n) + " records, " + str(r[2]) + " of them a record written again") else: print(" " + str(r[1]) + " writes for " + str(n) + " records, none written twice") def over_many(n, rate): print("100 batches of " + str(n) + " records, " + str(rate) + "% of writes failing, at most 50 attempts each:") print(" retry attempts writes written again not done") for row in batch.many(n, rate, 100, 50): print(" " + pad(row[0], 12) + lpad(str(row[1]), 10) + lpad(str(row[2]), 8) + lpad(str(row[3]), 15) + lpad(str(row[4]), 10)) print("Only retrying everything writes a record twice. It and all or none both need one attempt with no failure at all, rarer with every record added; retrying only the failed ones needs no such attempt.") def settings(n, rate): a = ask_int("records in a batch (2 to 12)> ", 2, 12) if a < 0: return [n, rate] b = ask_int("percent of writes failing (0 to 60)> ", 0, 60) if b < 0: return [n, rate] print("Batches of " + str(a) + " records, " + str(b) + "% of writes failing.") return [a, b] n = 8 rate = 20 print("== Batch retries ==") print("A batch of records written one by one, some writes failing at random, and three ways to retry.") running = True while running: print("") print("1) one batch 2) many batches 3) settings 4) quit") choice = plain(input("choice> ")) if choice == "1": one(n, rate) elif choice == "2": over_many(n, rate) elif choice == "3": s = settings(n, rate) n = s[0] rate = s[1] elif choice == "4": running = False else: print("Pick a number from 1 to 4.") print("Bye.") ``` ### batch.eml ```eml # P129 batch retries - a batch of records written one by one, some writes # failing at random. As in the corpus case partial-batch-failure-retries- # everything, three ways to finish the batch: # everything any failure retries the whole batch, so records already # written are written again; # failed only each retry writes only the records not yet written; # all or none as in partial-write-rollback, a failure rolls back the # writes of that attempt, and the whole batch is retried. # The batch is done when every record is written: for everything and all or # none, that takes one attempt with no failure at all. Whether a write fails # depends only on the draw, the attempt and the record, so the three meet the # same failures where they try the same writes. def chance(draw, attempt, record): # a number from 0 to 99, the same for the same draw, attempt and record # squaring modulo a prime mixes the three numbers; a linear step would # leave the records of one attempt in step with each other draw * 1000003 + attempt * 1009 + record * 7919 + 12345 => x for k in [1:4]: (x * x + 7) % 1000000007 => x return x % 100 def fails(draw, attempt, record, rate): return chance(draw, attempt, record) < rate def run(n, rate, draw, plan, most): # [attempts as [written, failed, rolled back], writes, duplicate writes, # done] [] => written for r in [1:n]: written + [0] => written [] => attempts 0 => writes 0 => dups False => done 0 => a while not done and a < most: a + 1 => a [] => ok [] => bad for r in [1:n]: if plan != "failed only" or written[r - 1] == 0: if fails(draw, a, r, rate): bad + [r] => bad else: ok + [r] => ok writes + 1 => writes if written[r - 1] > 0: dups + 1 => dups written[r - 1] + 1 => written[r - 1] [] => undone if plan == "all or none" and len(bad) > 0: for r in ok: written[r - 1] - 1 => written[r - 1] ok => undone attempts + [[ok, bad, undone]] => attempts if plan == "failed only": True => done for w in written: if w == 0: False => done else: len(bad) == 0 => done return [attempts, writes, dups, done] def many(n, rate, batches, most): # per plan [attempts, writes, duplicate writes, batches not done] over # draws 1 to batches [] => out for plan in ["everything", "failed only", "all or none"]: 0 => att 0 => wr 0 => du 0 => left for d in [1:batches]: run(n, rate, d, plan, most) => r att + len(r[0]) => att wr + r[1] => wr du + r[2] => du if not r[3]: left + 1 => left out + [[plan, att, wr, du, left]] => out return out ``` Python projection of batch.eml: ```python def chance(draw, attempt, record): x = draw * 1000003 + attempt * 1009 + record * 7919 + 12345 for k in range(1, 5): x = (x * x + 7) % 1000000007 return x % 100 def fails(draw, attempt, record, rate): return chance(draw, attempt, record) < rate def run(n, rate, draw, plan, most): written = [] for r in range(1, n+1): written = written + [0] attempts = [] writes = 0 dups = 0 done = False a = 0 while not done and a < most: a = a + 1 ok = [] bad = [] for r in range(1, n+1): if plan != "failed only" or written[r - 1] == 0: if fails(draw, a, r, rate): bad = bad + [r] else: ok = ok + [r] writes = writes + 1 if written[r - 1] > 0: dups = dups + 1 written[r - 1] = written[r - 1] + 1 undone = [] if plan == "all or none" and len(bad) > 0: for r in ok: written[r - 1] = written[r - 1] - 1 undone = ok attempts = attempts + [[ok, bad, undone]] if plan == "failed only": done = True for w in written: if w == 0: done = False else: done = len(bad) == 0 return [attempts, writes, dups, done] def many(n, rate, batches, most): out = [] for plan in ["everything", "failed only", "all or none"]: att = 0 wr = 0 du = 0 left = 0 for d in range(1, batches+1): r = run(n, rate, d, plan, most) att = att + len(r[0]) wr = wr + r[1] du = du + r[2] if not r[3]: left = left + 1 out = out + [[plan, att, wr, du, left]] return out ``` ## README # P129 - Batch retries A batch of records written one by one, some writes failing at random, and three ways to retry until every record is written. - `main.eml` - the menu, one batch attempt by attempt under the three ways, many batches counted, and the settings - `batch.eml` - whether a write fails, a run of one batch, and the counts over many How each part works: - As in the corpus case `partial-batch-failure-retries-everything`, a failure in one record can retry the whole batch, writing again the records that already succeeded. The three ways: - everything: any failure retries the whole batch; - failed only: each retry writes only the records not yet written; - all or none: as in the corpus case `partial-write-rollback`, a failure rolls back that attempt's writes, and the whole batch is retried. - Everything and all or none are done only by an attempt with no failure at all, which gets rarer with every record: 8 records at 20% take 503 attempts over 100 batches, against 205 for failed only. At 12 records and 30%, 48 of the 100 are still not done after 50 attempts. They make the same attempts; everything writes the records again, all or none rolls them back. Only failed only writes each record once and needs no clean attempt. - Whether a write fails depends on the draw, the attempt and the record, mixed by squaring modulo a prime: a linear mix kept the records of one attempt in step, and clean attempts almost never came. What is checked: menu choices 1 to 4; a draw from 1 to 999; 2 to 12 records and 0 to 60% failing. An empty answer cancels. Sessions: `sessions/basic.in` gives: - one batch of 8 at 20%, and 100 of them; - 100 batches of 12 at 30%, and one that neither everything nor all or none finishes in 50 attempts; - 8 records at 10%: one batch with nothing failing. `sessions/bad-input.in` gives: - menu choices 0 and x; - draws: a cancel, 0, 1000 and x; - settings: a cancel, 1 and 13 records, then 5 with a cancel and 61%, and 5 at 0%: one batch and 100 of them. Built on two verified corpus cases: `partial-batch-failure-retries-everything` (a failure in one item retries the whole batch) and `partial-write-rollback` (a failed batch rolled back to where it began).