{
  "id": "P107",
  "slug": "retry-planner",
  "key": "P107-retry-planner",
  "title": "Retry planner",
  "summary": "Waits between retries under fixed, exponential, full jitter, equal jitter and decorrelated policies; a crowd of clients coming back after an outage, with the busiest slot under each; and a job runner whose ledger checks that its finally block ran on every attempt.",
  "entry": "main.eml",
  "ui": "terminal",
  "readme": "# P107 - Retry planner\n\nHow long to wait between retries, under five policies. The program shows\none client's waits, a crowd of clients coming back after an outage, and\na job runner whose ledger checks its own retry loop.\n\n- `main.eml` - the menu, one client's schedule, the crowd table, retries\n  over time, jobs with a ledger, and the settings\n- `retry.eml` - the random numbers, the five policies, the crowd, the\n  busiest slot, and the job runner with its ledger\n\nHow each part works:\n\n- Waits are whole milliseconds, for retries 1, 2, ...:\n  - fixed: the first wait every time;\n  - exponential: the first wait doubled each time, up to the longest;\n  - full jitter: a random wait from 0 to the exponential one;\n  - equal jitter: half the exponential wait, plus a random wait up to the\n    other half;\n  - decorrelated: a random wait from the first wait to three times the\n    last one, up to the longest.\n\n  The random numbers come from a linear congruential generator seeded by\n  the seed and the client's number.\n- The crowd: every client fails when the outage starts, at 0 ms, and a\n  retry at or after its end gets through. Retries are counted per 100 ms.\n  As in the corpus case `backoff-thundering-herd`, fixed and exponential\n  waits are the same for every client, so each round of retries lands in\n  a single slot. A wait that depends on the client spreads each round over\n  the whole wait. That lowers the busiest slot only when the waits are\n  long against the slot. In the session, with a first wait of 100 ms,\n  full jitter's busiest slot holds more than exponential's.\n- Full and equal jitter wait on average about a half and three quarters of\n  the exponential wait. So with the same attempts, more clients can give\n  up before the service is back.\n- Jobs with a ledger, as in the corpus case `retry-with-backoff-ledger`: a\n  job fails a given number of times, or cannot be retried at all. Every\n  attempt is counted inside a try whose finally block counts again. A job\n  that cannot be retried stops at once, and one that runs out of attempts\n  raises. The ledger checks that finally ran on every attempt: after a\n  return, after a raise and after a failure alike.\n\nWhat is checked: menu choices 1 to 6; a policy name; a client number;\njobs written as name:failures or name:fatal, up to 8; settings in their\nranges (first wait 1 to 10,000 ms, longest from the first up to 600,000,\nattempts 2 to 12, clients 1 to 1,000, outage up to 600,000 ms, seed up to\n999,999). An empty answer cancels.\n\nSessions: `sessions/basic.in` gives:\n- client 1 under exponential, full jitter and decorrelated waits, and\n  client 2 under full jitter;\n- the crowd of 200: the busiest slot holds 200 under fixed and exponential\n  waits, and 28, 44 and 15 under the other three;\n- retries over time, second by second, for exponential and equal jitter;\n- four jobs: one done at once, one after two failures, one that gives up,\n  one that cannot be retried, with the ledger;\n- a first wait of 100 ms and the crowd again, where full jitter's busiest\n  slot is worse than exponential's.\n\n`sessions/bad-input.in` gives:\n- menu choices 0 and x;\n- a cancel, a policy slow, client numbers 0 and 201, then a cancel;\n- a cancel for retries over time;\n- jobs: a cancel, no colon, a word for the failures, no name, nine jobs;\n- settings: first waits 0 and 10,001, a longest wait below the first, one\n  attempt, and cancels.\n\nBuilt on the verified corpus cases `backoff-thundering-herd` (the same\nwaits for every client bring them back together) and\n`retry-with-backoff-ledger` (a retry loop audited by counting its\nfinally blocks).\n",
  "modules": [
    {
      "name": "main.eml",
      "eml": "# P107 retry planner: one client's waits under five policies, a crowd of\n# clients coming back after an outage, and jobs run with a ledger that\n# checks the retry loop.\nimport retry\n\ndef plain(s):\n    0 => i\n    len(s) => j\n    while i < j and s[i] == \" \":\n        i + 1 => i\n    while j > i and s[j - 1] == \" \":\n        j - 1 => j\n    return s[i:j]\n\ndef number(s, most):\n    if len(s) == 0 or len(s) > most:\n        return -1\n    0 => n\n    for c in s:\n        if not (c >= \"0\" and c <= \"9\"):\n            return -1\n        n * 10 + int(c) => n\n    return n\n\ndef pad(s, n):\n    while len(s) < n:\n        s + \" \" => s\n    return s\n\ndef lpad(s, n):\n    while len(s) < n:\n        \" \" + s => s\n    return s\n\ndef ask(prompt, low, high):\n    while True:\n        plain(input(prompt + \" (\" + str(low) + \" to \" + str(high) + \")> \")) => a\n        if a == \"\":\n            return -1\n        number(a, 7) => n\n        if n >= low and n <= high:\n            return n\n        (\"Type a whole number from \" + str(low) + \" to \" + str(high) + \".\") ^0\n\ndef policies():\n    return [\"fixed\", \"exponential\", \"full jitter\", \"equal jitter\", \"decorrelated\"]\n\ndef ask_policy():\n    plain(input(\"policy (fixed, exponential, full jitter, equal jitter or decorrelated)> \")) => p\n    if p == \"\":\n        return \"\"\n    for q in policies():\n        if q == p:\n            return p\n    \"Type fixed, exponential, full jitter, equal jitter or decorrelated.\" ^0\n    return \"!\"\n\ndef one_client(cfg):\n    ask_policy() => p\n    if p == \"\":\n        \"Cancelled.\" ^0\n        return\n    if p == \"!\":\n        return\n    ask(\"client number\", 1, cfg[3]) => c\n    if c == -1:\n        \"Cancelled.\" ^0\n        return\n    (\"Client \" + str(c) + \", \" + p + \": attempt 1 at 0 ms, then\") ^0\n    0 => t\n    2 => n\n    for w in retry.waits(p, cfg, c):\n        t + w => t\n        (\"  attempt \" + lpad(str(n), 2) + \"  after \" + lpad(str(w), 6) + \" ms, at \" + lpad(str(t), 7) + \" ms\") ^0\n        n + 1 => n\n    (\"In all \" + str(t) + \" ms before the last attempt.\") ^0\n\ndef crowd(cfg):\n    (str(cfg[3]) + \" clients fail at 0 ms; the service is back at \" + str(cfg[4]) + \" ms; each tries \" + str(cfg[2]) + \" times at most. Retries counted per \" + str(cfg[6]) + \" ms:\") ^0\n    \"  policy          retries   most in a slot   the slot from   all through by   gave up\" ^0\n    {} => gave\n    {} => top\n    for p in policies():\n        retry.herd(p, cfg) => h\n        h[4] => gave[p]\n        retry.peak(h[0])[0] => top[p]\n        retry.peak(h[0]) => m\n        \"-\" => last\n        if h[3] >= 0:\n            str(h[3]) + \" ms\" => last\n        (\"  \" + pad(p, 14) + lpad(str(h[1]), 9) + lpad(str(m[0]), 17) + lpad(str(m[1] * cfg[6]) + \" ms\", 16) + lpad(last, 17) + lpad(str(h[4]), 10)) ^0\n    (\"Fixed and exponential waits are the same for every client, so each round of retries lands in a single slot. A wait that depends on the client spreads each round over the whole wait, which lowers the busiest slot only when the waits are long against the slot: here \" + str(top[\"full jitter\"]) + \", \" + str(top[\"equal jitter\"]) + \" and \" + str(top[\"decorrelated\"]) + \" against \" + str(top[\"exponential\"]) + \".\") ^0\n    if gave[\"full jitter\"] > gave[\"exponential\"] or gave[\"equal jitter\"] > gave[\"exponential\"]:\n        (\"Full and equal jitter wait on average about a half and three quarters of the exponential wait, so with the same attempts \" + str(gave[\"full jitter\"]) + \" and \" + str(gave[\"equal jitter\"]) + \" clients gave up, against \" + str(gave[\"exponential\"]) + \" with exponential waits.\") ^0\n\ndef histogram(cfg):\n    ask_policy() => p\n    if p == \"\":\n        \"Cancelled.\" ^0\n        return\n    if p == \"!\":\n        return\n    [cfg[0], cfg[1], cfg[2], cfg[3], cfg[4], cfg[5], 1000] => by_second\n    retry.herd(p, by_second) => h\n    retry.peak(h[0]) => m\n    (p + \": retries in each second, # for every \" + str(scale(m[0])) + \":\") ^0\n    for i in [0:len(h[0]) - 1]:\n        if h[0][i] > 0:\n            \"\" => bar\n            for k in [1:bars(h[0][i], m[0])]:\n                bar + \"#\" => bar\n            (\"  \" + lpad(str(i * 1000), 7) + \" ms \" + lpad(str(h[0][i]), 5) + \"  \" + bar) ^0\n\ndef scale(most):\n    # one # stands for this many retries, so the longest bar fits in 40\n    1 => s\n    while most > 40 * s:\n        s + 1 => s\n    return s\n\ndef bars(n, most):\n    scale(most) => s\n    return int((n + s - 1 - (n + s - 1) % s) / s)\n\ndef jobs(cfg):\n    \"Jobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\" ^0\n    plain(input(\"jobs> \")) => t\n    if t == \"\":\n        \"Cancelled.\" ^0\n        return\n    [] => list\n    \"\" => w\n    for c in t + \" \":\n        if c == \" \":\n            if w != \"\":\n                list + [w] => list\n            \"\" => w\n        else:\n            w + c => w\n    [] => parsed\n    for item in list:\n        -1 => colon\n        for i in [0:len(item) - 1]:\n            if colon == -1 and item[i] == \":\":\n                i => colon\n        if colon < 1 or colon == len(item) - 1:\n            \"Type each job as name:number or name:fatal.\" ^0\n            return\n        item[0:colon] => name\n        item[colon + 1:len(item)] => rest\n        if rest == \"fatal\":\n            parsed + [[name, 0, True]] => parsed\n        else:\n            number(rest, 3) => n\n            if n < 0:\n                \"Type each job as name:number or name:fatal.\" ^0\n                return\n            parsed + [[name, n, False]] => parsed\n    if len(parsed) > 8:\n        \"Up to 8 jobs.\" ^0\n        return\n    retry.Ledger() => ledger\n    0 => done\n    0 => gave_up\n    0 => refused\n    for job in parsed:\n        try:\n            retry.run_job(ledger, cfg, job[0], job[1], job[2]) => result\n            done + 1 => done\n            (\"  \" + result) ^0\n        except TypeError as e:\n            refused + 1 => refused\n            (\"  \" + str(e)) ^0\n        except ValueError as e:\n            gave_up + 1 => gave_up\n            (\"  \" + str(e)) ^0\n    \"Ledger:\" ^0\n    for entry in ledger.log:\n        (\"  \" + entry) ^0\n    (\"Attempts \" + str(ledger.attempts) + \": \" + str(ledger.successes) + \" done, \" + str(ledger.failures) + \" failed. The finally block ran \" + str(ledger.finally_runs) + \" times. Waited \" + str(ledger.waited) + \" ms in all.\") ^0\n    (\"Jobs done, given up, refused: \" + str(done) + \", \" + str(gave_up) + \", \" + str(refused) + \".\") ^0\n    if ledger.finally_runs == ledger.attempts and ledger.successes + ledger.failures == ledger.attempts:\n        \"Every attempt was counted once, and finally ran on each - after a return, after a raise, and after a failure alike.\" ^0\n\ndef settings(cfg):\n    ask(\"first wait in ms\", 1, 10000) => b\n    if b == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    ask(\"longest wait in ms\", b, 600000) => c\n    if c == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    ask(\"attempts per client\", 2, 12) => a\n    if a == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    ask(\"clients\", 1, 1000) => n\n    if n == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    ask(\"outage in ms\", 0, 600000) => o\n    if o == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    ask(\"seed\", 0, 999999) => s\n    if s == -1:\n        \"Cancelled.\" ^0\n        return cfg\n    (\"First wait \" + str(b) + \" ms, longest \" + str(c) + \" ms, \" + str(a) + \" attempts, \" + str(n) + \" clients, outage \" + str(o) + \" ms, seed \" + str(s) + \".\") ^0\n    return [b, c, a, n, o, s, cfg[6]]\n\n[1000, 16000, 6, 200, 10000, 7, 100] => cfg\n\n\"== Retry planner ==\" ^0\n\"Waits between retries under five policies, a crowd coming back after an outage, and a job runner whose ledger checks itself.\" ^0\nTrue => running\nwhile running:\n    \"\" ^0\n    \"1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\" ^0\n    plain(input(\"choice> \")) => choice\n    if choice == \"1\":\n        one_client(cfg)\n    elif choice == \"2\":\n        crowd(cfg)\n    elif choice == \"3\":\n        histogram(cfg)\n    elif choice == \"4\":\n        jobs(cfg)\n    elif choice == \"5\":\n        settings(cfg) => cfg\n    elif choice == \"6\":\n        False => running\n    else:\n        \"Pick a number from 1 to 6.\" ^0\n\"Bye.\" ^0\n",
      "python": "import retry\n\ndef plain(s):\n    i = 0\n    j = len(s)\n    while i < j and s[i] == \" \":\n        i = i + 1\n    while j > i and s[j - 1] == \" \":\n        j = j - 1\n    return s[i:j]\n\ndef number(s, most):\n    if len(s) == 0 or len(s) > most:\n        return -1\n    n = 0\n    for c in s:\n        if not (c >= \"0\" and c <= \"9\"):\n            return -1\n        n = n * 10 + int(c)\n    return n\n\ndef pad(s, n):\n    while len(s) < n:\n        s = s + \" \"\n    return s\n\ndef lpad(s, n):\n    while len(s) < n:\n        s = \" \" + s\n    return s\n\ndef ask(prompt, low, high):\n    while True:\n        a = plain(input(prompt + \" (\" + str(low) + \" to \" + str(high) + \")> \"))\n        if a == \"\":\n            return -1\n        n = number(a, 7)\n        if n >= low and n <= high:\n            return n\n        print(\"Type a whole number from \" + str(low) + \" to \" + str(high) + \".\")\n\ndef policies():\n    return [\"fixed\", \"exponential\", \"full jitter\", \"equal jitter\", \"decorrelated\"]\n\ndef ask_policy():\n    p = plain(input(\"policy (fixed, exponential, full jitter, equal jitter or decorrelated)> \"))\n    if p == \"\":\n        return \"\"\n    for q in policies():\n        if q == p:\n            return p\n    print(\"Type fixed, exponential, full jitter, equal jitter or decorrelated.\")\n    return \"!\"\n\ndef one_client(cfg):\n    p = ask_policy()\n    if p == \"\":\n        print(\"Cancelled.\")\n        return\n    if p == \"!\":\n        return\n    c = ask(\"client number\", 1, cfg[3])\n    if c == -1:\n        print(\"Cancelled.\")\n        return\n    print(\"Client \" + str(c) + \", \" + p + \": attempt 1 at 0 ms, then\")\n    t = 0\n    n = 2\n    for w in retry.waits(p, cfg, c):\n        t = t + w\n        print(\"  attempt \" + lpad(str(n), 2) + \"  after \" + lpad(str(w), 6) + \" ms, at \" + lpad(str(t), 7) + \" ms\")\n        n = n + 1\n    print(\"In all \" + str(t) + \" ms before the last attempt.\")\n\ndef crowd(cfg):\n    print(str(cfg[3]) + \" clients fail at 0 ms; the service is back at \" + str(cfg[4]) + \" ms; each tries \" + str(cfg[2]) + \" times at most. Retries counted per \" + str(cfg[6]) + \" ms:\")\n    print(\"  policy          retries   most in a slot   the slot from   all through by   gave up\")\n    gave = {}\n    top = {}\n    for p in policies():\n        h = retry.herd(p, cfg)\n        gave[p] = h[4]\n        top[p] = retry.peak(h[0])[0]\n        m = retry.peak(h[0])\n        last = \"-\"\n        if h[3] >= 0:\n            last = str(h[3]) + \" ms\"\n        print(\"  \" + pad(p, 14) + lpad(str(h[1]), 9) + lpad(str(m[0]), 17) + lpad(str(m[1] * cfg[6]) + \" ms\", 16) + lpad(last, 17) + lpad(str(h[4]), 10))\n    print(\"Fixed and exponential waits are the same for every client, so each round of retries lands in a single slot. A wait that depends on the client spreads each round over the whole wait, which lowers the busiest slot only when the waits are long against the slot: here \" + str(top[\"full jitter\"]) + \", \" + str(top[\"equal jitter\"]) + \" and \" + str(top[\"decorrelated\"]) + \" against \" + str(top[\"exponential\"]) + \".\")\n    if gave[\"full jitter\"] > gave[\"exponential\"] or gave[\"equal jitter\"] > gave[\"exponential\"]:\n        print(\"Full and equal jitter wait on average about a half and three quarters of the exponential wait, so with the same attempts \" + str(gave[\"full jitter\"]) + \" and \" + str(gave[\"equal jitter\"]) + \" clients gave up, against \" + str(gave[\"exponential\"]) + \" with exponential waits.\")\n\ndef histogram(cfg):\n    p = ask_policy()\n    if p == \"\":\n        print(\"Cancelled.\")\n        return\n    if p == \"!\":\n        return\n    by_second = [cfg[0], cfg[1], cfg[2], cfg[3], cfg[4], cfg[5], 1000]\n    h = retry.herd(p, by_second)\n    m = retry.peak(h[0])\n    print(p + \": retries in each second, # for every \" + str(scale(m[0])) + \":\")\n    for i in range(0, len(h[0])):\n        if h[0][i] > 0:\n            bar = \"\"\n            for k in range(1, bars(h[0][i], m[0])+1):\n                bar = bar + \"#\"\n            print(\"  \" + lpad(str(i * 1000), 7) + \" ms \" + lpad(str(h[0][i]), 5) + \"  \" + bar)\n\ndef scale(most):\n    s = 1\n    while most > 40 * s:\n        s = s + 1\n    return s\n\ndef bars(n, most):\n    s = scale(most)\n    return int((n + s - 1 - (n + s - 1) % s) / s)\n\ndef jobs(cfg):\n    print(\"Jobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\")\n    t = plain(input(\"jobs> \"))\n    if t == \"\":\n        print(\"Cancelled.\")\n        return\n    lst = []\n    w = \"\"\n    for c in t + \" \":\n        if c == \" \":\n            if w != \"\":\n                lst = lst + [w]\n            w = \"\"\n        else:\n            w = w + c\n    parsed = []\n    for item in lst:\n        colon = -1\n        for i in range(0, len(item)):\n            if colon == -1 and item[i] == \":\":\n                colon = i\n        if colon < 1 or colon == len(item) - 1:\n            print(\"Type each job as name:number or name:fatal.\")\n            return\n        name = item[0:colon]\n        rest = item[colon + 1:len(item)]\n        if rest == \"fatal\":\n            parsed = parsed + [[name, 0, True]]\n        else:\n            n = number(rest, 3)\n            if n < 0:\n                print(\"Type each job as name:number or name:fatal.\")\n                return\n            parsed = parsed + [[name, n, False]]\n    if len(parsed) > 8:\n        print(\"Up to 8 jobs.\")\n        return\n    ledger = retry.Ledger()\n    done = 0\n    gave_up = 0\n    refused = 0\n    for job in parsed:\n        try:\n            result = retry.run_job(ledger, cfg, job[0], job[1], job[2])\n            done = done + 1\n            print(\"  \" + result)\n        except TypeError as e:\n            refused = refused + 1\n            print(\"  \" + str(e))\n        except ValueError as e:\n            gave_up = gave_up + 1\n            print(\"  \" + str(e))\n    print(\"Ledger:\")\n    for entry in ledger.log:\n        print(\"  \" + entry)\n    print(\"Attempts \" + str(ledger.attempts) + \": \" + str(ledger.successes) + \" done, \" + str(ledger.failures) + \" failed. The finally block ran \" + str(ledger.finally_runs) + \" times. Waited \" + str(ledger.waited) + \" ms in all.\")\n    print(\"Jobs done, given up, refused: \" + str(done) + \", \" + str(gave_up) + \", \" + str(refused) + \".\")\n    if ledger.finally_runs == ledger.attempts and ledger.successes + ledger.failures == ledger.attempts:\n        print(\"Every attempt was counted once, and finally ran on each - after a return, after a raise, and after a failure alike.\")\n\ndef settings(cfg):\n    b = ask(\"first wait in ms\", 1, 10000)\n    if b == -1:\n        print(\"Cancelled.\")\n        return cfg\n    c = ask(\"longest wait in ms\", b, 600000)\n    if c == -1:\n        print(\"Cancelled.\")\n        return cfg\n    a = ask(\"attempts per client\", 2, 12)\n    if a == -1:\n        print(\"Cancelled.\")\n        return cfg\n    n = ask(\"clients\", 1, 1000)\n    if n == -1:\n        print(\"Cancelled.\")\n        return cfg\n    o = ask(\"outage in ms\", 0, 600000)\n    if o == -1:\n        print(\"Cancelled.\")\n        return cfg\n    s = ask(\"seed\", 0, 999999)\n    if s == -1:\n        print(\"Cancelled.\")\n        return cfg\n    print(\"First wait \" + str(b) + \" ms, longest \" + str(c) + \" ms, \" + str(a) + \" attempts, \" + str(n) + \" clients, outage \" + str(o) + \" ms, seed \" + str(s) + \".\")\n    return [b, c, a, n, o, s, cfg[6]]\n\ncfg = [1000, 16000, 6, 200, 10000, 7, 100]\nprint(\"== Retry planner ==\")\nprint(\"Waits between retries under five policies, a crowd coming back after an outage, and a job runner whose ledger checks itself.\")\nrunning = True\nwhile running:\n    print(\"\")\n    print(\"1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\")\n    choice = plain(input(\"choice> \"))\n    if choice == \"1\":\n        one_client(cfg)\n    elif choice == \"2\":\n        crowd(cfg)\n    elif choice == \"3\":\n        histogram(cfg)\n    elif choice == \"4\":\n        jobs(cfg)\n    elif choice == \"5\":\n        cfg = settings(cfg)\n    elif choice == \"6\":\n        running = False\n    else:\n        print(\"Pick a number from 1 to 6.\")\nprint(\"Bye.\")\n"
    },
    {
      "name": "retry.eml",
      "eml": "# P107 retry planner - waits between retries, in whole milliseconds:\n#   fixed         the base every time\n#   exponential   base x 2^(k-1) for retry k, up to the cap\n#   full jitter   a random wait from 0 to the exponential one\n#   equal jitter  half the exponential one, plus a random wait up to the\n#                 other half\n#   decorrelated  a random wait from the base to three times the last one,\n#                 up to the cap\n# As in the corpus case backoff-thundering-herd, a policy that gives every\n# client the same waits sends them back all at once; only a wait that\n# depends on the client spreads them. A job runner with a ledger, as in the\n# corpus case retry-with-backoff-ledger, shows every attempt and checks\n# that its finally ran on every one.\n#\n# cfg is [base, cap, attempts, clients, outage, seed, bucket].\n\ndef next_random(state, lo, hi):\n    # a whole number from lo to hi, from a linear congruential generator\n    # (top 16 bits); state is a one-element list\n    (1664525 * state[0] + 1013904223) % 4294967296 => state[0]\n    return lo + int((state[0] - state[0] % 65536) / 65536) % (hi - lo + 1)\n\ndef exponential(cfg, k):\n    cfg[0] => w\n    for i in [2:k]:\n        if w < cfg[1]:\n            w * 2 => w\n    if w > cfg[1]:\n        cfg[1] => w\n    return w\n\ndef waits(policy, cfg, client):\n    # the waits before retries 1 .. attempts - 1\n    [cfg[5] * 1000 + client] => state\n    [] => out\n    cfg[0] => last\n    for k in [1:cfg[2] - 1]:\n        exponential(cfg, k) => e\n        if policy == \"fixed\":\n            cfg[0] => w\n        elif policy == \"exponential\":\n            e => w\n        elif policy == \"full jitter\":\n            next_random(state, 0, e) => w\n        elif policy == \"equal jitter\":\n            int((e - e % 2) / 2) => half\n            half + next_random(state, 0, e - half) => w\n        else:\n            next_random(state, cfg[0], last * 3) => w\n            if w > cfg[1]:\n                cfg[1] => w\n            w => last\n        out + [w] => out\n    return out\n\ndef herd(policy, cfg):\n    # every client fails at time 0 when the outage starts; a retry at or\n    # after the outage's end gets through. [bucket counts, retries,\n    # through, last through, gave up]\n    [] => counts\n    0 => retries\n    0 => through\n    -1 => last\n    0 => gave_up\n    for c in [1:cfg[3]]:\n        0 => t\n        False => ok\n        for w in waits(policy, cfg, c):\n            if not ok:\n                t + w => t\n                retries + 1 => retries\n                int((t - t % cfg[6]) / cfg[6]) => b\n                while len(counts) <= b:\n                    counts + [0] => counts\n                counts[b] + 1 => counts[b]\n                if t >= cfg[4]:\n                    True => ok\n                    through + 1 => through\n                    if t > last:\n                        t => last\n        if not ok:\n            gave_up + 1 => gave_up\n    return [counts, retries, through, last, gave_up]\n\ndef peak(counts):\n    # [most retries in one bucket, the first bucket with that many]\n    0 => most\n    -1 => at\n    for i in [0:len(counts) - 1]:\n        if counts[i] > most:\n            counts[i] => most\n            i => at\n    return [most, at]\n\nclass Ledger:\n    def __init__(self):\n        [] => self.log\n        0 => self.attempts\n        0 => self.successes\n        0 => self.failures\n        0 => self.finally_runs\n        0 => self.waited\n\n    def record(self, entry):\n        self.log + [entry] => self.log\n\ndef attempt(name, n, fails, fatal):\n    # the n-th call of a job that fails `fails` times, or for good\n    if fatal:\n        raise TypeError(name + \": cannot be retried\")\n    if n <= fails:\n        raise ValueError(name + \": failed on attempt \" + str(n))\n    return name + \": done on attempt \" + str(n)\n\ndef run_job(ledger, cfg, name, fails, fatal):\n    0 => n\n    while n < cfg[2]:\n        n + 1 => n\n        ledger.attempts + 1 => ledger.attempts\n        try:\n            attempt(name, n, fails, fatal) => result\n            ledger.successes + 1 => ledger.successes\n            ledger.record(name + \" attempt \" + str(n) + \": done\")\n            return result\n        except TypeError as e:\n            ledger.failures + 1 => ledger.failures\n            ledger.record(name + \" attempt \" + str(n) + \": cannot be retried, stopped\")\n            raise\n        except ValueError as e:\n            ledger.failures + 1 => ledger.failures\n            if n < cfg[2]:\n                exponential(cfg, n) => w\n                ledger.waited + w => ledger.waited\n                ledger.record(name + \" attempt \" + str(n) + \": failed, retry after \" + str(w) + \" ms\")\n            else:\n                ledger.record(name + \" attempt \" + str(n) + \": failed, no attempts left\")\n        finally:\n            ledger.finally_runs + 1 => ledger.finally_runs\n    raise ValueError(name + \": gave up after \" + str(cfg[2]) + \" attempts\")\n",
      "python": "def next_random(state, lo, hi):\n    state[0] = (1664525 * state[0] + 1013904223) % 4294967296\n    return lo + int((state[0] - state[0] % 65536) / 65536) % (hi - lo + 1)\n\ndef exponential(cfg, k):\n    w = cfg[0]\n    for i in range(2, k+1):\n        if w < cfg[1]:\n            w = w * 2\n    if w > cfg[1]:\n        w = cfg[1]\n    return w\n\ndef waits(policy, cfg, client):\n    state = [cfg[5] * 1000 + client]\n    out = []\n    last = cfg[0]\n    for k in range(1, cfg[2]):\n        e = exponential(cfg, k)\n        if policy == \"fixed\":\n            w = cfg[0]\n        elif policy == \"exponential\":\n            w = e\n        elif policy == \"full jitter\":\n            w = next_random(state, 0, e)\n        elif policy == \"equal jitter\":\n            half = int((e - e % 2) / 2)\n            w = half + next_random(state, 0, e - half)\n        else:\n            w = next_random(state, cfg[0], last * 3)\n            if w > cfg[1]:\n                w = cfg[1]\n            last = w\n        out = out + [w]\n    return out\n\ndef herd(policy, cfg):\n    counts = []\n    retries = 0\n    through = 0\n    last = -1\n    gave_up = 0\n    for c in range(1, cfg[3]+1):\n        t = 0\n        ok = False\n        for w in waits(policy, cfg, c):\n            if not ok:\n                t = t + w\n                retries = retries + 1\n                b = int((t - t % cfg[6]) / cfg[6])\n                while len(counts) <= b:\n                    counts = counts + [0]\n                counts[b] = counts[b] + 1\n                if t >= cfg[4]:\n                    ok = True\n                    through = through + 1\n                    if t > last:\n                        last = t\n        if not ok:\n            gave_up = gave_up + 1\n    return [counts, retries, through, last, gave_up]\n\ndef peak(counts):\n    most = 0\n    at = -1\n    for i in range(0, len(counts)):\n        if counts[i] > most:\n            most = counts[i]\n            at = i\n    return [most, at]\n\nclass Ledger:\n    def __init__(self):\n        self.log = []\n        self.attempts = 0\n        self.successes = 0\n        self.failures = 0\n        self.finally_runs = 0\n        self.waited = 0\n    def record(self, entry):\n        self.log = self.log + [entry]\n\ndef attempt(name, n, fails, fatal):\n    if fatal:\n        raise TypeError(name + \": cannot be retried\")\n    if n <= fails:\n        raise ValueError(name + \": failed on attempt \" + str(n))\n    return name + \": done on attempt \" + str(n)\n\ndef run_job(ledger, cfg, name, fails, fatal):\n    n = 0\n    while n < cfg[2]:\n        n = n + 1\n        ledger.attempts = ledger.attempts + 1\n        try:\n            result = attempt(name, n, fails, fatal)\n            ledger.successes = ledger.successes + 1\n            ledger.record(name + \" attempt \" + str(n) + \": done\")\n            return result\n        except TypeError as e:\n            ledger.failures = ledger.failures + 1\n            ledger.record(name + \" attempt \" + str(n) + \": cannot be retried, stopped\")\n            raise\n        except ValueError as e:\n            ledger.failures = ledger.failures + 1\n            if n < cfg[2]:\n                w = exponential(cfg, n)\n                ledger.waited = ledger.waited + w\n                ledger.record(name + \" attempt \" + str(n) + \": failed, retry after \" + str(w) + \" ms\")\n            else:\n                ledger.record(name + \" attempt \" + str(n) + \": failed, no attempts left\")\n        finally:\n            ledger.finally_runs = ledger.finally_runs + 1\n    raise ValueError(name + \": gave up after \" + str(cfg[2]) + \" attempts\")\n"
    }
  ],
  "sessions": [
    {
      "name": "bad-input",
      "input": "0\nx\n1\n\n1\nslow\n1\nexponential\n0\n201\n\n3\n\n4\n\n4\nalpha\n4\nalpha:x\n4\n:3\n4\na:1 b:1 c:1 d:1 e:1 f:1 g:1 h:1 i:1\n5\n0\n10001\n500\n499\n\n5\n1000\n16000\n1\n\n6\n",
      "screen": "== Retry planner ==\nWaits between retries under five policies, a crowd coming back after an outage, and a job runner whose ledger checks itself.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 0\nPick a number from 1 to 6.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> x\nPick a number from 1 to 6.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> slow\nType fixed, exponential, full jitter, equal jitter or decorrelated.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> exponential\nclient number (1 to 200)> 0\nType a whole number from 1 to 200.\nclient number (1 to 200)> 201\nType a whole number from 1 to 200.\nclient number (1 to 200)> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 3\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> alpha\nType each job as name:number or name:fatal.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> alpha:x\nType each job as name:number or name:fatal.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> :3\nType each job as name:number or name:fatal.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> a:1 b:1 c:1 d:1 e:1 f:1 g:1 h:1 i:1\nUp to 8 jobs.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 5\nfirst wait in ms (1 to 10000)> 0\nType a whole number from 1 to 10000.\nfirst wait in ms (1 to 10000)> 10001\nType a whole number from 1 to 10000.\nfirst wait in ms (1 to 10000)> 500\nlongest wait in ms (500 to 600000)> 499\nType a whole number from 500 to 600000.\nlongest wait in ms (500 to 600000)> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 5\nfirst wait in ms (1 to 10000)> 1000\nlongest wait in ms (1000 to 600000)> 16000\nattempts per client (2 to 12)> 1\nType a whole number from 2 to 12.\nattempts per client (2 to 12)> \nCancelled.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 6\nBye.\n",
      "interpreter": "equal"
    },
    {
      "name": "basic",
      "input": "1\nexponential\n1\n1\nfull jitter\n1\n1\nfull jitter\n2\n1\ndecorrelated\n1\n2\n3\nexponential\n3\nequal jitter\n4\nalpha:0 bravo:2 charlie:9 delta:fatal\n5\n100\n16000\n6\n200\n10000\n7\n2\n6\n",
      "screen": "== Retry planner ==\nWaits between retries under five policies, a crowd coming back after an outage, and a job runner whose ledger checks itself.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> exponential\nclient number (1 to 200)> 1\nClient 1, exponential: attempt 1 at 0 ms, then\n  attempt  2  after   1000 ms, at    1000 ms\n  attempt  3  after   2000 ms, at    3000 ms\n  attempt  4  after   4000 ms, at    7000 ms\n  attempt  5  after   8000 ms, at   15000 ms\n  attempt  6  after  16000 ms, at   31000 ms\nIn all 31000 ms before the last attempt.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> full jitter\nclient number (1 to 200)> 1\nClient 1, full jitter: attempt 1 at 0 ms, then\n  attempt  2  after    152 ms, at     152 ms\n  attempt  3  after   1365 ms, at    1517 ms\n  attempt  4  after   2280 ms, at    3797 ms\n  attempt  5  after   5697 ms, at    9494 ms\n  attempt  6  after   4600 ms, at   14094 ms\nIn all 14094 ms before the last attempt.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> full jitter\nclient number (1 to 200)> 2\nClient 2, full jitter: attempt 1 at 0 ms, then\n  attempt  2  after    178 ms, at     178 ms\n  attempt  3  after   1306 ms, at    1484 ms\n  attempt  4  after   1622 ms, at    3106 ms\n  attempt  5  after    122 ms, at    3228 ms\n  attempt  6  after    274 ms, at    3502 ms\nIn all 3502 ms before the last attempt.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 1\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> decorrelated\nclient number (1 to 200)> 1\nClient 1, decorrelated: attempt 1 at 0 ms, then\n  attempt  2  after   1183 ms, at    1183 ms\n  attempt  3  after   2524 ms, at    3707 ms\n  attempt  4  after   4995 ms, at    8702 ms\n  attempt  5  after  12745 ms, at   21447 ms\n  attempt  6  after  16000 ms, at   37447 ms\nIn all 37447 ms before the last attempt.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 2\n200 clients fail at 0 ms; the service is back at 10000 ms; each tries 6 times at most. Retries counted per 100 ms:\n  policy          retries   most in a slot   the slot from   all through by   gave up\n  fixed              1000              200         1000 ms                -       200\n  exponential         800              200         1000 ms         15000 ms         0\n  full jitter         962               28          400 ms         24563 ms        36\n  equal jitter        848               44          600 ms         25666 ms         0\n  decorrelated        770               15         2400 ms         25330 ms        19\nFixed and exponential waits are the same for every client, so each round of retries lands in a single slot. A wait that depends on the client spreads each round over the whole wait, which lowers the busiest slot only when the waits are long against the slot: here 28, 44 and 15 against 200.\nFull and equal jitter wait on average about a half and three quarters of the exponential wait, so with the same attempts 36 and 0 clients gave up, against 0 with exponential waits.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 3\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> exponential\nexponential: retries in each second, # for every 5:\n     1000 ms   200  ########################################\n     3000 ms   200  ########################################\n     7000 ms   200  ########################################\n    15000 ms   200  ########################################\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 3\npolicy (fixed, exponential, full jitter, equal jitter or decorrelated)> equal jitter\nequal jitter: retries in each second, # for every 5:\n        0 ms   200  ########################################\n     1000 ms    53  ###########\n     2000 ms   147  ##############################\n     3000 ms     4  #\n     4000 ms    72  ###############\n     5000 ms    96  ####################\n     6000 ms    28  ######\n     8000 ms    11  ###\n     9000 ms    37  ########\n    10000 ms    36  ########\n    11000 ms    50  ##########\n    12000 ms    42  #########\n    13000 ms    21  #####\n    14000 ms     3  #\n    16000 ms     1  #\n    17000 ms     4  #\n    18000 ms     8  ##\n    19000 ms     6  ##\n    20000 ms     7  ##\n    21000 ms     4  #\n    22000 ms     2  #\n    23000 ms     5  #\n    24000 ms     6  ##\n    25000 ms     5  #\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 4\nJobs as name:failures, or name:fatal for a job that cannot be retried, such as alpha:0 bravo:2 charlie:9 delta:fatal.\njobs> alpha:0 bravo:2 charlie:9 delta:fatal\n  alpha: done on attempt 1\n  bravo: done on attempt 3\n  charlie: gave up after 6 attempts\n  delta: cannot be retried\nLedger:\n  alpha attempt 1: done\n  bravo attempt 1: failed, retry after 1000 ms\n  bravo attempt 2: failed, retry after 2000 ms\n  bravo attempt 3: done\n  charlie attempt 1: failed, retry after 1000 ms\n  charlie attempt 2: failed, retry after 2000 ms\n  charlie attempt 3: failed, retry after 4000 ms\n  charlie attempt 4: failed, retry after 8000 ms\n  charlie attempt 5: failed, retry after 16000 ms\n  charlie attempt 6: failed, no attempts left\n  delta attempt 1: cannot be retried, stopped\nAttempts 11: 2 done, 9 failed. The finally block ran 11 times. Waited 34000 ms in all.\nJobs done, given up, refused: 2, 1, 1.\nEvery attempt was counted once, and finally ran on each - after a return, after a raise, and after a failure alike.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 5\nfirst wait in ms (1 to 10000)> 100\nlongest wait in ms (100 to 600000)> 16000\nattempts per client (2 to 12)> 6\nclients (1 to 1000)> 200\noutage in ms (0 to 600000)> 10000\nseed (0 to 999999)> 7\nFirst wait 100 ms, longest 16000 ms, 6 attempts, 200 clients, outage 10000 ms, seed 7.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 2\n200 clients fail at 0 ms; the service is back at 10000 ms; each tries 6 times at most. Retries counted per 100 ms:\n  policy          retries   most in a slot   the slot from   all through by   gave up\n  fixed              1000              200          100 ms                -       200\n  exponential        1000              200          100 ms                -       200\n  full jitter        1000              250            0 ms                -       200\n  equal jitter       1000              195            0 ms                -       200\n  decorrelated       1000              119          200 ms         20286 ms       192\nFixed and exponential waits are the same for every client, so each round of retries lands in a single slot. A wait that depends on the client spreads each round over the whole wait, which lowers the busiest slot only when the waits are long against the slot: here 250, 195 and 119 against 200.\n\n1) one client  2) the crowd  3) retries over time  4) jobs with a ledger  5) settings  6) quit\nchoice> 6\nBye.\n",
      "interpreter": "equal"
    }
  ],
  "builtOn": [
    {
      "slug": "backoff-thundering-herd",
      "caseId": "247-backoff-thundering-herd",
      "title": "Backoff and the thundering herd — a policy that synchronises load"
    },
    {
      "slug": "retry-with-backoff-ledger",
      "caseId": "195-retry-with-backoff-ledger",
      "title": "Retry with backoff, and the ledger that audits it"
    }
  ],
  "updated": "2026-10-16"
}