{
  "id": "P118",
  "slug": "cache-stampede",
  "key": "P118-cache-stampede",
  "title": "Cache stampede",
  "summary": "One hot key with a time to live: naive, single flight, early refresh and serve stale, tick by tick and side by side, counting fetches, waiting and stale answers as the traffic grows.",
  "entry": "main.eml",
  "ui": "terminal",
  "readme": "# P118 - Cache stampede\n\nOne hot key in a cache with a time to live (TTL). When the value expires,\nevery request that misses can start a fetch of its own from the origin,\nand a hot key has the most requests to miss.\n\n- `main.eml` - the menu, the settings, one policy tick by tick, the four\n  policies side by side, and the fetches as the traffic grows\n- `stampede.eml` - the four policies over the ticks, and the gap at which\n  naive and single flight run alike\n\nHow each part works:\n\n- Requests arrive a few at a time, every so many ticks, for 40 ticks. A\n  fetch takes a set number of ticks; when it lands, the value is fresh\n  for the TTL. The cache starts empty.\n- The four policies, as in the corpus case `cache-stampede-single-flight`:\n  - naive: every miss starts its own fetch and waits for it;\n  - single flight: a miss joins the fetch already running;\n  - early refresh: like single flight, and a hit near the end of the TTL\n    starts a fetch before the value expires;\n  - serve stale: a miss gets the expired value while one fetch runs; only\n    with no value at all does a request wait.\n- Each time the value expires, naive starts a fetch for every request\n  until the first one lands, where single flight starts one: at 5\n  requests a tick and a fetch of 4 ticks, 20 fetches against 1, and 20\n  running at once. This holds whenever a request comes at least once a\n  TTL; the program says it only then.\n- With single requests further apart, naive and single flight run alike,\n  tick by tick, from a gap of exactly the fetch time: each fetch lands\n  before the next request. Below that, naive has two or more running.\n- No policy is free: single flight still makes requests wait, early\n  refresh fetches before it must (and with a margin shorter than a fetch,\n  the value expires before the new one lands), and serve stale answers\n  with an expired value.\n\nWhat is checked: menu choices 1 to 7; one of the four policies; 1 to 20\nrequests at a time, every 1 to 9 ticks; a TTL from 1 to 30 ticks, a fetch\nand a margin from 1 to 9, the margin shorter than the TTL. An empty\nanswer cancels.\n\nSessions: `sessions/basic.in` gives:\n- the settings and the four policies: 60 fetches for naive, 3 for single\n  flight;\n- single flight tick by tick, and the fetches as the traffic grows;\n- an early refresh margin of 4, the fetch time: then requests wait only\n  at the start, and early refresh tick by tick;\n- 1 request every 3 ticks, and the four policies again.\n\n`sessions/bad-input.in` gives:\n- menu choices 0 and x;\n- policies: a cancel and fast;\n- traffic: a cancel, 0, 21 and x, then 5 with a cancel, 0 and 10;\n- the cache: a cancel, 0 and 31, then a TTL of 10 with a cancel, 0 and\n  10 for the fetch, then a fetch of 4 with a cancel, 0 and 10 for the\n  margin, and margins of 1 for a TTL of 1 and 5 for a TTL of 3;\n- 2 requests every 6 ticks with a TTL of 3, where the naive sentence is\n  left out.\n\nBuilt on the verified corpus case `cache-stampede-single-flight` (every\nmiss at an expiry recomputes the same value, and one shared fetch removes\nthe duplicates).\n",
  "modules": [
    {
      "name": "main.eml",
      "eml": "# P118 cache stampede: one hot key with a TTL, four policies tick by tick\n# and side by side, and the fetches as the traffic grows.\nimport stampede\n\ndef plain(s):\n    0 => i\n    len(s) => j\n    while i < j and s[i] == \" \":\n        i + 1 => i\n    while j > i and s[j - 1] == \" \":\n        j - 1 => j\n    return s[i:j]\n\ndef pad(s, n):\n    s => out\n    while len(out) < n:\n        out + \" \" => out\n    return out\n\ndef lpad(s, n):\n    s => out\n    while len(out) < n:\n        \" \" + out => out\n    return out\n\ndef is_digits(s):\n    if len(s) == 0:\n        return False\n    for c in s:\n        if c < \"0\" or c > \"9\":\n            return False\n    return True\n\ndef ticks_text(n):\n    if n == 1:\n        return \"1 tick\"\n    return str(n) + \" ticks\"\n\ndef requests_text(n):\n    if n == 1:\n        return \"1 request\"\n    return str(n) + \" requests\"\n\ndef show(st):\n    if st[1] == 1:\n        (\"Traffic: \" + requests_text(st[0]) + \" every tick, for \" + str(st[5]) + \" ticks.\") ^0\n    else:\n        (\"Traffic: \" + requests_text(st[0]) + \" every \" + ticks_text(st[1]) + \", for \" + str(st[5]) + \" ticks.\") ^0\n    (\"Cache: a TTL of \" + ticks_text(st[2]) + \", a fetch of \" + ticks_text(st[3]) + \", early refresh within \" + ticks_text(st[4]) + \" of the end of the TTL.\") ^0\n    \"The cache starts empty, and a fetch that lands makes the value fresh for the TTL.\" ^0\n\ndef tick_by_tick(st):\n    plain(input(\"policy (naive, single flight, early refresh or serve stale)> \")) => p\n    if p == \"\":\n        \"Cancelled.\" ^0\n        return\n    False => known\n    for x in stampede.policies():\n        if x == p:\n            True => known\n    if not known:\n        \"Type naive, single flight, early refresh or serve stale.\" ^0\n        return\n    stampede.run(p, st[0], st[1], st[2], st[3], st[4], st[5]) => res\n    (\"Tick by tick, \" + p + \":\") ^0\n    \"  tick  requests  hits  waited  stale  started  running\" ^0\n    for row in res[0]:\n        (\"  \" + lpad(str(row[0]), 4) + lpad(str(row[1]), 10) + lpad(str(row[2]), 6) + lpad(str(row[3]), 8) + lpad(str(row[4]), 7) + lpad(str(row[5]), 9) + lpad(str(row[6]), 9)) ^0\n    totals_line(p, res[1])\n\ndef totals_line(p, x):\n    (p + \": \" + str(x[0]) + \" fetches, at most \" + str(x[1]) + \" running at once; \" + str(x[2]) + \" requests waited, \" + str(x[3]) + \" ticks in all; \" + str(x[4]) + \" served stale.\") ^0\n\ndef compare(st):\n    (\"Over \" + str(st[5]) + \" ticks:\") ^0\n    \"  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\" ^0\n    [] => results\n    for p in stampede.policies():\n        stampede.run(p, st[0], st[1], st[2], st[3], st[4], st[5])[1] => x\n        results + [x] => results\n        (\"  \" + pad(p, 13) + lpad(str(x[0]), 9) + lpad(str(x[1]), 14) + lpad(str(x[2]), 8) + lpad(str(x[3]), 14) + lpad(str(x[4]), 14) + lpad(str(x[5]), 15)) ^0\n    \"  most past TTL: how many ticks the oldest stale answer had been expired.\" ^0\n    (\"Naive starts \" + str(results[0][0]) + \" fetches where single flight starts \" + str(results[1][0]) + \", with up to \" + str(results[0][1]) + \" running at once.\") ^0\n    0 => burst\n    for t in [0:st[3] - 1]:\n        if t % st[1] == 0:\n            burst + st[0] => burst\n    if st[1] <= st[2]:\n        (\"Each time the value expires, naive starts a fetch for every request until the first one lands: \" + str(burst) + \" fetches, where single flight starts 1.\") ^0\n    if st[4] < st[3]:\n        (\"Early refresh starts at most \" + ticks_text(st[4]) + \" before the end of the TTL, and a fetch takes \" + ticks_text(st[3]) + \", so the value expires before the new one lands.\") ^0\n\ndef more_traffic(st):\n    (\"Fetches in \" + str(st[5]) + \" ticks, with a TTL of \" + ticks_text(st[2]) + \" and a fetch of \" + ticks_text(st[3]) + \":\") ^0\n    \"  requests every tick    naive  at once  single flight\" ^0\n    for r in [1, 2, 5, 10, 20]:\n        stampede.run(\"naive\", r, 1, st[2], st[3], st[4], st[5])[1] => a\n        stampede.run(\"single flight\", r, 1, st[2], st[3], st[4], st[5])[1][0] => b\n        (\"  \" + lpad(str(r), 19) + lpad(str(a[0]), 9) + lpad(str(a[1]), 9) + lpad(str(b), 15)) ^0\n    \"  one request every      naive  at once  single flight\" ^0\n    for g in [1:9]:\n        stampede.run(\"naive\", 1, g, st[2], st[3], st[4], st[5])[1] => a\n        stampede.run(\"single flight\", 1, g, st[2], st[3], st[4], st[5])[1][0] => b\n        (\"  \" + lpad(ticks_text(g), 19) + lpad(str(a[0]), 9) + lpad(str(a[1]), 9) + lpad(str(b), 15)) ^0\n    stampede.agree_gap(st[3], st[2], st[5]) => gap\n    if gap == 0:\n        \"Naive and single flight run differently at every gap up to 9 ticks.\" ^0\n    elif gap == st[3]:\n        (\"Naive and single flight run alike, tick by tick, once requests are \" + ticks_text(gap) + \" apart, the time a fetch takes: then each fetch lands before the next request.\") ^0\n    else:\n        (\"Naive and single flight run alike, tick by tick, once requests are \" + ticks_text(gap) + \" apart.\") ^0\n\ndef ask_number(prompt, low, high):\n    # the number typed, 0 to cancel, -1 when it is not from low to high\n    plain(input(prompt)) => t\n    if t == \"\":\n        \"Cancelled.\" ^0\n        return 0\n    if not is_digits(t) or len(t) > 2 or int(t) < low or int(t) > high:\n        (\"Type a whole number from \" + str(low) + \" to \" + str(high) + \".\") ^0\n        return 0 - 1\n    return int(t)\n\ndef traffic(st):\n    ask_number(\"requests at a time (1 to 20)> \", 1, 20) => r\n    if r <= 0:\n        return st\n    ask_number(\"every (1 to 9 ticks)> \", 1, 9) => g\n    if g <= 0:\n        return st\n    [r, g, st[2], st[3], st[4], st[5]] => out\n    show(out)\n    return out\n\ndef cache(st):\n    ask_number(\"TTL (1 to 30 ticks)> \", 1, 30) => ttl\n    if ttl <= 0:\n        return st\n    ask_number(\"fetch time (1 to 9 ticks)> \", 1, 9) => f\n    if f <= 0:\n        return st\n    ask_number(\"early refresh margin (1 to 9 ticks)> \", 1, 9) => m\n    if m <= 0:\n        return st\n    if m >= ttl:\n        \"The margin must be shorter than the TTL.\" ^0\n        return st\n    [st[0], st[1], ttl, f, m, st[5]] => out\n    show(out)\n    return out\n\n# [requests at a time, every so many ticks, TTL, fetch time, early margin, ticks]\n[5, 1, 10, 4, 2, 40] => st\n\n\"== Cache stampede ==\" ^0\n\"One hot key in a cache with a time to live. When it expires, every request that misses can start a fetch of its own from the origin.\" ^0\nTrue => running\nwhile running:\n    \"\" ^0\n    \"1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\" ^0\n    plain(input(\"choice> \")) => choice\n    if choice == \"1\":\n        show(st)\n    elif choice == \"2\":\n        tick_by_tick(st)\n    elif choice == \"3\":\n        compare(st)\n    elif choice == \"4\":\n        more_traffic(st)\n    elif choice == \"5\":\n        traffic(st) => st\n    elif choice == \"6\":\n        cache(st) => st\n    elif choice == \"7\":\n        False => running\n    else:\n        \"Pick a number from 1 to 7.\" ^0\n\"Bye.\" ^0\n",
      "python": "import stampede\n\ndef plain(s):\n    i = 0\n    j = len(s)\n    while i < j and s[i] == \" \":\n        i = i + 1\n    while j > i and s[j - 1] == \" \":\n        j = j - 1\n    return s[i:j]\n\ndef pad(s, n):\n    out = s\n    while len(out) < n:\n        out = out + \" \"\n    return out\n\ndef lpad(s, n):\n    out = s\n    while len(out) < n:\n        out = \" \" + out\n    return out\n\ndef is_digits(s):\n    if len(s) == 0:\n        return False\n    for c in s:\n        if c < \"0\" or c > \"9\":\n            return False\n    return True\n\ndef ticks_text(n):\n    if n == 1:\n        return \"1 tick\"\n    return str(n) + \" ticks\"\n\ndef requests_text(n):\n    if n == 1:\n        return \"1 request\"\n    return str(n) + \" requests\"\n\ndef show(st):\n    if st[1] == 1:\n        print(\"Traffic: \" + requests_text(st[0]) + \" every tick, for \" + str(st[5]) + \" ticks.\")\n    else:\n        print(\"Traffic: \" + requests_text(st[0]) + \" every \" + ticks_text(st[1]) + \", for \" + str(st[5]) + \" ticks.\")\n    print(\"Cache: a TTL of \" + ticks_text(st[2]) + \", a fetch of \" + ticks_text(st[3]) + \", early refresh within \" + ticks_text(st[4]) + \" of the end of the TTL.\")\n    print(\"The cache starts empty, and a fetch that lands makes the value fresh for the TTL.\")\n\ndef tick_by_tick(st):\n    p = plain(input(\"policy (naive, single flight, early refresh or serve stale)> \"))\n    if p == \"\":\n        print(\"Cancelled.\")\n        return\n    known = False\n    for x in stampede.policies():\n        if x == p:\n            known = True\n    if not known:\n        print(\"Type naive, single flight, early refresh or serve stale.\")\n        return\n    res = stampede.run(p, st[0], st[1], st[2], st[3], st[4], st[5])\n    print(\"Tick by tick, \" + p + \":\")\n    print(\"  tick  requests  hits  waited  stale  started  running\")\n    for row in res[0]:\n        print(\"  \" + lpad(str(row[0]), 4) + lpad(str(row[1]), 10) + lpad(str(row[2]), 6) + lpad(str(row[3]), 8) + lpad(str(row[4]), 7) + lpad(str(row[5]), 9) + lpad(str(row[6]), 9))\n    totals_line(p, res[1])\n\ndef totals_line(p, x):\n    print(p + \": \" + str(x[0]) + \" fetches, at most \" + str(x[1]) + \" running at once; \" + str(x[2]) + \" requests waited, \" + str(x[3]) + \" ticks in all; \" + str(x[4]) + \" served stale.\")\n\ndef compare(st):\n    print(\"Over \" + str(st[5]) + \" ticks:\")\n    print(\"  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\")\n    results = []\n    for p in stampede.policies():\n        x = stampede.run(p, st[0], st[1], st[2], st[3], st[4], st[5])[1]\n        results = results + [x]\n        print(\"  \" + pad(p, 13) + lpad(str(x[0]), 9) + lpad(str(x[1]), 14) + lpad(str(x[2]), 8) + lpad(str(x[3]), 14) + lpad(str(x[4]), 14) + lpad(str(x[5]), 15))\n    print(\"  most past TTL: how many ticks the oldest stale answer had been expired.\")\n    print(\"Naive starts \" + str(results[0][0]) + \" fetches where single flight starts \" + str(results[1][0]) + \", with up to \" + str(results[0][1]) + \" running at once.\")\n    burst = 0\n    for t in range(0, st[3]):\n        if t % st[1] == 0:\n            burst = burst + st[0]\n    if st[1] <= st[2]:\n        print(\"Each time the value expires, naive starts a fetch for every request until the first one lands: \" + str(burst) + \" fetches, where single flight starts 1.\")\n    if st[4] < st[3]:\n        print(\"Early refresh starts at most \" + ticks_text(st[4]) + \" before the end of the TTL, and a fetch takes \" + ticks_text(st[3]) + \", so the value expires before the new one lands.\")\n\ndef more_traffic(st):\n    print(\"Fetches in \" + str(st[5]) + \" ticks, with a TTL of \" + ticks_text(st[2]) + \" and a fetch of \" + ticks_text(st[3]) + \":\")\n    print(\"  requests every tick    naive  at once  single flight\")\n    for r in [1, 2, 5, 10, 20]:\n        a = stampede.run(\"naive\", r, 1, st[2], st[3], st[4], st[5])[1]\n        b = stampede.run(\"single flight\", r, 1, st[2], st[3], st[4], st[5])[1][0]\n        print(\"  \" + lpad(str(r), 19) + lpad(str(a[0]), 9) + lpad(str(a[1]), 9) + lpad(str(b), 15))\n    print(\"  one request every      naive  at once  single flight\")\n    for g in range(1, 10):\n        a = stampede.run(\"naive\", 1, g, st[2], st[3], st[4], st[5])[1]\n        b = stampede.run(\"single flight\", 1, g, st[2], st[3], st[4], st[5])[1][0]\n        print(\"  \" + lpad(ticks_text(g), 19) + lpad(str(a[0]), 9) + lpad(str(a[1]), 9) + lpad(str(b), 15))\n    gap = stampede.agree_gap(st[3], st[2], st[5])\n    if gap == 0:\n        print(\"Naive and single flight run differently at every gap up to 9 ticks.\")\n    elif gap == st[3]:\n        print(\"Naive and single flight run alike, tick by tick, once requests are \" + ticks_text(gap) + \" apart, the time a fetch takes: then each fetch lands before the next request.\")\n    else:\n        print(\"Naive and single flight run alike, tick by tick, once requests are \" + ticks_text(gap) + \" apart.\")\n\ndef ask_number(prompt, low, high):\n    t = plain(input(prompt))\n    if t == \"\":\n        print(\"Cancelled.\")\n        return 0\n    if not is_digits(t) or len(t) > 2 or int(t) < low or int(t) > high:\n        print(\"Type a whole number from \" + str(low) + \" to \" + str(high) + \".\")\n        return 0 - 1\n    return int(t)\n\ndef traffic(st):\n    r = ask_number(\"requests at a time (1 to 20)> \", 1, 20)\n    if r <= 0:\n        return st\n    g = ask_number(\"every (1 to 9 ticks)> \", 1, 9)\n    if g <= 0:\n        return st\n    out = [r, g, st[2], st[3], st[4], st[5]]\n    show(out)\n    return out\n\ndef cache(st):\n    ttl = ask_number(\"TTL (1 to 30 ticks)> \", 1, 30)\n    if ttl <= 0:\n        return st\n    f = ask_number(\"fetch time (1 to 9 ticks)> \", 1, 9)\n    if f <= 0:\n        return st\n    m = ask_number(\"early refresh margin (1 to 9 ticks)> \", 1, 9)\n    if m <= 0:\n        return st\n    if m >= ttl:\n        print(\"The margin must be shorter than the TTL.\")\n        return st\n    out = [st[0], st[1], ttl, f, m, st[5]]\n    show(out)\n    return out\n\nst = [5, 1, 10, 4, 2, 40]\nprint(\"== Cache stampede ==\")\nprint(\"One hot key in a cache with a time to live. When it expires, every request that misses can start a fetch of its own from the origin.\")\nrunning = True\nwhile running:\n    print(\"\")\n    print(\"1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\")\n    choice = plain(input(\"choice> \"))\n    if choice == \"1\":\n        show(st)\n    elif choice == \"2\":\n        tick_by_tick(st)\n    elif choice == \"3\":\n        compare(st)\n    elif choice == \"4\":\n        more_traffic(st)\n    elif choice == \"5\":\n        st = traffic(st)\n    elif choice == \"6\":\n        st = cache(st)\n    elif choice == \"7\":\n        running = False\n    else:\n        print(\"Pick a number from 1 to 7.\")\nprint(\"Bye.\")\n"
    },
    {
      "name": "stampede.eml",
      "eml": "# P118 cache stampede - one hot key in a cache with a time to live (TTL).\n# Requests arrive every few ticks; a fetch from the origin takes a few\n# ticks, and when it lands the value is fresh for the TTL. As in the corpus\n# case cache-stampede-single-flight, every request that misses while the\n# value is being fetched starts a fetch of its own, unless the cache lets\n# one fetch serve them all.\n#\n# The policies:\n#   naive          every miss starts its own fetch and waits for it;\n#   single flight  a miss joins the fetch already running, if there is one;\n#   early refresh  like single flight, and a hit within the margin of the\n#                  end of the TTL starts a fetch before the value expires;\n#   serve stale    a miss is served the expired value while one fetch runs;\n#                  only with no value at all does a request wait.\n\ndef policies():\n    return [\"naive\", \"single flight\", \"early refresh\", \"serve stale\"]\n\ndef run(policy, r, g, ttl, fetch, margin, ticks):\n    # [rows, totals]: a row per tick [tick, requests, hits, waited, stale,\n    # fetches started, fetches running]; totals [fetches, most running at\n    # once, requests that waited, ticks waited in all, served stale, most\n    # ticks past the TTL]\n    False => has_value\n    0 => fresh_until\n    [] => flights\n    0 => fetches\n    0 => most\n    0 => waited\n    0 => wait_total\n    0 => stale\n    0 => past\n    [] => rows\n    for t in [0:ticks - 1]:\n        [] => left\n        for d in flights:\n            if d <= t:\n                True => has_value\n                if d + ttl > fresh_until:\n                    d + ttl => fresh_until\n            else:\n                left + [d] => left\n        left => flights\n        0 => n\n        0 => hits\n        0 => w\n        0 => s\n        0 => started\n        if t % g == 0:\n            r => n\n        for q in [1:n]:\n            if has_value and t < fresh_until:\n                hits + 1 => hits\n                if policy == \"early refresh\" and len(flights) == 0 and fresh_until - t <= margin:\n                    flights + [t + fetch] => flights\n                    started + 1 => started\n            elif policy == \"naive\":\n                flights + [t + fetch] => flights\n                started + 1 => started\n                w + 1 => w\n                wait_total + fetch => wait_total\n            elif policy == \"serve stale\" and has_value:\n                s + 1 => s\n                if t - fresh_until > past:\n                    t - fresh_until => past\n                if len(flights) == 0:\n                    flights + [t + fetch] => flights\n                    started + 1 => started\n            else:\n                if len(flights) == 0:\n                    flights + [t + fetch] => flights\n                    started + 1 => started\n                w + 1 => w\n                wait_total + flights[0] - t => wait_total\n        fetches + started => fetches\n        waited + w => waited\n        stale + s => stale\n        if len(flights) > most:\n            len(flights) => most\n        rows + [[t, n, hits, w, s, started, len(flights)]] => rows\n    return [rows, [fetches, most, waited, wait_total, stale, past]]\n\ndef agree_gap(fetch, ttl, ticks):\n    # the smallest gap between single requests from which naive and single\n    # flight run alike, tick by tick, at every larger gap to 9; 0 if none\n    0 => found\n    for g in [1:9]:\n        run(\"naive\", 1, g, ttl, fetch, 1, ticks)[0] == run(\"single flight\", 1, g, ttl, fetch, 1, ticks)[0] => same\n        if same and found == 0:\n            g => found\n        if not same:\n            0 => found\n    return found\n",
      "python": "def policies():\n    return [\"naive\", \"single flight\", \"early refresh\", \"serve stale\"]\n\ndef run(policy, r, g, ttl, fetch, margin, ticks):\n    has_value = False\n    fresh_until = 0\n    flights = []\n    fetches = 0\n    most = 0\n    waited = 0\n    wait_total = 0\n    stale = 0\n    past = 0\n    rows = []\n    for t in range(0, ticks):\n        left = []\n        for d in flights:\n            if d <= t:\n                has_value = True\n                if d + ttl > fresh_until:\n                    fresh_until = d + ttl\n            else:\n                left = left + [d]\n        flights = left\n        n = 0\n        hits = 0\n        w = 0\n        s = 0\n        started = 0\n        if t % g == 0:\n            n = r\n        for q in range(1, n+1):\n            if has_value and t < fresh_until:\n                hits = hits + 1\n                if policy == \"early refresh\" and len(flights) == 0 and fresh_until - t <= margin:\n                    flights = flights + [t + fetch]\n                    started = started + 1\n            elif policy == \"naive\":\n                flights = flights + [t + fetch]\n                started = started + 1\n                w = w + 1\n                wait_total = wait_total + fetch\n            elif policy == \"serve stale\" and has_value:\n                s = s + 1\n                if t - fresh_until > past:\n                    past = t - fresh_until\n                if len(flights) == 0:\n                    flights = flights + [t + fetch]\n                    started = started + 1\n            else:\n                if len(flights) == 0:\n                    flights = flights + [t + fetch]\n                    started = started + 1\n                w = w + 1\n                wait_total = wait_total + flights[0] - t\n        fetches = fetches + started\n        waited = waited + w\n        stale = stale + s\n        if len(flights) > most:\n            most = len(flights)\n        rows = rows + [[t, n, hits, w, s, started, len(flights)]]\n    return [rows, [fetches, most, waited, wait_total, stale, past]]\n\ndef agree_gap(fetch, ttl, ticks):\n    found = 0\n    for g in range(1, 10):\n        same = run(\"naive\", 1, g, ttl, fetch, 1, ticks)[0] == run(\"single flight\", 1, g, ttl, fetch, 1, ticks)[0]\n        if same and found == 0:\n            found = g\n        if not same:\n            found = 0\n    return found\n"
    }
  ],
  "sessions": [
    {
      "name": "bad-input",
      "input": "0\nx\n2\n\n2\nfast\n5\n\n5\n0\n5\n21\n5\nx\n5\n5\n\n5\n5\n0\n5\n5\n10\n6\n\n6\n0\n6\n31\n6\n10\n\n6\n10\n0\n6\n10\n10\n6\n10\n4\n\n6\n10\n4\n0\n6\n10\n4\n10\n6\n1\n4\n1\n6\n3\n4\n5\n5\n2\n6\n6\n3\n4\n2\n3\n7\n",
      "screen": "== Cache stampede ==\nOne hot key in a cache with a time to live. When it expires, every request that misses can start a fetch of its own from the origin.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 0\nPick a number from 1 to 7.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> x\nPick a number from 1 to 7.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 2\npolicy (naive, single flight, early refresh or serve stale)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 2\npolicy (naive, single flight, early refresh or serve stale)> fast\nType naive, single flight, early refresh or serve stale.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 0\nType a whole number from 1 to 20.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 21\nType a whole number from 1 to 20.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> x\nType a whole number from 1 to 20.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 5\nevery (1 to 9 ticks)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 5\nevery (1 to 9 ticks)> 0\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 5\nevery (1 to 9 ticks)> 10\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 0\nType a whole number from 1 to 30.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 31\nType a whole number from 1 to 30.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 0\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 10\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> \nCancelled.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 0\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 10\nType a whole number from 1 to 9.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 1\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 1\nThe margin must be shorter than the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 3\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 5\nThe margin must be shorter than the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 2\nevery (1 to 9 ticks)> 6\nTraffic: 2 requests every 6 ticks, for 40 ticks.\nCache: a TTL of 10 ticks, a fetch of 4 ticks, early refresh within 2 ticks of the end of the TTL.\nThe cache starts empty, and a fetch that lands makes the value fresh for the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 3\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 2\nTraffic: 2 requests every 6 ticks, for 40 ticks.\nCache: a TTL of 3 ticks, a fetch of 4 ticks, early refresh within 2 ticks of the end of the TTL.\nThe cache starts empty, and a fetch that lands makes the value fresh for the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 3\nOver 40 ticks:\n  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\n  naive                8             2       8            32             0              0\n  single flight        4             1       8            32             0              0\n  early refresh        7             1       2             8             0              0\n  serve stale          4             1       2             8             6              5\n  most past TTL: how many ticks the oldest stale answer had been expired.\nNaive starts 8 fetches where single flight starts 4, with up to 2 running at once.\nEarly refresh starts at most 2 ticks before the end of the TTL, and a fetch takes 4 ticks, so the value expires before the new one lands.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 7\nBye.\n",
      "interpreter": "equal"
    },
    {
      "name": "basic",
      "input": "1\n3\n2\nsingle flight\n4\n6\n10\n4\n4\n3\n2\nearly refresh\n5\n1\n3\n3\n7\n",
      "screen": "== Cache stampede ==\nOne hot key in a cache with a time to live. When it expires, every request that misses can start a fetch of its own from the origin.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 1\nTraffic: 5 requests every tick, for 40 ticks.\nCache: a TTL of 10 ticks, a fetch of 4 ticks, early refresh within 2 ticks of the end of the TTL.\nThe cache starts empty, and a fetch that lands makes the value fresh for the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 3\nOver 40 ticks:\n  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\n  naive               60            20      60           240             0              0\n  single flight        3             1      60           150             0              0\n  early refresh        4             1      50            95             0              0\n  serve stale          3             1      20            50            40              3\n  most past TTL: how many ticks the oldest stale answer had been expired.\nNaive starts 60 fetches where single flight starts 3, with up to 20 running at once.\nEach time the value expires, naive starts a fetch for every request until the first one lands: 20 fetches, where single flight starts 1.\nEarly refresh starts at most 2 ticks before the end of the TTL, and a fetch takes 4 ticks, so the value expires before the new one lands.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 2\npolicy (naive, single flight, early refresh or serve stale)> single flight\nTick by tick, single flight:\n  tick  requests  hits  waited  stale  started  running\n     0         5     0       5      0        1        1\n     1         5     0       5      0        0        1\n     2         5     0       5      0        0        1\n     3         5     0       5      0        0        1\n     4         5     5       0      0        0        0\n     5         5     5       0      0        0        0\n     6         5     5       0      0        0        0\n     7         5     5       0      0        0        0\n     8         5     5       0      0        0        0\n     9         5     5       0      0        0        0\n    10         5     5       0      0        0        0\n    11         5     5       0      0        0        0\n    12         5     5       0      0        0        0\n    13         5     5       0      0        0        0\n    14         5     0       5      0        1        1\n    15         5     0       5      0        0        1\n    16         5     0       5      0        0        1\n    17         5     0       5      0        0        1\n    18         5     5       0      0        0        0\n    19         5     5       0      0        0        0\n    20         5     5       0      0        0        0\n    21         5     5       0      0        0        0\n    22         5     5       0      0        0        0\n    23         5     5       0      0        0        0\n    24         5     5       0      0        0        0\n    25         5     5       0      0        0        0\n    26         5     5       0      0        0        0\n    27         5     5       0      0        0        0\n    28         5     0       5      0        1        1\n    29         5     0       5      0        0        1\n    30         5     0       5      0        0        1\n    31         5     0       5      0        0        1\n    32         5     5       0      0        0        0\n    33         5     5       0      0        0        0\n    34         5     5       0      0        0        0\n    35         5     5       0      0        0        0\n    36         5     5       0      0        0        0\n    37         5     5       0      0        0        0\n    38         5     5       0      0        0        0\n    39         5     5       0      0        0        0\nsingle flight: 3 fetches, at most 1 running at once; 60 requests waited, 150 ticks in all; 0 served stale.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 4\nFetches in 40 ticks, with a TTL of 10 ticks and a fetch of 4 ticks:\n  requests every tick    naive  at once  single flight\n                    1       12        4              3\n                    2       24        8              3\n                    5       60       20              3\n                   10      120       40              3\n                   20      240       80              3\n  one request every      naive  at once  single flight\n               1 tick       12        4              3\n              2 ticks        6        2              3\n              3 ticks        6        2              3\n              4 ticks        3        1              3\n              5 ticks        3        1              3\n              6 ticks        3        1              3\n              7 ticks        3        1              3\n              8 ticks        3        1              3\n              9 ticks        3        1              3\nNaive and single flight run alike, tick by tick, once requests are 4 ticks apart, the time a fetch takes: then each fetch lands before the next request.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 6\nTTL (1 to 30 ticks)> 10\nfetch time (1 to 9 ticks)> 4\nearly refresh margin (1 to 9 ticks)> 4\nTraffic: 5 requests every tick, for 40 ticks.\nCache: a TTL of 10 ticks, a fetch of 4 ticks, early refresh within 4 ticks of the end of the TTL.\nThe cache starts empty, and a fetch that lands makes the value fresh for the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 3\nOver 40 ticks:\n  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\n  naive               60            20      60           240             0              0\n  single flight        3             1      60           150             0              0\n  early refresh        4             1      20            50             0              0\n  serve stale          3             1      20            50            40              3\n  most past TTL: how many ticks the oldest stale answer had been expired.\nNaive starts 60 fetches where single flight starts 3, with up to 20 running at once.\nEach time the value expires, naive starts a fetch for every request until the first one lands: 20 fetches, where single flight starts 1.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 2\npolicy (naive, single flight, early refresh or serve stale)> early refresh\nTick by tick, early refresh:\n  tick  requests  hits  waited  stale  started  running\n     0         5     0       5      0        1        1\n     1         5     0       5      0        0        1\n     2         5     0       5      0        0        1\n     3         5     0       5      0        0        1\n     4         5     5       0      0        0        0\n     5         5     5       0      0        0        0\n     6         5     5       0      0        0        0\n     7         5     5       0      0        0        0\n     8         5     5       0      0        0        0\n     9         5     5       0      0        0        0\n    10         5     5       0      0        1        1\n    11         5     5       0      0        0        1\n    12         5     5       0      0        0        1\n    13         5     5       0      0        0        1\n    14         5     5       0      0        0        0\n    15         5     5       0      0        0        0\n    16         5     5       0      0        0        0\n    17         5     5       0      0        0        0\n    18         5     5       0      0        0        0\n    19         5     5       0      0        0        0\n    20         5     5       0      0        1        1\n    21         5     5       0      0        0        1\n    22         5     5       0      0        0        1\n    23         5     5       0      0        0        1\n    24         5     5       0      0        0        0\n    25         5     5       0      0        0        0\n    26         5     5       0      0        0        0\n    27         5     5       0      0        0        0\n    28         5     5       0      0        0        0\n    29         5     5       0      0        0        0\n    30         5     5       0      0        1        1\n    31         5     5       0      0        0        1\n    32         5     5       0      0        0        1\n    33         5     5       0      0        0        1\n    34         5     5       0      0        0        0\n    35         5     5       0      0        0        0\n    36         5     5       0      0        0        0\n    37         5     5       0      0        0        0\n    38         5     5       0      0        0        0\n    39         5     5       0      0        0        0\nearly refresh: 4 fetches, at most 1 running at once; 20 requests waited, 50 ticks in all; 0 served stale.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 5\nrequests at a time (1 to 20)> 1\nevery (1 to 9 ticks)> 3\nTraffic: 1 request every 3 ticks, for 40 ticks.\nCache: a TTL of 10 ticks, a fetch of 4 ticks, early refresh within 4 ticks of the end of the TTL.\nThe cache starts empty, and a fetch that lands makes the value fresh for the TTL.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 3\nOver 40 ticks:\n  policy         fetches  most at once  waited  ticks waited  served stale  most past TTL\n  naive                6             2       6            24             0              0\n  single flight        3             1       6            15             0              0\n  early refresh        4             1       5             8             0              0\n  serve stale          3             1       2             5             4              4\n  most past TTL: how many ticks the oldest stale answer had been expired.\nNaive starts 6 fetches where single flight starts 3, with up to 2 running at once.\nEach time the value expires, naive starts a fetch for every request until the first one lands: 2 fetches, where single flight starts 1.\n\n1) settings  2) tick by tick  3) compare policies  4) more traffic  5) traffic  6) cache  7) quit\nchoice> 7\nBye.\n",
      "interpreter": "equal"
    }
  ],
  "builtOn": [
    {
      "slug": "cache-stampede-single-flight",
      "caseId": "276-cache-stampede-single-flight",
      "title": "Cache stampede — the hotter the key, the bigger the crowd at expiry"
    }
  ],
  "updated": "2026-10-16"
}