{
  "id": "P153",
  "slug": "flaky-tests",
  "key": "P153-flaky-tests",
  "title": "Flaky tests",
  "summary": "A suite whose tests fail at random, retried and quarantined week by week: the pass rate retries report, and a suite that grows greener every week while it catches fewer defects.",
  "entry": "main.eml",
  "ui": "terminal",
  "readme": "# P153 - Flaky tests\n\nA test suite whose tests fail at random, retried and quarantined, and what\nits pass rate says while what it checks falls.\n\n- `main.eml` - the menu, the suite, one week's run at a number of\n  attempts, the weeks side by side, and the pass rates retries report\n- `flaky.eml` - the tests, powers and percentages in tenths computed\n  exactly, a test's pass within r attempts, the tests still gating after a\n  number of weeks, the suite's green share and the defects it catches\n\nHow each part works:\n\n- Six tests, each failing a share of runs on its own and each the only one\n  that would catch some of 18 defects. Nothing in the code changes from\n  week to week.\n- As in the corpus case `retry-masked-flakiness`, a test failing a share f\n  of runs passes 1 - f^r of them at r attempts: at 3 attempts a test that\n  fails 20% of the time reports 99.2%, and 7 of 10 rates from 0% to 50%\n  report 99.0% or more. Retries leave every test exactly as flaky; the\n  report becomes a function of the retry count.\n- As in the corpus case `flaky-quarantine-drift`, each week the flakiest\n  test still gating is put aside: upload, webhook, billing, search, export.\n  At one attempt the suite goes from green on 46.8% of runs to 100.0%,\n  while the defects it catches go from 18 to 3. The green share never\n  falls and the defects never rise, week to week; after week 5 nothing\n  gating fails, and nothing more is put aside.\n\nWhat is checked: menu choices 1 to 5; 1 to 3 attempts; a week from 0 to 6.\nAn empty answer cancels.\n\nSessions: `sessions/basic.in` gives:\n- the suite; week 0 at 1 attempt and week 2 at 3 attempts;\n- the weeks at 1 attempt and at 3; the pass rates retries report.\n\n`sessions/bad-input.in` gives:\n- menu choices 0 and x;\n- attempts: a cancel, 0, 4 and x; then 2 attempts with a cancel, 7 and x\n  for the week, and week 6;\n- the weeks with a cancel and 9 attempts.\n\nBuilt on the verified corpus cases `retry-masked-flakiness` (the observed\npass rate 1 - f^r, exactly, against the true rate) and\n`flaky-quarantine-drift` (the flakiest test quarantined each week, the\npass rate and the defects still gated side by side).\n",
  "modules": [
    {
      "name": "main.eml",
      "eml": "# P153 flaky tests: the suite, one week's run at a number of attempts, the\n# weeks of quarantine side by side, and the pass rates retries report.\nimport flaky\n\ndef plain(s):\n    0 => i\n    len(s) => j\n    while i < j and s[i] == \" \":\n        i + 1 => i\n    while j > i and s[j - 1] == \" \":\n        j - 1 => j\n    return s[i:j]\n\ndef pad(s, n):\n    s => out\n    while len(out) < n:\n        out + \" \" => out\n    return out\n\ndef lpad(s, n):\n    s => out\n    while len(out) < n:\n        \" \" + out => out\n    return out\n\ndef times(n, one, many):\n    if n == 1:\n        return \"1 \" + one\n    return str(n) + \" \" + many\n\ndef pct(t):\n    t % 10 => d\n    return str(int((t - d) / 10)) + \".\" + str(d) + \"%\"\n\ndef ask(prompt, low, high):\n    # a whole number from low to high, -1 for a cancel, -2 when not valid\n    plain(input(prompt)) => t\n    if t == \"\":\n        return 0 - 1\n    if len(t) > 2:\n        return 0 - 2\n    for c in t:\n        if c < \"0\" or c > \"9\":\n            return 0 - 2\n    if int(t) < low or int(t) > high:\n        return 0 - 2\n    return int(t)\n\ndef show_suite():\n    \"The tests, how often each fails on its own, and the defects only it would catch:\" ^0\n    0 => total\n    for t in flaky.tests():\n        (\"  \" + pad(t[0], 9) + lpad(pct(t[1]), 6) + \" of runs fail\" + \"   \" + times(t[2], \"defect\", \"defects\")) ^0\n        total + t[2] => total\n    (\"  \" + str(total) + \" defects in all; nothing in the code changes from week to week.\") ^0\n\ndef week_run():\n    ask(\"attempts per test (1 to 3)> \", 1, 3) => r\n    if r == 0 - 1:\n        \"Cancelled.\" ^0\n        return\n    if r < 0:\n        \"Type 1, 2 or 3.\" ^0\n        return\n    ask(\"week (0 to 6)> \", 0, 6) => w\n    if w == 0 - 1:\n        \"Cancelled.\" ^0\n        return\n    if w < 0:\n        \"Type a week from 0 to 6.\" ^0\n        return\n    flaky.active(w) => live\n    (\"Week \" + str(w) + \", \" + times(r, \"attempt\", \"attempts\") + \" each, \" + str(len(live)) + \" of \" + str(len(flaky.tests())) + \" tests still gating:\") ^0\n    for t in live:\n        flaky.passes(t[1], r) => p\n        (\"  \" + pad(t[0], 9) + \"fails \" + lpad(pct(t[1]), 6) + \" of runs, passes within \" + str(r) + \": \" + lpad(pct(flaky.tenths(p[0], p[1])), 6)) ^0\n    flaky.green(live, r) => g\n    (\"The suite is green on \" + pct(flaky.tenths(g[0], g[1])) + \" of runs and catches \" + str(flaky.defects(live)) + \" of \" + str(flaky.defects(flaky.tests())) + \" defects.\") ^0\n\ndef drift():\n    ask(\"attempts per test (1 to 3)> \", 1, 3) => r\n    if r == 0 - 1:\n        \"Cancelled.\" ^0\n        return\n    if r < 0:\n        \"Type 1, 2 or 3.\" ^0\n        return\n    (\"Each week the flakiest test still gating is put aside, \" + times(r, \"attempt\", \"attempts\") + \" each:\") ^0\n    (\"  \" + pad(\"week\", 6) + pad(\"put aside\", 11) + \"tests  green   defects caught\") ^0\n    0 => w\n    flaky.tests() => before\n    0 => low\n    0 => lastgreen\n    while w <= 6:\n        flaky.active(w) => live\n        \"-\" => gone\n        if len(live) < len(before):\n            for t in before:\n                False => still\n                for u in live:\n                    if u[0] == t[0]:\n                        True => still\n                if not still:\n                    t[0] => gone\n        flaky.green(live, r) => g\n        flaky.tenths(g[0], g[1]) => gr\n        if w == 0:\n            gr => low\n        gr => lastgreen\n        (\"  \" + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + \" of \" + str(flaky.defects(flaky.tests()))) ^0\n        live => before\n        w + 1 => w\n    flaky.active(6) => last\n    (\"Green went from \" + pct(low) + \" to \" + pct(lastgreen) + \" while the defects caught went from \" + str(flaky.defects(flaky.tests())) + \" to \" + str(flaky.defects(last)) + \": the dashboard's number rose as what the suite checks fell.\") ^0\n    \"After week 5 nothing gating fails, and nothing more is put aside.\" ^0\n\ndef sweep():\n    \"What a test reports as its pass rate, by how often it really fails and the attempts allowed:\" ^0\n    (\"  \" + pad(\"fails\", 8) + pad(\"1 attempt\", 12) + pad(\"2 attempts\", 12) + \"3 attempts\") ^0\n    [0, 10, 20, 50, 100, 150, 200, 300, 400, 500] => fs\n    for f in fs:\n        \"  \" + pad(pct(f), 8) => line\n        1 => r\n        while r <= 3:\n            flaky.passes(f, r) => p\n            if r < 3:\n                line + pad(pct(flaky.tenths(p[0], p[1])), 12) => line\n            else:\n                line + pct(flaky.tenths(p[0], p[1])) => line\n            r + 1 => r\n        line ^0\n    1 => r\n    while r <= 3:\n        0 => over\n        for f in fs:\n            flaky.passes(f, r) => p\n            if flaky.tenths(p[0], p[1]) >= 990:\n                over + 1 => over\n        (\"  at \" + str(r) + \": \" + str(over) + \" of the \" + str(len(fs)) + \" rates report 99.0% or more\") ^0\n        r + 1 => r\n    \"Retries leave every test exactly as flaky; they make the report a function of the retry count.\" ^0\n\n\"== Flaky tests ==\" ^0\n\"A suite with tests that fail at random, retried and quarantined, and what its pass rate says.\" ^0\nTrue => running\nwhile running:\n    \"\" ^0\n    \"1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\" ^0\n    plain(input(\"choice> \")) => choice\n    if choice == \"1\":\n        show_suite()\n    elif choice == \"2\":\n        week_run()\n    elif choice == \"3\":\n        drift()\n    elif choice == \"4\":\n        sweep()\n    elif choice == \"5\":\n        False => running\n    else:\n        \"Pick a number from 1 to 5.\" ^0\n\"Bye.\" ^0\n",
      "python": "import flaky\n\ndef plain(s):\n    i = 0\n    j = len(s)\n    while i < j and s[i] == \" \":\n        i = i + 1\n    while j > i and s[j - 1] == \" \":\n        j = j - 1\n    return s[i:j]\n\ndef pad(s, n):\n    out = s\n    while len(out) < n:\n        out = out + \" \"\n    return out\n\ndef lpad(s, n):\n    out = s\n    while len(out) < n:\n        out = \" \" + out\n    return out\n\ndef times(n, one, many):\n    if n == 1:\n        return \"1 \" + one\n    return str(n) + \" \" + many\n\ndef pct(t):\n    d = t % 10\n    return str(int((t - d) / 10)) + \".\" + str(d) + \"%\"\n\ndef ask(prompt, low, high):\n    t = plain(input(prompt))\n    if t == \"\":\n        return 0 - 1\n    if len(t) > 2:\n        return 0 - 2\n    for c in t:\n        if c < \"0\" or c > \"9\":\n            return 0 - 2\n    if int(t) < low or int(t) > high:\n        return 0 - 2\n    return int(t)\n\ndef show_suite():\n    print(\"The tests, how often each fails on its own, and the defects only it would catch:\")\n    total = 0\n    for t in flaky.tests():\n        print(\"  \" + pad(t[0], 9) + lpad(pct(t[1]), 6) + \" of runs fail\" + \"   \" + times(t[2], \"defect\", \"defects\"))\n        total = total + t[2]\n    print(\"  \" + str(total) + \" defects in all; nothing in the code changes from week to week.\")\n\ndef week_run():\n    r = ask(\"attempts per test (1 to 3)> \", 1, 3)\n    if r == 0 - 1:\n        print(\"Cancelled.\")\n        return\n    if r < 0:\n        print(\"Type 1, 2 or 3.\")\n        return\n    w = ask(\"week (0 to 6)> \", 0, 6)\n    if w == 0 - 1:\n        print(\"Cancelled.\")\n        return\n    if w < 0:\n        print(\"Type a week from 0 to 6.\")\n        return\n    live = flaky.active(w)\n    print(\"Week \" + str(w) + \", \" + times(r, \"attempt\", \"attempts\") + \" each, \" + str(len(live)) + \" of \" + str(len(flaky.tests())) + \" tests still gating:\")\n    for t in live:\n        p = flaky.passes(t[1], r)\n        print(\"  \" + pad(t[0], 9) + \"fails \" + lpad(pct(t[1]), 6) + \" of runs, passes within \" + str(r) + \": \" + lpad(pct(flaky.tenths(p[0], p[1])), 6))\n    g = flaky.green(live, r)\n    print(\"The suite is green on \" + pct(flaky.tenths(g[0], g[1])) + \" of runs and catches \" + str(flaky.defects(live)) + \" of \" + str(flaky.defects(flaky.tests())) + \" defects.\")\n\ndef drift():\n    r = ask(\"attempts per test (1 to 3)> \", 1, 3)\n    if r == 0 - 1:\n        print(\"Cancelled.\")\n        return\n    if r < 0:\n        print(\"Type 1, 2 or 3.\")\n        return\n    print(\"Each week the flakiest test still gating is put aside, \" + times(r, \"attempt\", \"attempts\") + \" each:\")\n    print(\"  \" + pad(\"week\", 6) + pad(\"put aside\", 11) + \"tests  green   defects caught\")\n    w = 0\n    before = flaky.tests()\n    low = 0\n    lastgreen = 0\n    while w <= 6:\n        live = flaky.active(w)\n        gone = \"-\"\n        if len(live) < len(before):\n            for t in before:\n                still = False\n                for u in live:\n                    if u[0] == t[0]:\n                        still = True\n                if not still:\n                    gone = t[0]\n        g = flaky.green(live, r)\n        gr = flaky.tenths(g[0], g[1])\n        if w == 0:\n            low = gr\n        lastgreen = gr\n        print(\"  \" + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + \" of \" + str(flaky.defects(flaky.tests())))\n        before = live\n        w = w + 1\n    last = flaky.active(6)\n    print(\"Green went from \" + pct(low) + \" to \" + pct(lastgreen) + \" while the defects caught went from \" + str(flaky.defects(flaky.tests())) + \" to \" + str(flaky.defects(last)) + \": the dashboard's number rose as what the suite checks fell.\")\n    print(\"After week 5 nothing gating fails, and nothing more is put aside.\")\n\ndef sweep():\n    print(\"What a test reports as its pass rate, by how often it really fails and the attempts allowed:\")\n    print(\"  \" + pad(\"fails\", 8) + pad(\"1 attempt\", 12) + pad(\"2 attempts\", 12) + \"3 attempts\")\n    fs = [0, 10, 20, 50, 100, 150, 200, 300, 400, 500]\n    for f in fs:\n        line = \"  \" + pad(pct(f), 8)\n        r = 1\n        while r <= 3:\n            p = flaky.passes(f, r)\n            if r < 3:\n                line = line + pad(pct(flaky.tenths(p[0], p[1])), 12)\n            else:\n                line = line + pct(flaky.tenths(p[0], p[1]))\n            r = r + 1\n        print(line)\n    r = 1\n    while r <= 3:\n        over = 0\n        for f in fs:\n            p = flaky.passes(f, r)\n            if flaky.tenths(p[0], p[1]) >= 990:\n                over = over + 1\n        print(\"  at \" + str(r) + \": \" + str(over) + \" of the \" + str(len(fs)) + \" rates report 99.0% or more\")\n        r = r + 1\n    print(\"Retries leave every test exactly as flaky; they make the report a function of the retry count.\")\n\nprint(\"== Flaky tests ==\")\nprint(\"A suite with tests that fail at random, retried and quarantined, and what its pass rate says.\")\nrunning = True\nwhile running:\n    print(\"\")\n    print(\"1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\")\n    choice = plain(input(\"choice> \"))\n    if choice == \"1\":\n        show_suite()\n    elif choice == \"2\":\n        week_run()\n    elif choice == \"3\":\n        drift()\n    elif choice == \"4\":\n        sweep()\n    elif choice == \"5\":\n        running = False\n    else:\n        print(\"Pick a number from 1 to 5.\")\nprint(\"Bye.\")\n"
    },
    {
      "name": "flaky.eml",
      "eml": "# P153 flaky tests - a test suite with tests that fail at random, retried\n# and quarantined. As in the corpus case retry-masked-flakiness, a test that\n# fails a share f of runs passes 1 - f^r of them at r attempts, so retries\n# fold every unhealthy test into the narrow band below 100% where the\n# healthy ones are. As in the corpus case flaky-quarantine-drift, putting\n# the flakiest test aside each week makes the suite greener every week while\n# it checks less every week, and the number that shows it - the defects the\n# active tests still catch - is not the one on the dashboard. Rates are in\n# per mille and every probability is an exact fraction.\n\ndef tests():\n    # [name, failures per thousand runs, defects only this test catches]\n    return [[\"auth\", 0, 3], [\"billing\", 120, 4], [\"search\", 40, 2], [\"upload\", 300, 5], [\"export\", 10, 1], [\"webhook\", 200, 3]]\n\ndef power(b, n):\n    1 => r\n    0 => k\n    while k < n:\n        r * b => r\n        k + 1 => k\n    return r\n\ndef tenths(n, d):\n    # n / d as a percentage in tenths, rounded half up: the float quotient\n    # corrected until it is the exact floor, for numerators of 50 digits\n    2000 * n + d => x\n    2 * d => y\n    int(x / y) => q\n    while q * y > x:\n        q - 1 => q\n    while (q + 1) * y <= x:\n        q + 1 => q\n    return q\n\ndef passes(f, r):\n    # [numerator, denominator] of a pass within r attempts: 1 - (f/1000)^r\n    return [power(1000, r) - power(f, r), power(1000, r)]\n\ndef active(week):\n    # the tests still gating after week weeks of putting the flakiest aside;\n    # nothing is put aside once no active test fails at all\n    tests() => live\n    0 => w\n    while w < week:\n        0 => worst\n        0 - 1 => at\n        0 => i\n        for t in live:\n            if t[1] > worst:\n                t[1] => worst\n                i => at\n            i + 1 => i\n        if at >= 0:\n            [] => keep\n            0 => i\n            for t in live:\n                if i != at:\n                    keep + [t] => keep\n                i + 1 => i\n            keep => live\n        w + 1 => w\n    return live\n\ndef green(live, r):\n    # [numerator, denominator] of every active test passing within r tries\n    1 => n\n    1 => d\n    for t in live:\n        passes(t[1], r) => p\n        n * p[0] => n\n        d * p[1] => d\n    return [n, d]\n\ndef defects(live):\n    0 => s\n    for t in live:\n        s + t[2] => s\n    return s\n",
      "python": "def tests():\n    return [[\"auth\", 0, 3], [\"billing\", 120, 4], [\"search\", 40, 2], [\"upload\", 300, 5], [\"export\", 10, 1], [\"webhook\", 200, 3]]\n\ndef power(b, n):\n    r = 1\n    k = 0\n    while k < n:\n        r = r * b\n        k = k + 1\n    return r\n\ndef tenths(n, d):\n    x = 2000 * n + d\n    y = 2 * d\n    q = int(x / y)\n    while q * y > x:\n        q = q - 1\n    while (q + 1) * y <= x:\n        q = q + 1\n    return q\n\ndef passes(f, r):\n    return [power(1000, r) - power(f, r), power(1000, r)]\n\ndef active(week):\n    live = tests()\n    w = 0\n    while w < week:\n        worst = 0\n        at = 0 - 1\n        i = 0\n        for t in live:\n            if t[1] > worst:\n                worst = t[1]\n                at = i\n            i = i + 1\n        if at >= 0:\n            keep = []\n            i = 0\n            for t in live:\n                if i != at:\n                    keep = keep + [t]\n                i = i + 1\n            live = keep\n        w = w + 1\n    return live\n\ndef green(live, r):\n    n = 1\n    d = 1\n    for t in live:\n        p = passes(t[1], r)\n        n = n * p[0]\n        d = d * p[1]\n    return [n, d]\n\ndef defects(live):\n    s = 0\n    for t in live:\n        s = s + t[2]\n    return s\n"
    }
  ],
  "sessions": [
    {
      "name": "bad-input",
      "input": "0\nx\n2\n\n2\n0\n2\n4\n2\nx\n2\n2\n\n2\n2\n7\n2\n2\nx\n2\n2\n6\n3\n\n3\n9\n5\n",
      "screen": "== Flaky tests ==\nA suite with tests that fail at random, retried and quarantined, and what its pass rate says.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 0\nPick a number from 1 to 5.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> x\nPick a number from 1 to 5.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> \nCancelled.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 0\nType 1, 2 or 3.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 4\nType 1, 2 or 3.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> x\nType 1, 2 or 3.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 2\nweek (0 to 6)> \nCancelled.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 2\nweek (0 to 6)> 7\nType a week from 0 to 6.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 2\nweek (0 to 6)> x\nType a week from 0 to 6.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 2\nweek (0 to 6)> 6\nWeek 6, 2 attempts each, 1 of 6 tests still gating:\n  auth     fails   0.0% of runs, passes within 2: 100.0%\nThe suite is green on 100.0% of runs and catches 3 of 18 defects.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 3\nattempts per test (1 to 3)> \nCancelled.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 3\nattempts per test (1 to 3)> 9\nType 1, 2 or 3.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 5\nBye.\n",
      "interpreter": "equal"
    },
    {
      "name": "basic",
      "input": "1\n2\n1\n0\n2\n3\n2\n3\n1\n3\n3\n4\n5\n",
      "screen": "== Flaky tests ==\nA suite with tests that fail at random, retried and quarantined, and what its pass rate says.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 1\nThe tests, how often each fails on its own, and the defects only it would catch:\n  auth       0.0% of runs fail   3 defects\n  billing   12.0% of runs fail   4 defects\n  search     4.0% of runs fail   2 defects\n  upload    30.0% of runs fail   5 defects\n  export     1.0% of runs fail   1 defect\n  webhook   20.0% of runs fail   3 defects\n  18 defects in all; nothing in the code changes from week to week.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 1\nweek (0 to 6)> 0\nWeek 0, 1 attempt each, 6 of 6 tests still gating:\n  auth     fails   0.0% of runs, passes within 1: 100.0%\n  billing  fails  12.0% of runs, passes within 1:  88.0%\n  search   fails   4.0% of runs, passes within 1:  96.0%\n  upload   fails  30.0% of runs, passes within 1:  70.0%\n  export   fails   1.0% of runs, passes within 1:  99.0%\n  webhook  fails  20.0% of runs, passes within 1:  80.0%\nThe suite is green on 46.8% of runs and catches 18 of 18 defects.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 2\nattempts per test (1 to 3)> 3\nweek (0 to 6)> 2\nWeek 2, 3 attempts each, 4 of 6 tests still gating:\n  auth     fails   0.0% of runs, passes within 3: 100.0%\n  billing  fails  12.0% of runs, passes within 3:  99.8%\n  search   fails   4.0% of runs, passes within 3: 100.0%\n  export   fails   1.0% of runs, passes within 3: 100.0%\nThe suite is green on 99.8% of runs and catches 10 of 18 defects.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 3\nattempts per test (1 to 3)> 1\nEach week the flakiest test still gating is put aside, 1 attempt each:\n  week  put aside  tests  green   defects caught\n  0     -              6   46.8%      18 of 18\n  1     upload         5   66.9%      13 of 18\n  2     webhook        4   83.6%      10 of 18\n  3     billing        3   95.0%       6 of 18\n  4     search         2   99.0%       4 of 18\n  5     export         1  100.0%       3 of 18\n  6     -              1  100.0%       3 of 18\nGreen went from 46.8% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.\nAfter week 5 nothing gating fails, and nothing more is put aside.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 3\nattempts per test (1 to 3)> 3\nEach week the flakiest test still gating is put aside, 3 attempts each:\n  week  put aside  tests  green   defects caught\n  0     -              6   96.3%      18 of 18\n  1     upload         5   99.0%      13 of 18\n  2     webhook        4   99.8%      10 of 18\n  3     billing        3  100.0%       6 of 18\n  4     search         2  100.0%       4 of 18\n  5     export         1  100.0%       3 of 18\n  6     -              1  100.0%       3 of 18\nGreen went from 96.3% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.\nAfter week 5 nothing gating fails, and nothing more is put aside.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 4\nWhat a test reports as its pass rate, by how often it really fails and the attempts allowed:\n  fails   1 attempt   2 attempts  3 attempts\n  0.0%    100.0%      100.0%      100.0%\n  1.0%    99.0%       100.0%      100.0%\n  2.0%    98.0%       100.0%      100.0%\n  5.0%    95.0%       99.8%       100.0%\n  10.0%   90.0%       99.0%       99.9%\n  15.0%   85.0%       97.8%       99.7%\n  20.0%   80.0%       96.0%       99.2%\n  30.0%   70.0%       91.0%       97.3%\n  40.0%   60.0%       84.0%       93.6%\n  50.0%   50.0%       75.0%       87.5%\n  at 1: 2 of the 10 rates report 99.0% or more\n  at 2: 5 of the 10 rates report 99.0% or more\n  at 3: 7 of the 10 rates report 99.0% or more\nRetries leave every test exactly as flaky; they make the report a function of the retry count.\n\n1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit\nchoice> 5\nBye.\n",
      "interpreter": "equal"
    }
  ],
  "builtOn": [
    {
      "slug": "retry-masked-flakiness",
      "caseId": "287-retry-masked-flakiness",
      "title": "Retry-masked flakiness — one failure in five reports as 99.2%"
    },
    {
      "slug": "flaky-quarantine-drift",
      "caseId": "280-flaky-quarantine-drift",
      "title": "Flaky quarantine drift — a green pipeline and a third of the gate gone"
    }
  ],
  "updated": "2026-10-18"
}