<!-- canonical: https://efficientnewlanguage.org/eml-p/projects/P153-flaky-tests/ | updated: 2026-10-18 -->

# P153 Flaky tests

A suite whose tests fail at random, retried and quarantined week by week: the pass rate retries report, and a suite that grows greener every week while it catches fewer defects.

EML-P project `projects/flaky-tests` in the EML language repo: 2 module(s), entry `main.eml`, terminal UI. There, `eml project run projects/flaky-tests` runs it and `eml project verify projects/flaky-tests` replays every session under CPython (two hash seeds) and in the interpreter; the site build replays every session in the interpreter again and publishes a session only if its screen matches.

Built on verified corpus cases: retry-masked-flakiness (https://efficientnewlanguage.org/cases/287-retry-masked-flakiness/), flaky-quarantine-drift (https://efficientnewlanguage.org/cases/280-flaky-quarantine-drift/).

## Sessions

### bad-input - interpreter: byte-equal to the golden

Input:

```text
0
x
2

2
0
2
4
2
x
2
2

2
2
7
2
2
x
2
2
6
3

3
9
5
```

Screen:

```text
== Flaky tests ==
A suite with tests that fail at random, retried and quarantined, and what its pass rate says.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 0
Pick a number from 1 to 5.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> x
Pick a number from 1 to 5.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 0
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 4
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> x
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 7
Type a week from 0 to 6.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> x
Type a week from 0 to 6.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 6
Week 6, 2 attempts each, 1 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 2: 100.0%
The suite is green on 100.0% of runs and catches 3 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 9
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 5
Bye.
```

### basic - interpreter: byte-equal to the golden

Input:

```text
1
2
1
0
2
3
2
3
1
3
3
4
5
```

Screen:

```text
== Flaky tests ==
A suite with tests that fail at random, retried and quarantined, and what its pass rate says.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 1
The tests, how often each fails on its own, and the defects only it would catch:
  auth       0.0% of runs fail   3 defects
  billing   12.0% of runs fail   4 defects
  search     4.0% of runs fail   2 defects
  upload    30.0% of runs fail   5 defects
  export     1.0% of runs fail   1 defect
  webhook   20.0% of runs fail   3 defects
  18 defects in all; nothing in the code changes from week to week.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 1
week (0 to 6)> 0
Week 0, 1 attempt each, 6 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 1: 100.0%
  billing  fails  12.0% of runs, passes within 1:  88.0%
  search   fails   4.0% of runs, passes within 1:  96.0%
  upload   fails  30.0% of runs, passes within 1:  70.0%
  export   fails   1.0% of runs, passes within 1:  99.0%
  webhook  fails  20.0% of runs, passes within 1:  80.0%
The suite is green on 46.8% of runs and catches 18 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 3
week (0 to 6)> 2
Week 2, 3 attempts each, 4 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 3: 100.0%
  billing  fails  12.0% of runs, passes within 3:  99.8%
  search   fails   4.0% of runs, passes within 3: 100.0%
  export   fails   1.0% of runs, passes within 3: 100.0%
The suite is green on 99.8% of runs and catches 10 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 1
Each week the flakiest test still gating is put aside, 1 attempt each:
  week  put aside  tests  green   defects caught
  0     -              6   46.8%      18 of 18
  1     upload         5   66.9%      13 of 18
  2     webhook        4   83.6%      10 of 18
  3     billing        3   95.0%       6 of 18
  4     search         2   99.0%       4 of 18
  5     export         1  100.0%       3 of 18
  6     -              1  100.0%       3 of 18
Green went from 46.8% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.
After week 5 nothing gating fails, and nothing more is put aside.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 3
Each week the flakiest test still gating is put aside, 3 attempts each:
  week  put aside  tests  green   defects caught
  0     -              6   96.3%      18 of 18
  1     upload         5   99.0%      13 of 18
  2     webhook        4   99.8%      10 of 18
  3     billing        3  100.0%       6 of 18
  4     search         2  100.0%       4 of 18
  5     export         1  100.0%       3 of 18
  6     -              1  100.0%       3 of 18
Green went from 96.3% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.
After week 5 nothing gating fails, and nothing more is put aside.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 4
What a test reports as its pass rate, by how often it really fails and the attempts allowed:
  fails   1 attempt   2 attempts  3 attempts
  0.0%    100.0%      100.0%      100.0%
  1.0%    99.0%       100.0%      100.0%
  2.0%    98.0%       100.0%      100.0%
  5.0%    95.0%       99.8%       100.0%
  10.0%   90.0%       99.0%       99.9%
  15.0%   85.0%       97.8%       99.7%
  20.0%   80.0%       96.0%       99.2%
  30.0%   70.0%       91.0%       97.3%
  40.0%   60.0%       84.0%       93.6%
  50.0%   50.0%       75.0%       87.5%
  at 1: 2 of the 10 rates report 99.0% or more
  at 2: 5 of the 10 rates report 99.0% or more
  at 3: 7 of the 10 rates report 99.0% or more
Retries leave every test exactly as flaky; they make the report a function of the retry count.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 5
Bye.
```

## Modules

### main.eml

```eml
# P153 flaky tests: the suite, one week's run at a number of attempts, the
# weeks of quarantine side by side, and the pass rates retries report.
import flaky

def plain(s):
    0 => i
    len(s) => j
    while i < j and s[i] == " ":
        i + 1 => i
    while j > i and s[j - 1] == " ":
        j - 1 => j
    return s[i:j]

def pad(s, n):
    s => out
    while len(out) < n:
        out + " " => out
    return out

def lpad(s, n):
    s => out
    while len(out) < n:
        " " + out => out
    return out

def times(n, one, many):
    if n == 1:
        return "1 " + one
    return str(n) + " " + many

def pct(t):
    t % 10 => d
    return str(int((t - d) / 10)) + "." + str(d) + "%"

def ask(prompt, low, high):
    # a whole number from low to high, -1 for a cancel, -2 when not valid
    plain(input(prompt)) => t
    if t == "":
        return 0 - 1
    if len(t) > 2:
        return 0 - 2
    for c in t:
        if c < "0" or c > "9":
            return 0 - 2
    if int(t) < low or int(t) > high:
        return 0 - 2
    return int(t)

def show_suite():
    "The tests, how often each fails on its own, and the defects only it would catch:" ^0
    0 => total
    for t in flaky.tests():
        ("  " + pad(t[0], 9) + lpad(pct(t[1]), 6) + " of runs fail" + "   " + times(t[2], "defect", "defects")) ^0
        total + t[2] => total
    ("  " + str(total) + " defects in all; nothing in the code changes from week to week.") ^0

def week_run():
    ask("attempts per test (1 to 3)> ", 1, 3) => r
    if r == 0 - 1:
        "Cancelled." ^0
        return
    if r < 0:
        "Type 1, 2 or 3." ^0
        return
    ask("week (0 to 6)> ", 0, 6) => w
    if w == 0 - 1:
        "Cancelled." ^0
        return
    if w < 0:
        "Type a week from 0 to 6." ^0
        return
    flaky.active(w) => live
    ("Week " + str(w) + ", " + times(r, "attempt", "attempts") + " each, " + str(len(live)) + " of " + str(len(flaky.tests())) + " tests still gating:") ^0
    for t in live:
        flaky.passes(t[1], r) => p
        ("  " + pad(t[0], 9) + "fails " + lpad(pct(t[1]), 6) + " of runs, passes within " + str(r) + ": " + lpad(pct(flaky.tenths(p[0], p[1])), 6)) ^0
    flaky.green(live, r) => g
    ("The suite is green on " + pct(flaky.tenths(g[0], g[1])) + " of runs and catches " + str(flaky.defects(live)) + " of " + str(flaky.defects(flaky.tests())) + " defects.") ^0

def drift():
    ask("attempts per test (1 to 3)> ", 1, 3) => r
    if r == 0 - 1:
        "Cancelled." ^0
        return
    if r < 0:
        "Type 1, 2 or 3." ^0
        return
    ("Each week the flakiest test still gating is put aside, " + times(r, "attempt", "attempts") + " each:") ^0
    ("  " + pad("week", 6) + pad("put aside", 11) + "tests  green   defects caught") ^0
    0 => w
    flaky.tests() => before
    0 => low
    0 => lastgreen
    while w <= 6:
        flaky.active(w) => live
        "-" => gone
        if len(live) < len(before):
            for t in before:
                False => still
                for u in live:
                    if u[0] == t[0]:
                        True => still
                if not still:
                    t[0] => gone
        flaky.green(live, r) => g
        flaky.tenths(g[0], g[1]) => gr
        if w == 0:
            gr => low
        gr => lastgreen
        ("  " + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + " of " + str(flaky.defects(flaky.tests()))) ^0
        live => before
        w + 1 => w
    flaky.active(6) => last
    ("Green went from " + pct(low) + " to " + pct(lastgreen) + " while the defects caught went from " + str(flaky.defects(flaky.tests())) + " to " + str(flaky.defects(last)) + ": the dashboard's number rose as what the suite checks fell.") ^0
    "After week 5 nothing gating fails, and nothing more is put aside." ^0

def sweep():
    "What a test reports as its pass rate, by how often it really fails and the attempts allowed:" ^0
    ("  " + pad("fails", 8) + pad("1 attempt", 12) + pad("2 attempts", 12) + "3 attempts") ^0
    [0, 10, 20, 50, 100, 150, 200, 300, 400, 500] => fs
    for f in fs:
        "  " + pad(pct(f), 8) => line
        1 => r
        while r <= 3:
            flaky.passes(f, r) => p
            if r < 3:
                line + pad(pct(flaky.tenths(p[0], p[1])), 12) => line
            else:
                line + pct(flaky.tenths(p[0], p[1])) => line
            r + 1 => r
        line ^0
    1 => r
    while r <= 3:
        0 => over
        for f in fs:
            flaky.passes(f, r) => p
            if flaky.tenths(p[0], p[1]) >= 990:
                over + 1 => over
        ("  at " + str(r) + ": " + str(over) + " of the " + str(len(fs)) + " rates report 99.0% or more") ^0
        r + 1 => r
    "Retries leave every test exactly as flaky; they make the report a function of the retry count." ^0

"== Flaky tests ==" ^0
"A suite with tests that fail at random, retried and quarantined, and what its pass rate says." ^0
True => running
while running:
    "" ^0
    "1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit" ^0
    plain(input("choice> ")) => choice
    if choice == "1":
        show_suite()
    elif choice == "2":
        week_run()
    elif choice == "3":
        drift()
    elif choice == "4":
        sweep()
    elif choice == "5":
        False => running
    else:
        "Pick a number from 1 to 5." ^0
"Bye." ^0
```

Python projection of main.eml:

```python
import flaky

def plain(s):
    i = 0
    j = len(s)
    while i < j and s[i] == " ":
        i = i + 1
    while j > i and s[j - 1] == " ":
        j = j - 1
    return s[i:j]

def pad(s, n):
    out = s
    while len(out) < n:
        out = out + " "
    return out

def lpad(s, n):
    out = s
    while len(out) < n:
        out = " " + out
    return out

def times(n, one, many):
    if n == 1:
        return "1 " + one
    return str(n) + " " + many

def pct(t):
    d = t % 10
    return str(int((t - d) / 10)) + "." + str(d) + "%"

def ask(prompt, low, high):
    t = plain(input(prompt))
    if t == "":
        return 0 - 1
    if len(t) > 2:
        return 0 - 2
    for c in t:
        if c < "0" or c > "9":
            return 0 - 2
    if int(t) < low or int(t) > high:
        return 0 - 2
    return int(t)

def show_suite():
    print("The tests, how often each fails on its own, and the defects only it would catch:")
    total = 0
    for t in flaky.tests():
        print("  " + pad(t[0], 9) + lpad(pct(t[1]), 6) + " of runs fail" + "   " + times(t[2], "defect", "defects"))
        total = total + t[2]
    print("  " + str(total) + " defects in all; nothing in the code changes from week to week.")

def week_run():
    r = ask("attempts per test (1 to 3)> ", 1, 3)
    if r == 0 - 1:
        print("Cancelled.")
        return
    if r < 0:
        print("Type 1, 2 or 3.")
        return
    w = ask("week (0 to 6)> ", 0, 6)
    if w == 0 - 1:
        print("Cancelled.")
        return
    if w < 0:
        print("Type a week from 0 to 6.")
        return
    live = flaky.active(w)
    print("Week " + str(w) + ", " + times(r, "attempt", "attempts") + " each, " + str(len(live)) + " of " + str(len(flaky.tests())) + " tests still gating:")
    for t in live:
        p = flaky.passes(t[1], r)
        print("  " + pad(t[0], 9) + "fails " + lpad(pct(t[1]), 6) + " of runs, passes within " + str(r) + ": " + lpad(pct(flaky.tenths(p[0], p[1])), 6))
    g = flaky.green(live, r)
    print("The suite is green on " + pct(flaky.tenths(g[0], g[1])) + " of runs and catches " + str(flaky.defects(live)) + " of " + str(flaky.defects(flaky.tests())) + " defects.")

def drift():
    r = ask("attempts per test (1 to 3)> ", 1, 3)
    if r == 0 - 1:
        print("Cancelled.")
        return
    if r < 0:
        print("Type 1, 2 or 3.")
        return
    print("Each week the flakiest test still gating is put aside, " + times(r, "attempt", "attempts") + " each:")
    print("  " + pad("week", 6) + pad("put aside", 11) + "tests  green   defects caught")
    w = 0
    before = flaky.tests()
    low = 0
    lastgreen = 0
    while w <= 6:
        live = flaky.active(w)
        gone = "-"
        if len(live) < len(before):
            for t in before:
                still = False
                for u in live:
                    if u[0] == t[0]:
                        still = True
                if not still:
                    gone = t[0]
        g = flaky.green(live, r)
        gr = flaky.tenths(g[0], g[1])
        if w == 0:
            low = gr
        lastgreen = gr
        print("  " + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + " of " + str(flaky.defects(flaky.tests())))
        before = live
        w = w + 1
    last = flaky.active(6)
    print("Green went from " + pct(low) + " to " + pct(lastgreen) + " while the defects caught went from " + str(flaky.defects(flaky.tests())) + " to " + str(flaky.defects(last)) + ": the dashboard's number rose as what the suite checks fell.")
    print("After week 5 nothing gating fails, and nothing more is put aside.")

def sweep():
    print("What a test reports as its pass rate, by how often it really fails and the attempts allowed:")
    print("  " + pad("fails", 8) + pad("1 attempt", 12) + pad("2 attempts", 12) + "3 attempts")
    fs = [0, 10, 20, 50, 100, 150, 200, 300, 400, 500]
    for f in fs:
        line = "  " + pad(pct(f), 8)
        r = 1
        while r <= 3:
            p = flaky.passes(f, r)
            if r < 3:
                line = line + pad(pct(flaky.tenths(p[0], p[1])), 12)
            else:
                line = line + pct(flaky.tenths(p[0], p[1]))
            r = r + 1
        print(line)
    r = 1
    while r <= 3:
        over = 0
        for f in fs:
            p = flaky.passes(f, r)
            if flaky.tenths(p[0], p[1]) >= 990:
                over = over + 1
        print("  at " + str(r) + ": " + str(over) + " of the " + str(len(fs)) + " rates report 99.0% or more")
        r = r + 1
    print("Retries leave every test exactly as flaky; they make the report a function of the retry count.")

print("== Flaky tests ==")
print("A suite with tests that fail at random, retried and quarantined, and what its pass rate says.")
running = True
while running:
    print("")
    print("1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit")
    choice = plain(input("choice> "))
    if choice == "1":
        show_suite()
    elif choice == "2":
        week_run()
    elif choice == "3":
        drift()
    elif choice == "4":
        sweep()
    elif choice == "5":
        running = False
    else:
        print("Pick a number from 1 to 5.")
print("Bye.")
```

### flaky.eml

```eml
# P153 flaky tests - a test suite with tests that fail at random, retried
# and quarantined. As in the corpus case retry-masked-flakiness, a test that
# fails a share f of runs passes 1 - f^r of them at r attempts, so retries
# fold every unhealthy test into the narrow band below 100% where the
# healthy ones are. As in the corpus case flaky-quarantine-drift, putting
# the flakiest test aside each week makes the suite greener every week while
# it checks less every week, and the number that shows it - the defects the
# active tests still catch - is not the one on the dashboard. Rates are in
# per mille and every probability is an exact fraction.

def tests():
    # [name, failures per thousand runs, defects only this test catches]
    return [["auth", 0, 3], ["billing", 120, 4], ["search", 40, 2], ["upload", 300, 5], ["export", 10, 1], ["webhook", 200, 3]]

def power(b, n):
    1 => r
    0 => k
    while k < n:
        r * b => r
        k + 1 => k
    return r

def tenths(n, d):
    # n / d as a percentage in tenths, rounded half up: the float quotient
    # corrected until it is the exact floor, for numerators of 50 digits
    2000 * n + d => x
    2 * d => y
    int(x / y) => q
    while q * y > x:
        q - 1 => q
    while (q + 1) * y <= x:
        q + 1 => q
    return q

def passes(f, r):
    # [numerator, denominator] of a pass within r attempts: 1 - (f/1000)^r
    return [power(1000, r) - power(f, r), power(1000, r)]

def active(week):
    # the tests still gating after week weeks of putting the flakiest aside;
    # nothing is put aside once no active test fails at all
    tests() => live
    0 => w
    while w < week:
        0 => worst
        0 - 1 => at
        0 => i
        for t in live:
            if t[1] > worst:
                t[1] => worst
                i => at
            i + 1 => i
        if at >= 0:
            [] => keep
            0 => i
            for t in live:
                if i != at:
                    keep + [t] => keep
                i + 1 => i
            keep => live
        w + 1 => w
    return live

def green(live, r):
    # [numerator, denominator] of every active test passing within r tries
    1 => n
    1 => d
    for t in live:
        passes(t[1], r) => p
        n * p[0] => n
        d * p[1] => d
    return [n, d]

def defects(live):
    0 => s
    for t in live:
        s + t[2] => s
    return s
```

Python projection of flaky.eml:

```python
def tests():
    return [["auth", 0, 3], ["billing", 120, 4], ["search", 40, 2], ["upload", 300, 5], ["export", 10, 1], ["webhook", 200, 3]]

def power(b, n):
    r = 1
    k = 0
    while k < n:
        r = r * b
        k = k + 1
    return r

def tenths(n, d):
    x = 2000 * n + d
    y = 2 * d
    q = int(x / y)
    while q * y > x:
        q = q - 1
    while (q + 1) * y <= x:
        q = q + 1
    return q

def passes(f, r):
    return [power(1000, r) - power(f, r), power(1000, r)]

def active(week):
    live = tests()
    w = 0
    while w < week:
        worst = 0
        at = 0 - 1
        i = 0
        for t in live:
            if t[1] > worst:
                worst = t[1]
                at = i
            i = i + 1
        if at >= 0:
            keep = []
            i = 0
            for t in live:
                if i != at:
                    keep = keep + [t]
                i = i + 1
            live = keep
        w = w + 1
    return live

def green(live, r):
    n = 1
    d = 1
    for t in live:
        p = passes(t[1], r)
        n = n * p[0]
        d = d * p[1]
    return [n, d]

def defects(live):
    s = 0
    for t in live:
        s = s + t[2]
    return s
```

## README

# P153 - Flaky tests

A test suite whose tests fail at random, retried and quarantined, and what
its pass rate says while what it checks falls.

- `main.eml` - the menu, the suite, one week's run at a number of
  attempts, the weeks side by side, and the pass rates retries report
- `flaky.eml` - the tests, powers and percentages in tenths computed
  exactly, a test's pass within r attempts, the tests still gating after a
  number of weeks, the suite's green share and the defects it catches

How each part works:

- Six tests, each failing a share of runs on its own and each the only one
  that would catch some of 18 defects. Nothing in the code changes from
  week to week.
- As in the corpus case `retry-masked-flakiness`, a test failing a share f
  of runs passes 1 - f^r of them at r attempts: at 3 attempts a test that
  fails 20% of the time reports 99.2%, and 7 of 10 rates from 0% to 50%
  report 99.0% or more. Retries leave every test exactly as flaky; the
  report becomes a function of the retry count.
- As in the corpus case `flaky-quarantine-drift`, each week the flakiest
  test still gating is put aside: upload, webhook, billing, search, export.
  At one attempt the suite goes from green on 46.8% of runs to 100.0%,
  while the defects it catches go from 18 to 3. The green share never
  falls and the defects never rise, week to week; after week 5 nothing
  gating fails, and nothing more is put aside.

What is checked: menu choices 1 to 5; 1 to 3 attempts; a week from 0 to 6.
An empty answer cancels.

Sessions: `sessions/basic.in` gives:
- the suite; week 0 at 1 attempt and week 2 at 3 attempts;
- the weeks at 1 attempt and at 3; the pass rates retries report.

`sessions/bad-input.in` gives:
- menu choices 0 and x;
- attempts: a cancel, 0, 4 and x; then 2 attempts with a cancel, 7 and x
  for the week, and week 6;
- the weeks with a cancel and 9 attempts.

Built on the verified corpus cases `retry-masked-flakiness` (the observed
pass rate 1 - f^r, exactly, against the true rate) and
`flaky-quarantine-drift` (the flakiest test quarantined each week, the
pass rate and the defects still gated side by side).