Project P153

Flaky tests

A suite whose tests fail at random, retried and quarantined week by week: the pass rate retries report, and a suite that grows greener every week while it catches fewer defects.

2 modules · 2 recorded sessionstext-menu UI in the terminalupdated 2026-10-18

Every screen below was recorded under CPython. When this page was built, the EML interpreter replayed each session from the same input and printed the same bytes.

Raw markdown (agent layer) →JSON →

About

A test suite whose tests fail at random, retried and quarantined, and what its pass rate says while what it checks falls.

main.eml - the menu, the suite, one week's run at a number of attempts, the weeks side by side, and the pass rates retries report
flaky.eml - the tests, powers and percentages in tenths computed exactly, a test's pass within r attempts, the tests still gating after a number of weeks, the suite's green share and the defects it catches

How each part works:

Six tests, each failing a share of runs on its own and each the only one that would catch some of 18 defects. Nothing in the code changes from week to week.
As in the corpus case retry-masked-flakiness, a test failing a share f of runs passes 1 - f^r of them at r attempts: at 3 attempts a test that fails 20% of the time reports 99.2%, and 7 of 10 rates from 0% to 50% report 99.0% or more. Retries leave every test exactly as flaky; the report becomes a function of the retry count.
As in the corpus case flaky-quarantine-drift, each week the flakiest test still gating is put aside: upload, webhook, billing, search, export. At one attempt the suite goes from green on 46.8% of runs to 100.0%, while the defects it catches go from 18 to 3. The green share never falls and the defects never rise, week to week; after week 5 nothing gating fails, and nothing more is put aside.

What is checked: menu choices 1 to 5; 1 to 3 attempts; a week from 0 to 6. An empty answer cancels.

Sessions: sessions/basic.in gives: - the suite; week 0 at 1 attempt and week 2 at 3 attempts; - the weeks at 1 attempt and at 3; the pass rates retries report.

sessions/bad-input.in gives: - menu choices 0 and x; - attempts: a cancel, 0, 4 and x; then 2 attempts with a cancel, 7 and x for the week, and week 6; - the weeks with a cancel and 9 attempts.

Built on the verified corpus cases retry-masked-flakiness (the observed pass rate 1 - f^r, exactly, against the true rate) and flaky-quarantine-drift (the flakiest test quarantined each week, the pass rate and the defects still gated side by side).

Recorded sessions

What the screen shows while someone uses the program. Each typed line appears after its prompt, the way a terminal shows it.

bad-input

interpreter: byte-equal

== Flaky tests ==
A suite with tests that fail at random, retried and quarantined, and what its pass rate says.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 0
Pick a number from 1 to 5.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> x
Pick a number from 1 to 5.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 0
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 4
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> x
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 7
Type a week from 0 to 6.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> x
Type a week from 0 to 6.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 2
week (0 to 6)> 6
Week 6, 2 attempts each, 1 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 2: 100.0%
The suite is green on 100.0% of runs and catches 3 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 
Cancelled.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 9
Type 1, 2 or 3.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 5
Bye.

What was typed (27 lines)

basic

interpreter: byte-equal

== Flaky tests ==
A suite with tests that fail at random, retried and quarantined, and what its pass rate says.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 1
The tests, how often each fails on its own, and the defects only it would catch:
  auth       0.0% of runs fail   3 defects
  billing   12.0% of runs fail   4 defects
  search     4.0% of runs fail   2 defects
  upload    30.0% of runs fail   5 defects
  export     1.0% of runs fail   1 defect
  webhook   20.0% of runs fail   3 defects
  18 defects in all; nothing in the code changes from week to week.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 1
week (0 to 6)> 0
Week 0, 1 attempt each, 6 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 1: 100.0%
  billing  fails  12.0% of runs, passes within 1:  88.0%
  search   fails   4.0% of runs, passes within 1:  96.0%
  upload   fails  30.0% of runs, passes within 1:  70.0%
  export   fails   1.0% of runs, passes within 1:  99.0%
  webhook  fails  20.0% of runs, passes within 1:  80.0%
The suite is green on 46.8% of runs and catches 18 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 2
attempts per test (1 to 3)> 3
week (0 to 6)> 2
Week 2, 3 attempts each, 4 of 6 tests still gating:
  auth     fails   0.0% of runs, passes within 3: 100.0%
  billing  fails  12.0% of runs, passes within 3:  99.8%
  search   fails   4.0% of runs, passes within 3: 100.0%
  export   fails   1.0% of runs, passes within 3: 100.0%
The suite is green on 99.8% of runs and catches 10 of 18 defects.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 1
Each week the flakiest test still gating is put aside, 1 attempt each:
  week  put aside  tests  green   defects caught
  0     -              6   46.8%      18 of 18
  1     upload         5   66.9%      13 of 18
  2     webhook        4   83.6%      10 of 18
  3     billing        3   95.0%       6 of 18
  4     search         2   99.0%       4 of 18
  5     export         1  100.0%       3 of 18
  6     -              1  100.0%       3 of 18
Green went from 46.8% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.
After week 5 nothing gating fails, and nothing more is put aside.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 3
attempts per test (1 to 3)> 3
Each week the flakiest test still gating is put aside, 3 attempts each:
  week  put aside  tests  green   defects caught
  0     -              6   96.3%      18 of 18
  1     upload         5   99.0%      13 of 18
  2     webhook        4   99.8%      10 of 18
  3     billing        3  100.0%       6 of 18
  4     search         2  100.0%       4 of 18
  5     export         1  100.0%       3 of 18
  6     -              1  100.0%       3 of 18
Green went from 96.3% to 100.0% while the defects caught went from 18 to 3: the dashboard's number rose as what the suite checks fell.
After week 5 nothing gating fails, and nothing more is put aside.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 4
What a test reports as its pass rate, by how often it really fails and the attempts allowed:
  fails   1 attempt   2 attempts  3 attempts
  0.0%    100.0%      100.0%      100.0%
  1.0%    99.0%       100.0%      100.0%
  2.0%    98.0%       100.0%      100.0%
  5.0%    95.0%       99.8%       100.0%
  10.0%   90.0%       99.0%       99.9%
  15.0%   85.0%       97.8%       99.7%
  20.0%   80.0%       96.0%       99.2%
  30.0%   70.0%       91.0%       97.3%
  40.0%   60.0%       84.0%       93.6%
  50.0%   50.0%       75.0%       87.5%
  at 1: 2 of the 10 rates report 99.0% or more
  at 2: 5 of the 10 rates report 99.0% or more
  at 3: 7 of the 10 rates report 99.0% or more
Retries leave every test exactly as flaky; they make the report a function of the retry count.

1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit
choice> 5
Bye.

What was typed (13 lines)

Modules

The program as written, entry module first. Each module transpiles to its own Python file, which is what eml project run executes.

main.eml(entry)

eml

# P153 flaky tests: the suite, one week's run at a number of attempts, the
# weeks of quarantine side by side, and the pass rates retries report.
import flaky

def plain(s):
    0 => i
    len(s) => j
    while i < j and s[i] == " ":
        i + 1 => i
    while j > i and s[j - 1] == " ":
        j - 1 => j
    return s[i:j]

def pad(s, n):
    s => out
    while len(out) < n:
        out + " " => out
    return out

def lpad(s, n):
    s => out
    while len(out) < n:
        " " + out => out
    return out

def times(n, one, many):
    if n == 1:
        return "1 " + one
    return str(n) + " " + many

def pct(t):
    t % 10 => d
    return str(int((t - d) / 10)) + "." + str(d) + "%"

def ask(prompt, low, high):
    # a whole number from low to high, -1 for a cancel, -2 when not valid
    plain(input(prompt)) => t
    if t == "":
        return 0 - 1
    if len(t) > 2:
        return 0 - 2
    for c in t:
        if c < "0" or c > "9":
            return 0 - 2
    if int(t) < low or int(t) > high:
        return 0 - 2
    return int(t)

def show_suite():
    "The tests, how often each fails on its own, and the defects only it would catch:" ^0
    0 => total
    for t in flaky.tests():
        ("  " + pad(t[0], 9) + lpad(pct(t[1]), 6) + " of runs fail" + "   " + times(t[2], "defect", "defects")) ^0
        total + t[2] => total
    ("  " + str(total) + " defects in all; nothing in the code changes from week to week.") ^0

def week_run():
    ask("attempts per test (1 to 3)> ", 1, 3) => r
    if r == 0 - 1:
        "Cancelled." ^0
        return
    if r < 0:
        "Type 1, 2 or 3." ^0
        return
    ask("week (0 to 6)> ", 0, 6) => w
    if w == 0 - 1:
        "Cancelled." ^0
        return
    if w < 0:
        "Type a week from 0 to 6." ^0
        return
    flaky.active(w) => live
    ("Week " + str(w) + ", " + times(r, "attempt", "attempts") + " each, " + str(len(live)) + " of " + str(len(flaky.tests())) + " tests still gating:") ^0
    for t in live:
        flaky.passes(t[1], r) => p
        ("  " + pad(t[0], 9) + "fails " + lpad(pct(t[1]), 6) + " of runs, passes within " + str(r) + ": " + lpad(pct(flaky.tenths(p[0], p[1])), 6)) ^0
    flaky.green(live, r) => g
    ("The suite is green on " + pct(flaky.tenths(g[0], g[1])) + " of runs and catches " + str(flaky.defects(live)) + " of " + str(flaky.defects(flaky.tests())) + " defects.") ^0

def drift():
    ask("attempts per test (1 to 3)> ", 1, 3) => r
    if r == 0 - 1:
        "Cancelled." ^0
        return
    if r < 0:
        "Type 1, 2 or 3." ^0
        return
    ("Each week the flakiest test still gating is put aside, " + times(r, "attempt", "attempts") + " each:") ^0
    ("  " + pad("week", 6) + pad("put aside", 11) + "tests  green   defects caught") ^0
    0 => w
    flaky.tests() => before
    0 => low
    0 => lastgreen
    while w <= 6:
        flaky.active(w) => live
        "-" => gone
        if len(live) < len(before):
            for t in before:
                False => still
                for u in live:
                    if u[0] == t[0]:
                        True => still
                if not still:
                    t[0] => gone
        flaky.green(live, r) => g
        flaky.tenths(g[0], g[1]) => gr
        if w == 0:
            gr => low
        gr => lastgreen
        ("  " + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + " of " + str(flaky.defects(flaky.tests()))) ^0
        live => before
        w + 1 => w
    flaky.active(6) => last
    ("Green went from " + pct(low) + " to " + pct(lastgreen) + " while the defects caught went from " + str(flaky.defects(flaky.tests())) + " to " + str(flaky.defects(last)) + ": the dashboard's number rose as what the suite checks fell.") ^0
    "After week 5 nothing gating fails, and nothing more is put aside." ^0

def sweep():
    "What a test reports as its pass rate, by how often it really fails and the attempts allowed:" ^0
    ("  " + pad("fails", 8) + pad("1 attempt", 12) + pad("2 attempts", 12) + "3 attempts") ^0
    [0, 10, 20, 50, 100, 150, 200, 300, 400, 500] => fs
    for f in fs:
        "  " + pad(pct(f), 8) => line
        1 => r
        while r <= 3:
            flaky.passes(f, r) => p
            if r < 3:
                line + pad(pct(flaky.tenths(p[0], p[1])), 12) => line
            else:
                line + pct(flaky.tenths(p[0], p[1])) => line
            r + 1 => r
        line ^0
    1 => r
    while r <= 3:
        0 => over
        for f in fs:
            flaky.passes(f, r) => p
            if flaky.tenths(p[0], p[1]) >= 990:
                over + 1 => over
        ("  at " + str(r) + ": " + str(over) + " of the " + str(len(fs)) + " rates report 99.0% or more") ^0
        r + 1 => r
    "Retries leave every test exactly as flaky; they make the report a function of the retry count." ^0

"== Flaky tests ==" ^0
"A suite with tests that fail at random, retried and quarantined, and what its pass rate says." ^0
True => running
while running:
    "" ^0
    "1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit" ^0
    plain(input("choice> ")) => choice
    if choice == "1":
        show_suite()
    elif choice == "2":
        week_run()
    elif choice == "3":
        drift()
    elif choice == "4":
        sweep()
    elif choice == "5":
        False => running
    else:
        "Pick a number from 1 to 5." ^0
"Bye." ^0

Python projection (main.py)

import flaky

def plain(s):
    i = 0
    j = len(s)
    while i < j and s[i] == " ":
        i = i + 1
    while j > i and s[j - 1] == " ":
        j = j - 1
    return s[i:j]

def pad(s, n):
    out = s
    while len(out) < n:
        out = out + " "
    return out

def lpad(s, n):
    out = s
    while len(out) < n:
        out = " " + out
    return out

def times(n, one, many):
    if n == 1:
        return "1 " + one
    return str(n) + " " + many

def pct(t):
    d = t % 10
    return str(int((t - d) / 10)) + "." + str(d) + "%"

def ask(prompt, low, high):
    t = plain(input(prompt))
    if t == "":
        return 0 - 1
    if len(t) > 2:
        return 0 - 2
    for c in t:
        if c < "0" or c > "9":
            return 0 - 2
    if int(t) < low or int(t) > high:
        return 0 - 2
    return int(t)

def show_suite():
    print("The tests, how often each fails on its own, and the defects only it would catch:")
    total = 0
    for t in flaky.tests():
        print("  " + pad(t[0], 9) + lpad(pct(t[1]), 6) + " of runs fail" + "   " + times(t[2], "defect", "defects"))
        total = total + t[2]
    print("  " + str(total) + " defects in all; nothing in the code changes from week to week.")

def week_run():
    r = ask("attempts per test (1 to 3)> ", 1, 3)
    if r == 0 - 1:
        print("Cancelled.")
        return
    if r < 0:
        print("Type 1, 2 or 3.")
        return
    w = ask("week (0 to 6)> ", 0, 6)
    if w == 0 - 1:
        print("Cancelled.")
        return
    if w < 0:
        print("Type a week from 0 to 6.")
        return
    live = flaky.active(w)
    print("Week " + str(w) + ", " + times(r, "attempt", "attempts") + " each, " + str(len(live)) + " of " + str(len(flaky.tests())) + " tests still gating:")
    for t in live:
        p = flaky.passes(t[1], r)
        print("  " + pad(t[0], 9) + "fails " + lpad(pct(t[1]), 6) + " of runs, passes within " + str(r) + ": " + lpad(pct(flaky.tenths(p[0], p[1])), 6))
    g = flaky.green(live, r)
    print("The suite is green on " + pct(flaky.tenths(g[0], g[1])) + " of runs and catches " + str(flaky.defects(live)) + " of " + str(flaky.defects(flaky.tests())) + " defects.")

def drift():
    r = ask("attempts per test (1 to 3)> ", 1, 3)
    if r == 0 - 1:
        print("Cancelled.")
        return
    if r < 0:
        print("Type 1, 2 or 3.")
        return
    print("Each week the flakiest test still gating is put aside, " + times(r, "attempt", "attempts") + " each:")
    print("  " + pad("week", 6) + pad("put aside", 11) + "tests  green   defects caught")
    w = 0
    before = flaky.tests()
    low = 0
    lastgreen = 0
    while w <= 6:
        live = flaky.active(w)
        gone = "-"
        if len(live) < len(before):
            for t in before:
                still = False
                for u in live:
                    if u[0] == t[0]:
                        still = True
                if not still:
                    gone = t[0]
        g = flaky.green(live, r)
        gr = flaky.tenths(g[0], g[1])
        if w == 0:
            low = gr
        lastgreen = gr
        print("  " + pad(str(w), 6) + pad(gone, 11) + lpad(str(len(live)), 5) + lpad(pct(gr), 8) + lpad(str(flaky.defects(live)), 8) + " of " + str(flaky.defects(flaky.tests())))
        before = live
        w = w + 1
    last = flaky.active(6)
    print("Green went from " + pct(low) + " to " + pct(lastgreen) + " while the defects caught went from " + str(flaky.defects(flaky.tests())) + " to " + str(flaky.defects(last)) + ": the dashboard's number rose as what the suite checks fell.")
    print("After week 5 nothing gating fails, and nothing more is put aside.")

def sweep():
    print("What a test reports as its pass rate, by how often it really fails and the attempts allowed:")
    print("  " + pad("fails", 8) + pad("1 attempt", 12) + pad("2 attempts", 12) + "3 attempts")
    fs = [0, 10, 20, 50, 100, 150, 200, 300, 400, 500]
    for f in fs:
        line = "  " + pad(pct(f), 8)
        r = 1
        while r <= 3:
            p = flaky.passes(f, r)
            if r < 3:
                line = line + pad(pct(flaky.tenths(p[0], p[1])), 12)
            else:
                line = line + pct(flaky.tenths(p[0], p[1]))
            r = r + 1
        print(line)
    r = 1
    while r <= 3:
        over = 0
        for f in fs:
            p = flaky.passes(f, r)
            if flaky.tenths(p[0], p[1]) >= 990:
                over = over + 1
        print("  at " + str(r) + ": " + str(over) + " of the " + str(len(fs)) + " rates report 99.0% or more")
        r = r + 1
    print("Retries leave every test exactly as flaky; they make the report a function of the retry count.")

print("== Flaky tests ==")
print("A suite with tests that fail at random, retried and quarantined, and what its pass rate says.")
running = True
while running:
    print("")
    print("1) the suite  2) one week's run  3) the weeks  4) retries and pass rates  5) quit")
    choice = plain(input("choice> "))
    if choice == "1":
        show_suite()
    elif choice == "2":
        week_run()
    elif choice == "3":
        drift()
    elif choice == "4":
        sweep()
    elif choice == "5":
        running = False
    else:
        print("Pick a number from 1 to 5.")
print("Bye.")

flaky.eml

eml

# P153 flaky tests - a test suite with tests that fail at random, retried
# and quarantined. As in the corpus case retry-masked-flakiness, a test that
# fails a share f of runs passes 1 - f^r of them at r attempts, so retries
# fold every unhealthy test into the narrow band below 100% where the
# healthy ones are. As in the corpus case flaky-quarantine-drift, putting
# the flakiest test aside each week makes the suite greener every week while
# it checks less every week, and the number that shows it - the defects the
# active tests still catch - is not the one on the dashboard. Rates are in
# per mille and every probability is an exact fraction.

def tests():
    # [name, failures per thousand runs, defects only this test catches]
    return [["auth", 0, 3], ["billing", 120, 4], ["search", 40, 2], ["upload", 300, 5], ["export", 10, 1], ["webhook", 200, 3]]

def power(b, n):
    1 => r
    0 => k
    while k < n:
        r * b => r
        k + 1 => k
    return r

def tenths(n, d):
    # n / d as a percentage in tenths, rounded half up: the float quotient
    # corrected until it is the exact floor, for numerators of 50 digits
    2000 * n + d => x
    2 * d => y
    int(x / y) => q
    while q * y > x:
        q - 1 => q
    while (q + 1) * y <= x:
        q + 1 => q
    return q

def passes(f, r):
    # [numerator, denominator] of a pass within r attempts: 1 - (f/1000)^r
    return [power(1000, r) - power(f, r), power(1000, r)]

def active(week):
    # the tests still gating after week weeks of putting the flakiest aside;
    # nothing is put aside once no active test fails at all
    tests() => live
    0 => w
    while w < week:
        0 => worst
        0 - 1 => at
        0 => i
        for t in live:
            if t[1] > worst:
                t[1] => worst
                i => at
            i + 1 => i
        if at >= 0:
            [] => keep
            0 => i
            for t in live:
                if i != at:
                    keep + [t] => keep
                i + 1 => i
            keep => live
        w + 1 => w
    return live

def green(live, r):
    # [numerator, denominator] of every active test passing within r tries
    1 => n
    1 => d
    for t in live:
        passes(t[1], r) => p
        n * p[0] => n
        d * p[1] => d
    return [n, d]

def defects(live):
    0 => s
    for t in live:
        s + t[2] => s
    return s

Python projection (flaky.py)

def tests():
    return [["auth", 0, 3], ["billing", 120, 4], ["search", 40, 2], ["upload", 300, 5], ["export", 10, 1], ["webhook", 200, 3]]

def power(b, n):
    r = 1
    k = 0
    while k < n:
        r = r * b
        k = k + 1
    return r

def tenths(n, d):
    x = 2000 * n + d
    y = 2 * d
    q = int(x / y)
    while q * y > x:
        q = q - 1
    while (q + 1) * y <= x:
        q = q + 1
    return q

def passes(f, r):
    return [power(1000, r) - power(f, r), power(1000, r)]

def active(week):
    live = tests()
    w = 0
    while w < week:
        worst = 0
        at = 0 - 1
        i = 0
        for t in live:
            if t[1] > worst:
                worst = t[1]
                at = i
            i = i + 1
        if at >= 0:
            keep = []
            i = 0
            for t in live:
                if i != at:
                    keep = keep + [t]
                i = i + 1
            live = keep
        w = w + 1
    return live

def green(live, r):
    n = 1
    d = 1
    for t in live:
        p = passes(t[1], r)
        n = n * p[0]
        d = d * p[1]
    return [n, d]

def defects(live):
    s = 0
    for t in live:
        s = s + t[2]
    return s