Project P134
Log sampling
A stream of requests and four ways to keep one record in ten - every 10th, rotated, random, and every error kept - each answering the same questions about latency, errors and phases, and getting different ones wrong.
2 modules · 2 recorded sessionstext-menu UI in the terminalupdated 2026-10-17
Every screen below was recorded under CPython. When this page was built, the EML interpreter replayed each session from the same input and printed the same bytes.
About
A stream of 1000 requests, each with a latency and an error flag, and four ways to keep about one record in ten. Each sample answers the same questions, and gets different ones wrong.
main.eml- the menu, the stream, the four samplers side by side, and the period of the latency patternsampling.eml- the stream, which records each sampler keeps and what each stands for, and the answers from a sample
How each part works:
- Latency follows a pattern that repeats every 10 requests (it can be set from 1 to 20); requests 400 to 439 are a burst of slow errors, and about 1 in 100 of the rest fail.
- The samplers: - every 10th keeps requests 10, 20, 30, ...; - rotated moves the kept place along by one every ten requests; - random keeps a record when a mixed number of its index ends in 0; - errors kept keeps every error, and the rest as rotated does.
- Each kept record counts for 10 requests, except that errors kept counts each error it keeps once, so what a sample stands for is about, not exactly, 1000.
- As in the corpus case
log-sampling-bias, every 10th aliases with the pattern: it sees 1 of the 10 phases, and reports a mean latency of 57.6 ms where all 1000 give 109.2. It sees period / gcd(period, 10) phases, all of them only when the period shares no factor with 10. - No sample finds the maximum unless it keeps that one request; the program names the samplers that miss it.
What is checked: menu choices 1 to 4, and a period from 1 to 20. An empty answer cancels.
Sessions: sessions/basic.in gives: - the stream, and the samplers at a period of 10; - a period of 7, where every 10th sees every phase, and of 5, where it sees 1.
sessions/bad-input.in gives: - menu choices 0 and x; - periods: a cancel, 0, 21 and x, then 1 and 20, with the samplers.
Built on the verified corpus case log-sampling-bias (two samplers that keep as many records and answer different questions).
Recorded sessions
What the screen shows while someone uses the program. Each typed line appears after its prompt, the way a terminal shows it.
bad-input
interpreter: byte-equal== Log sampling ==
A stream of requests and four ways to keep about one record in ten: which questions can each sample still answer?
1) the stream 2) compare samplers 3) period 4) quit
choice> 0
Pick a number from 1 to 4.
1) the stream 2) compare samplers 3) period 4) quit
choice> x
Pick a number from 1 to 4.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)>
Cancelled.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 0
Type a whole number from 1 to 20.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 21
Type a whole number from 1 to 20.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> x
Type a whole number from 1 to 20.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 1
The latency pattern now repeats every 1 requests.
1) the stream 2) compare samplers 3) period 4) quit
choice> 2
The same questions, from all 1000 and from each sample (period 1):
question all every 10th rotated random errors kept
records kept 1000 100 100 101 142
stand for 1000 1000 1000 1010 997
mean latency 57.3 57.6 57.4 56.8 57.3
p90 latency 44 44 44 44 44
max latency 449 445 449 428 449
error rate 4.7% 4.0% 5.0% 4.0% 4.7%
phases seen 1 of 1 1 of 1 1 of 1 1 of 1 1 of 1
Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.
The maximum, 449 ms, is missed by every 10th and random: a sample finds it only by keeping that one request.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 20
The latency pattern now repeats every 20 requests.
1) the stream 2) compare samplers 3) period 4) quit
choice> 2
The same questions, from all 1000 and from each sample (period 20):
question all every 10th rotated random errors kept
records kept 1000 100 100 101 142
stand for 1000 1000 1000 1010 997
mean latency 166.8 115.2 168.3 174.1 167.4
p90 latency 259 164 270 260 270
max latency 449 445 449 428 449
error rate 4.7% 4.0% 5.0% 4.0% 4.7%
phases seen 20 of 20 2 of 20 10 of 20 20 of 20 20 of 20
Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.
Every 10th sees 2 of the 20 phases of the pattern, and reports them as the whole: the pattern's period shares a factor with 10.
The maximum, 449 ms, is missed by every 10th and random: a sample finds it only by keeping that one request.
1) the stream 2) compare samplers 3) period 4) quit
choice> 4
Bye.
What was typed (17 lines)
0
x
3
3
0
3
21
3
x
3
1
2
3
20
2
4
basic
interpreter: byte-equal== Log sampling ==
A stream of requests and four ways to keep about one record in ten: which questions can each sample still answer?
1) the stream 2) compare samplers 3) period 4) quit
choice> 1
1000 requests; the first 12 (latency in ms, E for an error):
52 64 76 90 100 115 126 137 151 43 56 68
Latency follows a pattern that repeats every 10 requests; requests 400 to 439 are a burst of slow errors, and about 1 in 100 of the rest fail.
1) the stream 2) compare samplers 3) period 4) quit
choice> 2
The same questions, from all 1000 and from each sample (period 10):
question all every 10th rotated random errors kept
records kept 1000 100 100 101 142
stand for 1000 1000 1000 1010 997
mean latency 109.2 57.6 110.7 112.3 110.3
p90 latency 149 44 150 148 150
max latency 449 445 449 428 449
error rate 4.7% 4.0% 5.0% 4.0% 4.7%
phases seen 10 of 10 1 of 10 10 of 10 10 of 10 10 of 10
Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.
Every 10th sees 1 of the 10 phases of the pattern, and reports them as the whole: the pattern's period shares a factor with 10.
The maximum, 449 ms, is missed by every 10th and random: a sample finds it only by keeping that one request.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 7
The latency pattern now repeats every 7 requests.
1) the stream 2) compare samplers 3) period 4) quit
choice> 2
The same questions, from all 1000 and from each sample (period 7):
question all every 10th rotated random errors kept
records kept 1000 100 100 101 142
stand for 1000 1000 1000 1010 997
mean latency 91.9 93.0 92.3 89.7 91.8
p90 latency 114 114 113 114 113
max latency 449 445 449 428 449
error rate 4.7% 4.0% 5.0% 4.0% 4.7%
phases seen 7 of 7 7 of 7 7 of 7 7 of 7 7 of 7
Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.
The maximum, 449 ms, is missed by every 10th and random: a sample finds it only by keeping that one request.
1) the stream 2) compare samplers 3) period 4) quit
choice> 3
period of the latency pattern (1 to 20)> 5
The latency pattern now repeats every 5 requests.
1) the stream 2) compare samplers 3) period 4) quit
choice> 2
The same questions, from all 1000 and from each sample (period 5):
question all every 10th rotated random errors kept
records kept 1000 100 100 101 142
stand for 1000 1000 1000 1010 997
mean latency 80.4 57.6 80.7 79.6 80.5
p90 latency 91 44 90 90 91
max latency 449 445 449 428 449
error rate 4.7% 4.0% 5.0% 4.0% 4.7%
phases seen 5 of 5 1 of 5 5 of 5 5 of 5 5 of 5
Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.
Every 10th sees 1 of the 5 phases of the pattern, and reports them as the whole: the pattern's period shares a factor with 10.
The maximum, 449 ms, is missed by every 10th and random: a sample finds it only by keeping that one request.
1) the stream 2) compare samplers 3) period 4) quit
choice> 4
Bye.
What was typed (9 lines)
1
2
3
7
2
3
5
2
4
Modules
The program as written, entry module first. Each module transpiles to its own Python file, which is what eml project run executes.
main.eml(entry)
eml# P134 log sampling: the stream, four samplers answering the same questions
# side by side, and the period of the latency pattern.
import sampling
def plain(s):
0 => i
len(s) => j
while i < j and s[i] == " ":
i + 1 => i
while j > i and s[j - 1] == " ":
j - 1 => j
return s[i:j]
def pad(s, n):
s => out
while len(out) < n:
out + " " => out
return out
def lpad(s, n):
s => out
while len(out) < n:
" " + out => out
return out
def tenths_text(t):
return str(int(t / 10)) + "." + str(t % 10)
def rate_text(errs, total):
# errors as a percentage, one decimal, rounded half up
int((2000 * errs + total) / (2 * total)) => t
while t * 2 * total > 2000 * errs + total:
t - 1 => t
while (t + 1) * 2 * total <= 2000 * errs + total:
t + 1 => t
return str(int(t / 10)) + "." + str(t % 10) + "%"
def show(period):
sampling.stream(1000, period) => rs
"1000 requests; the first 12 (latency in ms, E for an error):" ^0
"" => line
for k in [0:11]:
rs[k] => r
if line != "":
line + " " => line
line + str(r[1]) => line
if r[2]:
line + "E" => line
(" " + line) ^0
("Latency follows a pattern that repeats every " + str(period) + " requests; requests 400 to 439 are a burst of slow errors, and about 1 in 100 of the rest fail.") ^0
def compare(period):
sampling.stream(1000, period) => rs
[sampling.truth(rs, period)] => cols
for plan in sampling.samplers():
cols + [sampling.answers(sampling.sample(rs, plan), period)] => cols
("The same questions, from all 1000 and from each sample (period " + str(period) + "):") ^0
" question all every 10th rotated random errors kept" ^0
[12, 7, 14, 11, 11, 13] => w
["records kept", "stand for", "mean latency", "p90 latency", "max latency", "error rate", "phases seen"] => qs
for q in [0:6]:
" " + pad(qs[q], 12) => line
for c in [0:4]:
cols[c] => a
if q == 0:
str(a[0]) => v
elif q == 1:
str(a[1]) => v
elif q == 2:
tenths_text(a[2]) => v
elif q == 3:
str(a[3]) => v
elif q == 4:
str(a[4]) => v
elif q == 5:
rate_text(a[5], a[1]) => v
else:
str(a[6]) + " of " + str(period) => v
line + lpad(v, w[c + 1]) => line
line ^0
" Each kept record counts for 10 requests, except that errors kept counts each error it keeps once." ^0
cols[1] => e
if e[6] < period:
("Every 10th sees " + str(e[6]) + " of the " + str(period) + " phases of the pattern, and reports them as the whole: the pattern's period shares a factor with 10.") ^0
[] => missed
for c in [1:4]:
if cols[c][4] != cols[0][4]:
missed + [sampling.samplers()[c - 1]] => missed
if len(missed) > 0:
"" => who
for k in [0:len(missed) - 1]:
if k > 0 and k == len(missed) - 1:
who + " and " => who
elif k > 0:
who + ", " => who
who + missed[k] => who
("The maximum, " + str(cols[0][4]) + " ms, is missed by " + who + ": a sample finds it only by keeping that one request.") ^0
def set_period(period):
plain(input("period of the latency pattern (1 to 20)> ")) => t
if t == "":
"Cancelled." ^0
return period
if len(t) > 2:
"Type a whole number from 1 to 20." ^0
return period
for c in t:
if c < "0" or c > "9":
"Type a whole number from 1 to 20." ^0
return period
if int(t) < 1 or int(t) > 20:
"Type a whole number from 1 to 20." ^0
return period
("The latency pattern now repeats every " + t + " requests.") ^0
return int(t)
10 => period
"== Log sampling ==" ^0
"A stream of requests and four ways to keep about one record in ten: which questions can each sample still answer?" ^0
True => running
while running:
"" ^0
"1) the stream 2) compare samplers 3) period 4) quit" ^0
plain(input("choice> ")) => choice
if choice == "1":
show(period)
elif choice == "2":
compare(period)
elif choice == "3":
set_period(period) => period
elif choice == "4":
False => running
else:
"Pick a number from 1 to 4." ^0
"Bye." ^0
Python projection (main.py)
import sampling
def plain(s):
i = 0
j = len(s)
while i < j and s[i] == " ":
i = i + 1
while j > i and s[j - 1] == " ":
j = j - 1
return s[i:j]
def pad(s, n):
out = s
while len(out) < n:
out = out + " "
return out
def lpad(s, n):
out = s
while len(out) < n:
out = " " + out
return out
def tenths_text(t):
return str(int(t / 10)) + "." + str(t % 10)
def rate_text(errs, total):
t = int((2000 * errs + total) / (2 * total))
while t * 2 * total > 2000 * errs + total:
t = t - 1
while (t + 1) * 2 * total <= 2000 * errs + total:
t = t + 1
return str(int(t / 10)) + "." + str(t % 10) + "%"
def show(period):
rs = sampling.stream(1000, period)
print("1000 requests; the first 12 (latency in ms, E for an error):")
line = ""
for k in range(0, 12):
r = rs[k]
if line != "":
line = line + " "
line = line + str(r[1])
if r[2]:
line = line + "E"
print(" " + line)
print("Latency follows a pattern that repeats every " + str(period) + " requests; requests 400 to 439 are a burst of slow errors, and about 1 in 100 of the rest fail.")
def compare(period):
rs = sampling.stream(1000, period)
cols = [sampling.truth(rs, period)]
for plan in sampling.samplers():
cols = cols + [sampling.answers(sampling.sample(rs, plan), period)]
print("The same questions, from all 1000 and from each sample (period " + str(period) + "):")
print(" question all every 10th rotated random errors kept")
w = [12, 7, 14, 11, 11, 13]
qs = ["records kept", "stand for", "mean latency", "p90 latency", "max latency", "error rate", "phases seen"]
for q in range(0, 7):
line = " " + pad(qs[q], 12)
for c in range(0, 5):
a = cols[c]
if q == 0:
v = str(a[0])
elif q == 1:
v = str(a[1])
elif q == 2:
v = tenths_text(a[2])
elif q == 3:
v = str(a[3])
elif q == 4:
v = str(a[4])
elif q == 5:
v = rate_text(a[5], a[1])
else:
v = str(a[6]) + " of " + str(period)
line = line + lpad(v, w[c + 1])
print(line)
print(" Each kept record counts for 10 requests, except that errors kept counts each error it keeps once.")
e = cols[1]
if e[6] < period:
print("Every 10th sees " + str(e[6]) + " of the " + str(period) + " phases of the pattern, and reports them as the whole: the pattern's period shares a factor with 10.")
missed = []
for c in range(1, 5):
if cols[c][4] != cols[0][4]:
missed = missed + [sampling.samplers()[c - 1]]
if len(missed) > 0:
who = ""
for k in range(0, len(missed)):
if k > 0 and k == len(missed) - 1:
who = who + " and "
elif k > 0:
who = who + ", "
who = who + missed[k]
print("The maximum, " + str(cols[0][4]) + " ms, is missed by " + who + ": a sample finds it only by keeping that one request.")
def set_period(period):
t = plain(input("period of the latency pattern (1 to 20)> "))
if t == "":
print("Cancelled.")
return period
if len(t) > 2:
print("Type a whole number from 1 to 20.")
return period
for c in t:
if c < "0" or c > "9":
print("Type a whole number from 1 to 20.")
return period
if int(t) < 1 or int(t) > 20:
print("Type a whole number from 1 to 20.")
return period
print("The latency pattern now repeats every " + t + " requests.")
return int(t)
period = 10
print("== Log sampling ==")
print("A stream of requests and four ways to keep about one record in ten: which questions can each sample still answer?")
running = True
while running:
print("")
print("1) the stream 2) compare samplers 3) period 4) quit")
choice = plain(input("choice> "))
if choice == "1":
show(period)
elif choice == "2":
compare(period)
elif choice == "3":
period = set_period(period)
elif choice == "4":
running = False
else:
print("Pick a number from 1 to 4.")
print("Bye.")
sampling.eml
eml# P134 log sampling - a stream of requests, each with a latency and an
# error flag, and four ways to keep about one record in ten. As in the
# corpus case log-sampling-bias, the samplers keep about as many records
# and answer different questions: keeping every 10th aliases with a
# latency pattern whose period shares a factor with 10, and a sample that
# keeps every error has to weigh the rest back up.
#
# A record is [index, latency in ms, error]. The latency follows a pattern
# of the given period; requests 400 to 439 are an error burst, slow too.
def mix(i, salt):
i * 7919 + salt * 104729 + 12345 => x
for k in [1:3]:
(x * x + 7) % 1000000007 => x
return x % 1000000
def stream(n, period):
[] => out
for i in [1:n]:
40 + 12 * (i % period) + mix(i, 1) % 5 => lat
False => err
if i >= 400 and i <= 439:
True => err
400 + mix(i, 2) % 50 => lat
elif mix(i, 3) % 100 == 0:
True => err
out + [[i, lat, err]] => out
return out
def samplers():
return ["every 10th", "rotated", "random", "errors kept"]
def keep(plan, r):
# [kept, weight]: how many records of the stream this one stands for
r[0] => i
if plan == "every 10th":
return [i % 10 == 0, 10]
if plan == "rotated":
return [i % 10 == int(i / 10) % 10, 10]
if plan == "random":
return [mix(i, 5) % 10 == 0, 10]
# errors kept: every error, and the rest as rotated keeps them
if r[2]:
return [True, 1]
return [i % 10 == int(i / 10) % 10, 10]
def sample(records, plan):
# [record, weight] for the records kept
[] => out
for r in records:
keep(plan, r) => k
if k[0]:
out + [[r, k[1]]] => out
return out
def answers(ws, period):
# [records kept, records they stand for, mean latency in tenths, p90
# latency, max latency, errors stood for, phases seen]
0 => n
0 => total
0 => lat
0 => errs
0 => most
{} => phases
for p in ws:
n + 1 => n
total + p[1] => total
lat + p[0][1] * p[1] => lat
if p[0][2]:
errs + p[1] => errs
if p[0][1] > most:
p[0][1] => most
1 => phases[p[0][0] % period]
0 => mean
0 => p90
if total > 0:
int((20 * lat + total) / (2 * total)) => mean
while mean * 2 * total > 20 * lat + total:
mean - 1 => mean
while (mean + 1) * 2 * total <= 20 * lat + total:
mean + 1 => mean
# the weight at each latency, then the smallest latency at or below
# which nine tenths of the weight lies
{} => at
for p in ws:
if p[0][1] in at:
at[p[0][1]] + p[1] => at[p[0][1]]
else:
p[1] => at[p[0][1]]
0 => run
False => found
for v in [0:most]:
if not found and v in at:
run + at[v] => run
if run * 10 >= total * 9:
v => p90
True => found
return [n, total, mean, p90, most, errs, len(phases)]
def truth(records, period):
[] => ws
for r in records:
ws + [[r, 1]] => ws
return answers(ws, period)
Python projection (sampling.py)
def mix(i, salt):
x = i * 7919 + salt * 104729 + 12345
for k in range(1, 4):
x = (x * x + 7) % 1000000007
return x % 1000000
def stream(n, period):
out = []
for i in range(1, n+1):
lat = 40 + 12 * (i % period) + mix(i, 1) % 5
err = False
if i >= 400 and i <= 439:
err = True
lat = 400 + mix(i, 2) % 50
elif mix(i, 3) % 100 == 0:
err = True
out = out + [[i, lat, err]]
return out
def samplers():
return ["every 10th", "rotated", "random", "errors kept"]
def keep(plan, r):
i = r[0]
if plan == "every 10th":
return [i % 10 == 0, 10]
if plan == "rotated":
return [i % 10 == int(i / 10) % 10, 10]
if plan == "random":
return [mix(i, 5) % 10 == 0, 10]
if r[2]:
return [True, 1]
return [i % 10 == int(i / 10) % 10, 10]
def sample(records, plan):
out = []
for r in records:
k = keep(plan, r)
if k[0]:
out = out + [[r, k[1]]]
return out
def answers(ws, period):
n = 0
total = 0
lat = 0
errs = 0
most = 0
phases = {}
for p in ws:
n = n + 1
total = total + p[1]
lat = lat + p[0][1] * p[1]
if p[0][2]:
errs = errs + p[1]
if p[0][1] > most:
most = p[0][1]
phases[p[0][0] % period] = 1
mean = 0
p90 = 0
if total > 0:
mean = int((20 * lat + total) / (2 * total))
while mean * 2 * total > 20 * lat + total:
mean = mean - 1
while (mean + 1) * 2 * total <= 20 * lat + total:
mean = mean + 1
at = {}
for p in ws:
if p[0][1] in at:
at[p[0][1]] = at[p[0][1]] + p[1]
else:
at[p[0][1]] = p[1]
run = 0
found = False
for v in range(0, most+1):
if not found and v in at:
run = run + at[v]
if run * 10 >= total * 9:
p90 = v
found = True
return [n, total, mean, p90, most, errs, len(phases)]
def truth(records, period):
ws = []
for r in records:
ws = ws + [[r, 1]]
return answers(ws, period)