The benchmark was isolated and the machine was shared

# Self-authored for the EML case corpus (no external origin). The benchmark
# pins a core, disables frequency scaling, warms the caches and reports a
# confidence interval of four parts in a thousand. What it measures is computed
# below.
#
# The methodology is careful. The process is pinned to one core, the governor is
# set to performance so no clock changes under it, a thousand iterations run
# after a warm-up that is discarded, outliers are reported rather than dropped,
# and the interval is computed properly rather than as a range. This is better
# than most benchmarks in most repositories.
#
# All of that isolates the measurement WITHIN the process. It runs on a shared
# runner, so the other tenants of that machine are outside every one of those
# controls — the pinned core shares a memory controller and a last-level cache
# with cores this benchmark does not own.
#
# The interval is tight and the same commit measures differently on Tuesday.

1000 => iterations
40 => reported_interval_per_myriad
12400 => nanoseconds_on_a_quiet_runner
19800 => nanoseconds_on_a_busy_runner
0 => methodology_faults_found_in_review

nanoseconds_on_a_busy_runner - nanoseconds_on_a_quiet_runner => spread_ns
int(spread_ns * 10000 / nanoseconds_on_a_quiet_runner) => between_run_spread_per_myriad
int(between_run_spread_per_myriad / reported_interval_per_myriad) => spread_over_interval

"iterations                   : " + str(iterations) ^0
"reported interval            : " + str(reported_interval_per_myriad) + " per ten thousand" ^0
"" ^0
"same commit, quiet runner, ns: " + str(nanoseconds_on_a_quiet_runner) ^0
"same commit, busy runner, ns : " + str(nanoseconds_on_a_busy_runner) ^0
"between-run spread           : " + str(between_run_spread_per_myriad) + " per ten thousand" ^0
"the spread is the interval times : " + str(spread_over_interval) ^0
"" ^0

# ---- what the methodology verified ----

"the benchmark's controls" ^0
"  core pinned          : yes" ^0
"  frequency scaling    : disabled" ^0
"  warm-up              : run and discarded" ^0
"  iterations           : " + str(iterations) ^0
"  outliers             : reported, not dropped" ^0
"  interval computed properly : yes" ^0
"  faults found in review : " + str(methodology_faults_found_in_review) ^0
"  verdict              : ISOLATED" ^0
"" ^0
"  every one of those is a real control and each removes a" ^0
"  real source of variance" ^0
"" ^0

# ---- what the interval is an interval of ----

"the reported " + str(reported_interval_per_myriad) + " per ten thousand" ^0
"  computed over        : " + str(iterations) + " iterations in one process" ^0
"  what it bounds       : how much this run's own samples" ^0
"    disagree with each other" ^0
"  what it does not bound : how much this run disagrees" ^0
"    with the next one" ^0
"" ^0
"  a tight interval over one population is not a claim" ^0
"  about a second population it never sampled" ^0
"" ^0

# ---- what is outside the pin ----

"the shared runner" ^0
"  cores this process owns : 1" ^0
"  cores the machine has   : shared with other jobs" ^0
"  last-level cache        : shared" ^0
"  memory bandwidth        : shared" ^0
"  a control for any of those : none, and pinning is what" ^0
"    makes it look as though there is" ^0
"" ^0

# ---- what the regression gate does with it ----

# The gate fails a change whose measurement moves more than the interval. The
# interval is 40 per ten thousand and the machine moves 5967, so the gate is
# deciding on which runner it landed.
int(between_run_spread_per_myriad / 2) => noise_in_either_direction

"the regression gate" ^0
"  fails a change moving more than : " + str(reported_interval_per_myriad) + " per ten thousand" ^0
"  machine noise, either direction : up to " + str(noise_in_either_direction) ^0
"  so the gate decides on           : which runner it got" ^0
"  a real regression it would catch : one larger than the" ^0
"    noise, which is a regression nobody needs a benchmark" ^0
"    to notice" ^0
"" ^0

# ---- null control ----

# The same benchmark, reported as the median of five runs on different runners
# with the between-run spread as the interval.
between_run_spread_per_myriad => nc_reported_interval_per_myriad
5 => nc_runs_across_runners

"null control - the interval computed across runners" ^0
"  methodology faults  : " + str(methodology_faults_found_in_review) + ", unchanged" ^0
"  runs across runners : " + str(nc_runs_across_runners) ^0
"  reported interval   : " + str(nc_reported_interval_per_myriad) + " per ten thousand" ^0
"  the benchmark did not get noisier; the interval started" ^0
"  covering the variance that decides the gate" ^0
"" ^0

# ---- the rule ----

"what a careful benchmark guarantees" ^0
"  this measurement is repeatable within itself : exactly" ^0
"  this measurement is comparable to the last one : not" ^0
"    addressed; every control it applies is inside the" ^0
"    process, and the variance that matters is outside" ^0
"" ^0
"an interval describes the population it was computed over;" ^0
"comparing two runs is a question about a population of runs," ^0
"and one run cannot sample it" ^0
"" ^0

"The methodology is careful and review found " + str(methodology_faults_found_in_review) + " faults: pinned core, no frequency" ^0
"scaling, discarded warm-up, " + str(iterations) + " iterations, outliers reported. Its interval of" ^0
str(reported_interval_per_myriad) + " per ten thousand covers disagreement among its own samples, while the same" ^0
"commit measures " + str(nanoseconds_on_a_quiet_runner) + " ns and " + str(nanoseconds_on_a_busy_runner) + " ns on two runners - a spread of" ^0
str(between_run_spread_per_myriad) + " per ten thousand, " + str(spread_over_interval) + " times the interval the gate compares against." ^0

iterations = 1000
reported_interval_per_myriad = 40
nanoseconds_on_a_quiet_runner = 12400
nanoseconds_on_a_busy_runner = 19800
methodology_faults_found_in_review = 0
spread_ns = nanoseconds_on_a_busy_runner - nanoseconds_on_a_quiet_runner
between_run_spread_per_myriad = int(spread_ns * 10000 / nanoseconds_on_a_quiet_runner)
spread_over_interval = int(between_run_spread_per_myriad / reported_interval_per_myriad)
print("iterations                   : " + str(iterations))
print("reported interval            : " + str(reported_interval_per_myriad) + " per ten thousand")
print("")
print("same commit, quiet runner, ns: " + str(nanoseconds_on_a_quiet_runner))
print("same commit, busy runner, ns : " + str(nanoseconds_on_a_busy_runner))
print("between-run spread           : " + str(between_run_spread_per_myriad) + " per ten thousand")
print("the spread is the interval times : " + str(spread_over_interval))
print("")
print("the benchmark's controls")
print("  core pinned          : yes")
print("  frequency scaling    : disabled")
print("  warm-up              : run and discarded")
print("  iterations           : " + str(iterations))
print("  outliers             : reported, not dropped")
print("  interval computed properly : yes")
print("  faults found in review : " + str(methodology_faults_found_in_review))
print("  verdict              : ISOLATED")
print("")
print("  every one of those is a real control and each removes a")
print("  real source of variance")
print("")
print("the reported " + str(reported_interval_per_myriad) + " per ten thousand")
print("  computed over        : " + str(iterations) + " iterations in one process")
print("  what it bounds       : how much this run's own samples")
print("    disagree with each other")
print("  what it does not bound : how much this run disagrees")
print("    with the next one")
print("")
print("  a tight interval over one population is not a claim")
print("  about a second population it never sampled")
print("")
print("the shared runner")
print("  cores this process owns : 1")
print("  cores the machine has   : shared with other jobs")
print("  last-level cache        : shared")
print("  memory bandwidth        : shared")
print("  a control for any of those : none, and pinning is what")
print("    makes it look as though there is")
print("")
noise_in_either_direction = int(between_run_spread_per_myriad / 2)
print("the regression gate")
print("  fails a change moving more than : " + str(reported_interval_per_myriad) + " per ten thousand")
print("  machine noise, either direction : up to " + str(noise_in_either_direction))
print("  so the gate decides on           : which runner it got")
print("  a real regression it would catch : one larger than the")
print("    noise, which is a regression nobody needs a benchmark")
print("    to notice")
print("")
nc_reported_interval_per_myriad = between_run_spread_per_myriad
nc_runs_across_runners = 5
print("null control - the interval computed across runners")
print("  methodology faults  : " + str(methodology_faults_found_in_review) + ", unchanged")
print("  runs across runners : " + str(nc_runs_across_runners))
print("  reported interval   : " + str(nc_reported_interval_per_myriad) + " per ten thousand")
print("  the benchmark did not get noisier; the interval started")
print("  covering the variance that decides the gate")
print("")
print("what a careful benchmark guarantees")
print("  this measurement is repeatable within itself : exactly")
print("  this measurement is comparable to the last one : not")
print("    addressed; every control it applies is inside the")
print("    process, and the variance that matters is outside")
print("")
print("an interval describes the population it was computed over;")
print("comparing two runs is a question about a population of runs,")
print("and one run cannot sample it")
print("")
print("The methodology is careful and review found " + str(methodology_faults_found_in_review) + " faults: pinned core, no frequency")
print("scaling, discarded warm-up, " + str(iterations) + " iterations, outliers reported. Its interval of")
print(str(reported_interval_per_myriad) + " per ten thousand covers disagreement among its own samples, while the same")
print("commit measures " + str(nanoseconds_on_a_quiet_runner) + " ns and " + str(nanoseconds_on_a_busy_runner) + " ns on two runners - a spread of")
print(str(between_run_spread_per_myriad) + " per ten thousand, " + str(spread_over_interval) + " times the interval the gate compares against.")

iterations                   : 1000
reported interval            : 40 per ten thousand

same commit, quiet runner, ns: 12400
same commit, busy runner, ns : 19800
between-run spread           : 5967 per ten thousand
the spread is the interval times : 149

the benchmark's controls
  core pinned          : yes
  frequency scaling    : disabled
  warm-up              : run and discarded
  iterations           : 1000
  outliers             : reported, not dropped
  interval computed properly : yes
  faults found in review : 0
  verdict              : ISOLATED

  every one of those is a real control and each removes a
  real source of variance

the reported 40 per ten thousand
  computed over        : 1000 iterations in one process
  what it bounds       : how much this run's own samples
    disagree with each other
  what it does not bound : how much this run disagrees
    with the next one

  a tight interval over one population is not a claim
  about a second population it never sampled

the shared runner
  cores this process owns : 1
  cores the machine has   : shared with other jobs
  last-level cache        : shared
  memory bandwidth        : shared
  a control for any of those : none, and pinning is what
    makes it look as though there is

the regression gate
  fails a change moving more than : 40 per ten thousand
  machine noise, either direction : up to 2983
  so the gate decides on           : which runner it got
  a real regression it would catch : one larger than the
    noise, which is a regression nobody needs a benchmark
    to notice

null control - the interval computed across runners
  methodology faults  : 0, unchanged
  runs across runners : 5
  reported interval   : 5967 per ten thousand
  the benchmark did not get noisier; the interval started
  covering the variance that decides the gate

what a careful benchmark guarantees
  this measurement is repeatable within itself : exactly
  this measurement is comparable to the last one : not
    addressed; every control it applies is inside the
    process, and the variance that matters is outside

an interval describes the population it was computed over;
comparing two runs is a question about a population of runs,
and one run cannot sample it

The methodology is careful and review found 0 faults: pinned core, no frequency
scaling, discarded warm-up, 1000 iterations, outliers reported. Its interval of
40 per ten thousand covers disagreement among its own samples, while the same
commit measures 12400 ns and 19800 ns on two runners - a spread of
5967 per ten thousand, 149 times the interval the gate compares against.

Trace event types