The demo case became the only tested path

# Self-authored for the EML case corpus (no external origin). The example in the
# README is one shape of input. It is the shape 14 of 18 tests exercise.
#
# The example was chosen well: it is short, it demonstrates the feature, and it
# is the first thing anyone runs. It became the shape everybody copies - into
# tests, into fixtures, into the mental model of what the input looks like -
# because it is the only concrete input the documentation contains.
#
# Nothing about that is negligent. Writing tests from the example is faster and
# safer than inventing inputs, and the example is known to be valid.
#
# Test coverage and real traffic are both counted per shape here, so the gap is
# measured against what the system actually receives.

# [shape, share of real traffic, tests exercising it, defects found in production]
[["flat, all fields present", 22, 14, 1], ["nested one level", 31, 2, 5], ["optional fields missing", 25, 1, 6], ["unicode in every field", 14, 1, 4], ["empty collections", 8, 0, 3]] => shapes

def total(col):
    0 => t
    for s in shapes:
        t + s[col] => t
    return t

"input shapes : " + str(len(shapes)) ^0
"  tests    : " + str(total(2)) ^0
"  production defects : " + str(total(3)) ^0
"" ^0

"shape                        traffic   tests   defects" ^0
for s in shapes:
    "  " + s[0] + "   " + str(s[1]) + "%      " + str(s[2]) + "       " + str(s[3]) ^0
"" ^0

# ---- the demo shape ----

0 => demo_at
"the shape the README example uses : " + shapes[demo_at][0] ^0
"  its share of traffic : " + str(shapes[demo_at][1]) + "%" ^0
"  its share of tests   : " + str(int(shapes[demo_at][2] * 100 / total(2))) + "%" ^0
"  its share of defects : " + str(int(shapes[demo_at][3] * 100 / total(3))) + "%" ^0
"" ^0

# ---- everything else ----

0 => other_traffic
0 => other_tests
0 => other_defects
0 => i
for s in shapes:
    if not (i == demo_at):
        other_traffic + s[1] => other_traffic
        other_tests + s[2] => other_tests
        other_defects + s[3] => other_defects
    i + 1 => i
"every other shape, together" ^0
"  traffic : " + str(other_traffic) + "%" ^0
"  tests   : " + str(other_tests) + "  (" + str(int(other_tests * 100 / total(2))) + "%)" ^0
"  defects : " + str(other_defects) + "  (" + str(int(other_defects * 100 / total(3))) + "%)" ^0
"" ^0

# ---- tests per unit of traffic ----

"tests per point of traffic share" ^0
for s in shapes:
    "  " + s[0] + " : " + str(int(s[2] * 100 / s[1])) ^0
"" ^0

# ---- defects per test ----
#
# The shape with the most tests has the fewest defects, which is what tests are
# for. The question is whether the tests went where the traffic is.

0 => best_covered
0 => best_at
0 => j
for s in shapes:
    if s[2] > best_covered:
        s[2] => best_covered
        j => best_at
    j + 1 => j
0 => worst_defects
0 => worst_at
0 => k
for s in shapes:
    if s[3] > worst_defects:
        s[3] => worst_defects
        k => worst_at
    k + 1 => k
"  most tested   : " + shapes[best_at][0] + " (" + str(shapes[best_at][2]) + " tests, " + str(shapes[best_at][3]) + " defects)" ^0
"  most defects  : " + shapes[worst_at][0] + " (" + str(shapes[worst_at][2]) + " tests, " + str(shapes[worst_at][3]) + " defects)" ^0
if not (best_at == worst_at):
    "  the tests worked where they were pointed, and they were pointed at the" ^0
    "  example rather than at the traffic" ^0
"" ^0

# ---- what a second example in the docs would change ----

0 => second_at
99 => least_tested
0 => m
for s in shapes:
    if s[2] < least_tested:
        s[2] => least_tested
        m => second_at
    m + 1 => m
"the shape with the fewest tests : " + shapes[second_at][0] ^0
"  its traffic share : " + str(shapes[second_at][1]) + "%" ^0
"  its defects       : " + str(shapes[second_at][3]) ^0
"  documenting it costs one more example" ^0
"" ^0

# ---- the control: a system whose example IS the common case ----
#
# Copying the example is only a problem when the example is unrepresentative.

[["the documented shape", 80, 12, 1], ["everything else", 20, 4, 1]] => aligned
"control - a system whose example matches the traffic" ^0
"  documented shape : " + str(aligned[0][1]) + "% of traffic, " + str(aligned[0][2]) + " tests" ^0
"  everything else  : " + str(aligned[1][1]) + "% of traffic, " + str(aligned[1][2]) + " tests" ^0
if aligned[0][1] > 50:
    "  here copying the example points the tests at the traffic" ^0
"" ^0

"The example is correct, valid and well chosen for teaching. It is also the" ^0
"only input anyone was handed, and a test suite is built from the inputs" ^0
"people have." ^0

shapes = [["flat, all fields present", 22, 14, 1], ["nested one level", 31, 2, 5], ["optional fields missing", 25, 1, 6], ["unicode in every field", 14, 1, 4], ["empty collections", 8, 0, 3]]

def total(col):
    t = 0
    for s in shapes:
        t = t + s[col]
    return t

print("input shapes : " + str(len(shapes)))
print("  tests    : " + str(total(2)))
print("  production defects : " + str(total(3)))
print("")
print("shape                        traffic   tests   defects")
for s in shapes:
    print("  " + s[0] + "   " + str(s[1]) + "%      " + str(s[2]) + "       " + str(s[3]))
print("")
demo_at = 0
print("the shape the README example uses : " + shapes[demo_at][0])
print("  its share of traffic : " + str(shapes[demo_at][1]) + "%")
print("  its share of tests   : " + str(int(shapes[demo_at][2] * 100 / total(2))) + "%")
print("  its share of defects : " + str(int(shapes[demo_at][3] * 100 / total(3))) + "%")
print("")
other_traffic = 0
other_tests = 0
other_defects = 0
i = 0
for s in shapes:
    if not i == demo_at:
        other_traffic = other_traffic + s[1]
        other_tests = other_tests + s[2]
        other_defects = other_defects + s[3]
    i = i + 1
print("every other shape, together")
print("  traffic : " + str(other_traffic) + "%")
print("  tests   : " + str(other_tests) + "  (" + str(int(other_tests * 100 / total(2))) + "%)")
print("  defects : " + str(other_defects) + "  (" + str(int(other_defects * 100 / total(3))) + "%)")
print("")
print("tests per point of traffic share")
for s in shapes:
    print("  " + s[0] + " : " + str(int(s[2] * 100 / s[1])))
print("")
best_covered = 0
best_at = 0
j = 0
for s in shapes:
    if s[2] > best_covered:
        best_covered = s[2]
        best_at = j
    j = j + 1
worst_defects = 0
worst_at = 0
k = 0
for s in shapes:
    if s[3] > worst_defects:
        worst_defects = s[3]
        worst_at = k
    k = k + 1
print("  most tested   : " + shapes[best_at][0] + " (" + str(shapes[best_at][2]) + " tests, " + str(shapes[best_at][3]) + " defects)")
print("  most defects  : " + shapes[worst_at][0] + " (" + str(shapes[worst_at][2]) + " tests, " + str(shapes[worst_at][3]) + " defects)")
if not best_at == worst_at:
    print("  the tests worked where they were pointed, and they were pointed at the")
    print("  example rather than at the traffic")
print("")
second_at = 0
least_tested = 99
m = 0
for s in shapes:
    if s[2] < least_tested:
        least_tested = s[2]
        second_at = m
    m = m + 1
print("the shape with the fewest tests : " + shapes[second_at][0])
print("  its traffic share : " + str(shapes[second_at][1]) + "%")
print("  its defects       : " + str(shapes[second_at][3]))
print("  documenting it costs one more example")
print("")
aligned = [["the documented shape", 80, 12, 1], ["everything else", 20, 4, 1]]
print("control - a system whose example matches the traffic")
print("  documented shape : " + str(aligned[0][1]) + "% of traffic, " + str(aligned[0][2]) + " tests")
print("  everything else  : " + str(aligned[1][1]) + "% of traffic, " + str(aligned[1][2]) + " tests")
if aligned[0][1] > 50:
    print("  here copying the example points the tests at the traffic")
print("")
print("The example is correct, valid and well chosen for teaching. It is also the")
print("only input anyone was handed, and a test suite is built from the inputs")
print("people have.")

input shapes : 5
  tests    : 18
  production defects : 19

shape                        traffic   tests   defects
  flat, all fields present   22%      14       1
  nested one level   31%      2       5
  optional fields missing   25%      1       6
  unicode in every field   14%      1       4
  empty collections   8%      0       3

the shape the README example uses : flat, all fields present
  its share of traffic : 22%
  its share of tests   : 77%
  its share of defects : 5%

every other shape, together
  traffic : 78%
  tests   : 4  (22%)
  defects : 18  (94%)

tests per point of traffic share
  flat, all fields present : 63
  nested one level : 6
  optional fields missing : 4
  unicode in every field : 7
  empty collections : 0

  most tested   : flat, all fields present (14 tests, 1 defects)
  most defects  : optional fields missing (1 tests, 6 defects)
  the tests worked where they were pointed, and they were pointed at the
  example rather than at the traffic

the shape with the fewest tests : empty collections
  its traffic share : 8%
  its defects       : 3
  documenting it costs one more example

control - a system whose example matches the traffic
  documented shape : 80% of traffic, 12 tests
  everything else  : 20% of traffic, 4 tests
  here copying the example points the tests at the traffic

The example is correct, valid and well chosen for teaching. It is also the
only input anyone was handed, and a test suite is built from the inputs
people have.

Trace event types