The default sort made the diff unreadable

# Self-authored for the EML case corpus (no external origin). A generated
# configuration file is regenerated on every build and committed. How much of
# each diff is a real change is computed below.
#
# Not sorting the output is the right default for the serializer and it was
# chosen deliberately. Sorting costs time on files that can be large, iteration
# order is not part of the format's contract so relying on it would be a bug,
# and for a file that is READ by a program the order carries no meaning at all.
# The serializer is used in many places where the output is never seen by a
# person, and for those it is exactly right.
#
# This file is committed, so it is also read by people, and by a diff. A diff
# does not know that order is meaningless. It reports every line whose position
# changed, because for most files a moved line is a fact worth showing.
#
# The generator is deterministic given its input, which is the property the team
# checked. It is not deterministic given a rebuild on a different process, which
# is where the hash seed lives.

240 => keys
3 => keys_changed_by_a_real_edit

# The expected number of fixed points of a random permutation is exactly 1,
# whatever n is. So of `keys` lines, one stays put on average and the rest move.
1 => expected_lines_in_place

keys - expected_lines_in_place => lines_moved
lines_moved * 2 => diff_lines_from_noise
keys_changed_by_a_real_edit * 2 => diff_lines_from_the_change

"keys in the file           : " + str(keys) ^0
"keys a real edit touches   : " + str(keys_changed_by_a_real_edit) ^0
"" ^0

"  lines expected to keep their position : " + str(expected_lines_in_place) ^0
"  lines expected to move                : " + str(lines_moved) ^0
"  diff lines from reordering            : " + str(diff_lines_from_noise) ^0
"  diff lines from the actual change     : " + str(diff_lines_from_the_change) ^0
"  signal to noise                       : 1 to " + str(int(diff_lines_from_noise / diff_lines_from_the_change)) ^0
"" ^0

# ---- what a reviewer does with that ----

"a review of this diff" ^0
"  lines to read            : " + str(diff_lines_from_noise + diff_lines_from_the_change) ^0
"  lines that matter        : " + str(diff_lines_from_the_change) ^0
"  time to find them by eye : longer than reading the source change itself" ^0
"  what reviewers do instead: approve the file unread" ^0
"" ^0
"  and once it is approved unread, the file is outside review entirely," ^0
"  including on the day the real change is three lines nobody wanted" ^0
"" ^0

# ---- the same file, sorted ----

"with the keys sorted before writing" ^0
"  lines that move from reordering : 0" ^0
"  diff lines from reordering      : 0" ^0
"  diff lines from the change      : " + str(diff_lines_from_the_change) ^0
"  signal to noise                 : 1 to 0" ^0
"  cost : one sort of " + str(keys) + " keys, once per build" ^0
"" ^0

# ---- what determinism was checked against ----
#
# The team did verify the generator is deterministic. They ran it twice in one
# process and compared. That comparison holds; the hash seed is per-process.

"the determinism check that was run" ^0
"  generate twice in one process, compare : identical" ^0
"  generate on two machines, compare      : not run" ^0
"  generate in two processes, compare     : not run" ^0
"  the property that was verified : same process, same output" ^0
"  the property that matters      : same input, same output" ^0
"" ^0
"  a second run in the same process shares the seed, so it cannot" ^0
"  distinguish the two" ^0
"" ^0

# ---- what the noise hides ----
#
# A real change of three lines inside 478 lines of movement is not visible.
# Neither is a change nobody made deliberately.

"changes this diff would hide equally well" ^0
"  the intended three-line edit          hidden" ^0
"  a value edited by a bad merge         hidden" ^0
"  a key removed by a generator bug      hidden" ^0
"  a credential accidentally interpolated hidden" ^0
"  the noise does not discriminate between what it hides" ^0
"" ^0

# ---- the control ----
#
# The generator. Same input, same process, same output, every time. It has no
# bug and its output is correct in every line. The file is right; the DIFF is
# unreadable.

"control - is the generated file correct" ^0
"  keys present            : " + str(keys) ^0
"  keys with correct values: " + str(keys) ^0
"  wrong or missing entries: 0" ^0
"  the file is correct on every line, in every build" ^0
"" ^0
"  correctness of the file and readability of the diff are different" ^0
"  properties, and only one of them was ever a requirement" ^0
"" ^0

# ---- the null control ----
#
# The same unsorted serializer on a file that is generated and consumed by a
# program without being committed. No diff, no reviewer, no cost. The default
# is right for that file and it is the same default.

0 => nc_reviewers

"null control - the same serializer for a file nobody commits" ^0
"  reviewers            : " + str(nc_reviewers) ^0
"  diffs produced       : " + str(nc_reviewers) ^0
"  cost of the ordering : 0" ^0
"  time saved by not sorting : real, on every build" ^0
"  same serializer, same default, and here it is simply better" ^0
"  the cost appears only when the output crosses into a place where" ^0
"  position carries meaning, and a git repository is such a place" ^0
"" ^0

# ---- the rule ----

"unspecified order, in an output that will be diffed" ^0
"  is the order part of the format's contract   no" ^0
"  is relying on it a bug                       yes" ^0
"  does a diff rely on it                       yes, unavoidably" ^0
"  a diff is a tool for a format whose order means something," ^0
"  pointed at a format whose order means nothing" ^0
"" ^0
"the fix is not to make the diff smarter" ^0
"it is one sort, in the writer, so the file has an order to be stable in" ^0
"" ^0

"Not sorting is right for a serializer whose output is read by programs, and" ^0
"relying on iteration order would be a genuine bug. This file is committed, so" ^0
"a diff reads it, and a diff reports position. A random permutation leaves " + str(expected_lines_in_place) ^0
"line in place on average, so " + str(lines_moved) + " of " + str(keys) + " move and produce " + str(diff_lines_from_noise) + " diff lines" ^0
"around the " + str(diff_lines_from_the_change) + " that anyone meant to make." ^0

keys = 240
keys_changed_by_a_real_edit = 3
expected_lines_in_place = 1
lines_moved = keys - expected_lines_in_place
diff_lines_from_noise = lines_moved * 2
diff_lines_from_the_change = keys_changed_by_a_real_edit * 2
print("keys in the file           : " + str(keys))
print("keys a real edit touches   : " + str(keys_changed_by_a_real_edit))
print("")
print("  lines expected to keep their position : " + str(expected_lines_in_place))
print("  lines expected to move                : " + str(lines_moved))
print("  diff lines from reordering            : " + str(diff_lines_from_noise))
print("  diff lines from the actual change     : " + str(diff_lines_from_the_change))
print("  signal to noise                       : 1 to " + str(int(diff_lines_from_noise / diff_lines_from_the_change)))
print("")
print("a review of this diff")
print("  lines to read            : " + str(diff_lines_from_noise + diff_lines_from_the_change))
print("  lines that matter        : " + str(diff_lines_from_the_change))
print("  time to find them by eye : longer than reading the source change itself")
print("  what reviewers do instead: approve the file unread")
print("")
print("  and once it is approved unread, the file is outside review entirely,")
print("  including on the day the real change is three lines nobody wanted")
print("")
print("with the keys sorted before writing")
print("  lines that move from reordering : 0")
print("  diff lines from reordering      : 0")
print("  diff lines from the change      : " + str(diff_lines_from_the_change))
print("  signal to noise                 : 1 to 0")
print("  cost : one sort of " + str(keys) + " keys, once per build")
print("")
print("the determinism check that was run")
print("  generate twice in one process, compare : identical")
print("  generate on two machines, compare      : not run")
print("  generate in two processes, compare     : not run")
print("  the property that was verified : same process, same output")
print("  the property that matters      : same input, same output")
print("")
print("  a second run in the same process shares the seed, so it cannot")
print("  distinguish the two")
print("")
print("changes this diff would hide equally well")
print("  the intended three-line edit          hidden")
print("  a value edited by a bad merge         hidden")
print("  a key removed by a generator bug      hidden")
print("  a credential accidentally interpolated hidden")
print("  the noise does not discriminate between what it hides")
print("")
print("control - is the generated file correct")
print("  keys present            : " + str(keys))
print("  keys with correct values: " + str(keys))
print("  wrong or missing entries: 0")
print("  the file is correct on every line, in every build")
print("")
print("  correctness of the file and readability of the diff are different")
print("  properties, and only one of them was ever a requirement")
print("")
nc_reviewers = 0
print("null control - the same serializer for a file nobody commits")
print("  reviewers            : " + str(nc_reviewers))
print("  diffs produced       : " + str(nc_reviewers))
print("  cost of the ordering : 0")
print("  time saved by not sorting : real, on every build")
print("  same serializer, same default, and here it is simply better")
print("  the cost appears only when the output crosses into a place where")
print("  position carries meaning, and a git repository is such a place")
print("")
print("unspecified order, in an output that will be diffed")
print("  is the order part of the format's contract   no")
print("  is relying on it a bug                       yes")
print("  does a diff rely on it                       yes, unavoidably")
print("  a diff is a tool for a format whose order means something,")
print("  pointed at a format whose order means nothing")
print("")
print("the fix is not to make the diff smarter")
print("it is one sort, in the writer, so the file has an order to be stable in")
print("")
print("Not sorting is right for a serializer whose output is read by programs, and")
print("relying on iteration order would be a genuine bug. This file is committed, so")
print("a diff reads it, and a diff reports position. A random permutation leaves " + str(expected_lines_in_place))
print("line in place on average, so " + str(lines_moved) + " of " + str(keys) + " move and produce " + str(diff_lines_from_noise) + " diff lines")
print("around the " + str(diff_lines_from_the_change) + " that anyone meant to make.")

keys in the file           : 240
keys a real edit touches   : 3

  lines expected to keep their position : 1
  lines expected to move                : 239
  diff lines from reordering            : 478
  diff lines from the actual change     : 6
  signal to noise                       : 1 to 79

a review of this diff
  lines to read            : 484
  lines that matter        : 6
  time to find them by eye : longer than reading the source change itself
  what reviewers do instead: approve the file unread

  and once it is approved unread, the file is outside review entirely,
  including on the day the real change is three lines nobody wanted

with the keys sorted before writing
  lines that move from reordering : 0
  diff lines from reordering      : 0
  diff lines from the change      : 6
  signal to noise                 : 1 to 0
  cost : one sort of 240 keys, once per build

the determinism check that was run
  generate twice in one process, compare : identical
  generate on two machines, compare      : not run
  generate in two processes, compare     : not run
  the property that was verified : same process, same output
  the property that matters      : same input, same output

  a second run in the same process shares the seed, so it cannot
  distinguish the two

changes this diff would hide equally well
  the intended three-line edit          hidden
  a value edited by a bad merge         hidden
  a key removed by a generator bug      hidden
  a credential accidentally interpolated hidden
  the noise does not discriminate between what it hides

control - is the generated file correct
  keys present            : 240
  keys with correct values: 240
  wrong or missing entries: 0
  the file is correct on every line, in every build

  correctness of the file and readability of the diff are different
  properties, and only one of them was ever a requirement

null control - the same serializer for a file nobody commits
  reviewers            : 0
  diffs produced       : 0
  cost of the ordering : 0
  time saved by not sorting : real, on every build
  same serializer, same default, and here it is simply better
  the cost appears only when the output crosses into a place where
  position carries meaning, and a git repository is such a place

unspecified order, in an output that will be diffed
  is the order part of the format's contract   no
  is relying on it a bug                       yes
  does a diff rely on it                       yes, unavoidably
  a diff is a tool for a format whose order means something,
  pointed at a format whose order means nothing

the fix is not to make the diff smarter
it is one sort, in the writer, so the file has an order to be stable in

Not sorting is right for a serializer whose output is read by programs, and
relying on iteration order would be a genuine bug. This file is committed, so
a diff reads it, and a diff reports position. A random permutation leaves 1
line in place on average, so 239 of 240 move and produce 478 diff lines
around the 6 that anyone meant to make.

Trace event types