The compression was lossless and the schema was not

# Self-authored for the EML case corpus (no external origin). Every archived
# byte comes back identical and ninety restores prove it. What cannot be
# recovered is computed below.
#
# The compression is lossless in the exact sense. The decompressor returns the
# input bit for bit, a checksum over the original is stored beside the archive
# and compared on every restore, and ninety restores across three years have
# produced zero mismatches. There is no approximation anywhere in the codec.
#
# Lossless is a property of the CODEC, over the bytes it was given. Producing
# those bytes was a separate step, and that step used an encoding where an
# omitted field and a field set to its default are the same on the wire.
#
# So the archive restores exactly what was serialised, and what was serialised
# already could not tell "nobody said" from "somebody said zero".

71000000 => records_archived
90 => restores_performed
0 => byte_mismatches
22400000 => records_where_the_field_was_omitted
5900000 => records_where_the_field_was_explicitly_default

records_where_the_field_was_omitted + records_where_the_field_was_explicitly_default => records_that_read_as_zero
records_archived - records_that_read_as_zero => records_with_a_nonzero_value

"records archived              : " + str(records_archived) ^0
"restores performed            : " + str(restores_performed) ^0
"byte mismatches               : " + str(byte_mismatches) ^0
"" ^0
"field omitted                 : " + str(records_where_the_field_was_omitted) ^0
"field explicitly the default  : " + str(records_where_the_field_was_explicitly_default) ^0
"records that read as zero     : " + str(records_that_read_as_zero) ^0
"records with a nonzero value  : " + str(records_with_a_nonzero_value) ^0
"" ^0

# ---- what lossless verified ----

"the codec" ^0
"  decompressed output equals input : bit for bit" ^0
"  checksum stored beside the archive : yes" ^0
"  restores performed  : " + str(restores_performed) ^0
"  byte mismatches     : " + str(byte_mismatches) ^0
"  approximation anywhere in the codec : none" ^0
"  verdict             : LOSSLESS" ^0
"" ^0
"  the guarantee is exact and the evidence for it is three" ^0
"  years of restores" ^0
"" ^0

# ---- what it was given ----

"serialising one record" ^0
"  field absent from the object : written as nothing" ^0
"  field present and equal to the default : also written" ^0
"    as nothing, because the encoding omits defaults" ^0
"  bytes produced by the two cases : identical" ^0
"  the codec receives            : one of them" ^0
"" ^0
"  the loss happened before the compressor was called, in a" ^0
"  step nobody describes as lossy because it is a schema" ^0
"" ^0

int(records_that_read_as_zero * 10000 / records_archived) => ambiguous_per_myriad
"share of records that read as zero : " + str(ambiguous_per_myriad) + " per ten thousand" ^0
"" ^0

# ---- what a reader can and cannot answer ----

"questions about the restored data" ^0
"  what did record 4,180,002 hold : answerable, exactly" ^0
"  is the archive intact          : answerable, " + str(byte_mismatches) + " mismatches" ^0
"  how many customers declined    : not answerable" ^0
"  how many were never asked      : not answerable" ^0
"  their sum                      : " + str(records_that_read_as_zero) + ", answerable" ^0
"" ^0
"  the pair is gone and the total is intact, which is the" ^0
"  signature of a lossy encoding rather than a lossy codec" ^0
"" ^0

# ---- null control ----

# The same codec and the same archive, with the schema making the field
# explicitly nullable so absence has its own representation.
0 => nc_records_that_read_as_zero_ambiguously
records_where_the_field_was_omitted => nc_records_recorded_as_absent
records_where_the_field_was_explicitly_default => nc_records_recorded_as_zero

"null control - absence given its own representation" ^0
"  byte mismatches       : " + str(byte_mismatches) + ", unchanged" ^0
"  recorded as absent    : " + str(nc_records_recorded_as_absent) ^0
"  recorded as zero      : " + str(nc_records_recorded_as_zero) ^0
"  ambiguous records     : " + str(nc_records_that_read_as_zero_ambiguously) ^0
"  the compression did not improve; the bytes handed to it" ^0
"  started carrying the distinction" ^0
"" ^0

# ---- the rule ----

"what lossless compression guarantees" ^0
"  the bytes are recovered exactly : exactly" ^0
"  the meaning is recovered        : not addressed; the" ^0
"    codec's input is already the output of an encoding," ^0
"    and no property of the codec reaches upstream of it" ^0
"" ^0
"'lossless' names the last hop; when a pipeline has a lossy" ^0
"step and a lossless one, the honest claim is about the" ^0
"composition, and only the second hop has a word for it" ^0
"" ^0

"The codec is lossless in the exact sense: bit-for-bit output, a checksum on" ^0
"every restore, " + str(restores_performed) + " restores and " + str(byte_mismatches) + " mismatches in three years. The encoding" ^0
"omits defaults, so " + str(records_where_the_field_was_omitted) + " records that never stated the field and " + str(records_where_the_field_was_explicitly_default) ^0
"that stated it as zero produce identical bytes - " + str(records_that_read_as_zero) + " records, " + str(ambiguous_per_myriad) + " per" ^0
"ten thousand - and the archive restores that ambiguity perfectly." ^0

records_archived = 71000000
restores_performed = 90
byte_mismatches = 0
records_where_the_field_was_omitted = 22400000
records_where_the_field_was_explicitly_default = 5900000
records_that_read_as_zero = records_where_the_field_was_omitted + records_where_the_field_was_explicitly_default
records_with_a_nonzero_value = records_archived - records_that_read_as_zero
print("records archived              : " + str(records_archived))
print("restores performed            : " + str(restores_performed))
print("byte mismatches               : " + str(byte_mismatches))
print("")
print("field omitted                 : " + str(records_where_the_field_was_omitted))
print("field explicitly the default  : " + str(records_where_the_field_was_explicitly_default))
print("records that read as zero     : " + str(records_that_read_as_zero))
print("records with a nonzero value  : " + str(records_with_a_nonzero_value))
print("")
print("the codec")
print("  decompressed output equals input : bit for bit")
print("  checksum stored beside the archive : yes")
print("  restores performed  : " + str(restores_performed))
print("  byte mismatches     : " + str(byte_mismatches))
print("  approximation anywhere in the codec : none")
print("  verdict             : LOSSLESS")
print("")
print("  the guarantee is exact and the evidence for it is three")
print("  years of restores")
print("")
print("serialising one record")
print("  field absent from the object : written as nothing")
print("  field present and equal to the default : also written")
print("    as nothing, because the encoding omits defaults")
print("  bytes produced by the two cases : identical")
print("  the codec receives            : one of them")
print("")
print("  the loss happened before the compressor was called, in a")
print("  step nobody describes as lossy because it is a schema")
print("")
ambiguous_per_myriad = int(records_that_read_as_zero * 10000 / records_archived)
print("share of records that read as zero : " + str(ambiguous_per_myriad) + " per ten thousand")
print("")
print("questions about the restored data")
print("  what did record 4,180,002 hold : answerable, exactly")
print("  is the archive intact          : answerable, " + str(byte_mismatches) + " mismatches")
print("  how many customers declined    : not answerable")
print("  how many were never asked      : not answerable")
print("  their sum                      : " + str(records_that_read_as_zero) + ", answerable")
print("")
print("  the pair is gone and the total is intact, which is the")
print("  signature of a lossy encoding rather than a lossy codec")
print("")
nc_records_that_read_as_zero_ambiguously = 0
nc_records_recorded_as_absent = records_where_the_field_was_omitted
nc_records_recorded_as_zero = records_where_the_field_was_explicitly_default
print("null control - absence given its own representation")
print("  byte mismatches       : " + str(byte_mismatches) + ", unchanged")
print("  recorded as absent    : " + str(nc_records_recorded_as_absent))
print("  recorded as zero      : " + str(nc_records_recorded_as_zero))
print("  ambiguous records     : " + str(nc_records_that_read_as_zero_ambiguously))
print("  the compression did not improve; the bytes handed to it")
print("  started carrying the distinction")
print("")
print("what lossless compression guarantees")
print("  the bytes are recovered exactly : exactly")
print("  the meaning is recovered        : not addressed; the")
print("    codec's input is already the output of an encoding,")
print("    and no property of the codec reaches upstream of it")
print("")
print("'lossless' names the last hop; when a pipeline has a lossy")
print("step and a lossless one, the honest claim is about the")
print("composition, and only the second hop has a word for it")
print("")
print("The codec is lossless in the exact sense: bit-for-bit output, a checksum on")
print("every restore, " + str(restores_performed) + " restores and " + str(byte_mismatches) + " mismatches in three years. The encoding")
print("omits defaults, so " + str(records_where_the_field_was_omitted) + " records that never stated the field and " + str(records_where_the_field_was_explicitly_default))
print("that stated it as zero produce identical bytes - " + str(records_that_read_as_zero) + " records, " + str(ambiguous_per_myriad) + " per")
print("ten thousand - and the archive restores that ambiguity perfectly.")

records archived              : 71000000
restores performed            : 90
byte mismatches               : 0

field omitted                 : 22400000
field explicitly the default  : 5900000
records that read as zero     : 28300000
records with a nonzero value  : 42700000

the codec
  decompressed output equals input : bit for bit
  checksum stored beside the archive : yes
  restores performed  : 90
  byte mismatches     : 0
  approximation anywhere in the codec : none
  verdict             : LOSSLESS

  the guarantee is exact and the evidence for it is three
  years of restores

serialising one record
  field absent from the object : written as nothing
  field present and equal to the default : also written
    as nothing, because the encoding omits defaults
  bytes produced by the two cases : identical
  the codec receives            : one of them

  the loss happened before the compressor was called, in a
  step nobody describes as lossy because it is a schema

share of records that read as zero : 3985 per ten thousand

questions about the restored data
  what did record 4,180,002 hold : answerable, exactly
  is the archive intact          : answerable, 0 mismatches
  how many customers declined    : not answerable
  how many were never asked      : not answerable
  their sum                      : 28300000, answerable

  the pair is gone and the total is intact, which is the
  signature of a lossy encoding rather than a lossy codec

null control - absence given its own representation
  byte mismatches       : 0, unchanged
  recorded as absent    : 22400000
  recorded as zero      : 5900000
  ambiguous records     : 0
  the compression did not improve; the bytes handed to it
  started carrying the distinction

what lossless compression guarantees
  the bytes are recovered exactly : exactly
  the meaning is recovered        : not addressed; the
    codec's input is already the output of an encoding,
    and no property of the codec reaches upstream of it

'lossless' names the last hop; when a pipeline has a lossy
step and a lossless one, the honest claim is about the
composition, and only the second hop has a word for it

The codec is lossless in the exact sense: bit-for-bit output, a checksum on
every restore, 90 restores and 0 mismatches in three years. The encoding
omits defaults, so 22400000 records that never stated the field and 5900000
that stated it as zero produce identical bytes - 28300000 records, 3985 per
ten thousand - and the archive restores that ambiguity perfectly.

Trace event types