The integrity check read the backups own manifest

# Self-authored for the EML case corpus (no external origin). Every backup is
# verified rather than assumed: each file's checksum is recomputed after upload
# and the count is compared against the manifest, which caught three truncated
# uploads. What the manifest is a list of is computed below.
#
# The verification is the kind most teams skip. A backup job that reports
# success has reported that it did not crash; this one recomputes every
# uploaded file's checksum from the object store and compares it to the value
# computed at read time, so a silent truncation or a corrupted transfer is
# caught rather than discovered during a restore. It found three.
#
# The count it compares against comes from the MANIFEST, and the manifest is
# written by the same job from the list of things it decided to include. A
# database the job never enumerated is not a missing file; it is not a file.
#
# The server has eighteen databases and the include list names fifteen.

41200 => files_in_the_manifest
41200 => files_whose_checksum_was_recomputed
3 => truncated_uploads_caught
18 => databases_on_the_server
15 => databases_in_the_include_list
0 => jobs_that_enumerate_the_server
0 => alerts_when_a_database_is_absent_from_the_include_list
1 => processes_that_write_the_manifest_and_check_against_it

databases_on_the_server - databases_in_the_include_list => databases_never_backed_up
int(databases_in_the_include_list * 10000 / databases_on_the_server) => covered_per_myriad
int(files_whose_checksum_was_recomputed * 10000 / files_in_the_manifest) => manifest_verified_per_myriad

"files in the manifest           : " + str(files_in_the_manifest) ^0
"  checksum recomputed           : " + str(files_whose_checksum_was_recomputed) ^0
"  share verified                : " + str(manifest_verified_per_myriad) + " per ten thousand" ^0
"truncated uploads caught        : " + str(truncated_uploads_caught) ^0
"" ^0
"databases on the server         : " + str(databases_on_the_server) ^0
"  in the include list           : " + str(databases_in_the_include_list) ^0
"  never backed up               : " + str(databases_never_backed_up) ^0
"  share covered                 : " + str(covered_per_myriad) + " per ten thousand" ^0
"" ^0
"processes writing the manifest and checking it : " + str(processes_that_write_the_manifest_and_check_against_it) ^0
"jobs that enumerate the server                 : " + str(jobs_that_enumerate_the_server) ^0
"alerts when a database is absent from the list : " + str(alerts_when_a_database_is_absent_from_the_include_list) ^0
"" ^0

# ---- what the check verified ----

"the integrity check" ^0
"  what a plain success report means : the job did not" ^0
"    crash" ^0
"  what this does instead : recomputes every uploaded" ^0
"    file's checksum from the object store" ^0
"  compared against : the value computed at read time" ^0
"  so a silent truncation is : caught, not discovered" ^0
"    during a restore" ^0
"  truncated uploads it found : " + str(truncated_uploads_caught) ^0
"  verdict : WHAT WAS UPLOADED IS INTACT" ^0
"" ^0
"  recomputing from the object store rather than trusting" ^0
"  the upload's own return code is the expensive half, and" ^0
"  it is the half that found the three" ^0
"" ^0

# ---- where the expected count comes from ----

"the two sides of the comparison" ^0
"  one  : the files present in the object store" ^0
"  other: the count in the manifest" ^0
"  who writes the manifest : this job, from its include" ^0
"    list" ^0
"  processes doing both : " + str(processes_that_write_the_manifest_and_check_against_it) ^0
"  what a database outside the include list looks like : " ^0
"    absent from both sides" ^0
"  databases in that state : " + str(databases_never_backed_up) ^0
"" ^0
"  the check can prove the job finished what it started" ^0
"  and cannot ask what it should have started" ^0
"" ^0

# ---- the include list is not wrong ----

# It was correct when it was written. Two of the three missing databases were
# created afterwards by a team that had no reason to know a list existed, and
# the third was renamed.
"how three fell outside" ^0
"  created after the list was written : two" ^0
"  renamed since : one" ^0
"  was the list wrong when written : no" ^0
"  did anyone edit it incorrectly : no" ^0
"  what would have to happen for the list to notice : it" ^0
"    would have to be derived rather than maintained" ^0
"  alerts on that : " + str(alerts_when_a_database_is_absent_from_the_include_list) ^0
"" ^0

# ---- what the report says ----

"the nightly report" ^0
"  files verified : " + str(files_whose_checksum_was_recomputed) + " of " + str(files_in_the_manifest) ^0
"  integrity failures : none since the three" ^0
"  what a reader concludes : the backup is complete" ^0
"  what is claimed : everything the job set out to copy" ^0
"    arrived intact" ^0
"  the difference : " + str(databases_never_backed_up) + " databases" ^0
"" ^0

# ---- null control ----

# The same check, with the expected set enumerated from the server's own
# catalog rather than from the job's include list.
databases_on_the_server => nc_databases_in_the_expected_set
0 => nc_databases_never_backed_up
1 => nc_jobs_that_enumerate_the_server

"null control - the expected set comes from the server" ^0
"  files verified : " + str(files_whose_checksum_was_recomputed) + ", unchanged" ^0
"  jobs that enumerate the server : " + str(nc_jobs_that_enumerate_the_server) ^0
"  databases in the expected set : " + str(nc_databases_in_the_expected_set) ^0
"  databases never backed up : " + str(nc_databases_never_backed_up) ^0
"  the checksum work did not change; the list it is checked" ^0
"  against stopped being written by the thing it checks" ^0
"" ^0

# ---- the rule ----

"what a verified backup guarantees" ^0
"  everything the job copied arrived intact : exactly," ^0
"    recomputed from the store, and it caught " + str(truncated_uploads_caught) ^0
"  everything is backed up : not addressed; the expected" ^0
"    set is written by the job, so a thing it never" ^0
"    enumerated cannot be found missing" ^0
"" ^0
"a completeness check needs an expectation from outside the" ^0
"process it checks; taking the expectation from the process's" ^0
"own record turns every omission into an absence, and an" ^0
"absence is what a passing check looks like" ^0
"" ^0

"The check recomputes every uploaded file's checksum from the object store" ^0
"rather than trusting a return code, and it caught " + str(truncated_uploads_caught) + " truncated uploads -" ^0
str(files_whose_checksum_was_recomputed) + " of " + str(files_in_the_manifest) + " files verified. It compares against a manifest the same job" ^0
"writes from its include list, which names " + str(databases_in_the_include_list) + " of " + str(databases_on_the_server) + " databases - " + str(covered_per_myriad) ^0
"per ten thousand - leaving " + str(databases_never_backed_up) + " that are absent from both sides of the comparison." ^0

files_in_the_manifest = 41200
files_whose_checksum_was_recomputed = 41200
truncated_uploads_caught = 3
databases_on_the_server = 18
databases_in_the_include_list = 15
jobs_that_enumerate_the_server = 0
alerts_when_a_database_is_absent_from_the_include_list = 0
processes_that_write_the_manifest_and_check_against_it = 1
databases_never_backed_up = databases_on_the_server - databases_in_the_include_list
covered_per_myriad = int(databases_in_the_include_list * 10000 / databases_on_the_server)
manifest_verified_per_myriad = int(files_whose_checksum_was_recomputed * 10000 / files_in_the_manifest)
print("files in the manifest           : " + str(files_in_the_manifest))
print("  checksum recomputed           : " + str(files_whose_checksum_was_recomputed))
print("  share verified                : " + str(manifest_verified_per_myriad) + " per ten thousand")
print("truncated uploads caught        : " + str(truncated_uploads_caught))
print("")
print("databases on the server         : " + str(databases_on_the_server))
print("  in the include list           : " + str(databases_in_the_include_list))
print("  never backed up               : " + str(databases_never_backed_up))
print("  share covered                 : " + str(covered_per_myriad) + " per ten thousand")
print("")
print("processes writing the manifest and checking it : " + str(processes_that_write_the_manifest_and_check_against_it))
print("jobs that enumerate the server                 : " + str(jobs_that_enumerate_the_server))
print("alerts when a database is absent from the list : " + str(alerts_when_a_database_is_absent_from_the_include_list))
print("")
print("the integrity check")
print("  what a plain success report means : the job did not")
print("    crash")
print("  what this does instead : recomputes every uploaded")
print("    file's checksum from the object store")
print("  compared against : the value computed at read time")
print("  so a silent truncation is : caught, not discovered")
print("    during a restore")
print("  truncated uploads it found : " + str(truncated_uploads_caught))
print("  verdict : WHAT WAS UPLOADED IS INTACT")
print("")
print("  recomputing from the object store rather than trusting")
print("  the upload's own return code is the expensive half, and")
print("  it is the half that found the three")
print("")
print("the two sides of the comparison")
print("  one  : the files present in the object store")
print("  other: the count in the manifest")
print("  who writes the manifest : this job, from its include")
print("    list")
print("  processes doing both : " + str(processes_that_write_the_manifest_and_check_against_it))
print("  what a database outside the include list looks like : ")
print("    absent from both sides")
print("  databases in that state : " + str(databases_never_backed_up))
print("")
print("  the check can prove the job finished what it started")
print("  and cannot ask what it should have started")
print("")
print("how three fell outside")
print("  created after the list was written : two")
print("  renamed since : one")
print("  was the list wrong when written : no")
print("  did anyone edit it incorrectly : no")
print("  what would have to happen for the list to notice : it")
print("    would have to be derived rather than maintained")
print("  alerts on that : " + str(alerts_when_a_database_is_absent_from_the_include_list))
print("")
print("the nightly report")
print("  files verified : " + str(files_whose_checksum_was_recomputed) + " of " + str(files_in_the_manifest))
print("  integrity failures : none since the three")
print("  what a reader concludes : the backup is complete")
print("  what is claimed : everything the job set out to copy")
print("    arrived intact")
print("  the difference : " + str(databases_never_backed_up) + " databases")
print("")
nc_databases_in_the_expected_set = databases_on_the_server
nc_databases_never_backed_up = 0
nc_jobs_that_enumerate_the_server = 1
print("null control - the expected set comes from the server")
print("  files verified : " + str(files_whose_checksum_was_recomputed) + ", unchanged")
print("  jobs that enumerate the server : " + str(nc_jobs_that_enumerate_the_server))
print("  databases in the expected set : " + str(nc_databases_in_the_expected_set))
print("  databases never backed up : " + str(nc_databases_never_backed_up))
print("  the checksum work did not change; the list it is checked")
print("  against stopped being written by the thing it checks")
print("")
print("what a verified backup guarantees")
print("  everything the job copied arrived intact : exactly,")
print("    recomputed from the store, and it caught " + str(truncated_uploads_caught))
print("  everything is backed up : not addressed; the expected")
print("    set is written by the job, so a thing it never")
print("    enumerated cannot be found missing")
print("")
print("a completeness check needs an expectation from outside the")
print("process it checks; taking the expectation from the process's")
print("own record turns every omission into an absence, and an")
print("absence is what a passing check looks like")
print("")
print("The check recomputes every uploaded file's checksum from the object store")
print("rather than trusting a return code, and it caught " + str(truncated_uploads_caught) + " truncated uploads -")
print(str(files_whose_checksum_was_recomputed) + " of " + str(files_in_the_manifest) + " files verified. It compares against a manifest the same job")
print("writes from its include list, which names " + str(databases_in_the_include_list) + " of " + str(databases_on_the_server) + " databases - " + str(covered_per_myriad))
print("per ten thousand - leaving " + str(databases_never_backed_up) + " that are absent from both sides of the comparison.")

files in the manifest           : 41200
  checksum recomputed           : 41200
  share verified                : 10000 per ten thousand
truncated uploads caught        : 3

databases on the server         : 18
  in the include list           : 15
  never backed up               : 3
  share covered                 : 8333 per ten thousand

processes writing the manifest and checking it : 1
jobs that enumerate the server                 : 0
alerts when a database is absent from the list : 0

the integrity check
  what a plain success report means : the job did not
    crash
  what this does instead : recomputes every uploaded
    file's checksum from the object store
  compared against : the value computed at read time
  so a silent truncation is : caught, not discovered
    during a restore
  truncated uploads it found : 3
  verdict : WHAT WAS UPLOADED IS INTACT

  recomputing from the object store rather than trusting
  the upload's own return code is the expensive half, and
  it is the half that found the three

the two sides of the comparison
  one  : the files present in the object store
  other: the count in the manifest
  who writes the manifest : this job, from its include
    list
  processes doing both : 1
  what a database outside the include list looks like : 
    absent from both sides
  databases in that state : 3

  the check can prove the job finished what it started
  and cannot ask what it should have started

how three fell outside
  created after the list was written : two
  renamed since : one
  was the list wrong when written : no
  did anyone edit it incorrectly : no
  what would have to happen for the list to notice : it
    would have to be derived rather than maintained
  alerts on that : 0

the nightly report
  files verified : 41200 of 41200
  integrity failures : none since the three
  what a reader concludes : the backup is complete
  what is claimed : everything the job set out to copy
    arrived intact
  the difference : 3 databases

null control - the expected set comes from the server
  files verified : 41200, unchanged
  jobs that enumerate the server : 1
  databases in the expected set : 18
  databases never backed up : 0
  the checksum work did not change; the list it is checked
  against stopped being written by the thing it checks

what a verified backup guarantees
  everything the job copied arrived intact : exactly,
    recomputed from the store, and it caught 3
  everything is backed up : not addressed; the expected
    set is written by the job, so a thing it never
    enumerated cannot be found missing

a completeness check needs an expectation from outside the
process it checks; taking the expectation from the process's
own record turns every omission into an absence, and an
absence is what a passing check looks like

The check recomputes every uploaded file's checksum from the object store
rather than trusting a return code, and it caught 3 truncated uploads -
41200 of 41200 files verified. It compares against a manifest the same job
writes from its include list, which names 15 of 18 databases - 8333
per ten thousand - leaving 3 that are absent from both sides of the comparison.

Trace event types