The summary was accurate and the reader inferred more

# Self-authored for the EML case corpus (no external origin). A security review
# is summarised in one line: reviewed the authentication flow, found three
# issues, all fixed. Every word of that is true. What a reader takes from it is
# computed below.
#
# The summary is honest and it was written carefully. Three issues were found,
# they were real, all three were fixed and the fixes were verified. The reviewer
# did not overstate the depth, did not claim completeness, and did not pad the
# finding count. Compared with most such summaries it is unusually plain.
#
# A summary is a compression, and compression works by dropping what a reader
# can reconstruct. What was NOT examined has no natural place in it: a finding
# has a sentence and a non-finding has nothing, so the space that was not
# looked at leaves no trace of its own size.
#
# The reader's inference is not careless either. Given "reviewed X, found three
# issues", the ordinary reading is that X was reviewed. That inference is what
# makes summaries useful, and it is wrong here only because a number the summary
# does not contain is much larger than the one it does.

52000 => endpoints_total
400 => endpoints_sampled
3 => issues_found

"endpoints in the service : " + str(endpoints_total) ^0
"endpoints examined       : " + str(endpoints_sampled) ^0
"issues found             : " + str(issues_found) ^0
"issues fixed             : " + str(issues_found) ^0
"" ^0

int(endpoints_sampled * 1000 / endpoints_total) => coverage_per_mille
endpoints_total - endpoints_sampled => not_examined

"  coverage        : " + str(coverage_per_mille) + " per mille" ^0
"  not examined    : " + str(not_examined) ^0
"" ^0

# ---- every claim, checked ----

"claim in the summary                    true" ^0
"  the authentication flow was reviewed    yes" ^0
"  three issues were found                 yes" ^0
"  all three were fixed                    yes" ^0
"  the fixes were verified                 yes" ^0
"  false claims : 0 of 4" ^0
"" ^0

"what a reader takes from it" ^0
"  the authentication flow has been reviewed   inferred" ^0
"  the flow now has three fewer issues         inferred, and true" ^0
"  the flow has been checked for this class    inferred, and not established" ^0
"  remaining issues of this class              inferred as low; not measured" ^0
"" ^0

# ---- what the same finding rate implies about the rest ----
#
# Three issues in 400 endpoints is a rate. Applied to the endpoints nobody
# looked at, it is an estimate the summary neither makes nor refutes.

int(not_examined * issues_found / endpoints_sampled) => expected_remaining

"  issue rate in the sample : " + str(issues_found) + " in " + str(endpoints_sampled) ^0
"  endpoints not examined   : " + str(not_examined) ^0
"  issues expected there    : " + str(expected_remaining) ^0
"  issues reported          : " + str(issues_found) ^0
"  issues fixed             : " + str(issues_found) ^0
"" ^0
"  the summary is a complete account of " + str(issues_found) + " and silent about " + str(expected_remaining) ^0
"  and the silence is not evasion; it is what a summary IS" ^0
"" ^0

# ---- the sentence that would have carried it ----
#
# One clause. It costs nothing, it makes no additional claim, and it removes
# the inference entirely.

"the same summary with its denominator" ^0
"  reviewed " + str(endpoints_sampled) + " of " + str(endpoints_total) + " endpoints in the authentication flow," ^0
"  found " + str(issues_found) + " issues, all fixed" ^0
"" ^0
"  words added        : six" ^0
"  claims added       : none" ^0
"  inferences removed : the one that mattered" ^0
"" ^0

# ---- what makes this hard to notice ----

"why nobody asked for the denominator" ^0
"  the finding count is specific        : " + str(issues_found) + ", not 'several'" ^0
"  each finding has a fix and a verifier: yes" ^0
"  the summary avoids the word complete : yes" ^0
"  precision on the findings reads as precision about the review" ^0
"  and the more careful the finding list, the stronger that reading" ^0
"" ^0
"  a vaguer summary would have been questioned" ^0
"" ^0

# ---- the control ----
#
# The review itself. It sampled properly, examined each endpoint carefully,
# found real issues and fixed them. There is no defect in the work and none in
# the report of it.

"control - was the review done well" ^0
"  endpoints examined thoroughly : yes, all " + str(endpoints_sampled) ^0
"  false findings                : 0" ^0
"  findings missed within the sample : none known" ^0
"  fixes verified                : " + str(issues_found) + " of " + str(issues_found) ^0
"  defects in the review or the summary : 0" ^0
"" ^0
"  the review is correct, the summary is correct, and the sentence a" ^0
"  reader forms from the summary is not" ^0
"" ^0

# ---- the null control ----
#
# The same summary after a review that covered everything. Identical wording,
# and now the reader's inference is exactly right. The words did not change;
# the number they omit did.

52000 => nc_sampled

"null control - the same sentence after a complete review" ^0
"  endpoints examined : " + str(nc_sampled) + " of " + str(endpoints_total) ^0
"  coverage           : " + str(int(nc_sampled * 1000 / endpoints_total)) + " per mille" ^0
"  summary wording    : identical" ^0
"  reader's inference : identical, and now correct" ^0
"  the summary cannot distinguish the two cases, and neither can the reader" ^0
"" ^0

# ---- the rule ----

"what a summary drops, and what that costs" ^0
"  findings        kept, they are what a summary is for" ^0
"  the population  dropped, an absence has no sentence" ^0
"  a reader supplies the missing population from context" ^0
"  and context suggests 'all of it', because that is the usual case" ^0
"" ^0
"the fix is not more caution in the wording" ^0
"it is one number: what was examined, out of what" ^0
"a finding count with no denominator is not an incomplete summary," ^0
"it is a summary of a different quantity" ^0
"" ^0

"The reviewer found " + str(issues_found) + " real issues, fixed all " + str(issues_found) + ", verified the fixes, and" ^0
"claimed nothing beyond that - which is more restraint than most such summaries" ^0
"show. " + str(endpoints_sampled) + " of " + str(endpoints_total) + " endpoints were examined, " + str(coverage_per_mille) + " per mille, and the same" ^0
"finding rate over the " + str(not_examined) + " nobody opened is about " + str(expected_remaining) + " more. Six words would" ^0
"have carried that, and they would have added no claim at all." ^0

endpoints_total = 52000
endpoints_sampled = 400
issues_found = 3
print("endpoints in the service : " + str(endpoints_total))
print("endpoints examined       : " + str(endpoints_sampled))
print("issues found             : " + str(issues_found))
print("issues fixed             : " + str(issues_found))
print("")
coverage_per_mille = int(endpoints_sampled * 1000 / endpoints_total)
not_examined = endpoints_total - endpoints_sampled
print("  coverage        : " + str(coverage_per_mille) + " per mille")
print("  not examined    : " + str(not_examined))
print("")
print("claim in the summary                    true")
print("  the authentication flow was reviewed    yes")
print("  three issues were found                 yes")
print("  all three were fixed                    yes")
print("  the fixes were verified                 yes")
print("  false claims : 0 of 4")
print("")
print("what a reader takes from it")
print("  the authentication flow has been reviewed   inferred")
print("  the flow now has three fewer issues         inferred, and true")
print("  the flow has been checked for this class    inferred, and not established")
print("  remaining issues of this class              inferred as low; not measured")
print("")
expected_remaining = int(not_examined * issues_found / endpoints_sampled)
print("  issue rate in the sample : " + str(issues_found) + " in " + str(endpoints_sampled))
print("  endpoints not examined   : " + str(not_examined))
print("  issues expected there    : " + str(expected_remaining))
print("  issues reported          : " + str(issues_found))
print("  issues fixed             : " + str(issues_found))
print("")
print("  the summary is a complete account of " + str(issues_found) + " and silent about " + str(expected_remaining))
print("  and the silence is not evasion; it is what a summary IS")
print("")
print("the same summary with its denominator")
print("  reviewed " + str(endpoints_sampled) + " of " + str(endpoints_total) + " endpoints in the authentication flow,")
print("  found " + str(issues_found) + " issues, all fixed")
print("")
print("  words added        : six")
print("  claims added       : none")
print("  inferences removed : the one that mattered")
print("")
print("why nobody asked for the denominator")
print("  the finding count is specific        : " + str(issues_found) + ", not 'several'")
print("  each finding has a fix and a verifier: yes")
print("  the summary avoids the word complete : yes")
print("  precision on the findings reads as precision about the review")
print("  and the more careful the finding list, the stronger that reading")
print("")
print("  a vaguer summary would have been questioned")
print("")
print("control - was the review done well")
print("  endpoints examined thoroughly : yes, all " + str(endpoints_sampled))
print("  false findings                : 0")
print("  findings missed within the sample : none known")
print("  fixes verified                : " + str(issues_found) + " of " + str(issues_found))
print("  defects in the review or the summary : 0")
print("")
print("  the review is correct, the summary is correct, and the sentence a")
print("  reader forms from the summary is not")
print("")
nc_sampled = 52000
print("null control - the same sentence after a complete review")
print("  endpoints examined : " + str(nc_sampled) + " of " + str(endpoints_total))
print("  coverage           : " + str(int(nc_sampled * 1000 / endpoints_total)) + " per mille")
print("  summary wording    : identical")
print("  reader's inference : identical, and now correct")
print("  the summary cannot distinguish the two cases, and neither can the reader")
print("")
print("what a summary drops, and what that costs")
print("  findings        kept, they are what a summary is for")
print("  the population  dropped, an absence has no sentence")
print("  a reader supplies the missing population from context")
print("  and context suggests 'all of it', because that is the usual case")
print("")
print("the fix is not more caution in the wording")
print("it is one number: what was examined, out of what")
print("a finding count with no denominator is not an incomplete summary,")
print("it is a summary of a different quantity")
print("")
print("The reviewer found " + str(issues_found) + " real issues, fixed all " + str(issues_found) + ", verified the fixes, and")
print("claimed nothing beyond that - which is more restraint than most such summaries")
print("show. " + str(endpoints_sampled) + " of " + str(endpoints_total) + " endpoints were examined, " + str(coverage_per_mille) + " per mille, and the same")
print("finding rate over the " + str(not_examined) + " nobody opened is about " + str(expected_remaining) + " more. Six words would")
print("have carried that, and they would have added no claim at all.")

endpoints in the service : 52000
endpoints examined       : 400
issues found             : 3
issues fixed             : 3

  coverage        : 7 per mille
  not examined    : 51600

claim in the summary                    true
  the authentication flow was reviewed    yes
  three issues were found                 yes
  all three were fixed                    yes
  the fixes were verified                 yes
  false claims : 0 of 4

what a reader takes from it
  the authentication flow has been reviewed   inferred
  the flow now has three fewer issues         inferred, and true
  the flow has been checked for this class    inferred, and not established
  remaining issues of this class              inferred as low; not measured

  issue rate in the sample : 3 in 400
  endpoints not examined   : 51600
  issues expected there    : 387
  issues reported          : 3
  issues fixed             : 3

  the summary is a complete account of 3 and silent about 387
  and the silence is not evasion; it is what a summary IS

the same summary with its denominator
  reviewed 400 of 52000 endpoints in the authentication flow,
  found 3 issues, all fixed

  words added        : six
  claims added       : none
  inferences removed : the one that mattered

why nobody asked for the denominator
  the finding count is specific        : 3, not 'several'
  each finding has a fix and a verifier: yes
  the summary avoids the word complete : yes
  precision on the findings reads as precision about the review
  and the more careful the finding list, the stronger that reading

  a vaguer summary would have been questioned

control - was the review done well
  endpoints examined thoroughly : yes, all 400
  false findings                : 0
  findings missed within the sample : none known
  fixes verified                : 3 of 3
  defects in the review or the summary : 0

  the review is correct, the summary is correct, and the sentence a
  reader forms from the summary is not

null control - the same sentence after a complete review
  endpoints examined : 52000 of 52000
  coverage           : 1000 per mille
  summary wording    : identical
  reader's inference : identical, and now correct
  the summary cannot distinguish the two cases, and neither can the reader

what a summary drops, and what that costs
  findings        kept, they are what a summary is for
  the population  dropped, an absence has no sentence
  a reader supplies the missing population from context
  and context suggests 'all of it', because that is the usual case

the fix is not more caution in the wording
it is one number: what was examined, out of what
a finding count with no denominator is not an incomplete summary,
it is a summary of a different quantity

The reviewer found 3 real issues, fixed all 3, verified the fixes, and
claimed nothing beyond that - which is more restraint than most such summaries
show. 400 of 52000 endpoints were examined, 7 per mille, and the same
finding rate over the 51600 nobody opened is about 387 more. Six words would
have carried that, and they would have added no claim at all.

Trace event types