The questionnaire was answered by the vendor

# Self-authored for the EML case corpus (no external origin). Every vendor that
# touches customer data completes a one-hundred-and-eighty-question security
# review before a contract is signed, and forty-one have been rejected or made
# to remediate. Who writes the answers is computed below.
#
# The programme is real. A review takes six hours of a reviewer's time rather
# than a checkbox; the questions were written by the security team and not
# bought as a template; a vendor cannot be onboarded while a review is open, and
# that block is in the procurement system rather than in a policy; and forty-one
# vendors in thirty-three months were stopped or made to change something.
#
# The answers are the vendor's. So is every document attached to them.

340 => vendors_reviewed
180 => questions_in_the_questionnaire
22 => questions_that_ask_for_a_document
0 => answers_checkable_without_the_vendor
6 => reviewer_hours_per_review
33 => months_the_programme_has_run
41 => vendors_rejected_or_made_to_remediate
0 => vendors_rejected_for_an_answer_shown_to_be_false
3 => breaches_at_a_reviewed_vendor
3 => those_breaches_in_an_area_the_questionnaire_covers

questions_in_the_questionnaire - questions_that_ask_for_a_document => questions_answered_by_assertion_alone
vendors_reviewed * questions_in_the_questionnaire => questions_asked_in_total
vendors_reviewed * reviewer_hours_per_review => reviewer_hours_spent
vendors_rejected_or_made_to_remediate - vendors_rejected_for_an_answer_shown_to_be_false => vendors_rejected_for_what_they_said
int(questions_that_ask_for_a_document * 10000 / questions_in_the_questionnaire) => document_backed_per_myriad
int(vendors_rejected_or_made_to_remediate * 10000 / vendors_reviewed) => rejected_per_myriad

"vendors reviewed                : " + str(vendors_reviewed) ^0
"questions per review            : " + str(questions_in_the_questionnaire) ^0
"  that ask for a document       : " + str(questions_that_ask_for_a_document) ^0
"  answered by assertion alone   : " + str(questions_answered_by_assertion_alone) ^0
"  document-backed               : " + str(document_backed_per_myriad) + " per ten thousand" ^0
"questions asked in total        : " + str(questions_asked_in_total) ^0
"answers checkable without them  : " + str(answers_checkable_without_the_vendor) ^0
"" ^0
"reviewer hours per review       : " + str(reviewer_hours_per_review) ^0
"reviewer hours spent            : " + str(reviewer_hours_spent) ^0
"months the programme has run    : " + str(months_the_programme_has_run) ^0
"" ^0
"vendors rejected or remediated  : " + str(vendors_rejected_or_made_to_remediate) ^0
"  rejected share                : " + str(rejected_per_myriad) + " per ten thousand" ^0
"  for what the vendor said      : " + str(vendors_rejected_for_what_they_said) ^0
"  for an answer shown to be false : " + str(vendors_rejected_for_an_answer_shown_to_be_false) ^0
"" ^0
"breaches at a reviewed vendor   : " + str(breaches_at_a_reviewed_vendor) ^0
"  in an area the questions cover: " + str(those_breaches_in_an_area_the_questionnaire_covers) ^0
"" ^0

# ---- what the programme verified ----

"the vendor security review" ^0
"  effort : " + str(reviewer_hours_per_review) + " reviewer hours each, not a checkbox" ^0
"  the questions : written by the security team, not a" ^0
"    bought template" ^0
"  the block : a vendor cannot be onboarded while a review" ^0
"    is open, and that is enforced in procurement rather" ^0
"    than written in a policy" ^0
"  stopped or changed in " + str(months_the_programme_has_run) + " months : " + str(vendors_rejected_or_made_to_remediate) ^0
"  verdict : REVIEWED" ^0
"" ^0
"  putting the block in the procurement system rather than" ^0
"  in a policy is the part almost nobody does, and it is" ^0
"  why the " + str(vendors_rejected_or_made_to_remediate) + " were actually stopped" ^0
"" ^0

# ---- whose word the evidence is ----

"where each answer comes from" ^0
"  who writes it : the vendor" ^0
"  who writes the attached document : the vendor" ^0
"  questions answered by assertion alone : " ^0
"    " + str(questions_answered_by_assertion_alone) + " of " + str(questions_in_the_questionnaire) ^0
"  answers checkable against something the vendor does" ^0
"    not control : " + str(answers_checkable_without_the_vendor) ^0
"  so what a completed review establishes : what they" ^0
"    said, in writing, on the record" ^0
"" ^0
"  the subject of the question and the author of the" ^0
"  answer are the same party" ^0
"" ^0

# ---- what the forty-one were rejected for ----

"the rejections" ^0
"  for answering no, or refusing to answer : " ^0
"    " + str(vendors_rejected_for_what_they_said) ^0
"  for an answer that was checked and found false : " ^0
"    " + str(vendors_rejected_for_an_answer_shown_to_be_false) ^0
"  what that pair means : the programme discriminates" ^0
"    between vendors who admit a gap and vendors who do" ^0
"    not, which is a real distinction and not the one the" ^0
"    questionnaire is about" ^0
"  a vendor with the same gap who answers yes : passes" ^0
"" ^0

# ---- what happened anyway ----

"breaches at vendors that passed" ^0
"  breaches : " + str(breaches_at_a_reviewed_vendor) ^0
"  in areas the questionnaire asks about : " ^0
"    " + str(those_breaches_in_an_area_the_questionnaire_covers) ^0
"  what those vendors had answered : yes" ^0
"  was the answer wrong when given : unknown; nothing" ^0
"    recorded the state at the time" ^0
"  did the review fail : no. It asked, and it recorded" ^0
"    the answer, and the answer is what it recorded" ^0
"" ^0

# ---- null control ----

# The same programme, plus a sample: twenty vendors, five answers each, checked
# against something the vendor does not control - their own public DNS and TLS
# configuration, published breach notifications, and a test we commission.
100 => nc_answers_checked_independently
9 => nc_answers_that_did_not_match
340 => nc_reviews_completed

"null control - check a sample against outside sources" ^0
"  reviews completed : " + str(nc_reviews_completed) + ", unchanged" ^0
"  answers checked independently : " + str(nc_answers_checked_independently) ^0
"  answers that did not match : " + str(nc_answers_that_did_not_match) ^0
"  the questionnaire did not get better; a second author" ^0
"  was found for a hundred of its answers" ^0
"" ^0

# ---- the rule ----

"what a completed vendor review guarantees" ^0
"  the vendor stated that they do these things : exactly," ^0
"    " + str(questions_in_the_questionnaire) + " questions, " + str(reviewer_hours_per_review) + " hours, on the record, and " ^0
"    " + str(vendors_rejected_or_made_to_remediate) + " were stopped over what they stated" ^0
"  the vendor does these things : not addressed; the" ^0
"    answer and the thing it is about have the same" ^0
"    author, and " + str(answers_checkable_without_the_vendor) + " answers are checkable without them" ^0
"" ^0
"asking carefully is not the same as finding out; a review" ^0
"whose every input is written by its subject measures what" ^0
"the subject is willing to write down" ^0
"" ^0

"The questions were written in-house, a review costs " + str(reviewer_hours_per_review) + " reviewer hours, onboarding is" ^0
"blocked in procurement while one is open, and " + str(vendors_rejected_or_made_to_remediate) + " vendors were stopped in " + str(months_the_programme_has_run) ^0
"months. Every answer and every attached document is the vendor's own - " ^0
"" + str(questions_answered_by_assertion_alone) + " of " + str(questions_in_the_questionnaire) + " questions carry no document at all, " + str(answers_checkable_without_the_vendor) + " answers are checkable" ^0
"without them, and " + str(vendors_rejected_for_an_answer_shown_to_be_false) + " of the " + str(vendors_rejected_or_made_to_remediate) + " were rejected for saying something untrue." ^0

vendors_reviewed = 340
questions_in_the_questionnaire = 180
questions_that_ask_for_a_document = 22
answers_checkable_without_the_vendor = 0
reviewer_hours_per_review = 6
months_the_programme_has_run = 33
vendors_rejected_or_made_to_remediate = 41
vendors_rejected_for_an_answer_shown_to_be_false = 0
breaches_at_a_reviewed_vendor = 3
those_breaches_in_an_area_the_questionnaire_covers = 3
questions_answered_by_assertion_alone = questions_in_the_questionnaire - questions_that_ask_for_a_document
questions_asked_in_total = vendors_reviewed * questions_in_the_questionnaire
reviewer_hours_spent = vendors_reviewed * reviewer_hours_per_review
vendors_rejected_for_what_they_said = vendors_rejected_or_made_to_remediate - vendors_rejected_for_an_answer_shown_to_be_false
document_backed_per_myriad = int(questions_that_ask_for_a_document * 10000 / questions_in_the_questionnaire)
rejected_per_myriad = int(vendors_rejected_or_made_to_remediate * 10000 / vendors_reviewed)
print("vendors reviewed                : " + str(vendors_reviewed))
print("questions per review            : " + str(questions_in_the_questionnaire))
print("  that ask for a document       : " + str(questions_that_ask_for_a_document))
print("  answered by assertion alone   : " + str(questions_answered_by_assertion_alone))
print("  document-backed               : " + str(document_backed_per_myriad) + " per ten thousand")
print("questions asked in total        : " + str(questions_asked_in_total))
print("answers checkable without them  : " + str(answers_checkable_without_the_vendor))
print("")
print("reviewer hours per review       : " + str(reviewer_hours_per_review))
print("reviewer hours spent            : " + str(reviewer_hours_spent))
print("months the programme has run    : " + str(months_the_programme_has_run))
print("")
print("vendors rejected or remediated  : " + str(vendors_rejected_or_made_to_remediate))
print("  rejected share                : " + str(rejected_per_myriad) + " per ten thousand")
print("  for what the vendor said      : " + str(vendors_rejected_for_what_they_said))
print("  for an answer shown to be false : " + str(vendors_rejected_for_an_answer_shown_to_be_false))
print("")
print("breaches at a reviewed vendor   : " + str(breaches_at_a_reviewed_vendor))
print("  in an area the questions cover: " + str(those_breaches_in_an_area_the_questionnaire_covers))
print("")
print("the vendor security review")
print("  effort : " + str(reviewer_hours_per_review) + " reviewer hours each, not a checkbox")
print("  the questions : written by the security team, not a")
print("    bought template")
print("  the block : a vendor cannot be onboarded while a review")
print("    is open, and that is enforced in procurement rather")
print("    than written in a policy")
print("  stopped or changed in " + str(months_the_programme_has_run) + " months : " + str(vendors_rejected_or_made_to_remediate))
print("  verdict : REVIEWED")
print("")
print("  putting the block in the procurement system rather than")
print("  in a policy is the part almost nobody does, and it is")
print("  why the " + str(vendors_rejected_or_made_to_remediate) + " were actually stopped")
print("")
print("where each answer comes from")
print("  who writes it : the vendor")
print("  who writes the attached document : the vendor")
print("  questions answered by assertion alone : ")
print("    " + str(questions_answered_by_assertion_alone) + " of " + str(questions_in_the_questionnaire))
print("  answers checkable against something the vendor does")
print("    not control : " + str(answers_checkable_without_the_vendor))
print("  so what a completed review establishes : what they")
print("    said, in writing, on the record")
print("")
print("  the subject of the question and the author of the")
print("  answer are the same party")
print("")
print("the rejections")
print("  for answering no, or refusing to answer : ")
print("    " + str(vendors_rejected_for_what_they_said))
print("  for an answer that was checked and found false : ")
print("    " + str(vendors_rejected_for_an_answer_shown_to_be_false))
print("  what that pair means : the programme discriminates")
print("    between vendors who admit a gap and vendors who do")
print("    not, which is a real distinction and not the one the")
print("    questionnaire is about")
print("  a vendor with the same gap who answers yes : passes")
print("")
print("breaches at vendors that passed")
print("  breaches : " + str(breaches_at_a_reviewed_vendor))
print("  in areas the questionnaire asks about : ")
print("    " + str(those_breaches_in_an_area_the_questionnaire_covers))
print("  what those vendors had answered : yes")
print("  was the answer wrong when given : unknown; nothing")
print("    recorded the state at the time")
print("  did the review fail : no. It asked, and it recorded")
print("    the answer, and the answer is what it recorded")
print("")
nc_answers_checked_independently = 100
nc_answers_that_did_not_match = 9
nc_reviews_completed = 340
print("null control - check a sample against outside sources")
print("  reviews completed : " + str(nc_reviews_completed) + ", unchanged")
print("  answers checked independently : " + str(nc_answers_checked_independently))
print("  answers that did not match : " + str(nc_answers_that_did_not_match))
print("  the questionnaire did not get better; a second author")
print("  was found for a hundred of its answers")
print("")
print("what a completed vendor review guarantees")
print("  the vendor stated that they do these things : exactly,")
print("    " + str(questions_in_the_questionnaire) + " questions, " + str(reviewer_hours_per_review) + " hours, on the record, and ")
print("    " + str(vendors_rejected_or_made_to_remediate) + " were stopped over what they stated")
print("  the vendor does these things : not addressed; the")
print("    answer and the thing it is about have the same")
print("    author, and " + str(answers_checkable_without_the_vendor) + " answers are checkable without them")
print("")
print("asking carefully is not the same as finding out; a review")
print("whose every input is written by its subject measures what")
print("the subject is willing to write down")
print("")
print("The questions were written in-house, a review costs " + str(reviewer_hours_per_review) + " reviewer hours, onboarding is")
print("blocked in procurement while one is open, and " + str(vendors_rejected_or_made_to_remediate) + " vendors were stopped in " + str(months_the_programme_has_run))
print("months. Every answer and every attached document is the vendor's own - ")
print("" + str(questions_answered_by_assertion_alone) + " of " + str(questions_in_the_questionnaire) + " questions carry no document at all, " + str(answers_checkable_without_the_vendor) + " answers are checkable")
print("without them, and " + str(vendors_rejected_for_an_answer_shown_to_be_false) + " of the " + str(vendors_rejected_or_made_to_remediate) + " were rejected for saying something untrue.")

vendors reviewed                : 340
questions per review            : 180
  that ask for a document       : 22
  answered by assertion alone   : 158
  document-backed               : 1222 per ten thousand
questions asked in total        : 61200
answers checkable without them  : 0

reviewer hours per review       : 6
reviewer hours spent            : 2040
months the programme has run    : 33

vendors rejected or remediated  : 41
  rejected share                : 1205 per ten thousand
  for what the vendor said      : 41
  for an answer shown to be false : 0

breaches at a reviewed vendor   : 3
  in an area the questions cover: 3

the vendor security review
  effort : 6 reviewer hours each, not a checkbox
  the questions : written by the security team, not a
    bought template
  the block : a vendor cannot be onboarded while a review
    is open, and that is enforced in procurement rather
    than written in a policy
  stopped or changed in 33 months : 41
  verdict : REVIEWED

  putting the block in the procurement system rather than
  in a policy is the part almost nobody does, and it is
  why the 41 were actually stopped

where each answer comes from
  who writes it : the vendor
  who writes the attached document : the vendor
  questions answered by assertion alone : 
    158 of 180
  answers checkable against something the vendor does
    not control : 0
  so what a completed review establishes : what they
    said, in writing, on the record

  the subject of the question and the author of the
  answer are the same party

the rejections
  for answering no, or refusing to answer : 
    41
  for an answer that was checked and found false : 
    0
  what that pair means : the programme discriminates
    between vendors who admit a gap and vendors who do
    not, which is a real distinction and not the one the
    questionnaire is about
  a vendor with the same gap who answers yes : passes

breaches at vendors that passed
  breaches : 3
  in areas the questionnaire asks about : 
    3
  what those vendors had answered : yes
  was the answer wrong when given : unknown; nothing
    recorded the state at the time
  did the review fail : no. It asked, and it recorded
    the answer, and the answer is what it recorded

null control - check a sample against outside sources
  reviews completed : 340, unchanged
  answers checked independently : 100
  answers that did not match : 9
  the questionnaire did not get better; a second author
  was found for a hundred of its answers

what a completed vendor review guarantees
  the vendor stated that they do these things : exactly,
    180 questions, 6 hours, on the record, and 
    41 were stopped over what they stated
  the vendor does these things : not addressed; the
    answer and the thing it is about have the same
    author, and 0 answers are checkable without them

asking carefully is not the same as finding out; a review
whose every input is written by its subject measures what
the subject is willing to write down

The questions were written in-house, a review costs 6 reviewer hours, onboarding is
blocked in procurement while one is open, and 41 vendors were stopped in 33
months. Every answer and every attached document is the vendor's own - 
158 of 180 questions carry no document at all, 0 answers are checkable
without them, and 0 of the 41 were rejected for saying something untrue.

Trace event types