The data grew ten fold and the job grew a hundred

# Self-authored for the EML case corpus (no external origin). A nightly job
# finds duplicate records by comparing every record with every other, it runs
# in a minute on a thousand records, and the plan for ten thousand records
# allows ten minutes. How the number of comparisons grows with the number of
# records is computed below.
#
# The plan is careful. The one-minute timing is real and measured; ten thousand
# is exactly ten times a thousand; the allowance is ten times the timing; and
# the intent is exactly 'the same job on ten times the data'.
#
# Every-with-every is a count of pairs, and pairs grow with the square of the
# records, so ten times the records is a hundred times the comparisons - the
# ten-minute window holds a hundred-minute job.

1000 => records_now
10000 => records_planned
1 => minutes_now
10 => minutes_allowed

records_now - 1 => records_now_minus_one
int(records_now * records_now_minus_one / 2) => comparisons_now
records_planned - 1 => records_planned_minus_one
int(records_planned * records_planned_minus_one / 2) => comparisons_planned
int(comparisons_planned / comparisons_now) => comparisons_growth_factor
int(records_planned / records_now) => records_growth_factor
minutes_now * comparisons_growth_factor => minutes_the_job_will_take
minutes_the_job_will_take - minutes_allowed => minutes_over_the_window

"records now                     : " + str(records_now) + ", " + str(comparisons_now) + " comparisons, " + str(minutes_now) + " minute" ^0
"records planned                 : " + str(records_planned) + ", " + str(records_growth_factor) + " times as many" ^0
"minutes allowed                 : " + str(minutes_allowed) + ", " + str(records_growth_factor) + " times the timing" ^0
"" ^0
"comparisons at the plan         : " + str(comparisons_planned) ^0
"comparisons growth              : " + str(comparisons_growth_factor) + " times" ^0
"minutes the job will take       : " + str(minutes_the_job_will_take) ^0
"minutes over the window         : " + str(minutes_over_the_window) ^0
"" ^0

# ---- what the plan verified ----

"the capacity plan" ^0
"  timing : real and measured, " + str(minutes_now) + " minute on " + str(records_now) ^0
"  data growth : exactly " + str(records_growth_factor) + " times" ^0
"  allowance : " + str(records_growth_factor) + " times the timing" ^0
"  intent : the same job on ten times the data" ^0
"  timings guessed : 0" ^0
"  verdict : TEN TIMES THE DATA, TEN TIMES THE WINDOW" ^0
"" ^0
"  scaling the window by the measured growth in data is the" ^0
"  part done right here, and it is why the allowance is" ^0
"  exactly proportionate to the records" ^0
"" ^0

# ---- how the comparisons grow ----

"pairs" ^0
"  what every-with-every counts : pairs of records" ^0
"  pairs in " + str(records_now) + " : " + str(comparisons_now) ^0
"  pairs in " + str(records_planned) + " : " + str(comparisons_planned) ^0
"  records grew : " + str(records_growth_factor) + " times; comparisons grew " + str(comparisons_growth_factor) + " times" ^0
"  so the job : " + str(minutes_the_job_will_take) + " minutes, not " + str(minutes_allowed) ^0
"  the window was scaled : with the records; the work" ^0
"    scales with their square" ^0
"" ^0

# ---- what the operators got ----

"the night" ^0
"  window : " + str(minutes_allowed) + " minutes" ^0
"  job : " + str(minutes_the_job_will_take) + " minutes, " + str(minutes_over_the_window) + " over" ^0
"  is the timing wrong : no; one minute is measured" ^0
"  is the work proportional to the data : no; it is" ^0
"    proportional to the pairs, and the pairs are the square" ^0
"" ^0

# ---- null control ----

# The same duplicates found by sorting or hashing first, so each record is
# compared only with its neighbours and the work grows with the records.
100 => nc_work_growth_every_with_every
10 => nc_work_growth_sort_or_hash_first
10 => nc_minutes_at_ten_thousand_with_sort_or_hash

"null control - sort or hash before comparing" ^0
"  work growth, every with every : " + str(nc_work_growth_every_with_every) + " times" ^0
"  work growth, sort or hash first : " + str(nc_work_growth_sort_or_hash_first) + " times" ^0
"  minutes at ten thousand, sort or hash first : " + str(nc_minutes_at_ten_thousand_with_sort_or_hash) ^0
"  no record changed; the job stopped touching every pair" ^0
"" ^0

# ---- the rule ----

"what a window scaled with the data guarantees" ^0
"  the window is ten times the measured minute : exactly," ^0
"    real timing, exact growth factor" ^0
"  the job fits the window : not addressed; every-with-every" ^0
"    is a count of pairs, which grows as the square, so ten" ^0
"    times the records is " + str(comparisons_growth_factor) + " times the work and " + str(minutes_the_job_will_take) + " minutes" ^0
"" ^0

"work that touches every pair grows with the square of what it touches; a" ^0
"plan that scales the window with the count has scaled it with the root of" ^0
"the work, and the gap between the two is a factor equal to the growth itself" ^0
"" ^0

"The window is exactly " + str(records_growth_factor) + " times the measured minute - proportionate to the" ^0
"records. But the job compares every pair, and pairs grow as the square: " + str(records_planned) ^0
"records are " + str(comparisons_planned) + " comparisons, " + str(comparisons_growth_factor) + " times the work, " + str(minutes_the_job_will_take) + " minutes in a " + str(minutes_allowed) + "-minute" ^0
"window, until the job stops touching every pair." ^0

records_now = 1000
records_planned = 10000
minutes_now = 1
minutes_allowed = 10
records_now_minus_one = records_now - 1
comparisons_now = int(records_now * records_now_minus_one / 2)
records_planned_minus_one = records_planned - 1
comparisons_planned = int(records_planned * records_planned_minus_one / 2)
comparisons_growth_factor = int(comparisons_planned / comparisons_now)
records_growth_factor = int(records_planned / records_now)
minutes_the_job_will_take = minutes_now * comparisons_growth_factor
minutes_over_the_window = minutes_the_job_will_take - minutes_allowed
print("records now                     : " + str(records_now) + ", " + str(comparisons_now) + " comparisons, " + str(minutes_now) + " minute")
print("records planned                 : " + str(records_planned) + ", " + str(records_growth_factor) + " times as many")
print("minutes allowed                 : " + str(minutes_allowed) + ", " + str(records_growth_factor) + " times the timing")
print("")
print("comparisons at the plan         : " + str(comparisons_planned))
print("comparisons growth              : " + str(comparisons_growth_factor) + " times")
print("minutes the job will take       : " + str(minutes_the_job_will_take))
print("minutes over the window         : " + str(minutes_over_the_window))
print("")
print("the capacity plan")
print("  timing : real and measured, " + str(minutes_now) + " minute on " + str(records_now))
print("  data growth : exactly " + str(records_growth_factor) + " times")
print("  allowance : " + str(records_growth_factor) + " times the timing")
print("  intent : the same job on ten times the data")
print("  timings guessed : 0")
print("  verdict : TEN TIMES THE DATA, TEN TIMES THE WINDOW")
print("")
print("  scaling the window by the measured growth in data is the")
print("  part done right here, and it is why the allowance is")
print("  exactly proportionate to the records")
print("")
print("pairs")
print("  what every-with-every counts : pairs of records")
print("  pairs in " + str(records_now) + " : " + str(comparisons_now))
print("  pairs in " + str(records_planned) + " : " + str(comparisons_planned))
print("  records grew : " + str(records_growth_factor) + " times; comparisons grew " + str(comparisons_growth_factor) + " times")
print("  so the job : " + str(minutes_the_job_will_take) + " minutes, not " + str(minutes_allowed))
print("  the window was scaled : with the records; the work")
print("    scales with their square")
print("")
print("the night")
print("  window : " + str(minutes_allowed) + " minutes")
print("  job : " + str(minutes_the_job_will_take) + " minutes, " + str(minutes_over_the_window) + " over")
print("  is the timing wrong : no; one minute is measured")
print("  is the work proportional to the data : no; it is")
print("    proportional to the pairs, and the pairs are the square")
print("")
nc_work_growth_every_with_every = 100
nc_work_growth_sort_or_hash_first = 10
nc_minutes_at_ten_thousand_with_sort_or_hash = 10
print("null control - sort or hash before comparing")
print("  work growth, every with every : " + str(nc_work_growth_every_with_every) + " times")
print("  work growth, sort or hash first : " + str(nc_work_growth_sort_or_hash_first) + " times")
print("  minutes at ten thousand, sort or hash first : " + str(nc_minutes_at_ten_thousand_with_sort_or_hash))
print("  no record changed; the job stopped touching every pair")
print("")
print("what a window scaled with the data guarantees")
print("  the window is ten times the measured minute : exactly,")
print("    real timing, exact growth factor")
print("  the job fits the window : not addressed; every-with-every")
print("    is a count of pairs, which grows as the square, so ten")
print("    times the records is " + str(comparisons_growth_factor) + " times the work and " + str(minutes_the_job_will_take) + " minutes")
print("")
print("work that touches every pair grows with the square of what it touches; a")
print("plan that scales the window with the count has scaled it with the root of")
print("the work, and the gap between the two is a factor equal to the growth itself")
print("")
print("The window is exactly " + str(records_growth_factor) + " times the measured minute - proportionate to the")
print("records. But the job compares every pair, and pairs grow as the square: " + str(records_planned))
print("records are " + str(comparisons_planned) + " comparisons, " + str(comparisons_growth_factor) + " times the work, " + str(minutes_the_job_will_take) + " minutes in a " + str(minutes_allowed) + "-minute")
print("window, until the job stops touching every pair.")

records now                     : 1000, 499500 comparisons, 1 minute
records planned                 : 10000, 10 times as many
minutes allowed                 : 10, 10 times the timing

comparisons at the plan         : 49995000
comparisons growth              : 100 times
minutes the job will take       : 100
minutes over the window         : 90

the capacity plan
  timing : real and measured, 1 minute on 1000
  data growth : exactly 10 times
  allowance : 10 times the timing
  intent : the same job on ten times the data
  timings guessed : 0
  verdict : TEN TIMES THE DATA, TEN TIMES THE WINDOW

  scaling the window by the measured growth in data is the
  part done right here, and it is why the allowance is
  exactly proportionate to the records

pairs
  what every-with-every counts : pairs of records
  pairs in 1000 : 499500
  pairs in 10000 : 49995000
  records grew : 10 times; comparisons grew 100 times
  so the job : 100 minutes, not 10
  the window was scaled : with the records; the work
    scales with their square

the night
  window : 10 minutes
  job : 100 minutes, 90 over
  is the timing wrong : no; one minute is measured
  is the work proportional to the data : no; it is
    proportional to the pairs, and the pairs are the square

null control - sort or hash before comparing
  work growth, every with every : 100 times
  work growth, sort or hash first : 10 times
  minutes at ten thousand, sort or hash first : 10
  no record changed; the job stopped touching every pair

what a window scaled with the data guarantees
  the window is ten times the measured minute : exactly,
    real timing, exact growth factor
  the job fits the window : not addressed; every-with-every
    is a count of pairs, which grows as the square, so ten
    times the records is 100 times the work and 100 minutes

work that touches every pair grows with the square of what it touches; a
plan that scales the window with the count has scaled it with the root of
the work, and the gap between the two is a factor equal to the growth itself

The window is exactly 10 times the measured minute - proportionate to the
records. But the job compares every pair, and pairs grow as the square: 10000
records are 49995000 comparisons, 100 times the work, 100 minutes in a 10-minute
window, until the job stops touching every pair.

Trace event types