The timestamp changed from event time to ingest time

# Self-authored for the EML case corpus (no external origin). A field called
# occurred_at changed from the time the event happened to the time the pipeline
# received it. Same name, same type, same format. What moved is computed below.
#
# The change was made for a good reason. Producers were sending clock-skewed
# event times, some of them from user devices, and 3% of events arrived dated
# in the future. Stamping at ingest gives a monotonic, trustworthy value that
# nothing upstream can corrupt, and the team that did it wrote a design note.
#
# Ingest time and event time are the same number whenever the pipeline is
# keeping up. They diverge exactly when it is not. So the field agrees with
# its old meaning on every ordinary day, and stops agreeing precisely on the
# days anybody looks at it closely.
#
# Both quantities are carried, per hour of one backlog day.

# [hour, events, minutes of pipeline lag, events dated into the next hour by ingest stamping]
[["09", 41000, 0, 0], ["10", 44000, 0, 0], ["11", 46000, 4, 3000], ["12", 52000, 31, 26000], ["13", 58000, 74, 58000], ["14", 55000, 92, 55000], ["15", 47000, 40, 31000], ["16", 43000, 6, 4000]] => hours

len(hours) => n
0 => total
0 => misdated
for h in hours:
    total + h[1] => total
    misdated + h[3] => misdated

"hour   events   pipeline lag   events stamped into a later hour" ^0
for h in hours:
    "  " + h[0] + "     " + str(h[1]) + "    " + str(h[2]) + " min        " + str(h[3]) ^0
"" ^0
"events that day        : " + str(total) ^0
"stamped into the wrong hour : " + str(misdated) + ", " + str(int(misdated * 100 / total)) + "%" ^0
"" ^0

# ---- when the two meanings agree ----

0 => agree_hours
for h in hours:
    if h[2] == 0:
        agree_hours + 1 => agree_hours
"hours in which the two meanings coincide" ^0
"  lag zero : " + str(agree_hours) + " of " + str(n) ^0
"  on those hours the field is identical under either definition" ^0
"  on the other " + str(n - agree_hours) + " it is not, and those are the hours with a backlog" ^0
"" ^0

# ---- what each consumer computes ----

"the hourly volume chart, both ways" ^0
"hour   by event time   by ingest time   difference" ^0
0 => prev_spill
for h in hours:
    h[1] - h[3] + prev_spill => by_ingest
    "  " + h[0] + "     " + str(h[1]) + "         " + str(by_ingest) + "        " + str(by_ingest - h[1]) ^0
    h[3] => prev_spill
"  the chart is smooth under both definitions and it is a different chart" ^0
"" ^0

# ---- the alert that reads it ----

50000 => alert_threshold
"an alert fires when an hour exceeds " + str(alert_threshold) + " events" ^0
0 => fires_event
0 => fires_ingest
0 => spill
for h in hours:
    if h[1] > alert_threshold:
        fires_event + 1 => fires_event
    h[1] - h[3] + spill => by_ingest
    if by_ingest > alert_threshold:
        fires_ingest + 1 => fires_ingest
    h[3] => spill
"  hours over the bar by event time  : " + str(fires_event) ^0
"  hours over the bar by ingest time : " + str(fires_ingest) ^0
"  same data, same threshold, same field name" ^0
"" ^0

# ---- the retention rule ----

"records are deleted 90 days after occurred_at" ^0
"  under event time  : 90 days after the thing happened" ^0
"  under ingest time : 90 days after we received it" ^0
"  for an event delayed " + str(hours[4][2]) + " minutes the difference is negligible" ^0
"  for a replayed backfill the difference is the age of the backfill" ^0
"  so a re-ingest of two-year-old data resets its deletion clock, and the" ^0
"  retention rule has no way to notice" ^0
"" ^0

# ---- what the change did and did not do ----

"the problem it solved" ^0
"  events dated in the future : 3% before, 0 after" ^0
"  clock skew from user devices : no longer reaches any consumer" ^0
"  monotonic ordering : guaranteed, which it was not before" ^0
"  all three of those are real and none of them came back" ^0
"" ^0
"what a consumer would need to notice" ^0
"  field name change : none" ^0
"  type change       : none" ^0
"  new field carrying the old meaning : not added" ^0
"  design note : written, and read by the two people in the review" ^0
"  consumers of this field : 11" ^0
"" ^0

# ---- the fix that keeps both ----

"carrying both stamps" ^0
"  occurred_at : event time, as the name says, skew and all" ^0
"  ingested_at : pipeline time, monotonic and trustworthy" ^0
"  bytes added per record : 8" ^0
"  bytes a day : " + str(total * 8) ^0
"  consumers that would then be choosing rather than inheriting : 11" ^0
"  the skew problem is solved by which field the ordering uses, not by" ^0
"  redefining the one that was already named after the other thing" ^0
"" ^0

# ---- the control: a field renamed with its meaning ----
#
# Where the new meaning got a new name, every consumer had to say which one it
# wanted, and the ones that did nothing kept the old behaviour.

[["received_at", 11, 11, 0]] => renamed
for r in renamed:
    "control - the same change shipped as a new field " + r[0] ^0
    "  consumers : " + str(r[1]) ^0
    "  consumers that had to make a choice : " + str(r[2]) ^0
    "  consumers that silently changed behaviour : " + str(r[3]) ^0
    "  the identical semantic change, and the rename is what converts a" ^0
    "  silent shift into a decision somebody makes" ^0
"" ^0

"Stamping at ingest removed real clock skew and 3% future-dated events." ^0
"The two meanings agree whenever the pipeline keeps up, so the field only" ^0
"disagrees with its own name on the " + str(n - agree_hours) + " hours anybody would investigate." ^0

hours = [["09", 41000, 0, 0], ["10", 44000, 0, 0], ["11", 46000, 4, 3000], ["12", 52000, 31, 26000], ["13", 58000, 74, 58000], ["14", 55000, 92, 55000], ["15", 47000, 40, 31000], ["16", 43000, 6, 4000]]
n = len(hours)
total = 0
misdated = 0
for h in hours:
    total = total + h[1]
    misdated = misdated + h[3]
print("hour   events   pipeline lag   events stamped into a later hour")
for h in hours:
    print("  " + h[0] + "     " + str(h[1]) + "    " + str(h[2]) + " min        " + str(h[3]))
print("")
print("events that day        : " + str(total))
print("stamped into the wrong hour : " + str(misdated) + ", " + str(int(misdated * 100 / total)) + "%")
print("")
agree_hours = 0
for h in hours:
    if h[2] == 0:
        agree_hours = agree_hours + 1
print("hours in which the two meanings coincide")
print("  lag zero : " + str(agree_hours) + " of " + str(n))
print("  on those hours the field is identical under either definition")
print("  on the other " + str(n - agree_hours) + " it is not, and those are the hours with a backlog")
print("")
print("the hourly volume chart, both ways")
print("hour   by event time   by ingest time   difference")
prev_spill = 0
for h in hours:
    by_ingest = h[1] - h[3] + prev_spill
    print("  " + h[0] + "     " + str(h[1]) + "         " + str(by_ingest) + "        " + str(by_ingest - h[1]))
    prev_spill = h[3]
print("  the chart is smooth under both definitions and it is a different chart")
print("")
alert_threshold = 50000
print("an alert fires when an hour exceeds " + str(alert_threshold) + " events")
fires_event = 0
fires_ingest = 0
spill = 0
for h in hours:
    if h[1] > alert_threshold:
        fires_event = fires_event + 1
    by_ingest = h[1] - h[3] + spill
    if by_ingest > alert_threshold:
        fires_ingest = fires_ingest + 1
    spill = h[3]
print("  hours over the bar by event time  : " + str(fires_event))
print("  hours over the bar by ingest time : " + str(fires_ingest))
print("  same data, same threshold, same field name")
print("")
print("records are deleted 90 days after occurred_at")
print("  under event time  : 90 days after the thing happened")
print("  under ingest time : 90 days after we received it")
print("  for an event delayed " + str(hours[4][2]) + " minutes the difference is negligible")
print("  for a replayed backfill the difference is the age of the backfill")
print("  so a re-ingest of two-year-old data resets its deletion clock, and the")
print("  retention rule has no way to notice")
print("")
print("the problem it solved")
print("  events dated in the future : 3% before, 0 after")
print("  clock skew from user devices : no longer reaches any consumer")
print("  monotonic ordering : guaranteed, which it was not before")
print("  all three of those are real and none of them came back")
print("")
print("what a consumer would need to notice")
print("  field name change : none")
print("  type change       : none")
print("  new field carrying the old meaning : not added")
print("  design note : written, and read by the two people in the review")
print("  consumers of this field : 11")
print("")
print("carrying both stamps")
print("  occurred_at : event time, as the name says, skew and all")
print("  ingested_at : pipeline time, monotonic and trustworthy")
print("  bytes added per record : 8")
print("  bytes a day : " + str(total * 8))
print("  consumers that would then be choosing rather than inheriting : 11")
print("  the skew problem is solved by which field the ordering uses, not by")
print("  redefining the one that was already named after the other thing")
print("")
renamed = [["received_at", 11, 11, 0]]
for r in renamed:
    print("control - the same change shipped as a new field " + r[0])
    print("  consumers : " + str(r[1]))
    print("  consumers that had to make a choice : " + str(r[2]))
    print("  consumers that silently changed behaviour : " + str(r[3]))
    print("  the identical semantic change, and the rename is what converts a")
    print("  silent shift into a decision somebody makes")
print("")
print("Stamping at ingest removed real clock skew and 3% future-dated events.")
print("The two meanings agree whenever the pipeline keeps up, so the field only")
print("disagrees with its own name on the " + str(n - agree_hours) + " hours anybody would investigate.")

hour   events   pipeline lag   events stamped into a later hour
  09     41000    0 min        0
  10     44000    0 min        0
  11     46000    4 min        3000
  12     52000    31 min        26000
  13     58000    74 min        58000
  14     55000    92 min        55000
  15     47000    40 min        31000
  16     43000    6 min        4000

events that day        : 386000
stamped into the wrong hour : 177000, 45%

hours in which the two meanings coincide
  lag zero : 2 of 8
  on those hours the field is identical under either definition
  on the other 6 it is not, and those are the hours with a backlog

the hourly volume chart, both ways
hour   by event time   by ingest time   difference
  09     41000         41000        0
  10     44000         44000        0
  11     46000         43000        -3000
  12     52000         29000        -23000
  13     58000         26000        -32000
  14     55000         58000        3000
  15     47000         71000        24000
  16     43000         70000        27000
  the chart is smooth under both definitions and it is a different chart

an alert fires when an hour exceeds 50000 events
  hours over the bar by event time  : 3
  hours over the bar by ingest time : 3
  same data, same threshold, same field name

records are deleted 90 days after occurred_at
  under event time  : 90 days after the thing happened
  under ingest time : 90 days after we received it
  for an event delayed 74 minutes the difference is negligible
  for a replayed backfill the difference is the age of the backfill
  so a re-ingest of two-year-old data resets its deletion clock, and the
  retention rule has no way to notice

the problem it solved
  events dated in the future : 3% before, 0 after
  clock skew from user devices : no longer reaches any consumer
  monotonic ordering : guaranteed, which it was not before
  all three of those are real and none of them came back

what a consumer would need to notice
  field name change : none
  type change       : none
  new field carrying the old meaning : not added
  design note : written, and read by the two people in the review
  consumers of this field : 11

carrying both stamps
  occurred_at : event time, as the name says, skew and all
  ingested_at : pipeline time, monotonic and trustworthy
  bytes added per record : 8
  bytes a day : 3088000
  consumers that would then be choosing rather than inheriting : 11
  the skew problem is solved by which field the ordering uses, not by
  redefining the one that was already named after the other thing

control - the same change shipped as a new field received_at
  consumers : 11
  consumers that had to make a choice : 11
  consumers that silently changed behaviour : 0
  the identical semantic change, and the rename is what converts a
  silent shift into a decision somebody makes

Stamping at ingest removed real clock skew and 3% future-dated events.
The two meanings agree whenever the pipeline keeps up, so the field only
disagrees with its own name on the 6 hours anybody would investigate.

Trace event types