Merge drops the differing field — a gap and a contradiction are the same shape in the code

# Self-authored for the EML case corpus (no external origin). "First non-empty
# wins" is a rule for filling gaps, and it is used to settle contradictions.
#
# Merging two records for the same entity, field by field, taking the first
# value that is not blank. It is the obvious rule and it is right about the
# case everybody has in mind: one source knows the phone number and the other
# does not, so take the one that knows.
#
# There is a second case that looks identical to the code and is not the same
# situation at all: both sources have a value and the values DISAGREE. A gap
# is missing information. A disagreement is a claim that one of the two
# systems is wrong, and it is the only part of the merge that carries news.
# "First non-empty" resolves it by argument position and says nothing.
#
# The fields where two systems disagree are not random. They are the fields
# that CHANGED - the address after a move, the phone after a switch, the
# status after a suspension - because that is what it takes for two systems to
# hold different values. So the rule discards precisely the recent facts.
#
# The measurement separates gaps from disagreements, then re-runs every merge
# with the sources swapped and counts how many merged fields change. A field
# that depends on argument order was decided by argument order.

def blank(v):
    return len(v) == 0

def merge_first_non_empty(a, b, fields):
    [] => out
    0 => i
    while i < len(fields):
        if blank(a[i]):
            out + [b[i]] => out
        else:
            out + [a[i]] => out
        i + 1 => i
    return out

def merge_flag_conflicts(a, b, fields):
    # The same rule for gaps, and a refusal for disagreements.
    [] => out
    0 => i
    while i < len(fields):
        if blank(a[i]):
            out + [b[i]] => out
        elif blank(b[i]):
            out + [a[i]] => out
        elif a[i] == b[i]:
            out + [a[i]] => out
        else:
            out + ["CONFLICT"] => out
        i + 1 => i
    return out

def classify(a, b, i):
    if blank(a[i]):
        if blank(b[i]):
            return "both-blank"
        return "gap"
    if blank(b[i]):
        return "gap"
    if a[i] == b[i]:
        return "agree"
    return "disagree"

["name", "email", "phone", "address", "status"] => FIELDS

# Pairs of records for the same entity, from two systems. crm first, billing
# second. The disagreements are the ordinary ones: someone moved, someone
# changed carrier, someone was suspended in one system and not the other.
[[["Jing Wu", "[email protected]", "5551234", "12 Oak St", "active"],
  ["Jing Wu", "[email protected]", "5559999", "48 Pine Ave", "active"]],
 [["Ana Diaz", "", "5552222", "9 Elm Rd", "active"],
  ["Ana Diaz", "[email protected]", "5552222", "9 Elm Rd", "suspended"]],
 [["Sam Roy", "[email protected]", "", "", "active"],
  ["Sam Roy", "[email protected]", "5553333", "77 Cedar Ln", "active"]],
 [["Lee Park", "[email protected]", "5554444", "3 Birch Way", "closed"],
  ["Lee Park", "[email protected]", "5554444", "3 Birch Way", "active"]]] => PAIRS

"pair  field     crm            billing        class"^0
"----- --------- -------------- -------------- --------"^0

0 => gaps
0 => disagreements
0 => agreements
0 => cells
0 => p
while p < len(PAIRS):
    PAIRS[p][0] => a
    PAIRS[p][1] => b
    0 => i
    while i < len(FIELDS):
        classify(a, b, i) => cls
        cells + 1 => cells
        if cls == "gap":
            gaps + 1 => gaps
        elif cls == "disagree":
            disagreements + 1 => disagreements
        elif cls == "agree":
            agreements + 1 => agreements
        if cls == "disagree":
            (("  " + str(p) + "     ")[0:6] + (FIELDS[i] + "          ")[0:10] + (a[i] + "               ")[0:15] + (b[i] + "               ")[0:15] + cls)^0
        i + 1 => i
    p + 1 => p

""^0
("cells compared: " + str(cells))^0
("  agree: " + str(agreements) + "   gap: " + str(gaps) + "   disagree: " + str(disagreements))^0

""^0
"what changes when the two sources are swapped"^0

# The test that separates a fill from a decision: run the merge both ways.
# A field whose merged value depends on which source was passed first was not
# merged, it was picked.
0 => order_dependent
0 => merged_cells
0 => p
while p < len(PAIRS):
    PAIRS[p][0] => a
    PAIRS[p][1] => b
    merge_first_non_empty(a, b, FIELDS) => ab
    merge_first_non_empty(b, a, FIELDS) => ba
    0 => i
    while i < len(FIELDS):
        merged_cells + 1 => merged_cells
        if not (ab[i] == ba[i]):
            order_dependent + 1 => order_dependent
            (("  pair " + str(p) + " " + FIELDS[i] + "            ")[0:20] + " crm-first: " + (ab[i] + "               ")[0:15] + " billing-first: " + ba[i])^0
        i + 1 => i
    p + 1 => p

("merged cells: " + str(merged_cells) + ", of which order-dependent: " + str(order_dependent))^0

""^0
"the conflict-flagging merge, on the same data"^0
0 => flagged
0 => p
while p < len(PAIRS):
    PAIRS[p][0] => a
    PAIRS[p][1] => b
    merge_flag_conflicts(a, b, FIELDS) => m
    0 => i
    while i < len(FIELDS):
        if m[i] == "CONFLICT":
            flagged + 1 => flagged
        i + 1 => i
    p + 1 => p
("fields raised for review: " + str(flagged))^0

# And it must agree with the silent merge everywhere there was no conflict -
# otherwise it is a different policy rather than the same policy with the
# decisions surfaced.
0 => divergent_outside_conflicts
0 => p
while p < len(PAIRS):
    PAIRS[p][0] => a
    PAIRS[p][1] => b
    merge_first_non_empty(a, b, FIELDS) => silent
    merge_flag_conflicts(a, b, FIELDS) => loud
    0 => i
    while i < len(FIELDS):
        if not (loud[i] == "CONFLICT"):
            if not (loud[i] == silent[i]):
                divergent_outside_conflicts + 1 => divergent_outside_conflicts
        i + 1 => i
    p + 1 => p
("cells where the two merges differ outside a conflict: " + str(divergent_outside_conflicts))^0

""^0
"which fields the disagreements land in"^0
0 => i
while i < len(FIELDS):
    0 => n
    0 => p
    while p < len(PAIRS):
        if classify(PAIRS[p][0], PAIRS[p][1], i) == "disagree":
            n + 1 => n
        p + 1 => p
    ((FIELDS[i] + "          ")[0:10] + " disagreements: " + str(n))^0
    i + 1 => i

""^0
0 => checked
0 => passed

# There must be both gaps and disagreements in the data, or the case cannot
# show that one rule is being asked to handle two situations.
checked + 1 => checked
if gaps > 0:
    if disagreements > 0:
        passed + 1 => passed

# The silent merge must be order-dependent, and exactly as often as there are
# disagreements - that equality is the whole claim, and it is computed on both
# sides rather than asserted.
checked + 1 => checked
if order_dependent == disagreements:
    passed + 1 => passed

# Gaps must NOT be order-dependent. Filling a gap is genuinely commutative,
# which is why the rule looks safe when it is tested on gaps.
checked + 1 => checked
0 => gap_cells_order_dependent
0 => p
while p < len(PAIRS):
    PAIRS[p][0] => a
    PAIRS[p][1] => b
    merge_first_non_empty(a, b, FIELDS) => ab
    merge_first_non_empty(b, a, FIELDS) => ba
    0 => i
    while i < len(FIELDS):
        if classify(a, b, i) == "gap":
            if not (ab[i] == ba[i]):
                gap_cells_order_dependent + 1 => gap_cells_order_dependent
        i + 1 => i
    p + 1 => p
if gap_cells_order_dependent == 0:
    passed + 1 => passed

# The flagging merge must raise exactly the disagreements - no more, no less.
checked + 1 => checked
if flagged == disagreements:
    passed + 1 => passed

# And be identical to the silent merge everywhere else, so the only change is
# that the decisions became visible.
checked + 1 => checked
if divergent_outside_conflicts == 0:
    passed + 1 => passed

# The disagreements must be spread over more than one field, so this is not
# one bad column.
checked + 1 => checked
0 => fields_with_disagreement
0 => i
while i < len(FIELDS):
    0 => n
    0 => p
    while p < len(PAIRS):
        if classify(PAIRS[p][0], PAIRS[p][1], i) == "disagree":
            n + 1 => n
        p + 1 => p
    if n > 0:
        fields_with_disagreement + 1 => fields_with_disagreement
    i + 1 => i
if fields_with_disagreement >= 3:
    passed + 1 => passed

("checks passed: " + str(passed) + "/" + str(checked))^0
if passed == checked:
    "Every contradiction was resolved by argument position and reported as a merge." => verdict
else:
    "FAILED - the merges did not behave as the checks describe." => verdict
verdict^0

""^0
"A gap and a contradiction are the same shape in the code and opposite"^0
"situations in the world. One is information the other system had; the"^0
"other is information that one of the two systems is out of date, which is"^0
"the only thing in the whole merge worth a human's attention. The rule"^0
"that handles both handles neither - it just stops the field from being"^0
"blank."^0

def blank(v):
    return len(v) == 0

def merge_first_non_empty(a, b, fields):
    out = []
    i = 0
    while i < len(fields):
        if blank(a[i]):
            out = out + [b[i]]
        else:
            out = out + [a[i]]
        i = i + 1
    return out

def merge_flag_conflicts(a, b, fields):
    out = []
    i = 0
    while i < len(fields):
        if blank(a[i]):
            out = out + [b[i]]
        elif blank(b[i]):
            out = out + [a[i]]
        elif a[i] == b[i]:
            out = out + [a[i]]
        else:
            out = out + ["CONFLICT"]
        i = i + 1
    return out

def classify(a, b, i):
    if blank(a[i]):
        if blank(b[i]):
            return "both-blank"
        return "gap"
    if blank(b[i]):
        return "gap"
    if a[i] == b[i]:
        return "agree"
    return "disagree"

FIELDS = ["name", "email", "phone", "address", "status"]
PAIRS = [[["Jing Wu", "[email protected]", "5551234", "12 Oak St", "active"], ["Jing Wu", "[email protected]", "5559999", "48 Pine Ave", "active"]], [["Ana Diaz", "", "5552222", "9 Elm Rd", "active"], ["Ana Diaz", "[email protected]", "5552222", "9 Elm Rd", "suspended"]], [["Sam Roy", "[email protected]", "", "", "active"], ["Sam Roy", "[email protected]", "5553333", "77 Cedar Ln", "active"]], [["Lee Park", "[email protected]", "5554444", "3 Birch Way", "closed"], ["Lee Park", "[email protected]", "5554444", "3 Birch Way", "active"]]]
print("pair  field     crm            billing        class")
print("----- --------- -------------- -------------- --------")
gaps = 0
disagreements = 0
agreements = 0
cells = 0
p = 0
while p < len(PAIRS):
    a = PAIRS[p][0]
    b = PAIRS[p][1]
    i = 0
    while i < len(FIELDS):
        cls = classify(a, b, i)
        cells = cells + 1
        if cls == "gap":
            gaps = gaps + 1
        elif cls == "disagree":
            disagreements = disagreements + 1
        elif cls == "agree":
            agreements = agreements + 1
        if cls == "disagree":
            print(("  " + str(p) + "     ")[0:6] + (FIELDS[i] + "          ")[0:10] + (a[i] + "               ")[0:15] + (b[i] + "               ")[0:15] + cls)
        i = i + 1
    p = p + 1
print("")
print("cells compared: " + str(cells))
print("  agree: " + str(agreements) + "   gap: " + str(gaps) + "   disagree: " + str(disagreements))
print("")
print("what changes when the two sources are swapped")
order_dependent = 0
merged_cells = 0
p = 0
while p < len(PAIRS):
    a = PAIRS[p][0]
    b = PAIRS[p][1]
    ab = merge_first_non_empty(a, b, FIELDS)
    ba = merge_first_non_empty(b, a, FIELDS)
    i = 0
    while i < len(FIELDS):
        merged_cells = merged_cells + 1
        if not ab[i] == ba[i]:
            order_dependent = order_dependent + 1
            print(("  pair " + str(p) + " " + FIELDS[i] + "            ")[0:20] + " crm-first: " + (ab[i] + "               ")[0:15] + " billing-first: " + ba[i])
        i = i + 1
    p = p + 1
print("merged cells: " + str(merged_cells) + ", of which order-dependent: " + str(order_dependent))
print("")
print("the conflict-flagging merge, on the same data")
flagged = 0
p = 0
while p < len(PAIRS):
    a = PAIRS[p][0]
    b = PAIRS[p][1]
    m = merge_flag_conflicts(a, b, FIELDS)
    i = 0
    while i < len(FIELDS):
        if m[i] == "CONFLICT":
            flagged = flagged + 1
        i = i + 1
    p = p + 1
print("fields raised for review: " + str(flagged))
divergent_outside_conflicts = 0
p = 0
while p < len(PAIRS):
    a = PAIRS[p][0]
    b = PAIRS[p][1]
    silent = merge_first_non_empty(a, b, FIELDS)
    loud = merge_flag_conflicts(a, b, FIELDS)
    i = 0
    while i < len(FIELDS):
        if not loud[i] == "CONFLICT":
            if not loud[i] == silent[i]:
                divergent_outside_conflicts = divergent_outside_conflicts + 1
        i = i + 1
    p = p + 1
print("cells where the two merges differ outside a conflict: " + str(divergent_outside_conflicts))
print("")
print("which fields the disagreements land in")
i = 0
while i < len(FIELDS):
    n = 0
    p = 0
    while p < len(PAIRS):
        if classify(PAIRS[p][0], PAIRS[p][1], i) == "disagree":
            n = n + 1
        p = p + 1
    print((FIELDS[i] + "          ")[0:10] + " disagreements: " + str(n))
    i = i + 1
print("")
checked = 0
passed = 0
checked = checked + 1
if gaps > 0:
    if disagreements > 0:
        passed = passed + 1
checked = checked + 1
if order_dependent == disagreements:
    passed = passed + 1
checked = checked + 1
gap_cells_order_dependent = 0
p = 0
while p < len(PAIRS):
    a = PAIRS[p][0]
    b = PAIRS[p][1]
    ab = merge_first_non_empty(a, b, FIELDS)
    ba = merge_first_non_empty(b, a, FIELDS)
    i = 0
    while i < len(FIELDS):
        if classify(a, b, i) == "gap":
            if not ab[i] == ba[i]:
                gap_cells_order_dependent = gap_cells_order_dependent + 1
        i = i + 1
    p = p + 1
if gap_cells_order_dependent == 0:
    passed = passed + 1
checked = checked + 1
if flagged == disagreements:
    passed = passed + 1
checked = checked + 1
if divergent_outside_conflicts == 0:
    passed = passed + 1
checked = checked + 1
fields_with_disagreement = 0
i = 0
while i < len(FIELDS):
    n = 0
    p = 0
    while p < len(PAIRS):
        if classify(PAIRS[p][0], PAIRS[p][1], i) == "disagree":
            n = n + 1
        p = p + 1
    if n > 0:
        fields_with_disagreement = fields_with_disagreement + 1
    i = i + 1
if fields_with_disagreement >= 3:
    passed = passed + 1
print("checks passed: " + str(passed) + "/" + str(checked))
if passed == checked:
    verdict = "Every contradiction was resolved by argument position and reported as a merge."
else:
    verdict = "FAILED - the merges did not behave as the checks describe."
print(verdict)
print("")
print("A gap and a contradiction are the same shape in the code and opposite")
print("situations in the world. One is information the other system had; the")
print("other is information that one of the two systems is out of date, which is")
print("the only thing in the whole merge worth a human's attention. The rule")
print("that handles both handles neither - it just stops the field from being")
print("blank.")

pair  field     crm            billing        class
----- --------- -------------- -------------- --------
  0   phone     5551234        5559999        disagree
  0   address   12 Oak St      48 Pine Ave    disagree
  1   status    active         suspended      disagree
  3   email     [email protected]@example.com disagree
  3   status    closed         active         disagree

cells compared: 20
  agree: 12   gap: 3   disagree: 5

what changes when the two sources are swapped
  pair 0 phone       crm-first: 5551234         billing-first: 5559999
  pair 0 address     crm-first: 12 Oak St       billing-first: 48 Pine Ave
  pair 1 status      crm-first: active          billing-first: suspended
  pair 3 email       crm-first: [email protected] billing-first: [email protected]
  pair 3 status      crm-first: closed          billing-first: active
merged cells: 20, of which order-dependent: 5

the conflict-flagging merge, on the same data
fields raised for review: 5
cells where the two merges differ outside a conflict: 0

which fields the disagreements land in
name       disagreements: 0
email      disagreements: 1
phone      disagreements: 1
address    disagreements: 1
status     disagreements: 2

checks passed: 6/6
Every contradiction was resolved by argument position and reported as a merge.

A gap and a contradiction are the same shape in the code and opposite
situations in the world. One is information the other system had; the
other is information that one of the two systems is out of date, which is
the only thing in the whole merge worth a human's attention. The rule
that handles both handles neither - it just stops the field from being
blank.

Trace event types