Project P144
Customer merge
Duplicate customers merged by a matcher that compares stored strings and one that normalises when it compares - merges the first job's own writes manufacture - and the orders that still name the rows merged away.
2 modules · 2 recorded sessionstext-menu UI in the terminalupdated 2026-10-18
Every screen below was recorded under CPython. When this page was built, the EML interpreter replayed each session from the same input and printed the same bytes.
About
Duplicate customers merged by two jobs, merge by merge, and the orders that still name the rows merged away.
main.eml- the menu, the rows and each job's groups, one merge at a time or until done, the orders joined three ways, and starting overdedupe.eml- the customers and orders, how two records match, the next match, a merge, whether original rows matched, and the orders a join still finds
How each part works:
- Eight rows, typed by different people: Ana's phone as
555-0101and5550101, Bo's as555 0202and5550202, Cy's as555-0303and5550303; some rows have only an email. - A merge keeps the newer row and writes the phone in its canonical form, digits only. The raw job's matcher compares the stored strings; the compare-time job normalises the phone when it compares.
- As in the corpus case
merge-invents-a-match, the raw job finds Ana's rows 1 and 3 by email, and the merge writes5550101- which now equals row 2 as typed. The next merge rests on that value: no original row of one side matches one of the other. Bo's rows go the same way. Cy has no email to start the chain, so the raw job never joins rows 7 and 8: 4 merges, 2 of them manufactured. The compare-time job makes 5, none manufactured. - As in the corpus case
survivor-id-still-referenced, the orders keep naming the row merged away. With the merged rows hard-deleted, a join finds 6 of the 12 orders after the raw job and 5 after the other. A redirect from each merged row to its survivor finds them all only when it is followed to the end: row 4 was merged into 5, and 5 into 6, so one redirect lands on a row that is gone.
What is checked: menu choices 1 to 6.
Sessions: sessions/basic.in gives: - the rows and groups; one merge each and the orders; - three merges more and the orders, with one redirect falling short; - merging until done, the orders, and the groups at the end.
sessions/bad-input.in gives: - menu choices 0, x and 7; - merging until done, then a next merge with nothing left to match; the orders; starting over, and the orders with nothing merged.
Built on the verified corpus cases merge-invents-a-match (a merge that writes a canonical value, and the match it manufactures for a matcher that compares stored strings) and survivor-id-still-referenced (the loser's id still named by the orders, under a hard delete and a redirect).
Recorded sessions
What the screen shows while someone uses the program. Each typed line appears after its prompt, the way a terminal shows it.
bad-input
interpreter: byte-equal== Customer merge ==
Duplicate customers merged by two matchers, and the orders that still name the rows merged away.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 0
Pick a number from 1 to 6.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> x
Pick a number from 1 to 6.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 7
Pick a number from 1 to 6.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 3
Raw matcher: merged 1 into 3 - email ana@x; an original row of each side matches: real.
Compare-time matcher: merged 1 into 2 - phone '555-0101' and '5550101', both 5550101; an original row of each side matches: real.
Raw matcher: merged 2 into 3 - phone '5550101' in both; no original row of one side matches one of the other: manufactured by a value a merge wrote.
Compare-time matcher: merged 2 into 3 - email ana@x; an original row of each side matches: real.
Raw matcher: merged 4 into 5 - email bo@x; an original row of each side matches: real.
Compare-time matcher: merged 4 into 5 - email bo@x; an original row of each side matches: real.
Raw matcher: merged 5 into 6 - phone '5550202' in both; no original row of one side matches one of the other: manufactured by a value a merge wrote.
Compare-time matcher: merged 5 into 6 - phone '5550202' in both; an original row of each side matches: real.
Compare-time matcher: merged 7 into 8 - phone '555-0303' and '5550303', both 5550303; an original row of each side matches: real.
Both jobs are done.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 2
Raw matcher: no two records match any more.
Compare-time matcher: no two records match any more.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 4
Orders, 12 in all, joined to their customer after the merges so far:
raw compare-time
counted without a join 12 12
joined, merged rows hard-deleted 6 5
joined, following one redirect 11 9
joined, following every redirect 12 12
Hard-deleted: an order still names the row merged away, and the join drops it without an error.
One redirect: an order whose row was merged into one that was merged again lands on a row that is gone.
Raw matcher: 4 merges so far, 2 of them manufactured.
Compare-time matcher: 5 merges so far, 0 of them manufactured.
Both jobs are done. Rows the compare-time matcher put together and the raw job never did: 7 and 8.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 5
Back to the rows as typed, nothing merged.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 4
Orders, 12 in all, joined to their customer after the merges so far:
raw compare-time
counted without a join 12 12
joined, merged rows hard-deleted 12 12
joined, following one redirect 12 12
joined, following every redirect 12 12
Raw matcher: 0 merges so far, 0 of them manufactured.
Compare-time matcher: 0 merges so far, 0 of them manufactured.
Merge until done to see where the two jobs end.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 6
Bye.
What was typed (9 lines)
0
x
7
3
2
4
5
4
6
basic
interpreter: byte-equal== Customer merge ==
Duplicate customers merged by two matchers, and the orders that still name the rows merged away.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 1
The rows, as typed:
1 Ana Silva '555-0101' 'ana@x'
2 Ana Silva '5550101' ''
3 Ana S. '' 'ana@x'
4 Bo Chen '555 0202' 'bo@x'
5 Bo Chen '' 'bo@x'
6 B. Chen '5550202' ''
7 Cy Diaz '555-0303' 'cy@x'
8 Cy Diaz '5550303' ''
Raw matcher (compares the stored strings), groups:
1: rows 1 phone '555-0101' email 'ana@x'
2: rows 2 phone '5550101' email ''
3: rows 3 phone '' email 'ana@x'
4: rows 4 phone '555 0202' email 'bo@x'
5: rows 5 phone '' email 'bo@x'
6: rows 6 phone '5550202' email ''
7: rows 7 phone '555-0303' email 'cy@x'
8: rows 8 phone '5550303' email ''
Compare-time matcher (normalises before comparing), groups:
1: rows 1 phone '555-0101' email 'ana@x'
2: rows 2 phone '5550101' email ''
3: rows 3 phone '' email 'ana@x'
4: rows 4 phone '555 0202' email 'bo@x'
5: rows 5 phone '' email 'bo@x'
6: rows 6 phone '5550202' email ''
7: rows 7 phone '555-0303' email 'cy@x'
8: rows 8 phone '5550303' email ''
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 2
Raw matcher: merged 1 into 3 - email ana@x; an original row of each side matches: real.
Compare-time matcher: merged 1 into 2 - phone '555-0101' and '5550101', both 5550101; an original row of each side matches: real.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 4
Orders, 12 in all, joined to their customer after the merges so far:
raw compare-time
counted without a join 12 12
joined, merged rows hard-deleted 10 10
joined, following one redirect 12 12
joined, following every redirect 12 12
Hard-deleted: an order still names the row merged away, and the join drops it without an error.
Raw matcher: 1 merge so far, 0 of them manufactured.
Compare-time matcher: 1 merge so far, 0 of them manufactured.
Merge until done to see where the two jobs end.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 2
Raw matcher: merged 2 into 3 - phone '5550101' in both; no original row of one side matches one of the other: manufactured by a value a merge wrote.
Compare-time matcher: merged 2 into 3 - email ana@x; an original row of each side matches: real.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 2
Raw matcher: merged 4 into 5 - email bo@x; an original row of each side matches: real.
Compare-time matcher: merged 4 into 5 - email bo@x; an original row of each side matches: real.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 2
Raw matcher: merged 5 into 6 - phone '5550202' in both; no original row of one side matches one of the other: manufactured by a value a merge wrote.
Compare-time matcher: merged 5 into 6 - phone '5550202' in both; an original row of each side matches: real.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 4
Orders, 12 in all, joined to their customer after the merges so far:
raw compare-time
counted without a join 12 12
joined, merged rows hard-deleted 6 6
joined, following one redirect 11 9
joined, following every redirect 12 12
Hard-deleted: an order still names the row merged away, and the join drops it without an error.
One redirect: an order whose row was merged into one that was merged again lands on a row that is gone.
Raw matcher: 4 merges so far, 2 of them manufactured.
Compare-time matcher: 4 merges so far, 0 of them manufactured.
Merge until done to see where the two jobs end.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 3
Compare-time matcher: merged 7 into 8 - phone '555-0303' and '5550303', both 5550303; an original row of each side matches: real.
Both jobs are done.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 4
Orders, 12 in all, joined to their customer after the merges so far:
raw compare-time
counted without a join 12 12
joined, merged rows hard-deleted 6 5
joined, following one redirect 11 9
joined, following every redirect 12 12
Hard-deleted: an order still names the row merged away, and the join drops it without an error.
One redirect: an order whose row was merged into one that was merged again lands on a row that is gone.
Raw matcher: 4 merges so far, 2 of them manufactured.
Compare-time matcher: 5 merges so far, 0 of them manufactured.
Both jobs are done. Rows the compare-time matcher put together and the raw job never did: 7 and 8.
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 1
The rows, as typed:
1 Ana Silva '555-0101' 'ana@x'
2 Ana Silva '5550101' ''
3 Ana S. '' 'ana@x'
4 Bo Chen '555 0202' 'bo@x'
5 Bo Chen '' 'bo@x'
6 B. Chen '5550202' ''
7 Cy Diaz '555-0303' 'cy@x'
8 Cy Diaz '5550303' ''
Raw matcher (compares the stored strings), groups:
3: rows 1, 3, 2 phone '5550101' email 'ana@x'
6: rows 4, 5, 6 phone '5550202' email 'bo@x'
7: rows 7 phone '555-0303' email 'cy@x'
8: rows 8 phone '5550303' email ''
Compare-time matcher (normalises before comparing), groups:
3: rows 1, 2, 3 phone '5550101' email 'ana@x'
6: rows 4, 5, 6 phone '5550202' email 'bo@x'
8: rows 7, 8 phone '5550303' email 'cy@x'
1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit
choice> 6
Bye.
What was typed (11 lines)
1
2
4
2
2
2
4
3
4
1
6
Modules
The program as written, entry module first. Each module transpiles to its own Python file, which is what eml project run executes.
main.eml(entry)
eml# P144 customer merge: the same duplicate customers merged by two jobs - a
# matcher that compares the stored strings and one that normalises when it
# compares - merge by merge, and the orders joined to their customers after
# a hard delete, one redirect, or every redirect.
import dedupe
def plain(s):
0 => i
len(s) => j
while i < j and s[i] == " ":
i + 1 => i
while j > i and s[j - 1] == " ":
j - 1 => j
return s[i:j]
def pad(s, n):
s => out
while len(out) < n:
out + " " => out
return out
def lpad(s, n):
s => out
while len(out) < n:
" " + out => out
return out
def q(s):
return "'" + s + "'"
def ids(xs):
"" => out
for x in xs:
if out == "":
str(x) => out
else:
out + ", " + str(x) => out
return out
def times(n, one, many):
if n == 1:
return "1 " + one
return str(n) + " " + many
def label(normalise):
if normalise:
return "Compare-time matcher"
return "Raw matcher"
def show_groups(normalise, gs):
if normalise:
(label(normalise) + " (normalises before comparing), groups:") ^0
else:
(label(normalise) + " (compares the stored strings), groups:") ^0
for g in gs:
(" " + pad(str(g[0]) + ": rows " + ids(g[1]), 20) + pad("phone " + q(g[2]), 18) + "email " + q(g[3])) ^0
def show(r, n):
"The rows, as typed:" ^0
for c in dedupe.customers():
(" " + str(c[0]) + " " + pad(c[1], 11) + pad(q(c[2]), 12) + q(c[3])) ^0
show_groups(False, r[0])
show_groups(True, n[0])
def step(normalise, w):
# one merge in world w = [groups, redirects, merges, manufactured];
# returns [the world, True when there was a match]
w[0] => gs
dedupe.next_pair(normalise, gs) => p
if len(p) == 0:
(label(normalise) + ": no two records match any more.") ^0
return [w, False]
p[0] => i
p[1] => j
dedupe.evidence(normalise, gs[i], gs[j]) => ev
if ev == "email":
"email " + gs[i][3] => said
elif normalise and gs[i][2] != gs[j][2]:
"phone " + q(gs[i][2]) + " and " + q(gs[j][2]) + ", both " + dedupe.digits(gs[i][2]) => said
else:
"phone " + q(gs[i][2]) + " in both" => said
dedupe.loser_of(gs, i, j) => lo
dedupe.survivor_of(gs, i, j) => su
w[3] => made
if dedupe.rows_match(normalise, gs[i][1], gs[j][1]):
"an original row of each side matches: real." => tail
else:
"no original row of one side matches one of the other: manufactured by a value a merge wrote." => tail
made + 1 => made
(label(normalise) + ": merged " + str(lo) + " into " + str(su) + " - " + said + "; " + tail) ^0
{} => red
for k in w[1]:
w[1][k] => red[k]
su => red[lo]
return [[dedupe.merge(gs, i, j), red, w[2] + 1, made], True]
def fresh():
return [dedupe.start(), {}, 0, 0]
def done(normalise, w):
return len(dedupe.next_pair(normalise, w[0])) == 0
def group_of(gs, id):
for g in gs:
for x in g[1]:
if x == id:
return g[0]
return 0
def reports(r, n):
len(dedupe.orders()) => total
("Orders, " + str(total) + " in all, joined to their customer after the merges so far:") ^0
" raw compare-time" ^0
(" " + pad("counted without a join", 38) + lpad(str(total), 5) + lpad(str(total), 15)) ^0
(" " + pad("joined, merged rows hard-deleted", 38) + lpad(str(dedupe.joined(r[0], r[1], 0)), 5) + lpad(str(dedupe.joined(n[0], n[1], 0)), 15)) ^0
(" " + pad("joined, following one redirect", 38) + lpad(str(dedupe.joined(r[0], r[1], 1)), 5) + lpad(str(dedupe.joined(n[0], n[1], 1)), 15)) ^0
(" " + pad("joined, following every redirect", 38) + lpad(str(dedupe.joined(r[0], r[1], 9)), 5) + lpad(str(dedupe.joined(n[0], n[1], 9)), 15)) ^0
if dedupe.joined(r[0], r[1], 0) < total or dedupe.joined(n[0], n[1], 0) < total:
"Hard-deleted: an order still names the row merged away, and the join drops it without an error." ^0
if dedupe.joined(r[0], r[1], 1) < total or dedupe.joined(n[0], n[1], 1) < total:
"One redirect: an order whose row was merged into one that was merged again lands on a row that is gone." ^0
("Raw matcher: " + times(r[2], "merge", "merges") + " so far, " + str(r[3]) + " of them manufactured.") ^0
("Compare-time matcher: " + times(n[2], "merge", "merges") + " so far, " + str(n[3]) + " of them manufactured.") ^0
if done(False, r) and done(True, n):
"" => apart
for a in dedupe.customers():
for b in dedupe.customers():
if a[0] < b[0] and group_of(n[0], a[0]) == group_of(n[0], b[0]) and group_of(r[0], a[0]) != group_of(r[0], b[0]):
if apart == "":
str(a[0]) + " and " + str(b[0]) => apart
else:
apart + "; " + str(a[0]) + " and " + str(b[0]) => apart
if apart == "":
"Both jobs are done and ended with the same groups." ^0
else:
("Both jobs are done. Rows the compare-time matcher put together and the raw job never did: " + apart + ".") ^0
else:
"Merge until done to see where the two jobs end." ^0
fresh() => r
fresh() => n
"== Customer merge ==" ^0
"Duplicate customers merged by two matchers, and the orders that still name the rows merged away." ^0
True => running
while running:
"" ^0
"1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit" ^0
plain(input("choice> ")) => choice
if choice == "1":
show(r, n)
elif choice == "2":
step(False, r)[0] => r
step(True, n)[0] => n
elif choice == "3":
True => more
while more:
False => more
if not done(False, r):
step(False, r)[0] => r
True => more
if not done(True, n):
step(True, n)[0] => n
True => more
"Both jobs are done." ^0
elif choice == "4":
reports(r, n)
elif choice == "5":
fresh() => r
fresh() => n
"Back to the rows as typed, nothing merged." ^0
elif choice == "6":
False => running
else:
"Pick a number from 1 to 6." ^0
"Bye." ^0
Python projection (main.py)
import dedupe
def plain(s):
i = 0
j = len(s)
while i < j and s[i] == " ":
i = i + 1
while j > i and s[j - 1] == " ":
j = j - 1
return s[i:j]
def pad(s, n):
out = s
while len(out) < n:
out = out + " "
return out
def lpad(s, n):
out = s
while len(out) < n:
out = " " + out
return out
def q(s):
return "'" + s + "'"
def ids(xs):
out = ""
for x in xs:
if out == "":
out = str(x)
else:
out = out + ", " + str(x)
return out
def times(n, one, many):
if n == 1:
return "1 " + one
return str(n) + " " + many
def label(normalise):
if normalise:
return "Compare-time matcher"
return "Raw matcher"
def show_groups(normalise, gs):
if normalise:
print(label(normalise) + " (normalises before comparing), groups:")
else:
print(label(normalise) + " (compares the stored strings), groups:")
for g in gs:
print(" " + pad(str(g[0]) + ": rows " + ids(g[1]), 20) + pad("phone " + q(g[2]), 18) + "email " + q(g[3]))
def show(r, n):
print("The rows, as typed:")
for c in dedupe.customers():
print(" " + str(c[0]) + " " + pad(c[1], 11) + pad(q(c[2]), 12) + q(c[3]))
show_groups(False, r[0])
show_groups(True, n[0])
def step(normalise, w):
gs = w[0]
p = dedupe.next_pair(normalise, gs)
if len(p) == 0:
print(label(normalise) + ": no two records match any more.")
return [w, False]
i = p[0]
j = p[1]
ev = dedupe.evidence(normalise, gs[i], gs[j])
if ev == "email":
said = "email " + gs[i][3]
elif normalise and gs[i][2] != gs[j][2]:
said = "phone " + q(gs[i][2]) + " and " + q(gs[j][2]) + ", both " + dedupe.digits(gs[i][2])
else:
said = "phone " + q(gs[i][2]) + " in both"
lo = dedupe.loser_of(gs, i, j)
su = dedupe.survivor_of(gs, i, j)
made = w[3]
if dedupe.rows_match(normalise, gs[i][1], gs[j][1]):
tail = "an original row of each side matches: real."
else:
tail = "no original row of one side matches one of the other: manufactured by a value a merge wrote."
made = made + 1
print(label(normalise) + ": merged " + str(lo) + " into " + str(su) + " - " + said + "; " + tail)
red = {}
for k in w[1]:
red[k] = w[1][k]
red[lo] = su
return [[dedupe.merge(gs, i, j), red, w[2] + 1, made], True]
def fresh():
return [dedupe.start(), {}, 0, 0]
def done(normalise, w):
return len(dedupe.next_pair(normalise, w[0])) == 0
def group_of(gs, id):
for g in gs:
for x in g[1]:
if x == id:
return g[0]
return 0
def reports(r, n):
total = len(dedupe.orders())
print("Orders, " + str(total) + " in all, joined to their customer after the merges so far:")
print(" raw compare-time")
print(" " + pad("counted without a join", 38) + lpad(str(total), 5) + lpad(str(total), 15))
print(" " + pad("joined, merged rows hard-deleted", 38) + lpad(str(dedupe.joined(r[0], r[1], 0)), 5) + lpad(str(dedupe.joined(n[0], n[1], 0)), 15))
print(" " + pad("joined, following one redirect", 38) + lpad(str(dedupe.joined(r[0], r[1], 1)), 5) + lpad(str(dedupe.joined(n[0], n[1], 1)), 15))
print(" " + pad("joined, following every redirect", 38) + lpad(str(dedupe.joined(r[0], r[1], 9)), 5) + lpad(str(dedupe.joined(n[0], n[1], 9)), 15))
if dedupe.joined(r[0], r[1], 0) < total or dedupe.joined(n[0], n[1], 0) < total:
print("Hard-deleted: an order still names the row merged away, and the join drops it without an error.")
if dedupe.joined(r[0], r[1], 1) < total or dedupe.joined(n[0], n[1], 1) < total:
print("One redirect: an order whose row was merged into one that was merged again lands on a row that is gone.")
print("Raw matcher: " + times(r[2], "merge", "merges") + " so far, " + str(r[3]) + " of them manufactured.")
print("Compare-time matcher: " + times(n[2], "merge", "merges") + " so far, " + str(n[3]) + " of them manufactured.")
if done(False, r) and done(True, n):
apart = ""
for a in dedupe.customers():
for b in dedupe.customers():
if a[0] < b[0] and group_of(n[0], a[0]) == group_of(n[0], b[0]) and group_of(r[0], a[0]) != group_of(r[0], b[0]):
if apart == "":
apart = str(a[0]) + " and " + str(b[0])
else:
apart = apart + "; " + str(a[0]) + " and " + str(b[0])
if apart == "":
print("Both jobs are done and ended with the same groups.")
else:
print("Both jobs are done. Rows the compare-time matcher put together and the raw job never did: " + apart + ".")
else:
print("Merge until done to see where the two jobs end.")
r = fresh()
n = fresh()
print("== Customer merge ==")
print("Duplicate customers merged by two matchers, and the orders that still name the rows merged away.")
running = True
while running:
print("")
print("1) rows and groups 2) next merge 3) merge until done 4) orders 5) start over 6) quit")
choice = plain(input("choice> "))
if choice == "1":
show(r, n)
elif choice == "2":
r = step(False, r)[0]
n = step(True, n)[0]
elif choice == "3":
more = True
while more:
more = False
if not done(False, r):
r = step(False, r)[0]
more = True
if not done(True, n):
n = step(True, n)[0]
more = True
print("Both jobs are done.")
elif choice == "4":
reports(r, n)
elif choice == "5":
r = fresh()
n = fresh()
print("Back to the rows as typed, nothing merged.")
elif choice == "6":
running = False
else:
print("Pick a number from 1 to 6.")
print("Bye.")
dedupe.eml
eml# P144 customer merge - duplicate customers found by a matcher and merged,
# and the orders that still name the rows merged away. A group is
# [survivor, [member rows], stored phone, stored email]: the record the
# matcher compares. The newer row survives, and a merge writes the phone in
# its canonical form, digits only. As in the corpus case
# merge-invents-a-match, a matcher that compares the stored strings can then
# find a match no two original rows have: the merge made the evidence. The
# matcher that normalises when it compares cannot. As in the corpus case
# survivor-id-still-referenced, the orders keep naming the loser: a hard
# delete drops them from every join, a redirect keeps them only if it is
# followed to the end.
def customers():
# [id, name, phone as typed, email]
return [[1, "Ana Silva", "555-0101", "ana@x"], [2, "Ana Silva", "5550101", ""], [3, "Ana S.", "", "ana@x"], [4, "Bo Chen", "555 0202", "bo@x"], [5, "Bo Chen", "", "bo@x"], [6, "B. Chen", "5550202", ""], [7, "Cy Diaz", "555-0303", "cy@x"], [8, "Cy Diaz", "5550303", ""]]
def orders():
# [order number, customer id]
return [[101, 1], [102, 2], [103, 3], [104, 3], [105, 4], [106, 5], [107, 6], [108, 6], [109, 7], [110, 8], [111, 1], [112, 5]]
def row(id):
for c in customers():
if c[0] == id:
return c
return []
def digits(s):
"" => out
for c in s:
if c >= "0" and c <= "9":
out + c => out
return out
def start():
[] => gs
for c in customers():
gs + [[c[0], [c[0]], c[2], c[3]]] => gs
return gs
def evidence(normalise, a, b):
# how two stored records match: "phone", "email" or ""
if normalise:
if digits(a[2]) != "" and digits(a[2]) == digits(b[2]):
return "phone"
elif a[2] != "" and a[2] == b[2]:
return "phone"
if a[3] != "" and a[3] == b[3]:
return "email"
return ""
def rows_match(normalise, ids1, ids2):
# does an original row of one side match an original row of the other
for x in ids1:
for y in ids2:
row(x) => a
row(y) => b
if evidence(normalise, [a[0], [], a[2], a[3]], [b[0], [], b[2], b[3]]) != "":
return True
return False
def next_pair(normalise, gs):
# positions [i, j] of the first two groups that match, or []
0 => i
while i < len(gs):
i + 1 => j
while j < len(gs):
if evidence(normalise, gs[i], gs[j]) != "":
return [i, j]
j + 1 => j
i + 1 => i
return []
def merge(gs, i, j):
# the groups with i and j made one; the newer survivor stays and the
# merge writes the phone digits only
gs[i] => a
gs[j] => b
if a[0] > b[0]:
a => keep
b => lose
else:
b => keep
a => lose
keep[2] => phone
if phone == "":
lose[2] => phone
keep[3] => email
if email == "":
lose[3] => email
[keep[0], a[1] + b[1], digits(phone), email] => g
[] => out
0 => k
for x in gs:
if k == i:
out + [g] => out
elif k != j:
out + [x] => out
k + 1 => k
return out
def loser_of(gs, i, j):
if gs[i][0] > gs[j][0]:
return gs[j][0]
return gs[i][0]
def survivor_of(gs, i, j):
if gs[i][0] > gs[j][0]:
return gs[i][0]
return gs[j][0]
def is_survivor(gs, id):
for g in gs:
if g[0] == id:
return True
return False
def joined(gs, redirect, hops):
# orders whose customer is found: hops 0 is a hard delete, 1 follows one
# redirect, more follows them that many times
0 => n
for o in orders():
o[1] => id
0 => h
while not is_survivor(gs, id) and h < hops and id in redirect:
redirect[id] => id
h + 1 => h
if is_survivor(gs, id):
n + 1 => n
return n
Python projection (dedupe.py)
def customers():
return [[1, "Ana Silva", "555-0101", "ana@x"], [2, "Ana Silva", "5550101", ""], [3, "Ana S.", "", "ana@x"], [4, "Bo Chen", "555 0202", "bo@x"], [5, "Bo Chen", "", "bo@x"], [6, "B. Chen", "5550202", ""], [7, "Cy Diaz", "555-0303", "cy@x"], [8, "Cy Diaz", "5550303", ""]]
def orders():
return [[101, 1], [102, 2], [103, 3], [104, 3], [105, 4], [106, 5], [107, 6], [108, 6], [109, 7], [110, 8], [111, 1], [112, 5]]
def row(id):
for c in customers():
if c[0] == id:
return c
return []
def digits(s):
out = ""
for c in s:
if c >= "0" and c <= "9":
out = out + c
return out
def start():
gs = []
for c in customers():
gs = gs + [[c[0], [c[0]], c[2], c[3]]]
return gs
def evidence(normalise, a, b):
if normalise:
if digits(a[2]) != "" and digits(a[2]) == digits(b[2]):
return "phone"
elif a[2] != "" and a[2] == b[2]:
return "phone"
if a[3] != "" and a[3] == b[3]:
return "email"
return ""
def rows_match(normalise, ids1, ids2):
for x in ids1:
for y in ids2:
a = row(x)
b = row(y)
if evidence(normalise, [a[0], [], a[2], a[3]], [b[0], [], b[2], b[3]]) != "":
return True
return False
def next_pair(normalise, gs):
i = 0
while i < len(gs):
j = i + 1
while j < len(gs):
if evidence(normalise, gs[i], gs[j]) != "":
return [i, j]
j = j + 1
i = i + 1
return []
def merge(gs, i, j):
a = gs[i]
b = gs[j]
if a[0] > b[0]:
keep = a
lose = b
else:
keep = b
lose = a
phone = keep[2]
if phone == "":
phone = lose[2]
email = keep[3]
if email == "":
email = lose[3]
g = [keep[0], a[1] + b[1], digits(phone), email]
out = []
k = 0
for x in gs:
if k == i:
out = out + [g]
elif k != j:
out = out + [x]
k = k + 1
return out
def loser_of(gs, i, j):
if gs[i][0] > gs[j][0]:
return gs[j][0]
return gs[i][0]
def survivor_of(gs, i, j):
if gs[i][0] > gs[j][0]:
return gs[i][0]
return gs[j][0]
def is_survivor(gs, id):
for g in gs:
if g[0] == id:
return True
return False
def joined(gs, redirect, hops):
n = 0
for o in orders():
id = o[1]
h = 0
while not is_survivor(gs, id) and h < hops and id in redirect:
id = redirect[id]
h = h + 1
if is_survivor(gs, id):
n = n + 1
return n