{
  "id": "P152",
  "slug": "import-pipeline",
  "key": "P152-import-pipeline",
  "title": "Import pipeline",
  "summary": "Order rows through parse, enrich, dedupe and window, the books closed only with a ledger of every drop, and a listing whose count lets a viewer read the amounts it cannot see.",
  "entry": "main.eml",
  "ui": "terminal",
  "readme": "# P152 - Import pipeline\n\nOrder rows through four import stages, the books closed with a ledger and\nwithout, and a listing of what came out that counts what its viewer cannot\nsee.\n\n- `main.eml` - the menu, the rows coming in, the import as it reports\n  itself, the import with a ledger, the listing as one team sees it, and\n  narrowing its filter\n- `pipeline.eml` - the rows, the account lookup, the four stages with the\n  reason each drop is recorded under, which stage reports itself, the\n  listing's query and what a viewer is shown\n\nHow each part works:\n\n- As in the corpus case `silent-drop-accounting`, each stage drops rows\n  for a reason that is right on its own: parse rejects a bad amount and\n  says so; enrich drops a row whose account it cannot look up; dedupe drops\n  repeats; window drops rows outside 09:00 to 09:10. As written, only parse\n  reports: 12 in, 6 out, 1 explained, and 5 rows gone with nothing to say\n  where.\n- With a ledger every drop is recorded with its stage and reason: 12 in - 6\n  out - 6 explained = 0. The rows out are the same either way; what changes\n  is what can be said about the rest.\n- As in the corpus case `count-leaks-hidden-items`, the listing counts in\n  its query and filters by team where it shows the rows, so the header\n  counts rows the viewer cannot see: sales is told 6 results and shown 3.\n- Narrowing \"amount at least\" one step at a time, the header minus the\n  rows shown drops each time a hidden row falls out: without reading one,\n  sales learns the other team's amounts, 2, 3 and 8. Counted after the\n  visibility filter the header would equal the rows shown and tell\n  nothing.\n\nWhat is checked: menu choices 1 to 6; the viewer sales or finance; an\namount from 0 to 99. An empty answer cancels.\n\nSessions: `sessions/basic.in` gives:\n- the rows in, the import and the import with a ledger;\n- the listing for sales from 0, for finance from 5 and for sales from 11,\n  where nothing hidden matches; sales narrowing the filter.\n\n`sessions/bad-input.in` gives:\n- menu choices 0 and x;\n- the listing: a cancel and hr for the viewer; then sales with a cancel,\n  100 and x for the amount;\n- narrowing: a cancel and marketing, then finance.\n\nBuilt on the verified corpus cases `silent-drop-accounting` (stages that\ndrop rows unreported, and a ledger that makes in - out - explained close)\nand `count-leaks-hidden-items` (a count taken before the visibility filter,\nand the filter that narrows it into an answer).\n",
  "modules": [
    {
      "name": "main.eml",
      "eml": "# P152 import pipeline: the rows coming in, the import as it reports itself,\n# the import with a ledger, the listing of what came out as one team sees\n# it, and narrowing its filter.\nimport pipeline\n\ndef plain(s):\n    0 => i\n    len(s) => j\n    while i < j and s[i] == \" \":\n        i + 1 => i\n    while j > i and s[j - 1] == \" \":\n        j - 1 => j\n    return s[i:j]\n\ndef pad(s, n):\n    s => out\n    while len(out) < n:\n        out + \" \" => out\n    return out\n\ndef lpad(s, n):\n    s => out\n    while len(out) < n:\n        \" \" + out => out\n    return out\n\ndef numbers(rs):\n    \"\" => out\n    for r in rs:\n        if out == \"\":\n            str(r[0]) => out\n        else:\n            out + \", \" + str(r[0]) => out\n    if out == \"\":\n        return \"none\"\n    return out\n\ndef show_rows():\n    \"The rows coming in:\" ^0\n    for r in pipeline.rows():\n        (\"  \" + lpad(str(r[0]), 2) + \"  \" + r[1] + \"  \" + pad(r[2], 4) + pad(r[3], 11) + r[4]) ^0\n\ndef plain_run():\n    pipeline.run(pipeline.rows()) => x\n    x[0] => out\n    0 => told\n    for d in x[1]:\n        if pipeline.said(d[0]):\n            (\"  parse rejected row \" + str(d[1]) + \": \" + d[2] + \".\") ^0\n            told + 1 => told\n    (\"Imported \" + str(len(out)) + \" rows: \" + numbers(out) + \".\") ^0\n    len(pipeline.rows()) => n\n    (str(n) + \" in, \" + str(len(out)) + \" out, \" + str(told) + \" explained: \" + str(n - len(out) - told) + \" rows are gone and nothing says where.\") ^0\n\ndef ledger_run():\n    pipeline.run(pipeline.rows()) => x\n    x[0] => out\n    x[1] => led\n    len(pipeline.rows()) => left\n    for st in [\"parse\", \"enrich\", \"dedupe\", \"window\"]:\n        [] => mine\n        for d in led:\n            if d[0] == st:\n                mine + [d] => mine\n        if pipeline.said(st):\n            \"reported\" => how\n        else:\n            \"silent as written\" => how\n        (\"  \" + pad(st, 7) + str(left) + \" in, \" + str(left - len(mine)) + \" out (\" + how + \")\") ^0\n        for d in mine:\n            (\"      row \" + str(d[1]) + \": \" + d[2]) ^0\n        left - len(mine) => left\n    len(pipeline.rows()) => n\n    (str(n) + \" in - \" + str(len(out)) + \" out - \" + str(len(led)) + \" explained = \" + str(n - len(out) - len(led)) + \": every row is accounted for.\") ^0\n    \"The rows out are the same as without the ledger; what changes is what can be said about the rest.\" ^0\n\ndef read_viewer():\n    plain(input(\"viewer (sales or finance)> \")) => v\n    return v\n\ndef listing():\n    read_viewer() => v\n    if v == \"\":\n        \"Cancelled.\" ^0\n        return\n    if v != \"sales\" and v != \"finance\":\n        \"Type sales or finance.\" ^0\n        return\n    plain(input(\"amount at least (0 to 99)> \")) => t\n    if t == \"\":\n        \"Cancelled.\" ^0\n        return\n    if len(t) > 2 or not pipeline.is_number(t):\n        \"Type a whole number from 0 to 99.\" ^0\n        return\n    int(t) => least\n    pipeline.run(pipeline.rows())[0] => out\n    pipeline.query(out, least) => m\n    pipeline.visible(m, v) => shown\n    (\"Header, counted by the query: \" + str(len(m)) + \" results.\") ^0\n    for r in shown:\n        (\"  \" + lpad(str(r[0]), 2) + \"  \" + pad(r[1], 4) + pad(r[2], 11) + lpad(str(r[3]), 3)) ^0\n    if len(shown) == 0:\n        \"  (no rows shown)\" ^0\n    if len(m) > len(shown):\n        (str(len(m) - len(shown)) + \" of the \" + str(len(m)) + \" are not for \" + v + \" to see, and the header counted them. Counted after the filter it says \" + str(len(shown)) + \".\") ^0\n    else:\n        \"The header and the rows agree here: nothing hidden matches.\" ^0\n\ndef narrow():\n    read_viewer() => v\n    if v == \"\":\n        \"Cancelled.\" ^0\n        return\n    if v != \"sales\" and v != \"finance\":\n        \"Type sales or finance.\" ^0\n        return\n    pipeline.run(pipeline.rows())[0] => out\n    0 => top\n    for r in out:\n        if r[3] > top:\n            r[3] => top\n    (\"As \" + v + \", amount at least 0, 1, 2 and up; the header minus the rows shown is what is hidden:\") ^0\n    (\"  \" + pad(\"at least\", 10) + \"header  shown  hidden\") ^0\n    [] => learned\n    0 => before\n    0 => n\n    while n <= top + 1:\n        pipeline.query(out, n) => m\n        pipeline.visible(m, v) => shown\n        len(m) - len(shown) => hidden\n        if n == 0 or hidden != before:\n            (\"  \" + pad(str(n), 10) + lpad(str(len(m)), 6) + lpad(str(len(shown)), 7) + lpad(str(hidden), 8)) ^0\n        if n > 0 and hidden < before:\n            0 => k\n            while k < before - hidden:\n                learned + [n - 1] => learned\n                k + 1 => k\n        hidden => before\n        n + 1 => n\n    \"\" => said\n    for a in learned:\n        if said == \"\":\n            str(a) => said\n        else:\n            said + \", \" + str(a) => said\n    (\"Without reading one, \" + v + \" learns the amounts of the rows it cannot see: \" + said + \".\") ^0\n    \"Counted after the visibility filter, the header would equal the rows shown at every step, and tell nothing.\" ^0\n\n\"== Import pipeline ==\" ^0\n\"Order rows through parse, enrich, dedupe and window, and a listing of what came out.\" ^0\nTrue => running\nwhile running:\n    \"\" ^0\n    \"1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\" ^0\n    plain(input(\"choice> \")) => choice\n    if choice == \"1\":\n        show_rows()\n    elif choice == \"2\":\n        plain_run()\n    elif choice == \"3\":\n        ledger_run()\n    elif choice == \"4\":\n        listing()\n    elif choice == \"5\":\n        narrow()\n    elif choice == \"6\":\n        False => running\n    else:\n        \"Pick a number from 1 to 6.\" ^0\n\"Bye.\" ^0\n",
      "python": "import pipeline\n\ndef plain(s):\n    i = 0\n    j = len(s)\n    while i < j and s[i] == \" \":\n        i = i + 1\n    while j > i and s[j - 1] == \" \":\n        j = j - 1\n    return s[i:j]\n\ndef pad(s, n):\n    out = s\n    while len(out) < n:\n        out = out + \" \"\n    return out\n\ndef lpad(s, n):\n    out = s\n    while len(out) < n:\n        out = \" \" + out\n    return out\n\ndef numbers(rs):\n    out = \"\"\n    for r in rs:\n        if out == \"\":\n            out = str(r[0])\n        else:\n            out = out + \", \" + str(r[0])\n    if out == \"\":\n        return \"none\"\n    return out\n\ndef show_rows():\n    print(\"The rows coming in:\")\n    for r in pipeline.rows():\n        print(\"  \" + lpad(str(r[0]), 2) + \"  \" + r[1] + \"  \" + pad(r[2], 4) + pad(r[3], 11) + r[4])\n\ndef plain_run():\n    x = pipeline.run(pipeline.rows())\n    out = x[0]\n    told = 0\n    for d in x[1]:\n        if pipeline.said(d[0]):\n            print(\"  parse rejected row \" + str(d[1]) + \": \" + d[2] + \".\")\n            told = told + 1\n    print(\"Imported \" + str(len(out)) + \" rows: \" + numbers(out) + \".\")\n    n = len(pipeline.rows())\n    print(str(n) + \" in, \" + str(len(out)) + \" out, \" + str(told) + \" explained: \" + str(n - len(out) - told) + \" rows are gone and nothing says where.\")\n\ndef ledger_run():\n    x = pipeline.run(pipeline.rows())\n    out = x[0]\n    led = x[1]\n    left = len(pipeline.rows())\n    for st in [\"parse\", \"enrich\", \"dedupe\", \"window\"]:\n        mine = []\n        for d in led:\n            if d[0] == st:\n                mine = mine + [d]\n        if pipeline.said(st):\n            how = \"reported\"\n        else:\n            how = \"silent as written\"\n        print(\"  \" + pad(st, 7) + str(left) + \" in, \" + str(left - len(mine)) + \" out (\" + how + \")\")\n        for d in mine:\n            print(\"      row \" + str(d[1]) + \": \" + d[2])\n        left = left - len(mine)\n    n = len(pipeline.rows())\n    print(str(n) + \" in - \" + str(len(out)) + \" out - \" + str(len(led)) + \" explained = \" + str(n - len(out) - len(led)) + \": every row is accounted for.\")\n    print(\"The rows out are the same as without the ledger; what changes is what can be said about the rest.\")\n\ndef read_viewer():\n    v = plain(input(\"viewer (sales or finance)> \"))\n    return v\n\ndef listing():\n    v = read_viewer()\n    if v == \"\":\n        print(\"Cancelled.\")\n        return\n    if v != \"sales\" and v != \"finance\":\n        print(\"Type sales or finance.\")\n        return\n    t = plain(input(\"amount at least (0 to 99)> \"))\n    if t == \"\":\n        print(\"Cancelled.\")\n        return\n    if len(t) > 2 or not pipeline.is_number(t):\n        print(\"Type a whole number from 0 to 99.\")\n        return\n    least = int(t)\n    out = pipeline.run(pipeline.rows())[0]\n    m = pipeline.query(out, least)\n    shown = pipeline.visible(m, v)\n    print(\"Header, counted by the query: \" + str(len(m)) + \" results.\")\n    for r in shown:\n        print(\"  \" + lpad(str(r[0]), 2) + \"  \" + pad(r[1], 4) + pad(r[2], 11) + lpad(str(r[3]), 3))\n    if len(shown) == 0:\n        print(\"  (no rows shown)\")\n    if len(m) > len(shown):\n        print(str(len(m) - len(shown)) + \" of the \" + str(len(m)) + \" are not for \" + v + \" to see, and the header counted them. Counted after the filter it says \" + str(len(shown)) + \".\")\n    else:\n        print(\"The header and the rows agree here: nothing hidden matches.\")\n\ndef narrow():\n    v = read_viewer()\n    if v == \"\":\n        print(\"Cancelled.\")\n        return\n    if v != \"sales\" and v != \"finance\":\n        print(\"Type sales or finance.\")\n        return\n    out = pipeline.run(pipeline.rows())[0]\n    top = 0\n    for r in out:\n        if r[3] > top:\n            top = r[3]\n    print(\"As \" + v + \", amount at least 0, 1, 2 and up; the header minus the rows shown is what is hidden:\")\n    print(\"  \" + pad(\"at least\", 10) + \"header  shown  hidden\")\n    learned = []\n    before = 0\n    n = 0\n    while n <= top + 1:\n        m = pipeline.query(out, n)\n        shown = pipeline.visible(m, v)\n        hidden = len(m) - len(shown)\n        if n == 0 or hidden != before:\n            print(\"  \" + pad(str(n), 10) + lpad(str(len(m)), 6) + lpad(str(len(shown)), 7) + lpad(str(hidden), 8))\n        if n > 0 and hidden < before:\n            k = 0\n            while k < before - hidden:\n                learned = learned + [n - 1]\n                k = k + 1\n        before = hidden\n        n = n + 1\n    said = \"\"\n    for a in learned:\n        if said == \"\":\n            said = str(a)\n        else:\n            said = said + \", \" + str(a)\n    print(\"Without reading one, \" + v + \" learns the amounts of the rows it cannot see: \" + said + \".\")\n    print(\"Counted after the visibility filter, the header would equal the rows shown at every step, and tell nothing.\")\n\nprint(\"== Import pipeline ==\")\nprint(\"Order rows through parse, enrich, dedupe and window, and a listing of what came out.\")\nrunning = True\nwhile running:\n    print(\"\")\n    print(\"1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\")\n    choice = plain(input(\"choice> \"))\n    if choice == \"1\":\n        show_rows()\n    elif choice == \"2\":\n        plain_run()\n    elif choice == \"3\":\n        ledger_run()\n    elif choice == \"4\":\n        listing()\n    elif choice == \"5\":\n        narrow()\n    elif choice == \"6\":\n        running = False\n    else:\n        print(\"Pick a number from 1 to 6.\")\nprint(\"Bye.\")\n"
    },
    {
      "name": "pipeline.eml",
      "eml": "# P152 import pipeline - order rows through four stages, then a listing of\n# what came out. As in the corpus case silent-drop-accounting, every stage\n# drops rows for a reason that is right on its own - parse rejects a bad\n# amount and says so, enrich drops a row whose account it cannot look up,\n# dedupe drops repeats, window drops rows outside the hour - and only a\n# ledger of those reasons closes the books: in - out - explained = 0. As in\n# the corpus case count-leaks-hidden-items, the listing counts in the query\n# and filters where it shows the rows, so the count includes rows the viewer\n# cannot see, and narrowing a filter tells the viewer what they hold.\n# A row is [number, time, account, product, amount as typed].\n\ndef rows():\n    return [[1, \"09:00\", \"a1\", \"widget\", \"10\"], [2, \"09:01\", \"a2\", \"gadget\", \"5\"], [3, \"09:02\", \"a1\", \"widget\", \"10\"], [4, \"09:03\", \"zz\", \"widget\", \"7\"], [5, \"09:04\", \"a3\", \"doohickey\", \"3\"], [6, \"09:05\", \"a2\", \"gadget\", \"x\"], [7, \"09:06\", \"a4\", \"widget\", \"2\"], [8, \"09:07\", \"a1\", \"widget\", \"10\"], [9, \"09:08\", \"a3\", \"gizmo\", \"8\"], [10, \"09:09\", \"a2\", \"widget\", \"4\"], [11, \"09:12\", \"a4\", \"gadget\", \"6\"], [12, \"09:15\", \"a1\", \"gizmo\", \"9\"]]\n\ndef team(account):\n    # the lookup enrich joins; \"\" for an account it does not have\n    if account == \"a1\" or account == \"a2\":\n        return \"sales\"\n    if account == \"a3\" or account == \"a4\":\n        return \"finance\"\n    return \"\"\n\ndef is_number(s):\n    if len(s) == 0:\n        return False\n    for c in s:\n        if c < \"0\" or c > \"9\":\n            return False\n    return True\n\ndef run(rs):\n    # [rows out as [number, account, product, amount, team], ledger of\n    # [stage, number, reason] for every row dropped]\n    [] => led\n    [] => a\n    for r in rs:\n        if is_number(r[4]):\n            a + [r] => a\n        else:\n            led + [[\"parse\", r[0], \"amount \" + r[4] + \" is not a number\"]] => led\n    [] => b\n    for r in a:\n        if team(r[2]) == \"\":\n            led + [[\"enrich\", r[0], \"no account \" + r[2]]] => led\n        else:\n            b + [r] => b\n    [] => c\n    for r in b:\n        0 => first\n        for s in c:\n            if s[2] == r[2] and s[3] == r[3] and s[4] == r[4]:\n                s[0] => first\n        if first > 0:\n            led + [[\"dedupe\", r[0], \"repeats row \" + str(first)]] => led\n        else:\n            c + [r] => c\n    [] => out\n    for r in c:\n        if r[1] >= \"09:00\" and r[1] < \"09:10\":\n            out + [[r[0], r[2], r[3], int(r[4]), team(r[2])]] => out\n        else:\n            led + [[\"window\", r[0], \"at \" + r[1] + \", outside 09:00 to 09:10\"]] => led\n    return [out, led]\n\ndef said(stage):\n    # whether the plain pipeline reports what the stage drops\n    return stage == \"parse\"\n\ndef query(out, least):\n    # the rows the listing's query matches: amount at least least\n    [] => m\n    for r in out:\n        if r[3] >= least:\n            m + [r] => m\n    return m\n\ndef visible(rs, viewer):\n    [] => v\n    for r in rs:\n        if r[4] == viewer:\n            v + [r] => v\n    return v\n",
      "python": "def rows():\n    return [[1, \"09:00\", \"a1\", \"widget\", \"10\"], [2, \"09:01\", \"a2\", \"gadget\", \"5\"], [3, \"09:02\", \"a1\", \"widget\", \"10\"], [4, \"09:03\", \"zz\", \"widget\", \"7\"], [5, \"09:04\", \"a3\", \"doohickey\", \"3\"], [6, \"09:05\", \"a2\", \"gadget\", \"x\"], [7, \"09:06\", \"a4\", \"widget\", \"2\"], [8, \"09:07\", \"a1\", \"widget\", \"10\"], [9, \"09:08\", \"a3\", \"gizmo\", \"8\"], [10, \"09:09\", \"a2\", \"widget\", \"4\"], [11, \"09:12\", \"a4\", \"gadget\", \"6\"], [12, \"09:15\", \"a1\", \"gizmo\", \"9\"]]\n\ndef team(account):\n    if account == \"a1\" or account == \"a2\":\n        return \"sales\"\n    if account == \"a3\" or account == \"a4\":\n        return \"finance\"\n    return \"\"\n\ndef is_number(s):\n    if len(s) == 0:\n        return False\n    for c in s:\n        if c < \"0\" or c > \"9\":\n            return False\n    return True\n\ndef run(rs):\n    led = []\n    a = []\n    for r in rs:\n        if is_number(r[4]):\n            a = a + [r]\n        else:\n            led = led + [[\"parse\", r[0], \"amount \" + r[4] + \" is not a number\"]]\n    b = []\n    for r in a:\n        if team(r[2]) == \"\":\n            led = led + [[\"enrich\", r[0], \"no account \" + r[2]]]\n        else:\n            b = b + [r]\n    c = []\n    for r in b:\n        first = 0\n        for s in c:\n            if s[2] == r[2] and s[3] == r[3] and s[4] == r[4]:\n                first = s[0]\n        if first > 0:\n            led = led + [[\"dedupe\", r[0], \"repeats row \" + str(first)]]\n        else:\n            c = c + [r]\n    out = []\n    for r in c:\n        if r[1] >= \"09:00\" and r[1] < \"09:10\":\n            out = out + [[r[0], r[2], r[3], int(r[4]), team(r[2])]]\n        else:\n            led = led + [[\"window\", r[0], \"at \" + r[1] + \", outside 09:00 to 09:10\"]]\n    return [out, led]\n\ndef said(stage):\n    return stage == \"parse\"\n\ndef query(out, least):\n    m = []\n    for r in out:\n        if r[3] >= least:\n            m = m + [r]\n    return m\n\ndef visible(rs, viewer):\n    v = []\n    for r in rs:\n        if r[4] == viewer:\n            v = v + [r]\n    return v\n"
    }
  ],
  "sessions": [
    {
      "name": "bad-input",
      "input": "0\nx\n4\n\n4\nhr\n4\nsales\n\n4\nsales\n100\n4\nsales\nx\n5\n\n5\nmarketing\n5\nfinance\n6\n",
      "screen": "== Import pipeline ==\nOrder rows through parse, enrich, dedupe and window, and a listing of what came out.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 0\nPick a number from 1 to 6.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> x\nPick a number from 1 to 6.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> \nCancelled.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> hr\nType sales or finance.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> sales\namount at least (0 to 99)> \nCancelled.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> sales\namount at least (0 to 99)> 100\nType a whole number from 0 to 99.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> sales\namount at least (0 to 99)> x\nType a whole number from 0 to 99.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 5\nviewer (sales or finance)> \nCancelled.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 5\nviewer (sales or finance)> marketing\nType sales or finance.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 5\nviewer (sales or finance)> finance\nAs finance, amount at least 0, 1, 2 and up; the header minus the rows shown is what is hidden:\n  at least  header  shown  hidden\n  0              6      3       3\n  5              3      1       2\n  6              2      1       1\n  11             0      0       0\nWithout reading one, finance learns the amounts of the rows it cannot see: 4, 5, 10.\nCounted after the visibility filter, the header would equal the rows shown at every step, and tell nothing.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 6\nBye.\n",
      "interpreter": "equal"
    },
    {
      "name": "basic",
      "input": "1\n2\n3\n4\nsales\n0\n4\nfinance\n5\n4\nsales\n11\n5\nsales\n6\n",
      "screen": "== Import pipeline ==\nOrder rows through parse, enrich, dedupe and window, and a listing of what came out.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 1\nThe rows coming in:\n   1  09:00  a1  widget     10\n   2  09:01  a2  gadget     5\n   3  09:02  a1  widget     10\n   4  09:03  zz  widget     7\n   5  09:04  a3  doohickey  3\n   6  09:05  a2  gadget     x\n   7  09:06  a4  widget     2\n   8  09:07  a1  widget     10\n   9  09:08  a3  gizmo      8\n  10  09:09  a2  widget     4\n  11  09:12  a4  gadget     6\n  12  09:15  a1  gizmo      9\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 2\n  parse rejected row 6: amount x is not a number.\nImported 6 rows: 1, 2, 5, 7, 9, 10.\n12 in, 6 out, 1 explained: 5 rows are gone and nothing says where.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 3\n  parse  12 in, 11 out (reported)\n      row 6: amount x is not a number\n  enrich 11 in, 10 out (silent as written)\n      row 4: no account zz\n  dedupe 10 in, 8 out (silent as written)\n      row 3: repeats row 1\n      row 8: repeats row 1\n  window 8 in, 6 out (silent as written)\n      row 11: at 09:12, outside 09:00 to 09:10\n      row 12: at 09:15, outside 09:00 to 09:10\n12 in - 6 out - 6 explained = 0: every row is accounted for.\nThe rows out are the same as without the ledger; what changes is what can be said about the rest.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> sales\namount at least (0 to 99)> 0\nHeader, counted by the query: 6 results.\n   1  a1  widget      10\n   2  a2  gadget       5\n  10  a2  widget       4\n3 of the 6 are not for sales to see, and the header counted them. Counted after the filter it says 3.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> finance\namount at least (0 to 99)> 5\nHeader, counted by the query: 3 results.\n   9  a3  gizmo        8\n2 of the 3 are not for finance to see, and the header counted them. Counted after the filter it says 1.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 4\nviewer (sales or finance)> sales\namount at least (0 to 99)> 11\nHeader, counted by the query: 0 results.\n  (no rows shown)\nThe header and the rows agree here: nothing hidden matches.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 5\nviewer (sales or finance)> sales\nAs sales, amount at least 0, 1, 2 and up; the header minus the rows shown is what is hidden:\n  at least  header  shown  hidden\n  0              6      3       3\n  3              5      3       2\n  4              4      3       1\n  9              1      1       0\nWithout reading one, sales learns the amounts of the rows it cannot see: 2, 3, 8.\nCounted after the visibility filter, the header would equal the rows shown at every step, and tell nothing.\n\n1) rows in  2) import  3) import with a ledger  4) the listing  5) narrow the filter  6) quit\nchoice> 6\nBye.\n",
      "interpreter": "equal"
    }
  ],
  "builtOn": [
    {
      "slug": "silent-drop-accounting",
      "caseId": "243-silent-drop-accounting",
      "title": "Same output, and only one can say what went missing"
    },
    {
      "slug": "count-leaks-hidden-items",
      "caseId": "292-count-leaks-hidden-items",
      "title": "Count leaks hidden items — the rows were filtered and the total was not"
    }
  ],
  "updated": "2026-10-18"
}