{ "id": "P149", "slug": "signup-funnel", "key": "P149-signup-funnel", "title": "Signup funnel", "summary": "Sign-up rates by segment with events over humans, clamped and matched, and the sign-ups credited to onboarding tips set against the ones they caused as more people see them.", "entry": "main.eml", "ui": "terminal", "readme": "# P149 - Signup funnel\n\nSign-up rates by traffic segment, worked out three ways, and the sign-ups\ncredited to an onboarding feature set against the ones it caused.\n\n- `main.eml` - the menu, the segments, the rates, the tips at one rollout,\n and at every rollout\n- `funnel.eml` - the segments, percentages in tenths rounded half up, the\n three rates, the buckets of visitors, and a rollout's counts\n\nHow each part works:\n\n- Four segments - search, ads, social, email - each with human visitors\n and sign-ups, and bot and test visitors and sign-ups. As in the corpus\n case `rate-numerator-outgrows-denominator`, the sign-ups come off the\n event table, which has every one, and the visitors off a view of humans.\n- Events over humans: ads reaches 135.0%, the one rate anyone notices, and\n the other three are high without looking wrong - social 35.0% against\n 10.0%. Clamped at 100%, ads reads 100.0%: all four are still wrong and\n nothing is left to notice. Humans over humans is the rate.\n- As in the corpus case `usage-reported-as-effect`, the tips are shown to\n the most engaged buckets first, and \"sign-ups by users who saw the tips\"\n counts everyone who passed through them. Shown to every bucket that\n count is 184, all the sign-ups; the tips caused 27. Down the buckets the\n credited count falls while the caused count never does: the most engaged\n sign up anyway.\n\nWhat is checked: menu choices 1 to 5; a number of buckets from 0 to 10. An\nempty answer cancels.\n\nSessions: `sessions/basic.in` gives:\n- the segments and the rates;\n- the tips shown to 5 buckets and to all 10; every rollout.\n\n`sessions/bad-input.in` gives:\n- menu choices 0 and x;\n- buckets: a cancel, 11, x and 123; then 0, where nobody saw the tips, and\n 1.\n\nBuilt on the verified corpus cases `rate-numerator-outgrows-denominator`\n(sign-ups and visitors from different populations, mismatched, clamped and\nmatched) and `usage-reported-as-effect` (a feature's usage count against\nits true lift as exposure widens).\n", "modules": [ { "name": "main.eml", "eml": "# P149 signup funnel: the segments, their sign-up rates three ways, the\n# onboarding tips at one rollout, and at every rollout.\nimport funnel\n\ndef plain(s):\n 0 => i\n len(s) => j\n while i < j and s[i] == \" \":\n i + 1 => i\n while j > i and s[j - 1] == \" \":\n j - 1 => j\n return s[i:j]\n\ndef pad(s, n):\n s => out\n while len(out) < n:\n out + \" \" => out\n return out\n\ndef lpad(s, n):\n s => out\n while len(out) < n:\n \" \" + out => out\n return out\n\ndef pct(t):\n # tenths of a percent\n t % 10 => d\n return str(int((t - d) / 10)) + \".\" + str(d) + \"%\"\n\ndef show_segments():\n (\" \" + pad(\"segment\", 9) + \"human visitors human sign-ups bot and test visitors bot and test sign-ups\") ^0\n for s in funnel.segments():\n (\" \" + pad(s[0], 9) + lpad(str(s[1]), 14) + lpad(str(s[2]), 16) + lpad(str(s[3]), 23) + lpad(str(s[4]), 23)) ^0\n \"The sign-ups come off the event table, which has them all; the visitors off a view of humans only.\" ^0\n\ndef rates():\n (\" \" + pad(\"segment\", 9) + \"events / humans clamped humans / humans\") ^0\n 0 => wrong1\n 0 => over1\n 0 => wrong2\n 0 => over2\n for s in funnel.segments():\n funnel.mismatched(s) => a\n funnel.clamped(s) => b\n funnel.matched(s) => c\n (\" \" + pad(s[0], 9) + lpad(pct(a), 15) + lpad(pct(b), 10) + lpad(pct(c), 18)) ^0\n if a != c:\n wrong1 + 1 => wrong1\n if a > 1000:\n over1 + 1 => over1\n if b != c:\n wrong2 + 1 => wrong2\n if b > 1000:\n over2 + 1 => over2\n len(funnel.segments()) => n\n (\"Events over humans: \" + str(wrong1) + \" of \" + str(n) + \" segments wrong, \" + str(over1) + \" of them over 100% - the one anyone notices.\") ^0\n (\"Clamped at 100%: \" + str(wrong2) + \" of \" + str(n) + \" still wrong, \" + str(over2) + \" over 100%: nothing is left to notice.\") ^0\n \"Humans over humans is the rate: both numbers from the same population.\" ^0\n\ndef read_rollout():\n plain(input(\"buckets shown the tips (0 to 10)> \")) => t\n if t == \"\":\n return 0 - 1\n if len(t) > 2:\n return 0 - 2\n for c in t:\n if c < \"0\" or c > \"9\":\n return 0 - 2\n if int(t) > 10:\n return 0 - 2\n return int(t)\n\ndef one_rollout():\n read_rollout() => r\n if r == 0 - 1:\n \"Cancelled.\" ^0\n return\n if r < 0:\n \"Type a number of buckets from 0 to 10.\" ^0\n return\n funnel.rollout(r) => x\n (\"Shown to \" + str(r) + \" of 10 buckets, \" + str(r * 100) + \" visitors: \" + str(x[0]) + \" sign-ups in all.\") ^0\n if r == 0:\n \" Nobody saw the tips: nothing to credit, and nothing they caused.\" ^0\n return\n (\" by users who saw the tips: \" + str(x[1]) + \", \" + pct(funnel.tenths(x[1], x[0])) + \" of the sign-ups\") ^0\n (\" caused by the tips: \" + str(x[2]) + \", \" + pct(funnel.tenths(x[2], x[0])) + \" of the sign-ups\") ^0\n (\" \" + pad(\"bucket\", 8) + \"signed up after seeing the tips of them only because of the tips\") ^0\n 0 => b\n for y in funnel.buckets():\n if b < r:\n (\" \" + pad(str(b + 1), 8) + lpad(str(y[0] + y[1]), 30) + lpad(str(y[1]), 35)) ^0\n b + 1 => b\n if r >= 2:\n \"The count is largest where the tips change least: the most engaged sign up anyway.\" ^0\n\ndef every_rollout():\n (\" \" + pad(\"shown\", 7) + \"sign-ups by users who saw the tips caused by the tips\") ^0\n 0 => r\n while r <= 10:\n funnel.rollout(r) => x\n (\" \" + pad(str(r), 7) + lpad(str(x[0]), 8) + lpad(str(x[1]) + \" \" + lpad(pct(funnel.tenths(x[1], x[0])), 6), 30) + lpad(str(x[2]) + \" \" + lpad(pct(funnel.tenths(x[2], x[0])), 6), 21)) ^0\n r + 1 => r\n funnel.rollout(1) => a\n funnel.rollout(10) => z\n (\"Shown to more buckets, the count credited to the tips goes from \" + str(a[1]) + \" to \" + str(z[1]) + \", \" + pct(funnel.tenths(z[1], z[0])) + \" of all sign-ups; the tips caused \" + str(z[2]) + \" of them.\") ^0\n \"The credited count measures who passed through the tips, not what they changed.\" ^0\n\n\"== Signup funnel ==\" ^0\n\"Sign-up rates by segment, and the sign-ups credited to the onboarding tips.\" ^0\nTrue => running\nwhile running:\n \"\" ^0\n \"1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\" ^0\n plain(input(\"choice> \")) => choice\n if choice == \"1\":\n show_segments()\n elif choice == \"2\":\n rates()\n elif choice == \"3\":\n one_rollout()\n elif choice == \"4\":\n every_rollout()\n elif choice == \"5\":\n False => running\n else:\n \"Pick a number from 1 to 5.\" ^0\n\"Bye.\" ^0\n", "python": "import funnel\n\ndef plain(s):\n i = 0\n j = len(s)\n while i < j and s[i] == \" \":\n i = i + 1\n while j > i and s[j - 1] == \" \":\n j = j - 1\n return s[i:j]\n\ndef pad(s, n):\n out = s\n while len(out) < n:\n out = out + \" \"\n return out\n\ndef lpad(s, n):\n out = s\n while len(out) < n:\n out = \" \" + out\n return out\n\ndef pct(t):\n d = t % 10\n return str(int((t - d) / 10)) + \".\" + str(d) + \"%\"\n\ndef show_segments():\n print(\" \" + pad(\"segment\", 9) + \"human visitors human sign-ups bot and test visitors bot and test sign-ups\")\n for s in funnel.segments():\n print(\" \" + pad(s[0], 9) + lpad(str(s[1]), 14) + lpad(str(s[2]), 16) + lpad(str(s[3]), 23) + lpad(str(s[4]), 23))\n print(\"The sign-ups come off the event table, which has them all; the visitors off a view of humans only.\")\n\ndef rates():\n print(\" \" + pad(\"segment\", 9) + \"events / humans clamped humans / humans\")\n wrong1 = 0\n over1 = 0\n wrong2 = 0\n over2 = 0\n for s in funnel.segments():\n a = funnel.mismatched(s)\n b = funnel.clamped(s)\n c = funnel.matched(s)\n print(\" \" + pad(s[0], 9) + lpad(pct(a), 15) + lpad(pct(b), 10) + lpad(pct(c), 18))\n if a != c:\n wrong1 = wrong1 + 1\n if a > 1000:\n over1 = over1 + 1\n if b != c:\n wrong2 = wrong2 + 1\n if b > 1000:\n over2 = over2 + 1\n n = len(funnel.segments())\n print(\"Events over humans: \" + str(wrong1) + \" of \" + str(n) + \" segments wrong, \" + str(over1) + \" of them over 100% - the one anyone notices.\")\n print(\"Clamped at 100%: \" + str(wrong2) + \" of \" + str(n) + \" still wrong, \" + str(over2) + \" over 100%: nothing is left to notice.\")\n print(\"Humans over humans is the rate: both numbers from the same population.\")\n\ndef read_rollout():\n t = plain(input(\"buckets shown the tips (0 to 10)> \"))\n if t == \"\":\n return 0 - 1\n if len(t) > 2:\n return 0 - 2\n for c in t:\n if c < \"0\" or c > \"9\":\n return 0 - 2\n if int(t) > 10:\n return 0 - 2\n return int(t)\n\ndef one_rollout():\n r = read_rollout()\n if r == 0 - 1:\n print(\"Cancelled.\")\n return\n if r < 0:\n print(\"Type a number of buckets from 0 to 10.\")\n return\n x = funnel.rollout(r)\n print(\"Shown to \" + str(r) + \" of 10 buckets, \" + str(r * 100) + \" visitors: \" + str(x[0]) + \" sign-ups in all.\")\n if r == 0:\n print(\" Nobody saw the tips: nothing to credit, and nothing they caused.\")\n return\n print(\" by users who saw the tips: \" + str(x[1]) + \", \" + pct(funnel.tenths(x[1], x[0])) + \" of the sign-ups\")\n print(\" caused by the tips: \" + str(x[2]) + \", \" + pct(funnel.tenths(x[2], x[0])) + \" of the sign-ups\")\n print(\" \" + pad(\"bucket\", 8) + \"signed up after seeing the tips of them only because of the tips\")\n b = 0\n for y in funnel.buckets():\n if b < r:\n print(\" \" + pad(str(b + 1), 8) + lpad(str(y[0] + y[1]), 30) + lpad(str(y[1]), 35))\n b = b + 1\n if r >= 2:\n print(\"The count is largest where the tips change least: the most engaged sign up anyway.\")\n\ndef every_rollout():\n print(\" \" + pad(\"shown\", 7) + \"sign-ups by users who saw the tips caused by the tips\")\n r = 0\n while r <= 10:\n x = funnel.rollout(r)\n print(\" \" + pad(str(r), 7) + lpad(str(x[0]), 8) + lpad(str(x[1]) + \" \" + lpad(pct(funnel.tenths(x[1], x[0])), 6), 30) + lpad(str(x[2]) + \" \" + lpad(pct(funnel.tenths(x[2], x[0])), 6), 21))\n r = r + 1\n a = funnel.rollout(1)\n z = funnel.rollout(10)\n print(\"Shown to more buckets, the count credited to the tips goes from \" + str(a[1]) + \" to \" + str(z[1]) + \", \" + pct(funnel.tenths(z[1], z[0])) + \" of all sign-ups; the tips caused \" + str(z[2]) + \" of them.\")\n print(\"The credited count measures who passed through the tips, not what they changed.\")\n\nprint(\"== Signup funnel ==\")\nprint(\"Sign-up rates by segment, and the sign-ups credited to the onboarding tips.\")\nrunning = True\nwhile running:\n print(\"\")\n print(\"1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\")\n choice = plain(input(\"choice> \"))\n if choice == \"1\":\n show_segments()\n elif choice == \"2\":\n rates()\n elif choice == \"3\":\n one_rollout()\n elif choice == \"4\":\n every_rollout()\n elif choice == \"5\":\n running = False\n else:\n print(\"Pick a number from 1 to 5.\")\nprint(\"Bye.\")\n" }, { "name": "funnel.eml", "eml": "# P149 signup funnel - sign-up rates by traffic segment, and a feature\n# credited with sign-ups. As in the corpus case\n# rate-numerator-outgrows-denominator, the sign-ups come off the event table,\n# which has every sign-up, and the visitors off a curated view of humans: the\n# ratio of two right numbers is not a rate, and clamping the one that goes\n# over 100% turns a loud impossible number into a quiet wrong one. As in the\n# corpus case usage-reported-as-effect, \"sign-ups by users who saw the tips\"\n# counts what passed through the feature, not what it changed: it grows as\n# the feature is shown to more people, and is largest where the feature\n# changes least.\n\ndef segments():\n # [name, human visitors, human sign-ups, bot and test visitors, bot and\n # test sign-ups]\n return [[\"search\", 120, 18, 30, 2], [\"ads\", 40, 4, 200, 50], [\"social\", 80, 8, 60, 20], [\"email\", 50, 15, 5, 5]]\n\ndef tenths(n, d):\n # n / d as a percentage in tenths, rounded half up\n return int((2000 * n + d) / (2 * d))\n\ndef mismatched(s):\n # every sign-up over the human visitors\n return tenths(s[2] + s[4], s[1])\n\ndef clamped(s):\n if mismatched(s) > 1000:\n return 1000\n return mismatched(s)\n\ndef matched(s):\n # both from the same population: humans over humans\n return tenths(s[2], s[1])\n\ndef buckets():\n # ten buckets of 100 visitors, the most engaged first, the feature shown\n # to the first ones: [would sign up anyway, sign up only with the tips]\n return [[40, 1], [32, 1], [25, 2], [19, 2], [14, 3], [10, 3], [7, 3], [5, 4], [3, 4], [2, 4]]\n\ndef rollout(r):\n # the tips shown to the first r buckets: [sign-ups in all, sign-ups by\n # users who saw the tips, sign-ups the tips caused]\n 0 => total\n 0 => saw\n 0 => caused\n 0 => b\n for x in buckets():\n total + x[0] => total\n if b < r:\n total + x[1] => total\n saw + x[0] + x[1] => saw\n caused + x[1] => caused\n b + 1 => b\n return [total, saw, caused]\n", "python": "def segments():\n return [[\"search\", 120, 18, 30, 2], [\"ads\", 40, 4, 200, 50], [\"social\", 80, 8, 60, 20], [\"email\", 50, 15, 5, 5]]\n\ndef tenths(n, d):\n return int((2000 * n + d) / (2 * d))\n\ndef mismatched(s):\n return tenths(s[2] + s[4], s[1])\n\ndef clamped(s):\n if mismatched(s) > 1000:\n return 1000\n return mismatched(s)\n\ndef matched(s):\n return tenths(s[2], s[1])\n\ndef buckets():\n return [[40, 1], [32, 1], [25, 2], [19, 2], [14, 3], [10, 3], [7, 3], [5, 4], [3, 4], [2, 4]]\n\ndef rollout(r):\n total = 0\n saw = 0\n caused = 0\n b = 0\n for x in buckets():\n total = total + x[0]\n if b < r:\n total = total + x[1]\n saw = saw + x[0] + x[1]\n caused = caused + x[1]\n b = b + 1\n return [total, saw, caused]\n" } ], "sessions": [ { "name": "bad-input", "input": "0\nx\n3\n\n3\n11\n3\nx\n3\n123\n3\n0\n3\n1\n5\n", "screen": "== Signup funnel ==\nSign-up rates by segment, and the sign-ups credited to the onboarding tips.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 0\nPick a number from 1 to 5.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> x\nPick a number from 1 to 5.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> \nCancelled.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 11\nType a number of buckets from 0 to 10.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> x\nType a number of buckets from 0 to 10.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 123\nType a number of buckets from 0 to 10.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 0\nShown to 0 of 10 buckets, 0 visitors: 157 sign-ups in all.\n Nobody saw the tips: nothing to credit, and nothing they caused.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 1\nShown to 1 of 10 buckets, 100 visitors: 158 sign-ups in all.\n by users who saw the tips: 41, 25.9% of the sign-ups\n caused by the tips: 1, 0.6% of the sign-ups\n bucket signed up after seeing the tips of them only because of the tips\n 1 41 1\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 5\nBye.\n", "interpreter": "equal" }, { "name": "basic", "input": "1\n2\n3\n5\n3\n10\n4\n5\n", "screen": "== Signup funnel ==\nSign-up rates by segment, and the sign-ups credited to the onboarding tips.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 1\n segment human visitors human sign-ups bot and test visitors bot and test sign-ups\n search 120 18 30 2\n ads 40 4 200 50\n social 80 8 60 20\n email 50 15 5 5\nThe sign-ups come off the event table, which has them all; the visitors off a view of humans only.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 2\n segment events / humans clamped humans / humans\n search 16.7% 16.7% 15.0%\n ads 135.0% 100.0% 10.0%\n social 35.0% 35.0% 10.0%\n email 40.0% 40.0% 30.0%\nEvents over humans: 4 of 4 segments wrong, 1 of them over 100% - the one anyone notices.\nClamped at 100%: 4 of 4 still wrong, 0 over 100%: nothing is left to notice.\nHumans over humans is the rate: both numbers from the same population.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 5\nShown to 5 of 10 buckets, 500 visitors: 166 sign-ups in all.\n by users who saw the tips: 139, 83.7% of the sign-ups\n caused by the tips: 9, 5.4% of the sign-ups\n bucket signed up after seeing the tips of them only because of the tips\n 1 41 1\n 2 33 1\n 3 27 2\n 4 21 2\n 5 17 3\nThe count is largest where the tips change least: the most engaged sign up anyway.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 3\nbuckets shown the tips (0 to 10)> 10\nShown to 10 of 10 buckets, 1000 visitors: 184 sign-ups in all.\n by users who saw the tips: 184, 100.0% of the sign-ups\n caused by the tips: 27, 14.7% of the sign-ups\n bucket signed up after seeing the tips of them only because of the tips\n 1 41 1\n 2 33 1\n 3 27 2\n 4 21 2\n 5 17 3\n 6 13 3\n 7 10 3\n 8 9 4\n 9 7 4\n 10 6 4\nThe count is largest where the tips change least: the most engaged sign up anyway.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 4\n shown sign-ups by users who saw the tips caused by the tips\n 0 157 0 0.0% 0 0.0%\n 1 158 41 25.9% 1 0.6%\n 2 159 74 46.5% 2 1.3%\n 3 161 101 62.7% 4 2.5%\n 4 163 122 74.8% 6 3.7%\n 5 166 139 83.7% 9 5.4%\n 6 169 152 89.9% 12 7.1%\n 7 172 162 94.2% 15 8.7%\n 8 176 171 97.2% 19 10.8%\n 9 180 178 98.9% 23 12.8%\n 10 184 184 100.0% 27 14.7%\nShown to more buckets, the count credited to the tips goes from 41 to 184, 100.0% of all sign-ups; the tips caused 27 of them.\nThe credited count measures who passed through the tips, not what they changed.\n\n1) segments 2) rates 3) the tips at one rollout 4) every rollout 5) quit\nchoice> 5\nBye.\n", "interpreter": "equal" } ], "builtOn": [ { "slug": "rate-numerator-outgrows-denominator", "caseId": "303-rate-numerator-outgrows-denominator", "title": "Rate numerator outgrows denominator — the clamp removed the evidence and kept the error" }, { "slug": "usage-reported-as-effect", "caseId": "320-usage-reported-as-effect", "title": "Usage reported as effect — the metric was largest where the effect was zero" } ], "updated": "2026-10-18" }