# Example 701 — The search index was fresh and the permissions were cached `the_search_index_was_fresh_and_the_permissions_were_cached.eml` - A document is searchable two seconds after it is written and the freshness is measured continuously. How old the access rules in the index are is computed below. ## EML ```eml # Self-authored for the EML case corpus (no external origin). A document is # searchable two seconds after it is written and the freshness is measured # continuously. How old the access rules in the index are is computed below. # # The indexing pipeline is fast and it is watched. Writes are streamed rather # than batched, the end-to-end lag from commit to searchable is measured on # every document rather than sampled, the alert fires above five seconds, and it # has fired twice this year for real reasons. Two seconds is a measured median # and the tail is bounded. # # What is indexed is the document AND a copy of who may see it, because # filtering results by permission at query time against the authorization # service would put a network call inside every hit. # # A document is re-indexed when its CONTENT changes. Permissions are not content. 41000000 => documents 2 => index_lag_seconds 18400 => permission_changes_per_day 240000 => documents_reindexed_per_day 2 => freshness_alerts_this_year int(documents / documents_reindexed_per_day) => mean_days_until_a_document_is_touched int(mean_days_until_a_document_is_touched / 2) => mean_days_a_permission_change_waits "documents : " + str(documents) ^0 "index lag, seconds : " + str(index_lag_seconds) ^0 "documents re-indexed per day : " + str(documents_reindexed_per_day) ^0 "permission changes per day : " + str(permission_changes_per_day) ^0 "" ^0 "days until a document is touched : " + str(mean_days_until_a_document_is_touched) ^0 "mean wait for a permission change: " + str(mean_days_a_permission_change_waits) + " days" ^0 "" ^0 # ---- what the freshness measurement verified ---- "the indexing pipeline" ^0 " writes streamed, not batched : yes" ^0 " lag measured on every document : yes, not sampled" ^0 " median lag, seconds : " + str(index_lag_seconds) ^0 " alert above five seconds : yes" ^0 " fired this year for real reasons : " + str(freshness_alerts_this_year) ^0 " verdict : FRESH" ^0 "" ^0 " measuring every document rather than a sample is the" ^0 " expensive choice and it was made deliberately" ^0 "" ^0 # ---- what freshness is about ---- "the two things in an index entry" ^0 " the content : re-indexed when it changes, in " + str(index_lag_seconds) + " s" ^0 " the access rule : copied in at index time, so that a" ^0 " query can filter without a network call per hit" ^0 " what re-indexing is triggered by : a content change" ^0 " what a permission change triggers : nothing" ^0 "" ^0 " the copy is there for a good reason and the trigger was" ^0 " written for the other field" ^0 "" ^0 int(permission_changes_per_day * mean_days_a_permission_change_waits) => stale_rules_in_flight "permission changes waiting in the index : about " + str(stale_rules_in_flight) ^0 "" ^0 # ---- what a revoked reader sees ---- # The document does not open: opening it goes through the authorization service, # which is current. The search RESULT is the leak - the title, the snippet, and # the fact that the document exists. "a reader whose access was revoked" ^0 " opening the document : refused, the service is current" ^0 " the document in results : present" ^0 " the title : shown" ^0 " the snippet : shown, and it is the matched text" ^0 " what that reveals : the existence, the name, and the" ^0 " passage containing the searched term" ^0 "" ^0 # ---- null control ---- # The same pipeline, with a permission change enqueued as a re-index of the # documents it covers. 0 => nc_stale_rules_in_flight index_lag_seconds => nc_permission_lag_seconds "null control - a permission change re-indexes its documents" ^0 " median content lag, seconds : " + str(index_lag_seconds) + ", unchanged" ^0 " permission lag, seconds : " + str(nc_permission_lag_seconds) ^0 " changes waiting : " + str(nc_stale_rules_in_flight) ^0 " the pipeline did not get faster; the second field in" ^0 " the entry got a trigger of its own" ^0 "" ^0 # ---- the rule ---- "what a fresh index guarantees" ^0 " the content is current : exactly, and measured per document" ^0 " the entry is current : not addressed; an entry has more" ^0 " than one field and only one of them has a trigger" ^0 "" ^0 "a freshness metric measures the field its trigger fires on;" ^0 "anything else copied into the same record ages silently, and" ^0 "the copy exists precisely because reading it live was too" ^0 "expensive" ^0 "" ^0 "The index is fresh and measured on every document: streamed writes, a " + str(index_lag_seconds) + " second" ^0 "median, an alert above five that has fired " + str(freshness_alerts_this_year) + " times this year for real reasons." ^0 "Re-indexing is triggered by a content change, so with " + str(documents_reindexed_per_day) + " documents touched a" ^0 "day out of " + str(documents) + ", a permission change waits about " + str(mean_days_a_permission_change_waits) + " days, roughly" ^0 str(stale_rules_in_flight) + " of them in flight, while the title and the matching snippet stay visible." ^0 ``` ## Python (deterministic transpilation) ```python documents = 41000000 index_lag_seconds = 2 permission_changes_per_day = 18400 documents_reindexed_per_day = 240000 freshness_alerts_this_year = 2 mean_days_until_a_document_is_touched = int(documents / documents_reindexed_per_day) mean_days_a_permission_change_waits = int(mean_days_until_a_document_is_touched / 2) print("documents : " + str(documents)) print("index lag, seconds : " + str(index_lag_seconds)) print("documents re-indexed per day : " + str(documents_reindexed_per_day)) print("permission changes per day : " + str(permission_changes_per_day)) print("") print("days until a document is touched : " + str(mean_days_until_a_document_is_touched)) print("mean wait for a permission change: " + str(mean_days_a_permission_change_waits) + " days") print("") print("the indexing pipeline") print(" writes streamed, not batched : yes") print(" lag measured on every document : yes, not sampled") print(" median lag, seconds : " + str(index_lag_seconds)) print(" alert above five seconds : yes") print(" fired this year for real reasons : " + str(freshness_alerts_this_year)) print(" verdict : FRESH") print("") print(" measuring every document rather than a sample is the") print(" expensive choice and it was made deliberately") print("") print("the two things in an index entry") print(" the content : re-indexed when it changes, in " + str(index_lag_seconds) + " s") print(" the access rule : copied in at index time, so that a") print(" query can filter without a network call per hit") print(" what re-indexing is triggered by : a content change") print(" what a permission change triggers : nothing") print("") print(" the copy is there for a good reason and the trigger was") print(" written for the other field") print("") stale_rules_in_flight = int(permission_changes_per_day * mean_days_a_permission_change_waits) print("permission changes waiting in the index : about " + str(stale_rules_in_flight)) print("") print("a reader whose access was revoked") print(" opening the document : refused, the service is current") print(" the document in results : present") print(" the title : shown") print(" the snippet : shown, and it is the matched text") print(" what that reveals : the existence, the name, and the") print(" passage containing the searched term") print("") nc_stale_rules_in_flight = 0 nc_permission_lag_seconds = index_lag_seconds print("null control - a permission change re-indexes its documents") print(" median content lag, seconds : " + str(index_lag_seconds) + ", unchanged") print(" permission lag, seconds : " + str(nc_permission_lag_seconds)) print(" changes waiting : " + str(nc_stale_rules_in_flight)) print(" the pipeline did not get faster; the second field in") print(" the entry got a trigger of its own") print("") print("what a fresh index guarantees") print(" the content is current : exactly, and measured per document") print(" the entry is current : not addressed; an entry has more") print(" than one field and only one of them has a trigger") print("") print("a freshness metric measures the field its trigger fires on;") print("anything else copied into the same record ages silently, and") print("the copy exists precisely because reading it live was too") print("expensive") print("") print("The index is fresh and measured on every document: streamed writes, a " + str(index_lag_seconds) + " second") print("median, an alert above five that has fired " + str(freshness_alerts_this_year) + " times this year for real reasons.") print("Re-indexing is triggered by a content change, so with " + str(documents_reindexed_per_day) + " documents touched a") print("day out of " + str(documents) + ", a permission change waits about " + str(mean_days_a_permission_change_waits) + " days, roughly") print(str(stale_rules_in_flight) + " of them in flight, while the title and the matching snippet stay visible.") ``` ## stdout (executed) ```text documents : 41000000 index lag, seconds : 2 documents re-indexed per day : 240000 permission changes per day : 18400 days until a document is touched : 170 mean wait for a permission change: 85 days the indexing pipeline writes streamed, not batched : yes lag measured on every document : yes, not sampled median lag, seconds : 2 alert above five seconds : yes fired this year for real reasons : 2 verdict : FRESH measuring every document rather than a sample is the expensive choice and it was made deliberately the two things in an index entry the content : re-indexed when it changes, in 2 s the access rule : copied in at index time, so that a query can filter without a network call per hit what re-indexing is triggered by : a content change what a permission change triggers : nothing the copy is there for a good reason and the trigger was written for the other field permission changes waiting in the index : about 1564000 a reader whose access was revoked opening the document : refused, the service is current the document in results : present the title : shown the snippet : shown, and it is the matched text what that reveals : the existence, the name, and the passage containing the searched term null control - a permission change re-indexes its documents median content lag, seconds : 2, unchanged permission lag, seconds : 2 changes waiting : 0 the pipeline did not get faster; the second field in the entry got a trigger of its own what a fresh index guarantees the content is current : exactly, and measured per document the entry is current : not addressed; an entry has more than one field and only one of them has a trigger a freshness metric measures the field its trigger fires on; anything else copied into the same record ages silently, and the copy exists precisely because reading it live was too expensive The index is fresh and measured on every document: streamed writes, a 2 second median, an alert above five that has fired 2 times this year for real reasons. Re-indexing is triggered by a content change, so with 240000 documents touched a day out of 41000000, a permission change waits about 85 days, roughly 1564000 of them in flight, while the title and the matching snippet stay visible. ``` ## Round-trip `ok: true` — round-trip fixpoint reached (python1 == python2) ## Trace event types eml:run:start · eml:assign · eml:output · eml:run:done