# Example 600 — The cache was warm and one request paid for all of them `the_cache_was_warm_and_one_request_paid_for_all_of_them.eml` - A computed page is cached with a sixty second expiry. The hit rate is above ninety-nine percent and mean latency is a few milliseconds. Who pays the recompute is computed below. ## EML ```eml # Self-authored for the EML case corpus (no external origin). A computed page # is cached with a sixty second expiry. The hit rate is above ninety-nine # percent and mean latency is a few milliseconds. Who pays the recompute is # computed below. # # The cache is correct and the design is the standard one. A sixty second entry # keeps the page fresh enough for what it shows, single-flight means one # recompute rather than a stampede, and the backend sees a tiny fraction of the # load it would otherwise. Every number the cache reports about itself is true # and every one of them is good. # # A hit rate is a statement about requests. Latency spent is a statement about # whoever was holding the request. The cache averages the first and assigns the # second, and averaging and assigning are not the same operation. # # The recompute does not get cheaper because it is rare. It gets rarer. 5000 => requests_per_minute 60 => ttl_seconds 1 => recomputes_per_expiry 4 => hit_ms 3200 => miss_ms requests_per_minute - recomputes_per_expiry => hits_per_minute "requests per minute : " + str(requests_per_minute) ^0 "entry lifetime : " + str(ttl_seconds) + " seconds" ^0 "recomputes per minute : " + str(recomputes_per_expiry) + ", single-flight" ^0 "a hit costs : " + str(hit_ms) + " ms" ^0 "a miss costs : " + str(miss_ms) + " ms" ^0 "" ^0 # ---- what the cache reports ---- int(hits_per_minute * 10000 / requests_per_minute) => hit_rate_per_myriad hits_per_minute * hit_ms => hit_ms_total recomputes_per_expiry * miss_ms => miss_ms_total hit_ms_total + miss_ms_total => total_ms int(total_ms * 100 / requests_per_minute) => mean_ms_hundredths "the cache's own numbers" ^0 " hits per minute : " + str(hits_per_minute) ^0 " misses : " + str(recomputes_per_expiry) ^0 " hit rate : " + str(hit_rate_per_myriad) + " per ten thousand" ^0 " mean latency : " + str(int(mean_ms_hundredths / 100)) + " point " + str(mean_ms_hundredths % 100) + " ms" ^0 " backend load : " + str(recomputes_per_expiry) + " of " + str(requests_per_minute) ^0 "" ^0 " every one of those is true and every one is good" ^0 "" ^0 # ---- the request that paid ---- int(miss_ms / hit_ms) => victim_ratio "the one request that missed" ^0 " its latency : " + str(miss_ms) + " ms" ^0 " a hit's latency : " + str(hit_ms) + " ms" ^0 " ratio : " + str(victim_ratio) + " times" ^0 " ratio to the mean : " + str(int(miss_ms * 100 / mean_ms_hundredths)) + " times" ^0 "" ^0 60 * 24 => minutes_per_day minutes_per_day * recomputes_per_expiry => victims_per_day " victims per hour : " + str(60 * recomputes_per_expiry) ^0 " victims per day : " + str(victims_per_day) ^0 " each of them a real person waiting " + str(miss_ms) + " ms" ^0 "" ^0 # ---- the same minute, from two directions ---- "minute requests hits misses ms spent on hits ms spent on the miss" ^0 for m in [1:4]: " " + str(m) + " " + str(requests_per_minute) + " " + str(hits_per_minute) + " " + str(recomputes_per_expiry) + " " + str(hit_ms_total) + " " + str(miss_ms_total) ^0 "" ^0 int(miss_ms_total * 1000 / total_ms) => miss_share_per_mille " share of all latency spent by " + str(recomputes_per_expiry) + " request of " + str(requests_per_minute) + " : " + str(miss_share_per_mille) + " per mille" ^0 "" ^0 # ---- what the percentiles say ---- "where the victim lands in the distribution" ^0 " p50 : " + str(hit_ms) + " ms" ^0 " p95 : " + str(hit_ms) + " ms" ^0 " p99 : " + str(hit_ms) + " ms" ^0 " max : " + str(miss_ms) + " ms" ^0 " the miss is " + str(int(recomputes_per_expiry * 10000 / requests_per_minute)) + " per ten thousand of requests, so no percentile" ^0 " below the very top can contain it" ^0 "" ^0 # ---- the control ---- # # The cache, against what it was put there to do. It was put there to keep the # backend from computing the page five thousand times a minute, and it has # reduced that to one. int(requests_per_minute / recomputes_per_expiry) => backend_reduction "control - is the cache doing its job" ^0 " backend computations without it : " + str(requests_per_minute) + " per minute" ^0 " backend computations with it : " + str(recomputes_per_expiry) + " per minute" ^0 " reduction : " + str(backend_reduction) + " times" ^0 " stampedes : 0, single-flight holds" ^0 " defects in the cache : 0" ^0 "" ^0 " removing the cache makes every request a " + str(miss_ms) + " ms request" ^0 "" ^0 # ---- the null control ---- # # The same cache, same TTL, same hit rate, refreshed in the background just # before the entry expires. Nothing about the caching changes. The recompute # still happens once a minute; it is no longer inside anybody's request. 0 => nc_victims_per_day "null control - the same cache refreshed ahead of expiry" ^0 " recomputes per minute : " + str(recomputes_per_expiry) ^0 " hit rate : " + str(hit_rate_per_myriad) + " per ten thousand, unchanged" ^0 " backend load : unchanged" ^0 " requests waiting " + str(miss_ms) + " ms : " + str(nc_victims_per_day) + " per day" ^0 " the work did not move or shrink; it left the request path" ^0 "" ^0 # ---- the rule ---- "what a hit rate averages and what it does not" ^0 " cost across requests : averaged, and the average is honest" ^0 " cost to a request : assigned, in full, to one of them" ^0 " and the assignment has no term in the hit rate" ^0 "" ^0 "a rare expensive path is not a small cost spread thin" ^0 "it is a full cost handed to somebody, on a schedule" ^0 "" ^0 "The cache turns " + str(requests_per_minute) + " backend computations a minute into " + str(recomputes_per_expiry) + ", a factor of" ^0 str(backend_reduction) + ", with 0 stampedes and a hit rate of " + str(hit_rate_per_myriad) + " per ten thousand. Mean" ^0 "latency reads " + str(int(mean_ms_hundredths / 100)) + " point " + str(mean_ms_hundredths % 100) + " ms because " + str(miss_share_per_mille) + " per mille of all latency is spent" ^0 "by " + str(recomputes_per_expiry) + " request in " + str(requests_per_minute) + ", and that request waits " + str(miss_ms) + " ms - " + str(victim_ratio) + " times a hit -" ^0 str(victims_per_day) + " times a day." ^0 ``` ## Python (deterministic transpilation) ```python requests_per_minute = 5000 ttl_seconds = 60 recomputes_per_expiry = 1 hit_ms = 4 miss_ms = 3200 hits_per_minute = requests_per_minute - recomputes_per_expiry print("requests per minute : " + str(requests_per_minute)) print("entry lifetime : " + str(ttl_seconds) + " seconds") print("recomputes per minute : " + str(recomputes_per_expiry) + ", single-flight") print("a hit costs : " + str(hit_ms) + " ms") print("a miss costs : " + str(miss_ms) + " ms") print("") hit_rate_per_myriad = int(hits_per_minute * 10000 / requests_per_minute) hit_ms_total = hits_per_minute * hit_ms miss_ms_total = recomputes_per_expiry * miss_ms total_ms = hit_ms_total + miss_ms_total mean_ms_hundredths = int(total_ms * 100 / requests_per_minute) print("the cache's own numbers") print(" hits per minute : " + str(hits_per_minute)) print(" misses : " + str(recomputes_per_expiry)) print(" hit rate : " + str(hit_rate_per_myriad) + " per ten thousand") print(" mean latency : " + str(int(mean_ms_hundredths / 100)) + " point " + str(mean_ms_hundredths % 100) + " ms") print(" backend load : " + str(recomputes_per_expiry) + " of " + str(requests_per_minute)) print("") print(" every one of those is true and every one is good") print("") victim_ratio = int(miss_ms / hit_ms) print("the one request that missed") print(" its latency : " + str(miss_ms) + " ms") print(" a hit's latency : " + str(hit_ms) + " ms") print(" ratio : " + str(victim_ratio) + " times") print(" ratio to the mean : " + str(int(miss_ms * 100 / mean_ms_hundredths)) + " times") print("") minutes_per_day = 60 * 24 victims_per_day = minutes_per_day * recomputes_per_expiry print(" victims per hour : " + str(60 * recomputes_per_expiry)) print(" victims per day : " + str(victims_per_day)) print(" each of them a real person waiting " + str(miss_ms) + " ms") print("") print("minute requests hits misses ms spent on hits ms spent on the miss") for m in range(1, 5): print(" " + str(m) + " " + str(requests_per_minute) + " " + str(hits_per_minute) + " " + str(recomputes_per_expiry) + " " + str(hit_ms_total) + " " + str(miss_ms_total)) print("") miss_share_per_mille = int(miss_ms_total * 1000 / total_ms) print(" share of all latency spent by " + str(recomputes_per_expiry) + " request of " + str(requests_per_minute) + " : " + str(miss_share_per_mille) + " per mille") print("") print("where the victim lands in the distribution") print(" p50 : " + str(hit_ms) + " ms") print(" p95 : " + str(hit_ms) + " ms") print(" p99 : " + str(hit_ms) + " ms") print(" max : " + str(miss_ms) + " ms") print(" the miss is " + str(int(recomputes_per_expiry * 10000 / requests_per_minute)) + " per ten thousand of requests, so no percentile") print(" below the very top can contain it") print("") backend_reduction = int(requests_per_minute / recomputes_per_expiry) print("control - is the cache doing its job") print(" backend computations without it : " + str(requests_per_minute) + " per minute") print(" backend computations with it : " + str(recomputes_per_expiry) + " per minute") print(" reduction : " + str(backend_reduction) + " times") print(" stampedes : 0, single-flight holds") print(" defects in the cache : 0") print("") print(" removing the cache makes every request a " + str(miss_ms) + " ms request") print("") nc_victims_per_day = 0 print("null control - the same cache refreshed ahead of expiry") print(" recomputes per minute : " + str(recomputes_per_expiry)) print(" hit rate : " + str(hit_rate_per_myriad) + " per ten thousand, unchanged") print(" backend load : unchanged") print(" requests waiting " + str(miss_ms) + " ms : " + str(nc_victims_per_day) + " per day") print(" the work did not move or shrink; it left the request path") print("") print("what a hit rate averages and what it does not") print(" cost across requests : averaged, and the average is honest") print(" cost to a request : assigned, in full, to one of them") print(" and the assignment has no term in the hit rate") print("") print("a rare expensive path is not a small cost spread thin") print("it is a full cost handed to somebody, on a schedule") print("") print("The cache turns " + str(requests_per_minute) + " backend computations a minute into " + str(recomputes_per_expiry) + ", a factor of") print(str(backend_reduction) + ", with 0 stampedes and a hit rate of " + str(hit_rate_per_myriad) + " per ten thousand. Mean") print("latency reads " + str(int(mean_ms_hundredths / 100)) + " point " + str(mean_ms_hundredths % 100) + " ms because " + str(miss_share_per_mille) + " per mille of all latency is spent") print("by " + str(recomputes_per_expiry) + " request in " + str(requests_per_minute) + ", and that request waits " + str(miss_ms) + " ms - " + str(victim_ratio) + " times a hit -") print(str(victims_per_day) + " times a day.") ``` ## stdout (executed) ```text requests per minute : 5000 entry lifetime : 60 seconds recomputes per minute : 1, single-flight a hit costs : 4 ms a miss costs : 3200 ms the cache's own numbers hits per minute : 4999 misses : 1 hit rate : 9998 per ten thousand mean latency : 4 point 63 ms backend load : 1 of 5000 every one of those is true and every one is good the one request that missed its latency : 3200 ms a hit's latency : 4 ms ratio : 800 times ratio to the mean : 691 times victims per hour : 60 victims per day : 1440 each of them a real person waiting 3200 ms minute requests hits misses ms spent on hits ms spent on the miss 1 5000 4999 1 19996 3200 2 5000 4999 1 19996 3200 3 5000 4999 1 19996 3200 4 5000 4999 1 19996 3200 share of all latency spent by 1 request of 5000 : 137 per mille where the victim lands in the distribution p50 : 4 ms p95 : 4 ms p99 : 4 ms max : 3200 ms the miss is 2 per ten thousand of requests, so no percentile below the very top can contain it control - is the cache doing its job backend computations without it : 5000 per minute backend computations with it : 1 per minute reduction : 5000 times stampedes : 0, single-flight holds defects in the cache : 0 removing the cache makes every request a 3200 ms request null control - the same cache refreshed ahead of expiry recomputes per minute : 1 hit rate : 9998 per ten thousand, unchanged backend load : unchanged requests waiting 3200 ms : 0 per day the work did not move or shrink; it left the request path what a hit rate averages and what it does not cost across requests : averaged, and the average is honest cost to a request : assigned, in full, to one of them and the assignment has no term in the hit rate a rare expensive path is not a small cost spread thin it is a full cost handed to somebody, on a schedule The cache turns 5000 backend computations a minute into 1, a factor of 5000, with 0 stampedes and a hit rate of 9998 per ten thousand. Mean latency reads 4 point 63 ms because 137 per mille of all latency is spent by 1 request in 5000, and that request waits 3200 ms - 800 times a hit - 1440 times a day. ``` ## Round-trip `ok: true` — round-trip fixpoint reached (python1 == python2) ## Trace event types eml:run:start · eml:assign · eml:output · eml:run:done