attention

std.nn.attention · Level L0

Scaled dot-product attention for one head.

softmax(Q·Kᵀ·s)·V

Signature

attention(Q: f64[m, d], K: f64[t, d], V: f64[t, e], scale: f64[]) → f64[m, e]

Structure

The function as NOVA stores it: one box per input, operation and output, and arrows that carry values. A double border marks another library function this one runs — called once, or by Scan once per element; select it to open that function.

Qf64[m, d]Kf64[t, d]scalef64[]Vf64[t, e]TransposeKtMatMulSMultiplySsSoftmaxPwMatMulOOf64[m, e]
  • input
  • operation
  • constant
  • call
  • output

Verification

  • Signature proven by NOVA’s shape solver, for every size.
  • Agrees with the reference softmax((Q @ K.T) * scale, axis=-1) @ V to 80 digits (100-digit arithmetic), on all 40 test cases.
  • All 680 float64 results inside the running error bound; the closest uses 23% of it.
  • Interpreter and NumPy backend return bit-identical results.
  • At float32 too: the 40 test cases rounded to float32, and 40 of them inside the running error bound at u = 2⁻²⁴; the closest uses 21% of it.
Accuracy in detail
correctly rounded (the float64 nearest the exact value)
54%
largest error, in units in the last place
175

Large ulp counts appear where a result is tiny next to the numbers it is computed from (after cancellation, for example), so one unit in the last place is tiny too; the absolute error is still inside the bound. Results within their own error of zero are not counted.

Note

The scale is an input (usually 1/√d): a square root of a symbolic size needs the Size and Sqrt primitives (Round 13.1).

Identity

Calls
—
Called by
sha256:26f7c94b7a02cfa34fdee56bb84e80ff4e812054a5492981eb23a2eab4a5330a

The semantic hash of the graph. It changes when the program changes, and never when only its documentation does.

Control handle

Symbol
Ω:std.nn.attention · Ω:attention
Pins
sha256:12bb0b195972424f7a07e4c178baf8bb214f0a8895a63205026ee95628db7e74this graph alone
Evidence
sha256:2a66a218044726076c03237f349ea3ae91d2b1fcecc02b2e49f6334865fd9c8athe hash of its verification record
Needs
no capability: a pure function

Through NOVA’s control layer, the symbol launches this function only while the program still matches what it pins: a change to this graph, or to any graph it reaches, needs a migration first.

ΣEML 2026

Efficient New Language — a high-density semantic overlay for humans and AI agents.

This site’s demos run on the real EML toolchain, in your browser.

© 2026 EveMissLab (一言諾科技有限公司) / Neo.K. All rights reserved.