What the graphs are made of.
Every library function is composed from these primitives. The last column counts the functions that use each one directly.
| Primitive | What it does | Inputs | Differentiable | Used by |
|---|---|---|---|---|
| Constant | A literal value inside the graph. | 0 | yes | 4 |
| Add | Elementwise sum, NumPy-style broadcasting. | 2 | yes | 7 |
| Subtract | Elementwise difference, broadcasting. | 2 | yes | 11 |
| Multiply | Elementwise product, broadcasting. | 2 | yes | 18 |
| Divide | Elementwise quotient, broadcasting. | 2 | yes | 3 |
| Negate | Elementwise negation. | 1 | yes | 2 |
| MatMul | Matrix product over the last two axes; leading axes broadcast. | 2 | yes | 6 |
| Transpose | Permute the axes. | 1 | yes | 3 |
| Reshape | Same elements, new shape; one −1 is inferred. | 1 | yes | 4 |
| ReduceSum | Sum along one axis, or over everything. | 1 | yes | 4 |
| Mean | Mean along one axis, or over everything. | 1 | yes | 7 |
| Relu | max(x, 0), elementwise. | 1 | yes | 4 |
| Sigmoid | 1 / (1 + e⁻ˣ), elementwise. | 1 | yes | 1 |
| Tanh | Hyperbolic tangent, elementwise. | 1 | yes | 1 |
| Softmax | Normalised exponentials along one axis. | 1 | yes | 1 |
| Identity | Pass a value through unchanged. | 1 | yes | — |
| StopGradient | Identity going forward, zero gradient going back. | 1 | yes | — |
| If | Choose between two values by a scalar condition. | 3 | not yet | — |
| BoundedLoop | A loop with a fixed bound; for now its body is one elementwise operation. | 1 | not yet | — |
| Call | Call another graph of the same module. This is how the library composes. | any | not yet | 11 |
The next primitives
Standard mathematics only. Each line lists what it unlocks.
- L1Sqrt, Exp, Log
Elementwise square root, exponential and logarithm.
Unlocks: norm, std, logsumexp, log_softmax, entropy, softplus, 1/√d in attention
- L1Maximum, Minimum
Exact elementwise max and min (a + relu(b − a) is not exact in floating point).
Unlocks: clip, relu6, hard_sigmoid, huber
- L1Less, Greater, Equal, Where
Elementwise comparison and selection.
Unlocks: masks, piecewise functions, sign
- L1ReduceMax, ReduceMin
Largest and smallest value along an axis.
Unlocks: stable logsumexp, max pooling, argmax (with Where)
- L1Size
The element count of a symbolic dimension, as a value.
Unlocks: sample variance and covariance (n − 1), covariance matrix
- L1Differentiation through Call
Reverse-mode AD that follows Call into the callee; today composed entries are not differentiable.
Unlocks: gradients of every composed entry (mlp2, bilinear, …)
- L2Slice, Concat, Gather
Indexing and joining along an axis.
Unlocks: finite differences, trapezoid rule, convolution, polygon area
- L2Scan
A general loop whose body is a graph.
Unlocks: iterative solvers, recurrences, cumulative sums