Lower non-MoE, workspace-free gemm flavors off one build-time recipe

docs/python_graph_and_execution_backends.md

@@ -102,10 +102,16 @@ caller does not control.
 themselves, held in a C type (`pygraph/variant_pack.cpp`) as one `DLTensor`
 each. That type both consumes `__dlpack_c_exchange_api__` — the C function
 table a producer publishes on its type, which is how one crossing reads the
-whole pack — and implements it, so a kernel reads a slot through the same fast
-path it has for a framework tensor. `address` is the pointer array
+whole pack — and implements it, so a kernel reads an operand through the same
+fast path it has for a framework tensor. `address` is the pointer array
 `_execute_with_raw_ptrs` takes.
 
+The pack's vocabulary distinguishes the position from the thing at it:
+`pack.index_of(tensor_or_uid)` gives an operand's POSITION, and
+`pack.operands(indices)` turns positions into `OperandBuffer`s — one caller
+buffer described (pointer, shape, stride, dtype), non-owning. Resolve positions
+once; ask for buffers per call.
+
 Each operand's OWN dim/stride/data_type is what the pack holds, deliberately
 not the graph's declaration: the two may differ and one engine relies on it —
 `frost_gemm` takes its M/N/K from the buffers, so a plan built for one problem
@@ -139,6 +145,83 @@ set it still receives the caller's `{uid: buffer}` map and reaches ports through
 has moved. `execute()` builds the `Tensor`s only for a plan that sets the flag —
 measured, normalizing for a plan that will not read the result costs more than
 it saves.
+
+#### What a per-execute path costs
+
+An engine owns its internals, and this section does not change that. It exists
+because the default outcome is expensive: an engine that re-derives its per-call
+facts lands around **40 µs of host time per execute**, and for a single-kernel
+op that is most of what the caller pays. The same kernel with those facts read
+once is **20**. Both numbers are `frost_gemm` at 256×256×128 bf16, host
+enqueue, min over 25 reps of a 64-call burst from a drained queue.
+
+The budget it has to fit in, all measured on SM100:
+
+| | µs |
+|---|---|
+| `cuLaunchKernelEx`, untraced | 1.85 |
+| one CuTe-DSL entry | ~3.6 |
+| `graph.execute()` entry + `_normalize` | ~8 |
+| **everything else is the engine's** | |
+
+Do not read a per-call cost out of an nsys trace: CUPTI adds ~2.2 µs per traced
+API call, which is more than the call.
+
+**Split the facts by when they are decided.** Operand roles and majors, packing
+factors, alignment requirements, output shape rules, which outputs need a seed —
+all fixed when the kernel compiled. M/N/K, strides and pointers arrive per call.
+Read the first set into a table at build (`gemm/frost/recipe.py` is the worked
+example) and let the call read the table. That alone is 44 → 35.
+
+**Then lower the table into one closure per plan**, with its constants captured
+and the operand structure flattened into the loop headers, so the call does no
+attribute lookup and takes no branch the build already settled. That is 35 → 20.
+Two rules make it safe:
+
+- **The lowered path never raises, and it is the only path that runs.** What it
+  refuses it hands to a checker that reads the same table, names the rule and
+  raises — it launches nothing. A graph the closure cannot serve at all is
+  declined when the engine is asked to support it, so it goes to the backend
+  rather than to a second executor.
+- **So a refusal is the answer, not a slower route.** The set of calls the
+  closure refuses should equal the set of illegal calls; a legal call it will
+  not serve is a bug. Keeping a reference executor instead would buy a
+  differential that catches divergence but never a misconception the two share —
+  which is exactly how an axis-order bug survived one here. The tests that
+  matter are against intended semantics and against the BACKEND, at the shapes
+  where two encodings coincide.
+
+**A loop over a flat table gets almost all of it, so do not hand-unroll per
+flavor.** Measured three ways on the same plan and buffers: interpreting the
+table 35.8, looping over it flattened 19.7, a hand-written straight line with the
+structure unrolled 17.5. The loop is worth 45%; unrolling adds 12% and costs one
+closure body per operand shape — six flavors, six bodies to keep in agreement.
+One loop over `arg_plan` (the launch argument order as data) serves aux, extra
+outputs, multi-GEMM and block scale at 22 µs each, down from 39–50. Source
+codegen off the same table is how to buy the last 12% back later, for every
+flavor at once rather than for the one that was worth hand-writing.
+
+This is a pattern to copy, not a framework to import. Sharing the code across
+engines would couple their kernels' ABIs, which is the thing engine autonomy
+buys; sharing the shape of the solution costs nothing.
+
+**Costs that are easy to miss, each measured:**
+
+- A `from x import y` inside a per-call function: **1.1 µs**. It was 65% of
+  what `_check_plan_device` cost.
+- `torch.Tensor.permute()`: **1.4 µs** per call, per operand.
+- Rebuilding a `{id(tensor): buffer}` map to look operands back up, when the
+  operand order was settled at build and a list index would do.
+- Recomputing a pure function of values every call. `tensor_alignment`'s
+  layout half is **1.5 µs** and memoizes on `(shape, stride, elem_bytes)` —
+  values, so there is nothing to invalidate; only the pointer half is per call.
+- Reading an operand through the exchange vtable is **0.08 µs** against 1.5 for
+  the python attribute walk. Framework neutrality is not what costs.
+
+**Measure from a drained queue, and sweep the burst size.** A number that is
+flat in the burst size is host-bound; one that climbs with it is the device
+rate, and back-to-back timing reads the device rate whenever host and device
+are close.
 - **An engine does not propose its own plans.** Which configs to try, in what
   order, and where the backend's entries belong is one comparison across every
   candidate, and no engine can make it from the inside — it sees neither its

python/cudnn/_pygraph.py

@@ -1882,12 +1882,21 @@ def _normalize(self, uid_to_data: Dict[int, Any], workspace: Any, override_uids=
         # caller's mistake. A python-only graph's layout is every wired port,
         # including optional ones, where a hole means "not requested".
         strict = self._lowered_graph is not None
+        from_graph = []
         for i in unread:
             data = uid_to_data.get(order[i])
             if data is None:
                 continue  # named below if this graph requires it
+            if type(data) is int:
+                # A bare address has no geometry of its own, so _describe lends
+                # it the graph's -- including the graph's AXIS ORDER, which for
+                # a matmul's B is [batch, K, N] where a caller allocates
+                # (batch, N, K). Nothing in the resulting description says which
+                # of the two it is (at N == K the two are bit-identical), so the
+                # slot that borrowed one is named here.
+                from_graph.append(i)
             ptr, tensor = self._describe(data, order[i])
-            native.set_slot(i, ptr, tuple(tensor.dim), tuple(tensor.stride), *_dlpack_code_bits(tensor.data_type))
+            native.set_operand(i, ptr, tuple(tensor.dim), tuple(tensor.stride), *_dlpack_code_bits(tensor.data_type))
         if strict:
             hole = native.first_unfilled()
             if hole >= 0:
@@ -1908,7 +1917,7 @@ def _normalize(self, uid_to_data: Dict[int, Any], workspace: Any, override_uids=
                 i = slot_of.get(uid)
                 if i is None:
                     raise ValueError(f"override_uids names tensor uid {uid}, which is not an operand of this graph")
-                native.override_slot(i, *_in_axis_order_of(tuple(override_shapes[j]), tuple(override_strides[j]), native.stride(i)))
+                native.override_operand(i, *_in_axis_order_of(tuple(override_shapes[j]), tuple(override_strides[j]), native.stride(i)))
         # The workspace has no uid, so it is not an operand — but an engine has
         # to bounds-check its carves, and reading its size here is the same read
         # every other buffer gets rather than a second probe further down.
@@ -1924,7 +1933,7 @@ def _normalize(self, uid_to_data: Dict[int, Any], workspace: Any, override_uids=
                 workspace_bytes = _byte_size(workspace_tensor)
             else:
                 workspace_ptr, workspace_bytes = extent
-        return VariantPack(tuple(order), native, workspace_ptr, workspace_bytes)
+        return VariantPack(tuple(order), native, workspace_ptr, workspace_bytes, tuple(from_graph))
 
     def _describe(self, data: Any, uid: int):
         """``(pointer, Tensor)`` for one caller buffer.

python/cudnn/engines/base.py

@@ -133,32 +133,37 @@ class VariantPack:
     pointers — silently, because every pointer in it is individually valid.
     """
 
-    __slots__ = ("uids", "native", "_slot_of", "workspace", "workspace_bytes", "_device")
+    __slots__ = ("uids", "native", "_index_of", "workspace", "workspace_bytes", "_device", "graph_described")
 
-    def __init__(self, uids, native, workspace_ptr: int = 0, workspace_bytes: int = 0):
+    def __init__(self, uids, native, workspace_ptr: int = 0, workspace_bytes: int = 0, graph_described=()):
         self.uids = uids
         self.native = native
         self.workspace = workspace_ptr
         self.workspace_bytes = workspace_bytes
-        self._slot_of = None  # built on first lookup: the backend never does one
+        # Slots whose dim/stride were lent by the graph because the caller
+        # passed a bare address. Usually empty. An engine that reads extents by
+        # axis position needs this: the graph and the caller order a matmul's B
+        # differently, and the description does not say which one it is.
+        self.graph_described = graph_described
+        self._index_of = None  # built on first lookup: the backend never does one
         self._device = None
 
     @property
     def address(self) -> int:
-        """The ``void*[]`` in slot order, for ``_execute_with_raw_ptrs``."""
+        """The ``void*[]`` in operand order, for ``_execute_with_raw_ptrs``."""
         return self.native.address
 
     def all_contiguous(self):
-        """``(ok, slot)`` over every filled operand, decided from the strides
+        """``(ok, index)`` over every filled operand, decided from the strides
         the native pack already holds."""
         ok, offender = self.native.all_contiguous()
         return ok, (int(offender) if offender else -1)
 
     @property
-    def slot_of(self):
-        if self._slot_of is None:
-            self._slot_of = {u: i for i, u in enumerate(self.uids)}
-        return self._slot_of
+    def index_of_uid(self):
+        if self._index_of is None:
+            self._index_of = {u: i for i, u in enumerate(self.uids)}
+        return self._index_of
 
     @property
     def device(self) -> int:
@@ -173,20 +178,20 @@ def device(self) -> int:
             self._device = current_device()
         return self._device
 
-    def slot(self, tensor_or_uid) -> int:
+    def index_of(self, tensor_or_uid) -> int:
         """Index of a tensor's operand. KeyError when it is not caller-filled
         (a virtual intermediate, or a value the graph itself supplies)."""
         uid = tensor_or_uid if isinstance(tensor_or_uid, int) else tensor_or_uid.uid
-        return self.slot_of[uid]
+        return self.index_of_uid[uid]
 
     def ptr(self, tensor_or_uid) -> int:
-        return self.native.pointer(self.slot(tensor_or_uid))
+        return self.native.pointer(self.index_of(tensor_or_uid))
 
-    def views(self, slots):
-        """The DLPack producers for ``slots``, in one crossing."""
-        return self.native.views(list(slots), self.device)
+    def operands(self, indices):
+        """The buffers for ``indices``, in one crossing."""
+        return self.native.operands(list(indices), self.device)
 
-    def view(self, slot: int):
+    def operand(self, index: int):
         """A DLPack producer over one operand, for a kernel that needs an
         object rather than an address.
 
@@ -198,40 +203,40 @@ def view(self, slot: int):
         which is an argument for making the producer a C type, not for keeping
         the caller's object.
         """
-        return self.native.view(slot, self.device)
+        return self.native.operand(index, self.device)
 
     def __len__(self) -> int:
         return len(self.uids)
 
 
 @dataclass(frozen=True)
-class PortSlots:
-    """Per-node ``{port_name: slot}``. A port with no caller operand — a
+class PortIndices:
+    """Per-node ``{port_name: operand index}``. A port with no caller operand — a
     virtual intermediate — is ABSENT, so ``.get(port) is None`` keeps meaning
     what it meant when these were buffers."""
 
     inputs: Dict[str, int]
     outputs: Dict[str, int]
 
 
-def bind_ports(graph: "pygraph", variant_pack: VariantPack) -> Dict[Any, PortSlots]:
+def bind_ports(graph: "pygraph", variant_pack: VariantPack) -> Dict[Any, PortIndices]:
     """Join each node's wired ports with the operand layout. Strict: every
     non-virtual port must have an operand."""
 
     def resolve(node, ports, direction):
-        slots = {}
+        indices = {}
         for port, t in ports.items():
             if t is None:
                 continue
-            slot = variant_pack.slot_of.get(t.uid)
-            if slot is None:
+            index = variant_pack.index_of_uid.get(t.uid)
+            if index is None:
                 if t.is_virtual:
                     continue  # engine-internal intermediate
                 raise ValueError(f"node {node.name!r}: no buffer for {direction} port {port!r} (tensor {t.name!r})")
-            slots[port] = slot
-        return slots
+            indices[port] = index
+        return indices
 
-    return {node: PortSlots(resolve(node, node.inputs, "input"), resolve(node, node.outputs, "output")) for node in graph.nodes}
+    return {node: PortIndices(resolve(node, node.inputs, "input"), resolve(node, node.outputs, "output")) for node in graph.nodes}
 
 
 def _view_over_address(address: int, tensor, node, port: str):

python/cudnn/frost/README.md

@@ -235,10 +235,31 @@ CompiledPlan.execute(graph, uid_to_data, ctx)          (the hot path)
 - **`build_plan` runs once per (graph, plan)** at `build_plans()` time and the
   compiled artifact lives on the graph, so one engine instance is safely
   reusable across graphs.
+- **What a runtime value cannot change belongs in a build-time table.** An
+  operand's role and major, each output's shape rule and required alignment,
+  which outputs are reductions -- all settled when the kernel compiled, and
+  deciding them again per call is most of what a python execute path costs
+  (measured: 40-50 -> 20-22 us for one gemm, across six flavors).
+  `gemm/frost/recipe.py` is the worked example: one table, captured into a
+  closure that loops over it flat. Even what the kernel's parameter list looks
+  like is a table entry (`arg_plan`), which is why one loop serves every flavor
+  -- a call path per flavor is how two of them disagree. That closure never
+  raises and it is the ONLY thing that launches: what it refuses goes to a
+  checker that reads the same table, names the rule and raises without running
+  anything, and a graph it cannot serve at all is declined at `check_support`.
+  A second executor kept for diagnostics is still a second answer to what the
+  graph computes, and a differential between two readings of one plan cannot
+  catch a misconception they share.
 - **`ExecutionContext` carries handle, stream and workspace explicitly.** No
   engine may hard-code a stream, reach into private graph state, or allocate
   hidden workspace. `uid_to_data` is the caller's variant pack (tensor uid ->
   device buffer), exactly as the classic backend receives it.
+- **The pack's vocabulary is `index` and `OperandBuffer`,** and the two are not
+  the same thing. `pack.index_of(tensor_or_uid)` gives an operand's POSITION in
+  the pack; `pack.operands(indices)` turns positions into `OperandBuffer`s --
+  one caller buffer described (pointer, shape, stride, dtype), non-owning, and
+  itself a DLPack producer. An engine resolves positions once at first execute
+  and asks for buffers per call.
 
 `python/cudnn/gemm/frost/engine.py` is the worked example, deliberately thin:
 `check_support` delegates to `probe_supported` and

python/cudnn/frost/buffers.py

@@ -180,7 +180,7 @@ def __dlpack__(self, *, stream=None, **_kwargs):
         is a use-after-free the moment a consumer outlives the view.
         """
         code, bits = DTYPES[self.dtype]
-        return _pybind_module.make_slot(self._ptr, list(self.shape), code, bits, self._device_id).__dlpack__()
+        return _pybind_module.make_operand_buffer(self._ptr, list(self.shape), code, bits, self._device_id).__dlpack__()
 
 
 class DeviceBuffer(DeviceView):

python/cudnn/frost/workspace.py

@@ -152,7 +152,7 @@ def view(self, offset: int, dtype: str, shape):
             count *= int(extent)
         self._check_span(offset, count * buffers.DTYPE_ITEMSIZE[dtype])
         code, bits = buffers.DTYPES[dtype]
-        return _pybind_module.make_slot(self._ptr + offset, list(shape), code, bits, self._device)
+        return _pybind_module.make_operand_buffer(self._ptr + offset, list(shape), code, bits, self._device)
 
     def carve(self, plan):
         """Every region a :func:`carve_plan` describes, in one crossing."""