Skip to content

Grammar API - #627

Closed
mhayes853 wants to merge 20 commits into
cactus-compute:mainfrom
mhayes853:initial-grammar
Closed

mhayes853 wants to merge 20 commits into
cactus-compute:mainfrom
mhayes853:initial-grammar

Conversation

@mhayes853

Copy link
Copy Markdown
Contributor

Introduces a Grammar API to enable structured generation in coming PRs.

For now, this isn't directly integrated with completion, but rather exposes all the necessary APIs to make that happen in the near future. The underlying state machine used is XGrammar, and for the most part the implementation directly wraps it.

There are 4 essential types:

  • Grammar an actual representation of a grammar, with conveniences for plain JSON, JSON schema, regex, empty, universal, structural tags, etc.
    • Also supports mathematical operations such as unioning and concatenating, as well as getting the underlying EBNF string.
  • GrammarMatcher the state machine type responsible for computing bitmasks.
    • accept and next_bitmask do most of the driving. However, there are also methods to rollback, and to check the termination state.
    • fork is an efficient way to create a copy of matcher (at it's current state) without having to go through GrammarEngine.
  • GrammarEngine responsible for compiling GrammarMatcher instances from a Grammar instance.
    • Under the hood, this type wraps both GrammarCompiler and CompilerGrammar from XGrammar. I felt it was a conceptually simpler API to keep these concepts bound together.
      • I also felt exposing the concept of a "compiled" grammar was a bit unecessary since it's only purpose is to be used in matcher construction, and the matcher already has fork which can be used for efficient copies.
    • The class exposes a singular compile_matcher method which takes a Grammar instance and returns a GrammarMatcher.
  • GrammarVocabulary tokenizer info required by the other grammar types.
    • This type merely wraps TokenizerInfo from XGrammar, there is also now a new tokenizer method to produce it.
    • At the FFI level, this is constructed from a model instance directly.

Among other things, I also extracted a helper for constructing the tokenizer from the model path. This was mainly so that I could use it directly for testing without loading model weights.

mhayes853 and others added 20 commits May 4, 2026 20:45
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
…ute#608)

* one line fix

* forgot i wasnt coding in cpp

Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* remaining ops

* fixed case with permute that serialization overlooked

* refactoring

* updated readme with documentation on end to end op contributing

Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* Add fused dense INT4 MLP op for Gemma-4 E2B decode

Fuses gate -> GeLU(tanh) -> up -> multiply -> requantize -> down into a
single thread-pool dispatch. Workers steal N-block tiles of d_ffn via
fetch_add and run the full chain end-to-end on their tile, accumulating
the down contribution into per-worker fp32 buffers that are reduced to
fp16 after the dispatch. Replaces the 5-barrier dense MLP chain.

On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 37 tok/s -> 44
tok/s (+19%). Default-on for INT4 INT8-grouped FFN weights with
group_size > 0; CACTUS_DENSE_MLP_FUSED=0 disables.

Per-tile activation rescaling (h = gate * up is requantized per tile)
slightly improves INT4 down-proj numerical fidelity vs a single global
h_scale.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Refactor fused dense INT4 MLP for bit-equal output and kernel reuse

Restructure the fused op as two phases with a single global h_scale:
phase 1 workers compute gate -> GeLU -> up -> multiply into a shared
fp16 h_full[d_ffn]; single-threaded max-abs and INT8 quantization
between phases; phase 2 workers do the down projection partitioned
by hidden_dim N-block tiles, writing directly to the output. Uses
cactus_gemv_int4_block_range three times (gate, up, down) and
eliminates the per-worker fp32 down accumulator and reduction.

Also refactor cactus_gemv_int4 to delegate its inner work to
cactus_gemv_int4_block_range with accumulate=false, removing ~120
lines of duplicated body code, and drop the now-unused
cactus_gemv_int4_st_kslice_fp32acc kernel.

On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 45-46 tok/s
fused vs 37 tok/s unfused (+22-24%), up from the previous 44 tok/s
per-tile-h_scale variant. The contiguous N-tile access pattern in
phase 2 (each worker reads small N-range x full K of down_w) is
more cache-friendly than the previous striped K-slice pattern.

Output is bit-equivalent to the unfused chain modulo floating-point
reduction-order non-determinism inherent to multi-threaded
inference; both paths show the same run-to-run divergence behavior
under greedy decoding.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Revert cactus_gemv_int4 to its original implementation

The previous refactor had cactus_gemv_int4 delegate to the new
cactus_gemv_int4_block_range helper. Functionally equivalent, but
modifies an existing kernel which is out of scope for this change
set. Restore the original body verbatim from main.

cactus_gemv_int4_block_range remains as a new function used only
by the new fused dense INT4 MLP op. Net diff vs main is now pure
additions (no modifications to existing kernels). Fused-MLP perf
unchanged at 45-46 tok/s (Gemma-4 E2B INT4, M4 Pro, 256-token
decode window) since the fused op calls the new helper directly,
not cactus_gemv_int4.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Match canonical kernel math in fused dense MLP without parallel_for cost

The phase-1 GELU and tile-multiply, and the once-per-layer h-vector
INT8 quantization, are now inline mirrors of cactus_gelu_f16,
cactus_multiply_f16, and cactus_fp16_to_int8 respectively. Same
SIMD bodies, same Eigen-style fast_tanh_f32x4 polynomial for the
GELU, same vmulq_f16 + scalar tail for the multiply, same
vcvtnq_s32_f32 + vqmovn quantize chain — but called inline so each
per-tile invocation skips the parallel_for setup and the
once-per-layer h_full -> h_int8 pass skips the parallel_for
barrier between phase 1 and phase 2.

Fixes a subtle math divergence in the previous homebrew GELU
helper: it used a Pade-style (z(27+z^2))/(27+9z^2) tanh
approximation clamped at +/-4.5, while cactus_gelu_f16 uses the
Eigen-style degree-6/3 polynomial clamped at +/-9.0. Output is
now bit-equivalent (modulo float reduction-order non-determinism
inherent to multi-threaded inference) to the unfused chain's
gb->gelu(matmul(...)) -> gb->multiply(...) -> gb->matmul(mul, ...)
pipeline, since both paths now flow through the exact same
arithmetic.

On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 43-45
tok/s fused on (median 44) vs 35-36 tok/s fused off (+22-26%).

Net diff vs main is still pure additions (no modifications to
existing code paths).

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Trim dead code from fused MLP path

Three small minimization passes that don't touch perf:

* cactus_gemv_int4_block_range: drop the unused accumulate flag (all
  three call sites in the fused op pass false) and the partial
  actual_n != 4 path (the op asserts hidden_dim % 4 == 0 and
  d_ffn % 4 == 0, so the fallback never fires for this kernel's only
  caller).

* compute_dense_mlp_int4_fused_node: factor the identical phase 1 and
  phase 2 thread-pool dispatch boilerplate (if num_threads <= 1,
  enqueue_n_threads + worker_task(0) + wait_all) into a single local
  lambda invoked twice.

Steady-state perf unchanged at 43-45 tok/s (Gemma-4 E2B INT4, M4 Pro,
256-token decode). Net diff vs main is now +373 / 0 (was +419), still
pure additions with no modifications to existing code paths.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Drop inline mirror helpers in fused MLP, use canonical kernels

The phase-1 GELU and tile-multiply, and the once-per-layer h-vector
INT8 quantization, now call cactus_gelu_f16, cactus_multiply_f16,
and cactus_fp16_to_int8 directly. The inline mirrors of those
kernels' SIMD bodies were tested empirically against the canonical
helpers on this hardware:

  * cactus_gelu_f16 / cactus_multiply_f16 with k_count=128 fall
    through parallel_for's threshold check and run inline anyway,
    so the inline mirrors saved nothing measurable.
  * cactus_fp16_to_int8 with d_ffn=12288 elements does parallelize,
    and the inline mirror saved ~1 tok/s by skipping its barrier.
    But that's within run-to-run noise on the 256-token bench, the
    magnitude depends on parallel_for cost (which varies per
    device), and ~22 lines of duplicated SIMD code is a poor trade
    when the savings don't generalize uniformly across smaller
    thread pools (e.g. mid-range Android phones).

Output is still bit-equivalent to the unfused chain (modulo
multi-threaded reduction-order non-determinism), since both paths
now flow through the exact same canonical kernels.

On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 42-43 tok/s
fused on vs 36 tok/s fused off (+17-19%).

Net diff vs main is now +306 / 0 (was +373), still pure additions
with no modifications to existing code paths.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Restore inline INT8 quantize for the once-per-layer h vector

cactus_fp16_to_int8 uses parallel_for with the ELEMENT_WISE config
({5000, 2500} on Android, same threshold-wise on Apple). At
d_ffn=12288 this dispatches ~5 worker threads, whose orchestration
cost (mutex acquire on enqueue, futex wakeups, std::future::wait
serialization, straggler waits if a slice lands on an A520-class
little-core) typically dominates the few-microsecond SIMD work.

Running the quantize inline on the dispatch thread skips that
overhead. The benefit applies on any device with non-trivial
thread-pool wakeup cost; on Android big.LITTLE configurations the
gap is wider than on Apple silicon because heterogeneous cores
amplify straggler-wait time.

The previous commit removed this helper along with the gelu/
multiply mirrors, but those latter two operate on k_count=128
which falls below SCALAR_EXPENSIVE/ELEMENT_WISE gates and
fast-paths to inline anyway -- correctly removed. The quantize
case is the only one of the three where parallel_for genuinely
dispatches and the inline mirror buys real time.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Trim comments in fused MLP

Reduce the long parallel_for-overhead explainer above
fp16_to_int8_inplace to a two-line WHY. No code changes.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Rename fused-MLP env var to a disable-style flag

CACTUS_DENSE_MLP_FUSED=0 (the previous escape hatch) inverted the
natural reading of the env var name -- "set to 0 to disable a
thing called *fused*" requires the reader to know the default.

Replace with CACTUS_DISABLE_DENSE_MLP_FUSED: any non-empty,
non-"0" value disables the fused path. Unset / "0" / empty leave
the default-on behavior untouched. Reads naturally and matches
the CACTUS_IGNORE_EOS-style convention already used elsewhere.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Address Copilot review: stable enum value + per-call scratch

* Move DENSE_MLP_INT4_FUSED to the end of OpType (and of
  op_type_names) so existing enum values aren't shifted. graph_io.cpp
  serializes OpType as raw uint32_t, so inserting in the middle would
  have broken compatibility for any persisted graph (or future
  cross-build scenarios).

* Replace the namespace-static dense_fused_scratch vector with a
  function-local std::vector<DenseFusedScratch>(num_workers)
  constructed per call. The previous static was concurrency-unsafe if
  two graphs ever executed this op concurrently (workers would race
  on the same shared buffers). Per-call construction is ~1-3 us with
  the freelist warm; net perf unchanged (44-45 tok/s).

The other four Copilot comments were either unreachable in practice
(group_size / d_ffn / hidden_dim alignment is always satisfied for
Gemma-4 weights, single internal call site for the kernel) or
pre-existing in main (op_type_names ordering for unrelated ops).

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

---------

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* inner 1 fast path

* added tests

Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
@mhayes853 mhayes853 closed this May 5, 2026
@mhayes853
mhayes853 deleted the initial-grammar branch May 5, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants