Repository navigation
Grammar API - #627
Closed
mhayes853 wants to merge 20 commits into
Closed
Grammar API#627mhayes853 wants to merge 20 commits into
mhayes853 wants to merge 20 commits into
Conversation
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
…ute#608) * one line fix * forgot i wasnt coding in cpp Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* remaining ops * fixed case with permute that serialization overlooked * refactoring * updated readme with documentation on end to end op contributing Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* Add fused dense INT4 MLP op for Gemma-4 E2B decode
Fuses gate -> GeLU(tanh) -> up -> multiply -> requantize -> down into a
single thread-pool dispatch. Workers steal N-block tiles of d_ffn via
fetch_add and run the full chain end-to-end on their tile, accumulating
the down contribution into per-worker fp32 buffers that are reduced to
fp16 after the dispatch. Replaces the 5-barrier dense MLP chain.
On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 37 tok/s -> 44
tok/s (+19%). Default-on for INT4 INT8-grouped FFN weights with
group_size > 0; CACTUS_DENSE_MLP_FUSED=0 disables.
Per-tile activation rescaling (h = gate * up is requantized per tile)
slightly improves INT4 down-proj numerical fidelity vs a single global
h_scale.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Refactor fused dense INT4 MLP for bit-equal output and kernel reuse
Restructure the fused op as two phases with a single global h_scale:
phase 1 workers compute gate -> GeLU -> up -> multiply into a shared
fp16 h_full[d_ffn]; single-threaded max-abs and INT8 quantization
between phases; phase 2 workers do the down projection partitioned
by hidden_dim N-block tiles, writing directly to the output. Uses
cactus_gemv_int4_block_range three times (gate, up, down) and
eliminates the per-worker fp32 down accumulator and reduction.
Also refactor cactus_gemv_int4 to delegate its inner work to
cactus_gemv_int4_block_range with accumulate=false, removing ~120
lines of duplicated body code, and drop the now-unused
cactus_gemv_int4_st_kslice_fp32acc kernel.
On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 45-46 tok/s
fused vs 37 tok/s unfused (+22-24%), up from the previous 44 tok/s
per-tile-h_scale variant. The contiguous N-tile access pattern in
phase 2 (each worker reads small N-range x full K of down_w) is
more cache-friendly than the previous striped K-slice pattern.
Output is bit-equivalent to the unfused chain modulo floating-point
reduction-order non-determinism inherent to multi-threaded
inference; both paths show the same run-to-run divergence behavior
under greedy decoding.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Revert cactus_gemv_int4 to its original implementation
The previous refactor had cactus_gemv_int4 delegate to the new
cactus_gemv_int4_block_range helper. Functionally equivalent, but
modifies an existing kernel which is out of scope for this change
set. Restore the original body verbatim from main.
cactus_gemv_int4_block_range remains as a new function used only
by the new fused dense INT4 MLP op. Net diff vs main is now pure
additions (no modifications to existing kernels). Fused-MLP perf
unchanged at 45-46 tok/s (Gemma-4 E2B INT4, M4 Pro, 256-token
decode window) since the fused op calls the new helper directly,
not cactus_gemv_int4.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Match canonical kernel math in fused dense MLP without parallel_for cost
The phase-1 GELU and tile-multiply, and the once-per-layer h-vector
INT8 quantization, are now inline mirrors of cactus_gelu_f16,
cactus_multiply_f16, and cactus_fp16_to_int8 respectively. Same
SIMD bodies, same Eigen-style fast_tanh_f32x4 polynomial for the
GELU, same vmulq_f16 + scalar tail for the multiply, same
vcvtnq_s32_f32 + vqmovn quantize chain — but called inline so each
per-tile invocation skips the parallel_for setup and the
once-per-layer h_full -> h_int8 pass skips the parallel_for
barrier between phase 1 and phase 2.
Fixes a subtle math divergence in the previous homebrew GELU
helper: it used a Pade-style (z(27+z^2))/(27+9z^2) tanh
approximation clamped at +/-4.5, while cactus_gelu_f16 uses the
Eigen-style degree-6/3 polynomial clamped at +/-9.0. Output is
now bit-equivalent (modulo float reduction-order non-determinism
inherent to multi-threaded inference) to the unfused chain's
gb->gelu(matmul(...)) -> gb->multiply(...) -> gb->matmul(mul, ...)
pipeline, since both paths now flow through the exact same
arithmetic.
On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 43-45
tok/s fused on (median 44) vs 35-36 tok/s fused off (+22-26%).
Net diff vs main is still pure additions (no modifications to
existing code paths).
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Trim dead code from fused MLP path
Three small minimization passes that don't touch perf:
* cactus_gemv_int4_block_range: drop the unused accumulate flag (all
three call sites in the fused op pass false) and the partial
actual_n != 4 path (the op asserts hidden_dim % 4 == 0 and
d_ffn % 4 == 0, so the fallback never fires for this kernel's only
caller).
* compute_dense_mlp_int4_fused_node: factor the identical phase 1 and
phase 2 thread-pool dispatch boilerplate (if num_threads <= 1,
enqueue_n_threads + worker_task(0) + wait_all) into a single local
lambda invoked twice.
Steady-state perf unchanged at 43-45 tok/s (Gemma-4 E2B INT4, M4 Pro,
256-token decode). Net diff vs main is now +373 / 0 (was +419), still
pure additions with no modifications to existing code paths.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Drop inline mirror helpers in fused MLP, use canonical kernels
The phase-1 GELU and tile-multiply, and the once-per-layer h-vector
INT8 quantization, now call cactus_gelu_f16, cactus_multiply_f16,
and cactus_fp16_to_int8 directly. The inline mirrors of those
kernels' SIMD bodies were tested empirically against the canonical
helpers on this hardware:
* cactus_gelu_f16 / cactus_multiply_f16 with k_count=128 fall
through parallel_for's threshold check and run inline anyway,
so the inline mirrors saved nothing measurable.
* cactus_fp16_to_int8 with d_ffn=12288 elements does parallelize,
and the inline mirror saved ~1 tok/s by skipping its barrier.
But that's within run-to-run noise on the 256-token bench, the
magnitude depends on parallel_for cost (which varies per
device), and ~22 lines of duplicated SIMD code is a poor trade
when the savings don't generalize uniformly across smaller
thread pools (e.g. mid-range Android phones).
Output is still bit-equivalent to the unfused chain (modulo
multi-threaded reduction-order non-determinism), since both paths
now flow through the exact same canonical kernels.
On Gemma-4 E2B INT4 (M4 Pro, 256-token decode window): 42-43 tok/s
fused on vs 36 tok/s fused off (+17-19%).
Net diff vs main is now +306 / 0 (was +373), still pure additions
with no modifications to existing code paths.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Restore inline INT8 quantize for the once-per-layer h vector
cactus_fp16_to_int8 uses parallel_for with the ELEMENT_WISE config
({5000, 2500} on Android, same threshold-wise on Apple). At
d_ffn=12288 this dispatches ~5 worker threads, whose orchestration
cost (mutex acquire on enqueue, futex wakeups, std::future::wait
serialization, straggler waits if a slice lands on an A520-class
little-core) typically dominates the few-microsecond SIMD work.
Running the quantize inline on the dispatch thread skips that
overhead. The benefit applies on any device with non-trivial
thread-pool wakeup cost; on Android big.LITTLE configurations the
gap is wider than on Apple silicon because heterogeneous cores
amplify straggler-wait time.
The previous commit removed this helper along with the gelu/
multiply mirrors, but those latter two operate on k_count=128
which falls below SCALAR_EXPENSIVE/ELEMENT_WISE gates and
fast-paths to inline anyway -- correctly removed. The quantize
case is the only one of the three where parallel_for genuinely
dispatches and the inline mirror buys real time.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Trim comments in fused MLP
Reduce the long parallel_for-overhead explainer above
fp16_to_int8_inplace to a two-line WHY. No code changes.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Rename fused-MLP env var to a disable-style flag
CACTUS_DENSE_MLP_FUSED=0 (the previous escape hatch) inverted the
natural reading of the env var name -- "set to 0 to disable a
thing called *fused*" requires the reader to know the default.
Replace with CACTUS_DISABLE_DENSE_MLP_FUSED: any non-empty,
non-"0" value disables the fused path. Unset / "0" / empty leave
the default-on behavior untouched. Reads naturally and matches
the CACTUS_IGNORE_EOS-style convention already used elsewhere.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Address Copilot review: stable enum value + per-call scratch
* Move DENSE_MLP_INT4_FUSED to the end of OpType (and of
op_type_names) so existing enum values aren't shifted. graph_io.cpp
serializes OpType as raw uint32_t, so inserting in the middle would
have broken compatibility for any persisted graph (or future
cross-build scenarios).
* Replace the namespace-static dense_fused_scratch vector with a
function-local std::vector<DenseFusedScratch>(num_workers)
constructed per call. The previous static was concurrency-unsafe if
two graphs ever executed this op concurrently (workers would race
on the same shared buffers). Per-call construction is ~1-3 us with
the freelist warm; net perf unchanged (44-45 tok/s).
The other four Copilot comments were either unreachable in practice
(group_size / d_ffn / hidden_dim alignment is always satisfied for
Gemma-4 weights, single internal call site for the kernel) or
pre-existing in main (op_type_names ordering for unrelated ops).
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
---------
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
* inner 1 fast path * added tests Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
Signed-off-by: Matthew Hayes <matthewha853@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a Grammar API to enable structured generation in coming PRs.
For now, this isn't directly integrated with completion, but rather exposes all the necessary APIs to make that happen in the near future. The underlying state machine used is XGrammar, and for the most part the implementation directly wraps it.
There are 4 essential types:
Grammaran actual representation of a grammar, with conveniences for plain JSON, JSON schema, regex, empty, universal, structural tags, etc.GrammarMatcherthe state machine type responsible for computing bitmasks.acceptandnext_bitmaskdo most of the driving. However, there are also methods to rollback, and to check the termination state.forkis an efficient way to create a copy of matcher (at it's current state) without having to go throughGrammarEngine.GrammarEngineresponsible for compilingGrammarMatcherinstances from aGrammarinstance.GrammarCompilerandCompilerGrammarfrom XGrammar. I felt it was a conceptually simpler API to keep these concepts bound together.forkwhich can be used for efficient copies.compile_matchermethod which takes aGrammarinstance and returns aGrammarMatcher.GrammarVocabularytokenizer info required by the other grammar types.TokenizerInfofrom XGrammar, there is also now a new tokenizer method to produce it.Among other things, I also extracted a helper for constructing the tokenizer from the model path. This was mainly so that I could use it directly for testing without loading model weights.