Repository navigation
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4c0b77d13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| struct TokenizeRequest<'a> { | ||
| inputs: &'a [String], | ||
| add_special_tokens: bool, |
There was a problem hiding this comment.
/tokenize wire shape
Standard TEI accepts a single string in inputs and returns one flat token array, but this request always sends an array and count_windows deserializes a nested array. Against the supported external TEI deployment, every probe therefore receives a 4xx, falls back to byte accounting, emits a warning, and repeats the failed request for every embedding call because 400/422 responses are not cached as unsupported; only the custom MLX endpoint added here implements this batched extension. Send/count individual TEI inputs or negotiate a genuinely supported batched shape.
AGENTS.md reference: AGENTS.md:L109-L112
Useful? React with 👍 / 👎.
This branch has not been deployed
1 participant
Short identified records could disappear during chunk-quality filtering, heuristic code splitting miscounted CRLF byte offsets, and partial-scope routing budgets did not affect emitted chunks. This change preserves meaningful record boundaries, calculates exact code offsets, and applies scope-adjusted character ceilings through splitting and final packing. Preparation schema 14 records the changed semantics.
The commit also includes the existing pipeline performance work: shared document metadata without per-chunk copies, borrowed deduplication identities, indexed structured-format source positions, bounded model-matched TEI tokenization with an observable byte-accounting fallback, MLX
/tokenizesupport, and rolling preparation/embedding/publication scheduling with bounded admission and ordered publication.Validation:
axon-documenttests passed after compiling the current crate sources directly with the pinned compiler against metadata-matched cached dependencies.Review considerations: scope ceilings use proportional character limits rather than claiming exact tokenizer enforcement. TEI tokenization can fall back to conservative UTF-8 byte accounting. Scheduling changes affect overlap, cancellation, and retained work admission; preparation semantics require normal refresh/reprocessing to update existing indexed content.