Repository navigation
Conversation
Chapter generation latency is dominated by reasoning tokens, not by input length: the same prompt produced 18,888-36,114 completion tokens and 282-514s per chapter, while shrinking the prompt from 208k to 54k chars left the reasoning untouched. Reasoning is therefore the only lever. Add `reasoning_effort` (low/medium/high, default medium) to chapter and batch generation requests, plumbed through AIService -> providers -> OpenAI client payload. Both retry recursions (stream_options rejection and learned output ceiling) forward the level, otherwise a retry silently falls back to the gateway default and the chapter takes 5-8 minutes again. Also append a length-floor clause (93% of the target) to the style tail block: low/medium effort returned 2,208-2,579 chars for a 3,000-char target, and 2,708-3,363 with the floor, at the same reasoning budget. Measured on one chapter of a real project (glm-5.3-flash): default 18,888-36,114 reasoning tokens, 282-514s low 100-303, 43-53s medium 49-661, 40-64s high 9,747, 209s refs #295 refs #279
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Chapter generation was taking 5–9 minutes per chapter. Measurements on one real project showed the cost is entirely reasoning tokens, not input size:
Cutting the prompt by 74% left the reasoning count unchanged, so context work cannot fix this. The gateway does accept a thinking-effort level, which was never sent.
What
ChapterGenerateRequest/BatchGenerateRequest: newreasoning_effort: low | medium | high, defaultmedium. Explicitnullkeeps the gateway default (the field is then not injected — asserted by test).AIService.generate_text_stream → providers → OpenAIClient payload. Anthropic/Gemini accept and ignore it; OpenAI forwards it, including the tool-call round.Measurements (same chapter, same prompt, glm-5.3-flash)
Style metrics stay inside the range of the drafts already delivered for this project (sentence length 22.4–25.5 vs 21–29; short-sentence ratio 0.29–0.37 vs 0.21–0.38). Lower effort does not make the short-sentence habit worse, but it does not fix it either — that remains a separate style issue.
finish_reasonwasstopin every sample, so the truncation gate is unaffected.Test plan
pytest tests/ -qin a clean checkout: 1684 passed, 0 failed (baseline 1669 + 15 new).tests/test_reasoning_effort.py(15 cases): schema default/validation, payload injection, no field when unset, retry recursion keeps the level, length-floor text (zh/en/absent).test_every_generation_path_appends_the_style_tail_blockto match the new 3-argument call form (still guards all 5 wiring sites).AIService.generate_text_stream(reasoning_effort="medium"), real gateway, 213k-char prompt): 96.6s total, 1,875 reasoning tokens, 3,420 chars,finish_reason=stop.refs #295