Skip to content

feat: control chapter generation reasoning budget - #296

Open
andyhsu6 wants to merge 1 commit into
mainfrom
feat/reasoning-effort-budget
Open

andyhsu6 wants to merge 1 commit into
mainfrom
feat/reasoning-effort-budget

Conversation

@andyhsu6

Copy link
Copy Markdown
Owner

Why

Chapter generation was taking 5–9 minutes per chapter. Measurements on one real project showed the cost is entirely reasoning tokens, not input size:

input chars reasoning tokens first token total
208,363 18,888 252s 282s
207,907 19,675 280s 316s
207,782 36,114 477s 514s
53,921 (context cut 4x) 33,546 476s 508s

Cutting the prompt by 74% left the reasoning count unchanged, so context work cannot fix this. The gateway does accept a thinking-effort level, which was never sent.

What

  • ChapterGenerateRequest / BatchGenerateRequest: new reasoning_effort: low | medium | high, default medium. Explicit null keeps the gateway default (the field is then not injected — asserted by test).
  • Plumbed AIService.generate_text_stream → providers → OpenAIClient payload. Anthropic/Gemini accept and ignore it; OpenAI forwards it, including the tool-call round.
  • Both retry recursions forward the level (stream_options rejection, learned output ceiling). Without that a retry silently drops back to the gateway default and the chapter takes 5–8 minutes again — covered by a dedicated test.
  • Style tail block gained a length-floor clause (93% of the target, zh/en), because lower effort returns shorter chapters.

Measurements (same chapter, same prompt, glm-5.3-flash)

level n reasoning tokens total output chars
default (not sent) 4 18,888–36,114 282–514s 3,380–4,402
low 4 100–303 43–53s 2,208–2,579
low + length floor 2 92–426 46–54s 2,708–2,909
medium + length floor 2 294–661 46–64s 2,726–3,363
high 1 9,747 209s 2,646

Style metrics stay inside the range of the drafts already delivered for this project (sentence length 22.4–25.5 vs 21–29; short-sentence ratio 0.29–0.37 vs 0.21–0.38). Lower effort does not make the short-sentence habit worse, but it does not fix it either — that remains a separate style issue.

finish_reason was stop in every sample, so the truncation gate is unaffected.

Test plan

  • pytest tests/ -q in a clean checkout: 1684 passed, 0 failed (baseline 1669 + 15 new).
  • New tests/test_reasoning_effort.py (15 cases): schema default/validation, payload injection, no field when unset, retry recursion keeps the level, length-floor text (zh/en/absent).
  • Updated test_every_generation_path_appends_the_style_tail_block to match the new 3-argument call form (still guards all 5 wiring sites).
  • End-to-end through the production call chain (AIService.generate_text_stream(reasoning_effort="medium"), real gateway, 213k-char prompt): 96.6s total, 1,875 reasoning tokens, 3,420 chars, finish_reason=stop.

refs #295

Chapter generation latency is dominated by reasoning tokens, not by input
length: the same prompt produced 18,888-36,114 completion tokens and
282-514s per chapter, while shrinking the prompt from 208k to 54k chars
left the reasoning untouched. Reasoning is therefore the only lever.

Add `reasoning_effort` (low/medium/high, default medium) to chapter and
batch generation requests, plumbed through AIService -> providers ->
OpenAI client payload. Both retry recursions (stream_options rejection and
learned output ceiling) forward the level, otherwise a retry silently
falls back to the gateway default and the chapter takes 5-8 minutes again.

Also append a length-floor clause (93% of the target) to the style tail
block: low/medium effort returned 2,208-2,579 chars for a 3,000-char
target, and 2,708-3,363 with the floor, at the same reasoning budget.

Measured on one chapter of a real project (glm-5.3-flash):
  default  18,888-36,114 reasoning tokens, 282-514s
  low          100-303,  43-53s
  medium        49-661,  40-64s
  high         9,747,   209s

refs #295
refs #279
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant