[zhanghaohit]

What

• New post-processing handler strip_actual_usage: removes the top-level actual_usage object that some sglang builds return next to usage, from SSE frames and non-stream JSON bodies (2xx only). Nothing else in the response is touched.

• vllm_format drops actual_usage as well.

• The SSE/JSON plumbing of vllm_format moves into json_rewrite_handler(), shared by both handlers. No behavior change for vllm_format apart from the two items here.

• bodylog-listener keeps actual_usage in resp_meta.actual_usage. The listener stores a success response as its content plus resp_meta, and resp_meta took usage but no other top-level field, so actual_usage never reached the bodylog. Once openresty strips it from the client response, the bodylog is the only place it survives.

• Test mock (test/mock_vllm_sse.py): --actual-usage (or {"emit_actual_usage": true} per request) returns an actual_usage; --host lets it run in a pod. Unchanged wire without the flags.

• Fix: rewriting a frame with the shared cjson instance turned "choices":[] into "choices":{} (cjson encodes every empty table as an object). That is exactly the final usage chunk of a stream. Post-processing now uses a private cjson instance with decode_array_with_array_mt(true), so arrays round-trip as arrays. The setting does not leak into other modules.

• Rename the rate-limit master switch option rate_limit → load_rate_limit (and _G.RATE_LIMIT_ENABLED → _G.LOAD_RATE_LIMIT_ENABLED). On a ModelRoute, nginx.values.rate_limit is already a video route's download speed limit, and the CRD rejects it on llm routes, so the documented rate_limit: false could not be applied. The endpoints (/_rate_limit_status, /_rate_limit_toggle) and their fields are unchanged; no alias for the old name (released in v0.1.29, set nowhere).

• CI runs the Lua unit tests. New test/run_lua_unit_tests.sh runs all test/utest_*.lua with the resty CLI and exits non-zero on a failure or when it finds none; a new lua job in test.yml runs it on jammy with the OpenResty version the Dockerfile pins.

• README: a "Response post-processing" section (opting in, the two handlers, which responses are touched, buffering / Content-Length, fail-open, what the bodylog keeps); Testing points at the runner.

Why

actual_usage is a second token count, e.g. with a different cached_tokens than usage:

"usage":        {"prompt_tokens": 4096, "prompt_tokens_details": {"cached_tokens": 2048, "cache_write_tokens": 1792}},
"actual_usage": {"prompt_tokens": 4096, "prompt_tokens_details": {"cached_tokens": 3072}}

Clients and billing should see one count, usage. openresty hands the bodylog the raw upstream chunks (bodylog runs before post-processing), and with the listener change above the bodylog record keeps actual_usage.

Cost

A plain substring test ("actual_usage") runs before any decoding. Frames without it pass through byte-identical and undecoded. In practice only the final usage chunk of a stream gets decoded and re-encoded once.

Enabling

Off unless a route opts in (one handler per route):

# ModelRoute
spec:
  nginx:
    values:
      postprocess: strip_actual_usage

Tests

End to end on a test cluster (openresty 0.1.30-rc1 + bodylog 0.1.16-rc1):

• A mock backend returns actual_usage; it is there when the mock is called directly.
• Through openresty, on both replicas: gone from the stream and the non-stream response; usage is unchanged; "choices":[] is still an array.
• All 4 bodylog records keep it in resp_meta.actual_usage.
• With the previous image (0.1.29, no handler) the same checks fail.
• The rate-limit switch e2e passes after the rename.
• load_rate_limit: "false" passes the ModelRoute CRD (server dry-run); rate_limit is rejected.

New test/utest_postprocess_strip_actual_usage.lua covers:

• streams split whole, into 3 and 7 pieces, and per byte;
• the content frame and [DONE] stay byte-identical;
• usage stays unchanged and "choices":[] stays an array;
• non-stream JSON split across chunks;
• no field present → byte-identical;
• a nested actual_usage is left alone;
• non-2xx and non-JSON media types pass through, and malformed JSON fails open;
• vllm_format drops the field and keeps choices an array.

Results:

• resty test/utest_postprocess_strip_actual_usage.lua: PASS.
• resty test/utest_postprocess_vllm.lua: PASS (unchanged).
• Against the previous lua/postprocess.lua, the new test fails, and the old vllm_format emits "choices":{} for the usage frame.
• go test ./... in bodylog-listener-go: PASS. New TestRespMetaKeepsActualUsage (stream and non-stream; absent upstream means absent) fails on the previous main.go.

[zoboth]

I read the whole diff. I found no blockers.

What I checked

• strip_actual_usage_json: it only checks for the substring "actual_usage" before decoding. An escaped \"actual_usage\" inside a string value doesn't match, so content frames stay byte-identical. Only the top-level key is removed, and it fails open on decode and encode errors. The handler is limited to 2xx responses and to text/event-stream / application/json, as described.
• json_rewrite_handler(): the move is mechanical. The JSON buffering, the overflow fail-open and the SSE leftover/line splitting all match the old vllm_format body line for line, with rewrite_json passed in. Each M.register call gets its own handler table.
• Private cjson instance: require("cjson.safe").new() together with decode_array_with_array_mt(true) keeps the setting on this instance only, so other modules' cjson stay as they are. json_null (cjson.null) is the same lightuserdata in every instance, so the null checks in vllm_format still work. The "choices":[] → {} fix is real and is covered by test 5.
• bodylog-listener: actualUsage follows the same path as usage (taken from whichever frame carries it, and only added to resp_meta when present). The test also covers the case where upstream sends none.
• Mock: the wire is unchanged without the new flags. --host defaults to 127.0.0.1 as before.

Non-blocking notes

1. The rate_limit → load_rate_limit rename isn't in the PR description. This PR also renames the route option rate_limit (merged in #29) to load_rate_limit, and _G.RATE_LIMIT_ENABLED to _G.LOAD_RATE_LIMIT_ENABLED, across route.lua, directives.lua, debug_endpoints.lua, session_base.conf, the README and test_rate_limit_switch.sh. The reason makes sense: the ModelRoute CRD rejects rate_limit on llm routes. But there is no fallback for the old names. A Lua route conf with rate_limit = false, or an overridden session_base.conf with _G.RATE_LIMIT_ENABLED = false, will be ignored without any message after this change, and load-based 429s come back on. That's the safe direction, so I'm not blocking on it. Still, I'd suggest either reading the old names as a deprecated alias with an ngx.log(ngx.WARN, ...), or at least logging when opts.rate_limit ~= nil on an llm route. I'd also add a line to the PR description, or split the rename into its own PR, so it shows up in the changelog.
2. session_base.conf still says "Per route: rate_limit = false in the route conf" in the comment right above _G.LOAD_RATE_LIMIT_ENABLED. It should say load_rate_limit.
3. The directive key rate_limit → load_rate_limit also changes what /_route_opts returns. If the console, or anything else that reads directives, looks for the key rate_limit by name, it needs to change at the same time. I couldn't check this from here.
4. The debug_endpoints.lua comment alignment on route_value moved by a few columns (cosmetic).
5. Optional: test 3 covers strip_actual_usage_json on a nested actual_usage directly, but not through the handler. It's fine as it is. One SSE case through run() would cover the end-to-end path too.