Repository navigation
Conversation
The handler started wkhtmltopdf with no deadline and without watching the request context, so a render that never finished held the connection and the child process for as long as it took. When io.Copy failed, the handler returned without cmd.Wait(), and the child was left behind. Killing the direct child is not enough: its own children inherit the stdout pipe and keep the copy blocked until the last one exits, so the render runs in its own process group and the group is killed. Add KWKHTMLTOPDF_TIMEOUT, in seconds, default 120, 0 to disable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is about
When a render takes too long, it currently takes the caller down with it. The request never comes back, and the process that was rendering stays alive with nobody reading its output — so the memory and the file descriptors stay taken too. Nothing reports it: the server keeps answering
/statuswhile this happens, so a pod in this state still looks healthy.We hit this on a fleet of Odoo tenants that share one deployment of this server. On one cluster we counted 9 episodes over two and a half days, all during working hours, none of which ever surfaced as an error. Each one left the tenant with a few minutes of failed requests.
The defects
No deadline, no cancellation.
exec.Commandis started without a context, and the request context is never watched. If the render hangs, the copy blocks for as long as the renderer lives, and nothing notices that the caller already left.The child is not reaped on the error path. When
io.Copyfails, the handler callshttpAbortand returns withoutcmd.Wait(). The child is left behind — and because nobody drains its stdout pipe any more, it can block on a full pipe and stay there indefinitely, rather than exiting and being collected later.Killing the direct child is not enough. Its own children inherit the stdout pipe and hold it open, so the copy stays blocked until the last of them exits. A timeout that only kills the direct child does not release the caller — verified below, it took 6s to return with a 2s timeout.
What this PR does
r.Context(), with a timeout fromKWKHTMLTOPDF_TIMEOUT(seconds, default120,0disables it).render timed out after 2s: signal: killed.The default of 120s is meant to sit below the timeouts callers usually have; a render that outlives it has no reader left anyway.
How it was verified
A fake renderer that writes and then hangs, against binaries built from
masterand from this branch:mastersh+dd, stateS), never reapedgofmt,go vetand builds for linux, windows and darwin are clean.The fake renderer and the check
For the second row, use a renderer that only does
sleep 8, setKWKHTMLTOPDF_TIMEOUT=2,and let the client wait:
masterreturns after 8s, this branch after 2s.Notes for the reviewer
server/now has three files, sogo run server/kwkhtmltopdf_server.gono longer works — the README was updated togo run ./server.SysProcAttr.Setpgiddoes not exist on windows; the windows path falls back to killing the direct child only.w.WriteHeader(http.StatusOK)still runs before the copy, so a timeout aborts the response instead of returning504. Deferring the header until the first byte would allow a real status code, but it changes observable behaviour for every client — happy to do it here if you prefer.HISTORY.mdwas left untouched, assuming release notes are yours to write.cc @az-adhoc