Fix server-side prepared statement leak from pooled connections - #13
Merged
Merged
Conversation
Statements a request leaves in cycle garbage were destroyed after the connection re-entered the pool. mysqlnd skips COM_STMT_CLOSE while a connection is busy in another coroutine and never retries, so every such statement leaked server-side until MySQL's max_prepared_stmt_count tripped (error 1461). - Worker now detaches pooled connections, tears down request state, runs gc_collect_cycles() at that safe point, and only then releases the connections back to their pools. - DatabasePool recycles connections past a configurable max_lifetime (default 300s) on release and in the pruner, bounding any residual leak from paths the worker cannot see.
- Detach pooled connections first in the finally block so a throw from the Livewire mutex or Redis release can never orphan a borrowed slot. - Sweep connections that destructors borrow into context during teardown or gc, and fold flush-time borrows in before Context::clear(). - Unset the caught worker exception before releasing: its trace args reference the request graph, including PDO statements. - Keep per-request pruning across ALL pools (previously done by DatabaseManager::releaseConnections, which the Worker no longer calls). - Gate the teardown gc on gc_status()['roots']. - Roll back abandoned transactions before recycling an expired connection: disconnect() only drops the PDO reference, so a leaked statement could keep the session and its row locks alive until a future gc. - Restart the lifetime clock when the reconnector swaps in a fresh PDO. - Close (instead of silently dropping) connections the pruner fails to re-pool, and trim the '.pool' suffix instead of str_replace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production MySQL hit
max_prepared_stmt_count(error 1461) on Aug 19. Themstudioservice held 10,086 of ~10,800 open server-side prepared statements — up to 1,220 on a single pooled connection, thousands of copies of the same two queries.Mechanism (reproduced deterministically): a PDOStatement destroyed while its connection is mid-query in another coroutine leaks its server-side prepared statement permanently — mysqlnd skips COM_STMT_CLOSE when the connection is not READY and never retries. The Worker released each request's pooled connection back to the pool before the request's object graph died, so statements held in cycle garbage (unfinished generators, exception traces) were destroyed after another coroutine had already borrowed the connection. 100% reproduction: 20/20 statements leak when destroyed against a busy connection; 0/20 when destroyed against an idle one.
Fix
flush,Context::clear,unsetincluding the caught exception), rungc_collect_cycles()at that safe point while the connections are still idle and unborrowable, and only then release them. A second sweep catches connections that destructors borrow during teardown, and per-request all-pool pruning is preserved.max_lifetime(default 300s, 0 disables): connections are recycled on release and in the pruner, bounding any residual leak. Abandoned transactions are rolled back before closing, sincedisconnect()only drops the PDO reference.End-to-end validation with the real pool against MySQL
performance_schema: old order leaks 20/20 statements, new order 0/20. All 160 package tests pass; every new guard is mutation-tested (fix removed → test red).Immediate prod mitigation already applied on the DB (
SET PERSIST max_prepared_stmt_count = 65536) and the leaking service was rolled. This PR is the root-cause fix.Known residuals (pre-existing, out of scope)
max_lifetimebounds this residue.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.