Occasional increase in slow queries and memory usage, worsened a lot in 3.0.X

[MarleDK]

Description of our setup

We are using proxysql in kubernetes, mainly using the connection multiplexing feature to reduce the number of connections created/shut down in MySQL, as one of our applications does not support thread pooling, but instead creates quite short lived connections to MySQL.
We are also using proxysql to route to read replicas, based on which user connects, and equally split traffic between read replicas.

We have 2 read replicas in one host group in proxysql, and a primary MySQL instance in the other host group.

Description of issue

We have now on multiple occasions tried to upgrade ProxySQL from 2.7.2, first to 3.0.2 and recently to 3.0.8, but in both cases, we ran into an issue, where it seems like the new proxysql pods would get into a weird state, and use way more memory than usual, and the number of slow queries reported by ProxySQL would also increase a lot, even though there are no changes to the metrics from the MySQL server, so it's not because the MySQL server behind proxysql got slower.

It does seem like this also happens to a smaller degree for ProxySQL 2.7.2, I can see that the number of slow queries in a single proxysql pod is occasionally worse (most pods are within 15-35, but the bad pods are above 200 slow queries per minute) than the rest for about a minute, maybe a bit less. And at the same time the memory usage of the pod is also higher (usually the higher memory usage stays for a bit longer than the slow requests).
It's not causing issues for us when running 2.7.2, so to me it seems like there is something that can cause occasional slowness in a single proxysql pod, and that in version 3.0.X there is some performance change, that is making this effect way worse.

We are running the proxysql/proxysql docker image

Some of the proxysql pods showed this error, when they had issues:
MySQL_Session.cpp:1635:handler_again___status_PINGING_SERVER(): [ERROR] Ping timeout during ping on famly-core-read-5a.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1344757us (timeout 500ms)
But only for some of our MySQL servers.

Full logs for a proxysql pod with the issue

2026-05-05T08:20:57.168603095Z stderr F 2026-05-05 08:20:57 [INFO] Using OpenSSL version: OpenSSL 3.5.5 27 Jan 2026
2026-05-05T08:20:57.330961964Z stderr F 2026-05-05 08:20:57 [INFO] ProxySQL version 3.0.8-481-g0ef544c
2026-05-05T08:20:57.330977603Z stderr F 2026-05-05 08:20:57 [INFO] ProxySQL SHA1 checksum: 5100cfdede83890db7fe8a300fc445a26fd16a90
2026-05-05T08:20:57.333662473Z stderr F Standard ProxySQL PgSQL Logger rev. 2.5.0421 -- PgSQL_Logger.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.335144232Z stderr F Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.335441076Z stderr F Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.37429416Z stderr F 2026-05-05 08:20:57 [INFO] Using UUID: 04356aa8-e9e1-4619-99aa-347a83c3575d , randomly generated. Writing it to database
2026-05-05T08:20:57.422307413Z stderr F 2026-05-05 08:20:57 Admin_FlushVariables.cpp:915:flush_pgsql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable ffto_enabled with value """". Deleting. If the variable name is correct, this version doesn't support it
2026-05-05T08:20:57.423097247Z stderr F Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.423244018Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD MYSQL USERS TO RUNTIME' was '0x70B7BE80A90C2619', with epoch '1777969257'
2026-05-05T08:20:57.425222866Z stderr F 2026-05-05 08:20:57 [INFO] Generating runtime mysql servers and mysql servers v2 records.
2026-05-05T08:20:57.425384047Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.42553258Z stderr F 2026-05-05 08:20:57 [INFO] Creating new server in HG 0 : famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 , gtid_port=0, weight=1, status=0
2026-05-05T08:20:57.425544576Z stderr F 2026-05-05 08:20:57 [INFO] Creating new server in HG 1 : famly-core-read-5a.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 , gtid_port=0, weight=50, status=0
2026-05-05T08:20:57.425556389Z stderr F 2026-05-05 08:20:57 [INFO] Creating new server in HG 1 : famly-core-read-5b.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 , gtid_port=0, weight=50, status=0
2026-05-05T08:20:57.425846627Z stderr F 2026-05-05 08:20:57 [INFO] New mysql_hostgroup_attributes table
2026-05-05T08:20:57.425925486Z stderr F 2026-05-05 08:20:57 [INFO] Checksum for table mysql_servers_v2 is 0xAA404C4766F8A1F0
2026-05-05T08:20:57.426514861Z stderr F | hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
2026-05-05T08:20:57.42652719Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.426614423Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.426686779Z stderr F 2026-05-05 08:20:57 [INFO] New pgsql_servers_ssl_params table
2026-05-05T08:20:57.426878682Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD PROXYSQL SERVERS TO RUNTIME' was '0x0000000000000000', with epoch '1777969257'
2026-05-05T08:20:57.427047191Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD PGSQL QUERY RULES TO RUNTIME' was '0x0000000000000000', with epoch '1777969257'
2026-05-05T08:20:57.428898912Z stderr F In memory Standard Query Cache (SQC) rev. 2.0.0385 -- Query_Cache.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.428969007Z stderr F In memory Standard Query Cache (SQC) rev. 2.0.0385 -- Query_Cache.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.429585761Z stderr F Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.429848032Z stderr F 2026-05-05 08:20:57 [INFO] For consultancy visit: https://proxysql.com/services/consulting/
2026-05-05T08:20:57.76016587Z stderr F 
2026-05-05T08:20:57.330974888Z stderr F 2026-05-05 08:20:57 [INFO] Detected OS: Linux proxysql-86f467c85d-7klw8 6.12.79 #1 SMP Tue Apr 14 00:50:43 UTC 2026 aarch64
2026-05-05T08:20:57.422325565Z stderr F 2026-05-05 08:20:57 Admin_FlushVariables.cpp:915:flush_pgsql_variables___database_to_runtime(): [WARNING] Impossible to set not existing variable ffto_max_buffer_size with value """". Deleting. If the variable name is correct, this version doesn't support it
2026-05-05T08:20:57.42310833Z stderr F Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.423116866Z stderr F Standard PgSQL Authentication rev. 0.1.0001 -- PgSQL_Authentication.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.425341814Z stderr F 2026-05-05 08:20:57 [INFO] Dumping mysql_servers_incoming
2026-05-05T08:20:57.425347238Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.425377472Z stderr F 2026-05-05 08:20:57 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
2026-05-05T08:20:57.425379485Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.425380967Z stderr F | mem_pointer | hostgroup_id | hostname | port |
2026-05-05T08:20:57.425382497Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.425477534Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.425516187Z stderr F | 1            | famly-core-read-5a.cebi1dmz86mh.eu-central-1.rds.amazonaws.com | 3306 | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         | 0           | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425523688Z stderr F | 1            | famly-core-read-5b.cebi1dmz86mh.eu-central-1.rds.amazonaws.com | 3306 | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         | 0           | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425525571Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.425619868Z stderr F 2026-05-05 08:20:57 [INFO] New mysql_group_replication_hostgroups table
2026-05-05T08:20:57.425710432Z stderr F 2026-05-05 08:20:57 [INFO] New mysql_galera_hostgroups table
2026-05-05T08:20:57.425772747Z stderr F 2026-05-05 08:20:57 [INFO] New mysql_aws_aurora_hostgroups table
2026-05-05T08:20:57.426110482Z stderr F 2026-05-05 08:20:57 [INFO] Checksum for table mysql_servers is 0xAA404C4766F8A1F0
2026-05-05T08:20:57.426113419Z stderr F 2026-05-05 08:20:57 [INFO] Rebuilding 'Hostgroup_Manager_Mapping' due to checksums change - mysql_servers { old: 0x0, new: 0x766F8A1F0AA404C4 }, mysql_replication_hostgroups { old:0x0, new:0x0 }
2026-05-05T08:20:57.426272058Z stderr F 2026-05-05 08:20:57 [INFO] MySQL_HostGroups_Manager::commit() locked for 1ms
2026-05-05T08:20:57.42640703Z stderr F 2026-05-05 08:20:57 [INFO] Generating runtime pgsql servers and pgsql servers v2 records.
2026-05-05T08:20:57.426511637Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.426517479Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.426519352Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.426524203Z stderr F 2026-05-05 08:20:57 [INFO] Dumping pgsql_servers LEFT JOIN pgsql_servers_incoming
2026-05-05T08:20:57.426542297Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.426907425Z stderr F Standard Query Processor rev. 3.0.0.0004 -- Query_Processor.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.426982426Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD MYSQL QUERY RULES TO RUNTIME' was '0x0000000000000000', with epoch '1777969257'
2026-05-05T08:20:57.429656573Z stderr F 2026-05-05 08:20:57 [INFO] Started Monitor scheduler thread for PgSQL servers
2026-05-05T08:20:57.426833774Z stderr F 2026-05-05 08:20:57 [INFO] Checksum for table pgsql_servers is 0x0000000000000000
2026-05-05T08:20:57.426872273Z stderr F 2026-05-05 08:20:57 [INFO] PgSQL_HostGroups_Manager::commit() locked for 0ms
2026-05-05T08:20:57.169976876Z stderr F 2026-05-05 08:20:57 [INFO] Using jemalloc with MALLOC_CONF: config.xmalloc:1, lg_tcache_max:16, opt.prof_accum:1, opt.prof_leak:1, opt.lg_prof_sample:20, opt.lg_prof_interval:30, rc:0
2026-05-05T08:20:57.171490055Z stderr F 2026-05-05 08:20:57 [INFO] No SSL keys/certificates found in datadir (/var/lib/proxysql). Generating new keys/certificates.
2026-05-05T08:20:57.333555215Z stderr F Standard ProxySQL MySQL Logger rev. 2.5.0421 -- MySQL_Logger.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.422857016Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD PGSQL VARIABLES TO RUNTIME' was '0x59576017ADF65469', with epoch '1777969257'
2026-05-05T08:20:57.423115187Z stderr F Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.423275355Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD PGSQL USERS TO RUNTIME' was '0x0000000000000000', with epoch '1777969257'
2026-05-05T08:20:57.42535313Z stderr F | 0            | famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com     | 3306 | 0         | 1      | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425357776Z stderr F | 1            | famly-core-read-5a.cebi1dmz86mh.eu-central-1.rds.amazonaws.com | 3306 | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425359608Z stderr F | 1            | famly-core-read-5b.cebi1dmz86mh.eu-central-1.rds.amazonaws.com | 3306 | 0         | 50     | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425475262Z stderr F 2026-05-05 08:20:57 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
2026-05-05T08:20:57.425511933Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.425867898Z stderr F 2026-05-05 08:20:57 [INFO] New mysql_servers_ssl_params table
2026-05-05T08:20:57.426543814Z stderr F +-------------+--------------+----------+------+
2026-05-05T08:20:57.426607609Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.426609254Z stderr F | hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
2026-05-05T08:20:57.426662308Z stderr F 2026-05-05 08:20:57 [INFO] New pgsql_hostgroup_attributes table
2026-05-05T08:20:57.426713595Z stderr F 2026-05-05 08:20:57 [INFO] Checksum for table pgsql_servers_v2 is 0x0000000000000000
2026-05-05T08:20:57.426740543Z stderr F 2026-05-05 08:20:57 [INFO] New computed global checksum for 'pgsql_servers_v2' is '0x0000000000000000'
2026-05-05T08:20:57.429822382Z stderr F 2026-05-05 08:20:57 [INFO] For information about products and services visit: https://proxysql.com/
2026-05-05T08:20:57.429842525Z stderr F 2026-05-05 08:20:57 [INFO] For support visit: https://proxysql.com/services/support/
2026-05-05T08:20:57.169829614Z stderr F 2026-05-05 08:20:57 [INFO] Using config file /etc/proxysql.cnf
2026-05-05T08:20:57.169984822Z stderr F 2026-05-05 08:20:57 [INFO] Current RLIMIT_NOFILE: 65536
2026-05-05T08:20:57.330991182Z stderr F 2026-05-05 08:20:57 [INFO] SSL keys/certificates found in datadir (/var/lib/proxysql): loading them.
2026-05-05T08:20:57.421327491Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD ADMIN VARIABLES TO RUNTIME' was '0xFC7DE114D48BCA72', with epoch '1777969257'
2026-05-05T08:20:57.422110379Z stderr F 2026-05-05 08:20:57 [INFO] Computed checksum for 'LOAD MYSQL VARIABLES TO RUNTIME' was '0x1637FB48E674A641', with epoch '1777969257'
2026-05-05T08:20:57.423106184Z stderr F 2026-05-05 08:20:57 [INFO] ProxySQL SHA1 checksum: 5100cfdede83890db7fe8a300fc445a26fd16a90
2026-05-05T08:20:57.425349474Z stderr F | hostgroup_id | hostname                                                       | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
2026-05-05T08:20:57.425351306Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.42536109Z stderr F +--------------+----------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.425502188Z stderr F | hostgroup_id | hostname                                                       | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
2026-05-05T08:20:57.425514332Z stderr F | 0            | famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com     | 3306 | 0         | 1      | 0      | 0           | 2000            | 0                   | 1       | 0              |         | 0           | 0         | 1      | 0      | 0           | 2000            | 0                   | 1       | 0              |         |
2026-05-05T08:20:57.425973885Z stderr F 2026-05-05 08:20:57 [INFO] New computed global checksum for 'mysql_servers_v2' is '0x7F9FB5406980D3B2'
2026-05-05T08:20:57.426506442Z stderr F 2026-05-05 08:20:57 [INFO] Dumping pgsql_servers_incoming
2026-05-05T08:20:57.426538131Z stderr F | mem_pointer | hostgroup_id | hostname | port |
2026-05-05T08:20:57.426604483Z stderr F 2026-05-05 08:20:57 [INFO] Dumping pgsql_servers JOIN pgsql_servers_incoming
2026-05-05T08:20:57.426610672Z stderr F +--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2026-05-05T08:20:57.42692718Z stderr F Standard Query Processor rev. 3.0.0.0004 -- Query_Processor.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.429573175Z stderr F Standard MySQL Monitor (StdMyMon) rev. 2.0.1226 -- MySQL_Monitor.cpp -- Fri Apr 24 09:12:41 2026
2026-05-05T08:20:57.429831178Z stderr F 2026-05-05 08:20:57 [INFO] For online documentation visit: https://proxysql.com/documentation/
2026-05-05T08:20:57.760138865Z stderr F 2026-05-05 08:20:57 [INFO] Latest ProxySQL version available: 3.0.8
2026-05-05T08:22:10.199028045Z stderr F 2026-05-05 08:22:10 MySQL_Session.cpp:1635:handler_again___status_PINGING_SERVER(): [ERROR] Ping timeout during ping on famly-core-read-5a.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1344757us (timeout 500ms)
2026-05-05T08:22:10.342187123Z stderr F 2026-05-05 08:22:10 MySQL_Session.cpp:1635:handler_again___status_PINGING_SERVER(): [ERROR] Ping timeout during ping on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)
2026-05-05T08:22:10.349236193Z stderr F 2026-05-05 08:22:10 MySQL_Session.cpp:1635:handler_again___status_PINGING_SERVER(): [ERROR] Ping timeout during ping on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)
2026-05-05T08:22:10.364924644Z stderr F 2026-05-05 08:22:10 MySQL_Session.cpp:1635:handler_again___status_PINGING_SERVER(): [ERROR] Ping timeout during ping on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)
2026-05-05T08:22:16.127898111Z stderr F 2026-05-05 08:22:16 MySQL_Session.cpp:5839:handler_minus1_LogErrorDuringQuery(): [WARNING] Error during query on (0,famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com,3306,6105930): 1062, Duplicate entry 'a6faf544-e038-496c-865d-15b70f59a456' for key 'child_checkin_active.childId'
2026-05-05T08:23:54.212427279Z stderr F 2026-05-05 08:23:54 [INFO] Received PROXYSQL PAUSE command
2026-05-05T08:24:00.311899036Z stderr F 2026-05-05 08:24:00 [INFO] Received command set runtime_global wait_timeout=0
2026-05-05T08:25:00.323892775Z stderr F 2026-05-05 08:25:00 ProxySQL_GloVars.cpp:32:term_handler(): [WARNING] Received TERM signal: shutdown in progress...

proxysql config

mysql_variables=
{
    threads=4 # Default
    max_connections=15000 # For each backend server we allow 2000 connections. We have 3 backend servers in total
    default_query_timeout=3600000 # This is 1 hour, we expect the mysql max execution time to kill queries after 1 minute in most cases
    interfaces="0.0.0.0:6033"
    server_version="8.0.44"
    wait_timeout=630000
    monitor_username="REDACTED"
    monitor_password="REDACTED"
    monitor_ping_interval=10000
    monitor_read_only_interval=15000 
    ping_timeout_server=500 
    log_unhealthy_connections=false
    default_charset="utf8mb4"
    default_collation_connection="utf8mb4_unicode_ci"
    free_connections_pct=1 
}

I know the description might be a bit vague, as this is not a simple reproducible bug, and I of course understand that you have limited time for this, so please let me know if there is anything else I can provide which might help, I have tried to investigate this quite a lot, to see if there might be some change in default variables or similar. I have also tried to have a long conversation with Claude, having it try to read through the diff of code in the two versions, but with my limited C++ knowledge, and how this works, I'm unfortunately unable to figure more our myself.

According to Claude this might be the issue:

Why it's worse in 3.x specifically

The root function is the same in both versions, but 3.x introduced new code paths that call RequestEnd on connection failure where 2.7.2 didn't. Specifically:

• The Base_Session refactoring (the shared MySQL/PostgreSQL session infrastructure added in 3.x) added calls to RequestEnd in failure branches inside
  handler_again___status_CONNECTING_SERVER that simply didn't exist in 2.7.2's simpler session handler
• Three partial fixes shipped in 3.0.8 (ae07745, de0a9b9, 41564ac) address crash safety around these calls — a double-RequestEnd on error 9001, a use-after-free on the myconn
  pointer, a null-pointer guard — but none of them prevent the state corruption itself

So in 2.7.2: connection fails → session cleaned up → client connection returns to idle.
In 3.0.8: connection fails → RequestEnd runs ProcessQueryAndSetStatusFlags on the failed connection → multiplexing state toggled incorrectly → session stuck permanently non-idle.

Under concentrated load driving 97 new connection attempts per second (many of which fail), this bug fires hundreds of times per minute on the overloaded pod. Each firing leaves
one more session permanently non-idle. The count ratchets up and never comes back down until the pod is replaced.

But I can't verify it in any way, due to lack of knowledge

[coderabbitai]

🔗 Related PRs

#5416 - Handle late COPY protocol messages after session exits fast_forward mode - P0 [merged]
#5528 - Run ProxySQL cluster inside single container + CI stability fixes [merged]
#5538 - CI test fixes: cluster sync, plan counts, test group assignments, and diagnostics [merged]
#5654 - fix(pgsql): surface mid-transaction backend errors instead of silently replaying [merged]
#5809 - Integration batch 2026-05-23: bundle of 6 fixes [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[NathanChou-at]

We attempted to upgrade from 2.6.6 -> 3.0.8 and noticed that memory + CPU seemed to increase with similar symptoms and also saw increased number of proxysql.slow_queries.

[renecannao]

Few points worth highlighting:

1. "there are no changes to the metrics from the MySQL server, so it's not because the MySQL server behind proxysql got slower"
2. "use way more memory than usual"
3. "Ping timeout during ping ... after 1344757us (timeout 500ms)"

Point 3 suggests either a DNS issue , or a stall.
Although, this exact time lapse suggests it is more likely a stall than a DNS issue:

... 08:22:10 ... Ping timeout ... on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)
... 08:22:10 ... Ping timeout ... on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)
...08:22:10 ... Ping timeout ... on famly-core-vpc.cebi1dmz86mh.eu-central-1.rds.amazonaws.com:3306 after 1067722us (timeout 500ms)

Right now my guess is that proxysql is starving on CPU.
Kubernetes is not giving enough CPU to proxysql , and it stalls.
Higher memory usage could be the trigger for CPU starvation.
Higher memory usage -> aggressive memory eviction -> CPU starvation -> timeouts and slow queries.

At rest, without any traffic at all, proxysql 3.x uses more memory than proxysql 2.7 because it has more modules (PostgreSQL for example) and features.
(In 4.0 we are planning to make it more modular, so the core will have less memory footprint).
Probably the switch from 2.7 to 3.0 was enough to goes over the available memory, cascading into CPU starvation and all the other side effects.
Try to assign more memory to proxysql, hopefully this will be enough to restore its operation.

[peppy]

We are also seeing issues after attempted upgrading to 3.x, but we cannot see slow query count increase, nor memory usage.

We have ProxySQL running outside of kubernetes, but when we move load across from 2.x to 3.x, queries begin to take much longer than usual after hitting peak server load. We have plenty of spare CPU, network usage is only at about 400 mbit, and we don't actually get any error reports – queries still complete eventually.

But across the board we start to see client timeout failures due to how long queries are taking.

Not really sure how to investigate further. Of note, we can hit the same issue on 2.x (it's one of the reasons we attempted an upgrade to see if things improved), but it requires much higher load levels.