[BUG] salt-syndic memory leak when using multi-master syndic mode in 3006.6

[johnl2323]

Description
We run salt masters in mult-master syndic mode (3 syndics and 1 master of master). We are seeing a memory leak in the salt-syndic process to the point that we have to restart all of the masters on a weekly basis to keep them up and running. We have about 8000 minions.

Setup
1 Master of Master (hostname: salt-0.foo.com)
3 Syndic Masters (hostnames: salt-1.foo.com, salt-2.foo.com, salt-3.foo.com). We use an EFS share to store shared keys between the 3 syndics

Please be as specific as possible and give set-up details.

• on-prem machine
• VM running on EC2 instances
• container docker. Masters run inside a docker container

Steps to Reproduce the behavior

Running the following command you can see the salt-syndic process memory leak. This was the memory usage after about 12 hours after our weekly restart of all 4 masters.

$ ps -o pid,start,time,%mem,rss,sz,vsz,size,command --sort -rss -p $(pgrep -f salt) | head -10
    PID  STARTED     TIME %MEM   RSS    SZ    VSZ  SIZE COMMAND
4063633 19:36:07 00:11:46 76.6 12204992 3678350 14713400 14695440 /usr/local/bin/python /usr/local/bin/salt-syndic MainProcess
4063807 19:36:09 00:08:48  0.8 140972 279547 1118188 1098160 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorkerQueue
4063794 19:36:09 00:01:32  0.4 77496 53359 213436 193672 /usr/local/bin/python /usr/local/bin/salt-master EventPublisher
4063793 19:36:09 00:01:20  0.4 73260 43617 174468 154688 /usr/local/bin/python /usr/local/bin/salt-master PubServerChannel._publish_daemon
4063946 19:36:10 00:03:38  0.4 64680 24299  97196 77400 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorker-29
4063822 19:36:09 00:03:44  0.4 64108 24110  96440 76644 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorker-9
4063952 19:36:10 00:03:35  0.3 63100 24373  97492 77696 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorker-31
4063819 19:36:09 00:03:49  0.3 63076 24419  97676 77880 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorker-7
4063842 19:36:09 00:03:26  0.3 63004 24091  96364 76568 /usr/local/bin/python /usr/local/bin/salt-master ReqServer MWorker-11

/var/log/salt/syndic file shows these repetitive messages when it is leaking memory

...
...
024-02-26 15:12:35,715 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:36,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:36,732 [salt.minion      :2406][WARNING ][291] The minion failed to return the job information for job 20240226014343249133. This is often due to the master being shut down or overloaded. If the master is running, consider increasing the worker_threads value.
2024-02-26 15:12:37,212 [salt.minion      :3685][ERROR   ][291] Unable to call _return_pub_multi on salt-0.foo.com, trying another...
2024-02-26 15:12:37,236 [salt.minion      :3482][INFO    ][291] Minion is ready to receive requests!
2024-02-26 15:12:37,237 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:37,714 [salt.minion      :2359][INFO    ][291] Returning information for job: 20240226014343249133
2024-02-26 15:12:38,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:39,715 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:40,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:40,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:40,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:41,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:42,218 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:43,723 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:44,715 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:45,216 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:46,728 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:47,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:49,724 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:50,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:51,216 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:51,724 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:52,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:53,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:53,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:54,715 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:54,717 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:56,215 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:56,216 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
2024-02-26 15:12:56,716 [salt.minion      :1740][INFO    ][291] fire_master failed: master could not be contacted. Request timed out.
...
...

The master log on the salt-0.foo.com Master of Master shows this repetitive set of messages:

024-02-26 15:12:35,877 [salt.utils.job   :86  ][INFO    ][305] Got return from ubuntu-rJ10kUQU for job 20240226211034426429
2024-02-26 15:12:35,878 [salt.utils.event :912 ][ERROR   ][305] Event iteration failed with exception: 'list' object has no attribute 'items'
2024-02-26 15:12:35,879 [salt.loaded.int.returner.local_cache:148 ][ERROR   ][305] An extra return was detected from minion ubuntu-rJ10kUQU, please verify the minion, this could be a replay attac
2024-02-26 15:12:35,879 [salt.utils.job   :86  ][INFO    ][305] Got return from ubuntu-rJ10kUQU for job 20240226211034585877
2024-02-26 15:12:35,880 [salt.utils.event :912 ][ERROR   ][305] Event iteration failed with exception: 'list' object has no attribute 'items'
2024-02-26 15:12:35,881 [salt.master      :1933][ERROR   ][305] Error in function _syndic_return:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/returners/local_cache.py", line 309, in get_load
    with salt.utils.files.fopen(minions_path, "rb") as rfh:
  File "/usr/local/lib/python3.10/site-packages/salt/utils/files.py", line 393, in fopen
    f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage
FileNotFoundError: [Errno 2] No such file or directory: '/srv/nfs/cache/jobs/ab/88c94ddc9c9d8bb8fcfca963a5fd1d65fc99022963534fe58c2e43f60eaf68/.minions.p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1927, in run_func
    ret = getattr(self, func)(load)
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1774, in _syndic_return
    self._return(ret)
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1718, in _return
    salt.utils.job.store_job(
  File "/usr/local/lib/python3.10/site-packages/salt/utils/job.py", line 129, in store_job
    if job_cache == "local_cache" and mminion.returners[getfstr](load.get("jid", "")):
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/returners/local_cache.py", line 312, in get_load
    salt.utils.files.process_read_exception(exc, minions_path)
  File "/usr/local/lib/python3.10/site-packages/salt/utils/files.py", line 225, in process_read_exception
    raise CommandExecutionError("{} does not exist".format(path))
salt.exceptions.CommandExecutionError: /srv/nfs/cache/jobs/ab/88c94ddc9c9d8bb8fcfca963a5fd1d65fc99022963534fe58c2e43f60eaf68/.minions.p does not exist
2024-02-26 15:12:42,830 [salt.utils.job   :86  ][INFO    ][325] Got return from W0372921.foo.com for job 20240226211134336027
2024-02-26 15:12:47,326 [salt.utils.job   :86  ][INFO    ][309] Got return from isg-vulcan-17 for job 20240226211236195019
2024-02-26 15:12:50,878 [salt.utils.job   :86  ][INFO    ][267] Got return from ubuntu-rJ10kUQU for job 20240226211034426429
2024-02-26 15:12:50,879 [salt.utils.event :912 ][ERROR   ][267] Event iteration failed with exception: 'list' object has no attribute 'items'
2024-02-26 15:12:50,880 [salt.loaded.int.returner.local_cache:148 ][ERROR   ][267] An extra return was detected from minion ubuntu-rJ10kUQU, please verify the minion, this could be a replay attac
2024-02-26 15:12:51,385 [salt.utils.job   :86  ][INFO    ][330] Got return from ubuntu-rJ10kUQU for job 20240226210859863844
2024-02-26 15:12:51,884 [salt.utils.job   :86  ][INFO    ][346] Got return from ubuntu-rJ10kUQU for job 20240226210859863844
2024-02-26 15:12:51,895 [salt.loaded.int.returner.local_cache:148 ][ERROR   ][346] An extra return was detected from minion ubuntu-rJ10kUQU, please verify the minion, this could be a replay attac
2024-02-26 15:13:05,395 [salt.utils.job   :86  ][INFO    ][264] Got return from salt-2 for job 20240226211305152704
2024-02-26 15:13:48,321 [salt.utils.job   :86  ][INFO    ][331] Got return from salt-0 for job 20240226211337422670
2024-02-26 15:14:03,393 [salt.utils.job   :86  ][INFO    ][321] Got return from ubuntu-Zm8JjirV for job 20240226211132408018
2024-02-26 15:14:03,398 [salt.master      :1933][ERROR   ][321] Error in function _syndic_return:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/returners/local_cache.py", line 309, in get_load
    with salt.utils.files.fopen(minions_path, "rb") as rfh:
  File "/usr/local/lib/python3.10/site-packages/salt/utils/files.py", line 393, in fopen
    f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage
FileNotFoundError: [Errno 2] No such file or directory: '/srv/nfs/cache/jobs/63/33f78ed3e6410369e8f4cda662ca7533983bae45ad4a46ce64c77cc7399f7d/.minions.p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1927, in run_func
    ret = getattr(self, func)(load)
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1774, in _syndic_return
    self._return(ret)
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 1718, in _return
    salt.utils.job.store_job(
  File "/usr/local/lib/python3.10/site-packages/salt/utils/job.py", line 129, in store_job
    if job_cache == "local_cache" and mminion.returners[getfstr](load.get("jid", "")):
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/returners/local_cache.py", line 312, in get_load
    salt.utils.files.process_read_exception(exc, minions_path)
  File "/usr/local/lib/python3.10/site-packages/salt/utils/files.py", line 225, in process_read_exception
    raise CommandExecutionError("{} does not exist".format(path))
salt.exceptions.CommandExecutionError: /srv/nfs/cache/jobs/63/33f78ed3e6410369e8f4cda662ca7533983bae45ad4a46ce64c77cc7399f7d/.minions.p does not exist
2024-02-26 15:14:14,388 [salt.utils.job   :86  ][INFO    ][321] Got return from salt-0 for job 20240226211413944119
2024-02-26 15:14:26,874 [salt.master      :2520][INFO    ][337] User salt Published command state.sls with jid 20240226211426871895
2024-02-26 15:15:38,326 [salt.utils.job   :86  ][INFO    ][265] Got return from WT5M3.foo.com for job 20240226211426871895
2024-02-26 15:15:38,327 [salt.utils.event :912 ][ERROR   ][265] Event iteration failed with exception: 'list' object has no attribute 'items'
2024-02-26 15:15:50,352 [salt.master      :2520][INFO    ][328] User salt Published command state.sls with jid 20240226211550349880
2024-02-26 15:15:54,883 [salt.utils.job   :86  ][INFO    ][348] Got return from ubuntu-eo9QsiYZ for job 20240226211550349880
2024-02-26 15:15:54,884 [salt.utils.event :912 ][ERROR   ][348] Event iteration failed with exception: 'list' object has no attribute 'items'
2024-02-26 15:15:56,338 [salt.master      :2520][INFO    ][344] User salt Published command state.sls with jid 20240226211556336553
2024-02-26 15:16:55,829 [salt.utils.job   :86  ][INFO    ][349] Got return from J12TMQ0XAV_WLYD3.foo.com for job 20240226211601616308
...
...
...

**Expected behavior**
salt masters should not leak memory and we can turn off our weekly application restart

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Versions Report**

Majority of the minions (7000) are running 3003.4.  About half are Ubuntu Linux the other half Windows 10.

This is the versions report from the syndic masters and master of masters:
```yaml
Salt Version:
          Salt: 3006.6

Python Version:
        Python: 3.10.4 (main, Apr 20 2022, 01:21:48) [GCC 10.3.1 20210424]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: Not Installed
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: alpine 3.14.6
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1034-aws
        system: Linux
       version: Alpine Linux 3.14.6

[johnl2323]

Here is the syndic masters main config/syndic related config:

/etc/salt/master

default_include: /etc/salt/master.d/*.conf
autosign_file: /etc/salt/autosign.conf

interface: 0.0.0.0
publish_port: 4505
ret_port: 4506

max_open_files: 250000
worker_threads: 32
pidfile: /var/run/salt-master.pid
timeout: 300
gather_job_timeout: 60
http_connect_timeout: 120
sock_pool_size: 48

tcp_keepalive: True
tcp_keepalive_idle: 180
tcp_keepalive_intvl: 60

syndic_master: salt-0.foo.com

/etc/salt/master.d/syndic-master.conf:

{
    "syndic_master_port": "45006"
}

[johnl2323]

It looks like the "fire_master failed: master could not be contacted. Request timed out." messages are because for some reason the syndic master can't communicate with the Master of Master (MoM).

Restarting the syndic master causes these messages to go away. Looks like the syndic process is not able to re-establish a connection to the MoM on its own without restarting salt master.

Now that the fire_master messages have gone away and the master and syndic logs look clean, we are still seeing a slow memory leak in this graph. The gap in the graph was when the syndic master we restarted.

[johnl2323]

We upgraded to 3006.7 today to see if that affects/improves the memory leak. We are seeing the number of open TCP connections on port 4506 growing over time. About 80,000 open connections on port 4506 in a week.

[Rocky-Web-Art]

I have setup like below but time to time I see memory consumption is too much and syndic nodes breaks communication to salt Master. But when we restart syndic service it release memory and start responding. We are using version 3006.8
Salt-Master - > Salt Master +Syndic -> Salt Master + Syndic -> Minion

[romanbakaleyko]

3006.8 same problem, salt-syndic leaks memory when that happens syndic are not responsive from MoM

[romanbakaleyko]

3006.9 problem still there

[dwoz]

Not reproducible locally: setup requires 3 syndics + MoM + ~8000 minions + shared EFS + days of runtime.

Since 3006.6 the following memory/resource-leak fixes landed on 3006.x/3007.x/3008.x:

• d4e2e07 Fix multiple memory leaks across master, minion and IPC transport (Maintenance loop LoadAuth/MasterMinion caching, IPC framing, ZMQ callback dedup)
• 33ad623 Fix resource leaks in master, minion, and API
• 42a0aba Fix process and file descriptor leaks in Salt Master
• 55d8a3b Fix transport zeromq test hang and syndic OOM in scenario tests
• 8a01f20 Quiet misleading replay-attack ERROR on duplicate minion returns (the '''extra return / list has no attribute items''' spam in the master log)

The '''fire_master failed: master could not be contacted''' symptom (syndic not reconnecting after MoM restart, driving the growing 4506 connections) is the subject of open PR #68923.

Closing as not-planned. Please retry on a current 3006.x/3007.x/3008.x release and, if the leak persists, open a fresh issue with the current version, exact rate of RSS growth, and gc/tracemalloc output from the leaking process.