Skip to content

bug: orphaned Camoufox process groups leak on macOS/Linux after abnormal exit or restart, and RemoveStaleProfiles() only unlinks directories without killing them #98

Closed

Description

@jianli9158

Summary

On macOS (and Linux), when aistudio2api (or a host process embedding it) exits without completing a graceful Close()—such as during an unexpected crash, SIGKILL, force-quit, or portable update/restart—all active camoufox worker process groups (1 camoufox + 2 plugin-container processes per warm account) are reparented to PID 1 (launchd / init) and remain running indefinitely in the background.

On the next startup, camoufoxnative.RemoveStaleProfiles() deletes their temporary profile directories (aistudio-camoufox-*), but does not terminate the orphaned camoufox processes still running against those unlinked profiles. Each restart after an unclean exit leaks another ~1.5 GB of RAM (for the default WARM_WORKER_LIMIT=5).


Environment

  • OS: macOS 27 (Darwin 27.2.0, arm64) — also affects Linux (//go:build !windows)
  • Backend: WAA_BACKEND=camoufox (default), WARM_WORKER_LIMIT=5

Observed Behavior (Evidence)

After an application update/restart, ps shows two full sets of 5 Camoufox workers (30 processes total, ~3.1 GB RSS):

  1. Orphaned set from previous run (PPID=1, 5 camoufox + 10 plugin-container processes, ~1.6 GB RSS):
    PID   PPID  PGID  ELAPSED   RSS     COMMAND
    61513    1  61513 15:50:43  206816  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-4000314028
    61535    1  61535 15:50:42  199904  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3967826041
    61536    1  61536 15:50:42  202816  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-429003522
    61537    1  61537 15:50:42  180000  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-1441533317
    61538    1  61538 15:50:42  200112  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3930056266
    
  2. Active set from current run (PPID=18047, 5 camoufox + 10 plugin-container processes, ~1.5 GB RSS):
    PID   PPID  PGID  ELAPSED   RSS     COMMAND
    18060 18047 18060 11:35     177248  camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3627381987
    ...
    

Inspecting the orphaned processes with lsof shows:

  • Their WebDriver BiDi TCP ports (127.0.0.1:50357, etc.) are in LISTEN state with no connected clients.
  • Their temporary profile directories (aistudio-camoufox-4000314028, etc.) have already been unlinked from os.TempDir() by RemoveStaleProfiles(), while camoufox still holds open the unlinked .parentlock inode:
    camoufox 61513 ... 8w REG 1,15 0 41174995 .../T/aistudio-camoufox-4000314028/.parentlock
    camoufox 61513 ... 9u IPv4 ...          TCP localhost:50357 (LISTEN)
    

Root Cause Analysis

1. No parent-death lifecycle binding on non-Windows platforms (internal/camoufoxnative/process_other.go)

  • On Windows (internal/camoufoxnative/process_windows.go:36-55), serviceBrowserJob() assigns every Camoufox process to a Windows Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. If the Go host process terminates for any reason, the OS kernel automatically kills all Camoufox process trees.
  • On macOS and Linux (internal/camoufoxnative/process_other.go:14-24), configureBrowserProcess only sets Setpgid: true, and attachBrowserProcess is a no-op:
    func configureBrowserProcess(command *exec.Cmd, _ bool) {
        command.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    }
    
    func attachBrowserProcess(*exec.Cmd) (browserTree, error) {
        return browserTree{}, nil
    }
    Because each Camoufox worker is placed in its own process group (PGID == PID), signals sent to the parent process group do not reach Camoufox, and if the parent process exits without completing Close(), the Camoufox process group is reparented to PID 1 and runs forever.

2. RemoveStaleProfiles() unlinks profile directories without terminating the orphaned process group (internal/camoufoxnative/profile.go)

  • In createProfile() (internal/camoufoxnative/profile.go:20-33), flock.New(filepath.Join(profile, ".aistudio2api-profile.lock")).TryLock() opens the lock file in the parent Go process with O_CLOEXEC. The child camoufox process does not inherit this file descriptor (it locks its own .parentlock file instead).
  • When the parent Go process exits abnormally, the OS releases the lock on .aistudio2api-profile.lock, even though the orphaned camoufox process group is still alive.
  • On the next startup, RemoveStaleProfiles() (internal/camoufoxnative/profile.go:56-85) succeeds in lock.TryLock() on .aistudio2api-profile.lock and immediately calls os.RemoveAll(profile):
    lock := flock.New(lockPath)
    locked, err := lock.TryLock()
    if err != nil || !locked {
        continue
    }
    if err := removeProfile(profile, lock); err != nil { ... }
    On Unix, os.RemoveAll unlinks the directory entries while the running camoufox processes keep their open file descriptors and continue running in memory indefinitely.

Suggested Fix

  1. Record and kill stale process groups in RemoveStaleProfiles():
    • After command.Start() in launchBrowser(), write the child PID/PGID (and/or executable identity) into the profile directory (e.g. .aistudio2api-profile.pid, or inside .aistudio2api-profile.lock), or inspect .parentlock.
    • In RemoveStaleProfiles(), after acquiring .aistudio2api-profile.lock, if a recorded PGID exists (and matches a camoufox process running with -profile <profile>), send syscall.Kill(-pgid, syscall.SIGKILL) and wait briefly for exit before calling removeProfile(profile, lock).
  2. Bind child lifetime to parent process where supported:
    • On Linux, set Pdeathsig: syscall.SIGKILL on SysProcAttr.
    • On macOS/Unix, either clean up via the recorded PGID in RemoveStaleProfiles() or monitor parent liveness so orphaned workers cannot accumulate across restarts.

Activity

  1. RyoKusnadi commented on Nov 4, 2026

    @RyoKusnadi

    @Mag1cFall I opened #106 for this. Could you please review it when you have a moment? Thank you!

  2. jianli9158 commented on Nov 4, 2026

    @jianli9158
    Author

    Thanks @RyoKusnadi for putting together #106!

    I reviewed the changes in #106—recording the process group ID in .aistudio2api-profile.pid at launch, verifying pgid + -profile <profile> via ps, and sending SIGKILL to -pid inside RemoveStaleProfiles() before unlinking the directory directly solves the core leak on macOS and Linux.

    Two small edge cases worth considering in terminateStaleBrowser():

    1. Wait briefly for the process group to exit after syscall.Kill(-pid, syscall.SIGKILL) before removeProfile():
      Since SIGKILL teardown is asynchronous and TMPDIR points to profile, calling os.RemoveAll(profile) immediately after syscall.Kill can occasionally race with dying camoufox / plugin-container processes flushing files and return ENOTEMPTY. Polling syscall.Kill(-pid, 0) every ~10ms (up to e.g. 500ms) until ESRCH before removeProfile(profile, lock) guarantees clean directory removal.
    2. Linux Xorg virtual display process:
      On headless Linux without DISPLAY (launcher.go:130), startVirtualDisplay() spawns an Xorg process before command.Start() in its own process group (Setpgid: true). If the Go service exits abnormally, that Xorg process is not in camoufox's process group (-pid), though its socket lives under /tmp. Recording or grouping it (if applicable) could be a nice follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions