Repository navigation
Description
Summary
On macOS (and Linux), when aistudio2api (or a host process embedding it) exits without completing a graceful Close()—such as during an unexpected crash, SIGKILL, force-quit, or portable update/restart—all active camoufox worker process groups (1 camoufox + 2 plugin-container processes per warm account) are reparented to PID 1 (launchd / init) and remain running indefinitely in the background.
On the next startup, camoufoxnative.RemoveStaleProfiles() deletes their temporary profile directories (aistudio-camoufox-*), but does not terminate the orphaned camoufox processes still running against those unlinked profiles. Each restart after an unclean exit leaks another ~1.5 GB of RAM (for the default WARM_WORKER_LIMIT=5).
Environment
- OS: macOS 27 (Darwin 27.2.0, arm64) — also affects Linux (
//go:build !windows) - Backend:
WAA_BACKEND=camoufox(default),WARM_WORKER_LIMIT=5
Observed Behavior (Evidence)
After an application update/restart, ps shows two full sets of 5 Camoufox workers (30 processes total, ~3.1 GB RSS):
- Orphaned set from previous run (
PPID=1, 5camoufox+ 10plugin-containerprocesses, ~1.6 GB RSS):PID PPID PGID ELAPSED RSS COMMAND 61513 1 61513 15:50:43 206816 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-4000314028 61535 1 61535 15:50:42 199904 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3967826041 61536 1 61536 15:50:42 202816 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-429003522 61537 1 61537 15:50:42 180000 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-1441533317 61538 1 61538 15:50:42 200112 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3930056266 - Active set from current run (
PPID=18047, 5camoufox+ 10plugin-containerprocesses, ~1.5 GB RSS):PID PPID PGID ELAPSED RSS COMMAND 18060 18047 18060 11:35 177248 camoufox --remote-debugging-port=0 -no-remote -headless -profile .../T/aistudio-camoufox-3627381987 ...
Inspecting the orphaned processes with lsof shows:
- Their WebDriver BiDi TCP ports (
127.0.0.1:50357, etc.) are inLISTENstate with no connected clients. - Their temporary profile directories (
aistudio-camoufox-4000314028, etc.) have already been unlinked fromos.TempDir()byRemoveStaleProfiles(), whilecamoufoxstill holds open the unlinked.parentlockinode:camoufox 61513 ... 8w REG 1,15 0 41174995 .../T/aistudio-camoufox-4000314028/.parentlock camoufox 61513 ... 9u IPv4 ... TCP localhost:50357 (LISTEN)
Root Cause Analysis
1. No parent-death lifecycle binding on non-Windows platforms (internal/camoufoxnative/process_other.go)
- On Windows (
internal/camoufoxnative/process_windows.go:36-55),serviceBrowserJob()assigns every Camoufox process to a Windows Job Object configured withJOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. If the Go host process terminates for any reason, the OS kernel automatically kills all Camoufox process trees. - On macOS and Linux (
internal/camoufoxnative/process_other.go:14-24),configureBrowserProcessonly setsSetpgid: true, andattachBrowserProcessis a no-op:Because each Camoufox worker is placed in its own process group (func configureBrowserProcess(command *exec.Cmd, _ bool) { command.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} } func attachBrowserProcess(*exec.Cmd) (browserTree, error) { return browserTree{}, nil }
PGID == PID), signals sent to the parent process group do not reach Camoufox, and if the parent process exits without completingClose(), the Camoufox process group is reparented toPID 1and runs forever.
2. RemoveStaleProfiles() unlinks profile directories without terminating the orphaned process group (internal/camoufoxnative/profile.go)
- In
createProfile()(internal/camoufoxnative/profile.go:20-33),flock.New(filepath.Join(profile, ".aistudio2api-profile.lock")).TryLock()opens the lock file in the parent Go process withO_CLOEXEC. The childcamoufoxprocess does not inherit this file descriptor (it locks its own.parentlockfile instead). - When the parent Go process exits abnormally, the OS releases the lock on
.aistudio2api-profile.lock, even though the orphanedcamoufoxprocess group is still alive. - On the next startup,
RemoveStaleProfiles()(internal/camoufoxnative/profile.go:56-85) succeeds inlock.TryLock()on.aistudio2api-profile.lockand immediately callsos.RemoveAll(profile):On Unix,lock := flock.New(lockPath) locked, err := lock.TryLock() if err != nil || !locked { continue } if err := removeProfile(profile, lock); err != nil { ... }
os.RemoveAllunlinks the directory entries while the runningcamoufoxprocesses keep their open file descriptors and continue running in memory indefinitely.
Suggested Fix
- Record and kill stale process groups in
RemoveStaleProfiles():- After
command.Start()inlaunchBrowser(), write the child PID/PGID (and/or executable identity) into the profile directory (e.g..aistudio2api-profile.pid, or inside.aistudio2api-profile.lock), or inspect.parentlock. - In
RemoveStaleProfiles(), after acquiring.aistudio2api-profile.lock, if a recorded PGID exists (and matches acamoufoxprocess running with-profile <profile>), sendsyscall.Kill(-pgid, syscall.SIGKILL)and wait briefly for exit before callingremoveProfile(profile, lock).
- After
- Bind child lifetime to parent process where supported:
- On Linux, set
Pdeathsig: syscall.SIGKILLonSysProcAttr. - On macOS/Unix, either clean up via the recorded PGID in
RemoveStaleProfiles()or monitor parent liveness so orphaned workers cannot accumulate across restarts.
- On Linux, set
Activity
RyoKusnadi commentedon Nov 4, 2026
@Mag1cFall I opened #106 for this. Could you please review it when you have a moment? Thank you!
jianli9158 commentedon Nov 4, 2026
Thanks @RyoKusnadi for putting together #106!
I reviewed the changes in #106—recording the process group ID in
.aistudio2api-profile.pidat launch, verifyingpgid+-profile <profile>viaps, and sendingSIGKILLto-pidinsideRemoveStaleProfiles()before unlinking the directory directly solves the core leak on macOS and Linux.Two small edge cases worth considering in
terminateStaleBrowser():syscall.Kill(-pid, syscall.SIGKILL)beforeremoveProfile():Since
SIGKILLteardown is asynchronous andTMPDIRpoints toprofile, callingos.RemoveAll(profile)immediately aftersyscall.Killcan occasionally race with dyingcamoufox/plugin-containerprocesses flushing files and returnENOTEMPTY. Pollingsyscall.Kill(-pid, 0)every ~10ms (up to e.g. 500ms) untilESRCHbeforeremoveProfile(profile, lock)guarantees clean directory removal.Xorgvirtual display process:On headless Linux without
DISPLAY(launcher.go:130),startVirtualDisplay()spawns anXorgprocess beforecommand.Start()in its own process group (Setpgid: true). If the Go service exits abnormally, thatXorgprocess is not incamoufox's process group (-pid), though its socket lives under/tmp. Recording or grouping it (if applicable) could be a nice follow-up.