[AdaAibaby]

Fixes #3548.

Problem

CachingHandler in go-nfs maps NFS file handles (UUID) to (filesystem, path) via an LRU. The LRU size was hardcoded at const cacheLimit = 1024. When a sandbox installs a large package — for example npm i -g @openai/codex, which pulls 6 platform-specific optional packages simultaneously — the cache fills and oldest entries are evicted. Any subsequent RPC using an evicted handle returns NFS3ERR_STALE, which Linux translates to ESTALE (errno 116):

npm error code UNKNOWN
npm error syscall write
npm error errno -116
npm error UNKNOWN: unknown error, write

The problem is amplified by the fact that the single 1024-slot LRU was shared across all sandboxes on the node.

Fix

Replace the hardcoded constant with a configurable env var so operators can tune the cache without recompiling the orchestrator:

	Before	After
Cache size	const cacheLimit = 1024 (hardcoded)	NFS_PROXY_CACHE_LIMIT (env var, default 16384)
Memory cost at default	~150 KB	~2.4 MB (+2.25 MB per node)
Tunable at runtime	No	Yes — change Nomad job env and redeploy

Changes

• pkg/cfg/model.go: add NFSProxyCacheLimit int with env:"NFS_PROXY_CACHE_LIMIT" envDefault:"16384"
• pkg/nfsproxy/cfg/model.go: add CacheLimit int field
• pkg/nfsproxy/proxy.go: remove const cacheLimit = 1024; use config.CacheLimit
• pkg/factories/run.go: thread config.NFSProxyCacheLimit → nfscfg.Config.CacheLimit

Not in scope

Per-sandbox cache isolation (each mount gets its own CachingHandler) would eliminate cross-sandbox LRU contention entirely — tracked in #3548 as a longer-term improvement.

[Merlin0220]

We encountered the same issue. As a temporary workaround, we increased the cache size to 16 * 1024. We later tried switching to a per-sandbox cache, but found that it increased sandbox creation time. We have not yet identified the exact cause.

[leonmeijer]

NFS_PROXY_CACHE_LIMIT accepts zero or negative values and NewProxy passes them directly to helpers.NewCachingHandler. The pinned go-nfs helper ignores the LRU constructor errors, leaves nil caches, and then panics on the first handle operation; zero-value Config callers have the same failure. Validate a positive limit and give nfsproxy.Config a safe default before constructing the handler.

[ryan-sana]

Thanks a lot, @AdaAibaby. The npm i -g @openai/codex reproduction in #3548 made the hardcoded 1024-entry handle cache very easy to see.

Main now goes a step further than a single configurable limit: 5457911 gives every mounted volume its own file-handle cache, defaulting to 16384 entries and tunable with NFS_PROXY_HANDLE_CACHE_LIMIT. Large installs on one volume can no longer evict handles another volume is still using.

I'm closing this PR because main covers the cache limit. It also carried a few unrelated changes (the NBD inspection tool and the kill handler); if any of those still matter to you, a small focused PR or issue for each is the easiest way for us to look at them.