[maleadt]

Since binary archives were enabled by default (#930), every harvested kernel leaves a gpuarchiver-XXXXXX directory of ~300–400 KB in $(getconf DARWIN_USER_CACHE_DIR)com.apple.gpuarchiver. Metal assembles each MTLBinaryArchive that functions are added to in such a directory and never removes it: not on serializeToURL:, on release, at exit or on reboot. The Metal CI runners built up ~700k of them each (~290 GB) since late August, and macmini-aarch64-5 ran out of disk.

This isn't specific to Julia. It reproduces in plain Objective-C on macOS 15.8 (Intel), 26.6 (M3) and 27.0 (M2), from inside an app bundle too (which then uses <cache>/<bundle id>/com.apple.gpuarchiver), and with Metal 4's MTL4PipelineDataSetSerializer. It costs one directory per archive object, regardless of how many functions it holds. Apple's guidance is to harvest into a few archives and serialize them once, and that pattern leaks only a few directories, which is presumably why I couldn't find any report of this. Our one-archive-per-kernel design turns it into one leak per cache miss.

The directory is the archive's working state until serializeToURL: returns; removing it earlier makes serialization fail with "cannot open …/PersistentState". After that it is unused, and the archive still loads and hits. The private -[_MTLBinaryArchive getArchiveIDWithError:] returns its path, so each harvest now:

1. queries it after adding the function (the query creates the directory if there isn't one yet, and serialization switches to a new one, so the order matters),
2. serializes,
3. removes it.

The path is only used if the method exists and the result looks like a gpuarchiver-* directory under com.apple.gpuarchiver in the user cache. Otherwise we fall back to sweeping such directories older than five minutes every 256 harvests, which leaves in-progress harvests of concurrent sessions alone. The same sweep at exit cleans up after killed sessions and the backlog from earlier versions. A machine with hundreds of thousands of these directories will spend a while on that first sweep; for the CI runners we're cleaning them up separately.

[maleadt]

Hmm, we should probably reconsider if using binary archives is even worth it, given how buggy they are. We may be able to get most of the value from the automatic system shader cache ($(getconf DARWIN_USER_CACHE_DIR)com.apple.metal) already, now that we've made IR emission deterministic: the cache is keyed on the library contents, so identical metallibs across sessions should just hit.

Quick measurement on an M3 Pro, backend compile only (link_pipeline), 10 non-trivial kernels, each case a fresh process:

Case	ms/kernel
Cold, archives off	28.0
System cache warm, archives off	1.1
Archive miss (harvest), system cache warm	12.0
Archive miss (harvest), cold	34.5
Archive hit	2.1–2.5

So with a warm system cache, an archive hit is actually slower than not using archives, and every miss pays 7–11 ms extra for the harvest, plus the leaked directory this PR works around. As far as I can tell, that's how everyone else does it too:

• MLX, PyTorch MPS, llama.cpp, Dawn/ANGLE, WebKit and MoltenVK rely on the system cache.
• Games ship archives compiled ahead of time with metal-tt.
• Blender Cycles is the only runtime harvester I found, and it only harvests about once per kernel variant per release.

Where archives could still help is when the system cache is cold: after an OS update, or when Metal evicts entries because a workload compiles a lot of kernels. Before merging this I'd like to benchmark something realistic (the test suite, or a large KA application) with archives on and off. If the numbers look like the table above, I'd rather flip the default back to off and keep archives as opt-in, instead of relying on a private method to clean up after Metal.

[github-actions]

Metal Benchmarks

Details

Benchmark suite	Current: eb8d1c2	Previous: 6171f86	Ratio
array/accumulate/Float32/1d	391000 ns	391208 ns	1.00
array/accumulate/Float32/dims=1	366125 ns	368041 ns	0.99
array/accumulate/Float32/dims=1L	8845209 ns	8868625 ns	1.00
array/accumulate/Float32/dims=2	428125 ns	426375 ns	1.00
array/accumulate/Float32/dims=2L	2284167 ns	3018834 ns	0.76
array/accumulate/Int64/1d	852167 ns	839458 ns	1.02
array/accumulate/Int64/dims=1	912208 ns	907833 ns	1.00
array/accumulate/Int64/dims=1L	9574541 ns	9554791 ns	1.00
array/accumulate/Int64/dims=2	1206084 ns	1203667 ns	1.00
array/accumulate/Int64/dims=2L	6543625 ns	6556500 ns	1.00
array/broadcast	227333 ns	240750 ns	0.94
array/construct	2250 ns	2250 ns	1
array/permutedims/2d	454292 ns	456292 ns	1.00
array/permutedims/3d	1023792 ns	1033792 ns	0.99
array/permutedims/4d	1103750 ns	1118042 ns	0.99
array/private/copy	221458 ns	224542 ns	0.99
array/private/copyto!/cpu_to_gpu	211000 ns	212500 ns	0.99
array/private/copyto!/gpu_to_cpu	213291 ns	209250 ns	1.02
array/private/copyto!/gpu_to_gpu	216917 ns	220875 ns	0.98
array/private/iteration/findall/bool	1058125 ns	1055208 ns	1.00
array/private/iteration/findall/int	1215542 ns	1224875 ns	0.99
array/private/iteration/findfirst/bool	1152083 ns	1153042 ns	1.00
array/private/iteration/findfirst/int	1151125 ns	1158250 ns	0.99
array/private/iteration/findmin/1d	1196583 ns	1191250 ns	1.00
array/private/iteration/findmin/2d	1027666 ns	1031833 ns	1.00
array/private/iteration/logical	1539208 ns	1666875 ns	0.92
array/private/iteration/scalar	1399666 ns	1339708 ns	1.04
array/random/rand/Float32	416000 ns	416542 ns	1.00
array/random/rand/Int64	497125 ns	496250 ns	1.00
array/random/rand!/Float32	401375 ns	402667 ns	1.00
array/random/rand!/Int64	428000 ns	430833 ns	0.99
array/random/randn/Float32	386250 ns	387084 ns	1.00
array/random/randn!/Float32	370792 ns	374708 ns	0.99
array/reductions/mapreduce/Float32/1d	452458 ns	448542 ns	1.01
array/reductions/mapreduce/Float32/dims=1	353500 ns	351583 ns	1.01
array/reductions/mapreduce/Float32/dims=1L	606959 ns	618334 ns	0.98
array/reductions/mapreduce/Float32/dims=2	355166 ns	361167 ns	0.98
array/reductions/mapreduce/Float32/dims=2L	1218667 ns	1213334 ns	1.00
array/reductions/mapreduce/Int64/1d	635334 ns	638125 ns	1.00
array/reductions/mapreduce/Int64/dims=1	634417 ns	634125 ns	1.00
array/reductions/mapreduce/Int64/dims=1L	1011917 ns	1011583 ns	1.00
array/reductions/mapreduce/Int64/dims=2	786083 ns	790083 ns	0.99
array/reductions/mapreduce/Int64/dims=2L	2194667 ns	2200458 ns	1.00
array/reductions/reduce/Float32/1d	399625 ns	450083 ns	0.89
array/reductions/reduce/Float32/dims=1	356708 ns	353625 ns	1.01
array/reductions/reduce/Float32/dims=1L	610458 ns	615000 ns	0.99
array/reductions/reduce/Float32/dims=2	250542 ns	251792 ns	1.00
array/reductions/reduce/Float32/dims=2L	462334 ns	468917 ns	0.99
array/reductions/reduce/Int64/1d	635584 ns	634542 ns	1.00
array/reductions/reduce/Int64/dims=1	637375 ns	637875 ns	1.00
array/reductions/reduce/Int64/dims=1L	1012125 ns	1001083 ns	1.01
array/reductions/reduce/Int64/dims=2	260667 ns	262958 ns	0.99
array/reductions/reduce/Int64/dims=2L	674083 ns	673167 ns	1.00
array/shared/copy	129750 ns	129583 ns	1.00
array/shared/copyto!/cpu_to_gpu	38375 ns	37375 ns	1.03
array/shared/copyto!/gpu_to_cpu	37709 ns	36750 ns	1.03
array/shared/copyto!/gpu_to_gpu	38167 ns	37250 ns	1.02
array/shared/iteration/findall/bool	1057500 ns	1056167 ns	1.00
array/shared/iteration/findall/int	1223083 ns	1229333 ns	0.99
array/shared/iteration/findfirst/bool	980459 ns	978916 ns	1.00
array/shared/iteration/findfirst/int	988125 ns	988875 ns	1.00
array/shared/iteration/findmin/1d	1050000 ns	1062167 ns	0.99
array/shared/iteration/findmin/2d	1037084 ns	1035625 ns	1.00
array/shared/iteration/logical	1528792 ns	1524417 ns	1.00
array/shared/iteration/scalar	4535.714285714285 ns	4505.857142857143 ns	1.01
array/sorting/1d	1782333 ns	2095500 ns	0.85
array/sorting/2d	8488459 ns	8439000 ns	1.01
integration/byval/reference	1121625 ns	1118125 ns	1.00
integration/byval/slices=1	1123500 ns	1112042 ns	1.01
integration/byval/slices=2	1997167 ns	2028083 ns	0.98
integration/byval/slices=3	6786417 ns	6803417 ns	1.00
integration/metaldevrt	393542 ns	395375 ns	1.00
kernel/indexing	214958 ns	216958 ns	0.99
kernel/indexing_checked	395250 ns	396500 ns	1.00
kernel/launch	2115.6666666666665 ns	2060.222222222222 ns	1.03
kernel/rand	403042 ns	405375 ns	0.99
latency/import	1874368083 ns	1856280917 ns	1.01
latency/precompile	33860359042 ns	33572182250 ns	1.01
latency/ttfp	2296757334 ns	2280424459 ns	1.01
metal/synchronization/context	524.5631578947368 ns	522.2513089005236 ns	1.00
metal/synchronization/stream	421.2713567839196 ns	418.13065326633165 ns	1.01

This comment was automatically generated by workflow using github-action-benchmark.

[christiangnrd]

Good catch it was using up 36GiB on my laptop

[christiangnrd]

If the numbers look like the table above, I'd rather flip the default back to off and keep archives as opt-in, instead of relying on a private method to clean up after Metal.

Hard agree agree here. I was a bit disappointed in the lack of perceived CI speedup after archives were originally implemented, and I'm not a huge fan of having private methods as default behaviour. If there is no or minimal speed-up, I think we should also not really advertise it in versioninfo unless it's explicitly on.

I suspect the binary archives are designed for graphical applications where the shaders are finite and unchanging.