git drs ls-files -l Performance Notes
A report showed git drs ls-files -l taking about 1 minute 24 seconds to list roughly 900 files. The current implementation has several O(number of repository files) and O(number of DRS files) steps that explain that runtime on large or slow working trees.
Current execution path
For the default local inventory path, cmd/lsfiles.collectRows calls loadLFSInventory with no remote refs, which delegates to lfs.GetTrackedLfsFiles. That function:
- runs
git ls-files -z for every tracked file in the repository;
- runs one batch
git check-attr -z --stdin filter over that full path list;
- for every LFS/DRS-tracked path, tries to read the worktree file and parse it as a pointer;
- for hydrated files, falls back to
git show :<path> once per path to read the index pointer;
- after rows are collected,
collectRows calls isLocalized once per output row, which reads the same worktree path again and parses it again.
The -l flag only controls whether the full object ID is printed. It does not change inventory collection, so it still pays all of the above costs.
Inefficiencies found
Full-repository enumeration before path filtering
collectRows receives include/pathspec patterns, but those patterns are applied after loadLFSInventory has already scanned all tracked files. A narrow command such as git drs ls-files data/foo.bam still starts from git ls-files -z for the whole repository and then filters in Go.
Suggested improvement: pass user pathspecs into the inventory layer and use git ls-files -z -- <pathspec...> where possible. Glob-only patterns that are not valid Git pathspecs can still fall back to the existing Go matcher.
Attribute filtering over every tracked file
GetTrackedLfsFiles checks attributes for every tracked file, even though the command only needs LFS/DRS pointer candidates. On repositories with many ordinary source files, this makes command time proportional to total repository size rather than DRS inventory size.
Suggested improvement: use Git attributes or index metadata to narrow candidates earlier. Options include:
- run
git ls-files -z -- ':(attr:filter=lfs)' ':(attr:filter=drs)' if the supported Git versions in this project handle the required attr pathspecs consistently;
- maintain a small repo-local cache of DRS-tracked paths updated by
git drs track, git drs rm, and clean/pre-commit flows;
- at minimum, accept pathspecs in
listTrackedWorktreeFiles so large unrelated subtrees are skipped.
One git show process per hydrated pointer
When a DRS/LFS path is localized, readWorktreePointerInfo cannot parse the hydrated payload, so GetTrackedLfsFiles calls readIndexPointerInfo, which runs git show :<path> for that single path. In a checkout with 900 hydrated files, this can spawn about 900 Git subprocesses.
Suggested improvement: replace per-file git show calls with a single batch operation. Existing code already has readRefBlobsBatch for refs; a similar helper can batch index blobs with git cat-file --batch and specs such as :<path> or by first collecting object IDs from git ls-files -s -z.
Worktree files are read twice
For pointer files, readWorktreePointerInfo reads and parses the worktree file to populate object metadata. Later, isLocalized reads the same path again to decide whether to print * or -. Hydrated files are also read once in readWorktreePointerInfo and again in isLocalized.
Suggested improvement: carry localization state out of the inventory layer. LfsFileInfo already has IsPointer; collectRows can set Localized from that instead of rereading the file. If IsPointer == false for index-derived metadata, the row is localized.
Regex compilation happens during row filtering
matchesPattern compiles glob regexes each time it tests a row. With multiple patterns and many rows this repeats the same compilation work.
Suggested improvement: compile include/pathspec matchers once before iterating over rows, then reuse them for DRS lookup filtering and final row construction.
Remote fallback may scan remote refs unexpectedly
If the local inventory is empty and no explicit --git-remote was passed, collectRows resolves a default remote, lists its remote refs, and scans those refs. That fallback is useful when pointer files are absent from the working tree, but it can surprise users because a local listing may become a history/ref scan.
Suggested improvement: make the fallback explicit via a flag, or emit debug/trace timing so users can see when the command changes from worktree inventory to remote-ref inventory.
Recommended implementation order
- Stop rereading worktree files for localization by deriving
Localized from LfsFileInfo.IsPointer.
- Batch index pointer reads for hydrated files using one
git cat-file --batch or git ls-files -s -z plus cat-file --batch pipeline.
- Push pathspecs into
git ls-files before full inventory collection.
- Compile pattern matchers once per command invocation.
- Add optional trace timing around each inventory phase so future reports identify whether time is spent in
git ls-files, check-attr, worktree reads, index blob reads, DRS lookups, or output formatting.
git drs ls-files -lPerformance NotesA report showed
git drs ls-files -ltaking about 1 minute 24 seconds to list roughly 900 files. The current implementation has several O(number of repository files) and O(number of DRS files) steps that explain that runtime on large or slow working trees.Current execution path
For the default local inventory path,
cmd/lsfiles.collectRowscallsloadLFSInventorywith no remote refs, which delegates tolfs.GetTrackedLfsFiles. That function:git ls-files -zfor every tracked file in the repository;git check-attr -z --stdin filterover that full path list;git show :<path>once per path to read the index pointer;collectRowscallsisLocalizedonce per output row, which reads the same worktree path again and parses it again.The
-lflag only controls whether the full object ID is printed. It does not change inventory collection, so it still pays all of the above costs.Inefficiencies found
Full-repository enumeration before path filtering
collectRowsreceives include/pathspec patterns, but those patterns are applied afterloadLFSInventoryhas already scanned all tracked files. A narrow command such asgit drs ls-files data/foo.bamstill starts fromgit ls-files -zfor the whole repository and then filters in Go.Suggested improvement: pass user pathspecs into the inventory layer and use
git ls-files -z -- <pathspec...>where possible. Glob-only patterns that are not valid Git pathspecs can still fall back to the existing Go matcher.Attribute filtering over every tracked file
GetTrackedLfsFileschecks attributes for every tracked file, even though the command only needs LFS/DRS pointer candidates. On repositories with many ordinary source files, this makes command time proportional to total repository size rather than DRS inventory size.Suggested improvement: use Git attributes or index metadata to narrow candidates earlier. Options include:
git ls-files -z -- ':(attr:filter=lfs)' ':(attr:filter=drs)'if the supported Git versions in this project handle the required attr pathspecs consistently;git drs track,git drs rm, and clean/pre-commit flows;listTrackedWorktreeFilesso large unrelated subtrees are skipped.One
git showprocess per hydrated pointerWhen a DRS/LFS path is localized,
readWorktreePointerInfocannot parse the hydrated payload, soGetTrackedLfsFilescallsreadIndexPointerInfo, which runsgit show :<path>for that single path. In a checkout with 900 hydrated files, this can spawn about 900 Git subprocesses.Suggested improvement: replace per-file
git showcalls with a single batch operation. Existing code already hasreadRefBlobsBatchfor refs; a similar helper can batch index blobs withgit cat-file --batchand specs such as:<path>or by first collecting object IDs fromgit ls-files -s -z.Worktree files are read twice
For pointer files,
readWorktreePointerInforeads and parses the worktree file to populate object metadata. Later,isLocalizedreads the same path again to decide whether to print*or-. Hydrated files are also read once inreadWorktreePointerInfoand again inisLocalized.Suggested improvement: carry localization state out of the inventory layer.
LfsFileInfoalready hasIsPointer;collectRowscan setLocalizedfrom that instead of rereading the file. IfIsPointer == falsefor index-derived metadata, the row is localized.Regex compilation happens during row filtering
matchesPatterncompiles glob regexes each time it tests a row. With multiple patterns and many rows this repeats the same compilation work.Suggested improvement: compile include/pathspec matchers once before iterating over rows, then reuse them for DRS lookup filtering and final row construction.
Remote fallback may scan remote refs unexpectedly
If the local inventory is empty and no explicit
--git-remotewas passed,collectRowsresolves a default remote, lists its remote refs, and scans those refs. That fallback is useful when pointer files are absent from the working tree, but it can surprise users because a local listing may become a history/ref scan.Suggested improvement: make the fallback explicit via a flag, or emit debug/trace timing so users can see when the command changes from worktree inventory to remote-ref inventory.
Recommended implementation order
LocalizedfromLfsFileInfo.IsPointer.git cat-file --batchorgit ls-files -s -zpluscat-file --batchpipeline.git ls-filesbefore full inventory collection.git ls-files,check-attr, worktree reads, index blob reads, DRS lookups, or output formatting.