refactor/Migrate file discovery from MANIFEST-based discovery to Git LFS-based discovery

[bwalsh]

Migrate file discovery from MANIFEST-based discovery to Git LFS-based discovery

--
 

• Status: Proposed
• Date: 2026-02-12
• Deciders: gen3_util maintainers
• Technical area: file tracking, ingest, commit pipeline
   

Context

 
Today, file discovery is centered around scanning MANIFEST/**/*.dvc entries and deriving file metadata from those descriptors.
This has worked well for explicit file registration, but it creates operational friction and scaling constraints:
 

1. Dual-source complexity
   Users manage data files and independent .dvc descriptors, which introduces synchronization errors (data moved without descriptor updates, stale realpath, etc.).
2. Discovery cost and coupling
   Recursive manifest scans couple discovery to repository layout conventions (MANIFEST/) and custom parsing code paths.
3. Poor alignment with Git-native workflows
   Teams already rely on Git object lifecycle primitives (add/commit/push/pull), while current discovery requires a separate conceptual model.
4. Interoperability limits
   External tools and CI systems understand Git and Git LFS pointers natively, but not project-specific MANIFEST/*.dvc semantics.
    
   The project is moving toward a Git-first experience, so file discovery should use Git LFS pointer tracking as the canonical source of discovered files.
    

Decision

 
We will migrate file discovery from MANIFEST/*.dvc scanning to Git LFS pointer-based discovery.
 

What becomes canonical

 

• Canonical source for discovered files: Git index/tree entries that are Git LFS pointer files.
• Canonical identifiers: LFS oid + pointer path (augmented with existing project/object mapping logic).
• Compatibility layer (transitional): Existing manifest discovery remains available behind a fallback/feature flag during rollout.
   

High-level behavior

 

1. Discover tracked files by reading Git-tracked paths and resolving which blobs are valid Git LFS pointers.
2. Parse pointer metadata (version, oid, size) and map to existing ingest/index data model.
3. Preserve current upload/index side effects where possible so downstream APIs remain stable.
4. Emit deprecation messaging for manifest-only paths and remove manifest discovery after migration exit criteria are met.
    

Rationale

 
Adopting Git LFS discovery:
 

• Reduces metadata duplication by letting Git/LFS tracking act as the system of record.
• Simplifies repository ergonomics (fewer generated manifest artifacts to reason about).
• Improves scalability by using Git operations rather than custom manifest tree walks.
• Aligns with common tooling (GitHub, GitLab, CI agents, developer IDE integrations).
• Enables cleaner future enhancements (change detection from Git diff, pointer validation in pre-commit/CI).
   

Considered options

 

Option A: Keep MANIFEST-based discovery only (current)

 

• Pros: No migration work; no behavior change.
• Cons: Preserves current complexity and long-term maintenance burden.
   

Option B: Hybrid forever (MANIFEST + Git LFS indefinitely)

 

• Pros: Maximum compatibility.
• Cons: Permanent dual-path complexity, divergent behavior risk, higher support burden.
   

Option C: Migrate to Git LFS with time-boxed compatibility window (chosen)

 

• Pros: Strategic simplification with controlled rollout risk.
• Cons: Requires migration tooling, communication, and short-term dual support.
   

Migration plan

 

Phase 0: Foundations

 

• Add an internal discovery abstraction (FileDiscoveryProvider) so callers are agnostic to source.
• Implement ManifestDiscoveryProvider (existing behavior) and GitLfsDiscoveryProvider.
• Add a feature flag (e.g., G3T_FILE_DISCOVERY=manifest\|git-lfs\|auto) with default auto.
   

Phase 1: Auto mode + parity validation

 

• In auto, prefer Git LFS discovery and fallback to manifest if no pointers are present.
• Add parity checks/telemetry in CI and integration tests to compare discovery outputs between providers.
• Define parity criteria: same discovered file count, stable identifiers, and equivalent required metadata coverage.
   

Phase 2: Default switch

 

• Switch default to git-lfs.
• Keep manifest mode as an explicit override with deprecation warnings.
• Publish migration guidance for users to track files via Git LFS and stop relying on generated manifest files.
   

Phase 3: Manifest retirement

 

• Remove manifest discovery provider and related fallback logic.
• Remove manifest-mode documentation and tests after one deprecation cycle.
   

Backward compatibility and deprecation policy

 

• One deprecation cycle will be maintained where manifest mode is still callable.
• Commands dependent on discovery must continue to function in repositories that are either:
• already migrated to Git LFS pointers, or
• not yet migrated but still using MANIFEST/*.dvc files.
• User-facing warnings must include clear remediation steps.
   

Risks and mitigations

 

1. Mixed repositories (some files in LFS, some in MANIFEST)

• Mitigation: auto mode and explicit conflict reporting.

2. Pointer parsing edge cases

• Mitigation: strict parser with informative errors and fixtures for malformed pointers.

3. Identifier drift

• Mitigation: parity tests and mapping rules that preserve existing object identity semantics.

4. Operational confusion during transition

• Mitigation: CLI warnings, release notes, and migration docs with examples.
   

Acceptance criteria

 
The migration is considered complete when all are true:
 

1. Default discovery mode is Git LFS.
2. Integration tests pass without requiring manifest discovery in standard workflows.
3. Parity checks show no critical divergence for supported repository states.
4. Manifest discovery code path has been removed.
    

Follow-up work

 

• Add a user-facing migration guide with step-by-step commands for enabling Git LFS tracking.
• Update CLI help text and docs to describe discovery mode behavior.
• Add a troubleshooting section for common migration failures (untracked files, stale pointers, missing LFS objects).
   

Addendum: refactor targets in current codebase

 
This addendum lists concrete classes and methods that should be refactored to implement the migration safely.
 

1) Discovery and parsing (primary)

 

• gen3_tracker/git/__init__.py::data_file_changes(manifest_path, update=False)
• Current role: walks MANIFEST/**/*.dvc and compares source file mtimes.
• Refactor: move behavior behind a provider interface (ManifestDiscoveryProvider) and add a parallel GitLfsDiscoveryProvider that reads Git-tracked LFS pointers.
• Outcome: callers stop depending directly on MANIFEST tree walking.
   
• gen3_tracker/git/__init__.py::dvc_data(committed_files)
• Current role: filters *.dvc paths and yields parsed DVC objects.
• Refactor: replace with a source-agnostic iterator (e.g., discovered_file_data(committed_files, mode)) that can emit records from LFS pointers and, during transition, from DVC.
   
• gen3_tracker/git/__init__.py::to_dvc(path)
• Current role: YAML parser for .dvc content.
• Refactor: keep as manifest compatibility parser; add LFS pointer parser (to_lfs_pointer) and normalization method to the common discovery model used by upload/index operations.
   

2) File registration path (creation/update semantics)

 

• gen3_tracker/git/adder.py::add_file(ctx, target)
• Current role: creates/updates MANIFEST/<target>.dvc and checks for existing tracked descriptors.
• Refactor: add a Git LFS-first path for tracking files (git lfs track/pointer verification) while retaining optional manifest compatibility mode.
   
• gen3_tracker/git/adder.py::add_url(ctx, target) and url_path(url)
• Current role: derives MANIFEST paths and writes descriptor metadata for URL-backed objects.
• Refactor: decouple URL import identity from MANIFEST path derivation and normalize to discovery records compatible with LFS mode.
   
• gen3_tracker/git/adder.py::write_dvc_file(target, yaml_data)
• Current role: central writer for MANIFEST/*.dvc files.
• Refactor: isolate as legacy compatibility utility; new default flow should not require this method for local file discovery.
   

3) CLI command surfaces that assume MANIFEST

 

• gen3_tracker/git/cli.py::add flow (uses data_file_changes, glob("MANIFEST/**/*.dvc"))
• Refactor: switch to discovery provider abstraction and avoid direct globbing of MANIFEST for primary behavior.
   
• gen3_tracker/git/cli.py::push flow (constructs dvc_objects via dvc_data(committed_files), then calls to_indexd/to_remote)
• Refactor: consume provider-normalized discovery records so push/index/upload behavior stays stable regardless of manifest vs LFS source.
   
• gen3_tracker/git/cli.py::ls, cat, and rm flows (object lookup and deletion paths under MANIFEST/...*.dvc)
• Refactor: implement object resolution via discovery provider first, with manifest path fallback only during deprecation window.
   

4) Upload/index adapters expecting DVC objects

 

• gen3_tracker/git/__init__.py::to_indexd(...)
• Current role: writes indexd entries from DVC objects.
• Refactor: accept a normalized discovery model (not strictly DVC) so LFS pointer-derived entries are first-class.
   
• gen3_tracker/git/__init__.py::{S3RemoteWriter, Gen3ClientRemoteWriter}.save(...)
• Current role: derive upload manifest rows from DVC and to_manifest(...).
• Refactor: map normalized discovery records to upload payloads; keep to_manifest(dvc) as compatibility adapter until manifest retirement.
   

Suggested sequencing for refactor execution

 

1. Introduce discovery interface and normalized record model.
2. Add Git LFS provider and parser with parity tests against existing manifest fixtures.
3. Migrate CLI read paths (ls, cat, push discovery) to provider abstraction.
4. Migrate write/add paths (add_file, add_url) to LFS-first behavior.
5. Retire direct MANIFEST assumptions and remove compatibility methods at end of deprecation cycle.