demonstration/anvil-gregor

[bwalsh]

Use Case: Gregor AnVIL Repository Managed with git-drs

--
 

Goal

 
Create a Git repository for the Gregor project that stores reviewable project structure, manifests, analysis code, and DRS pointer files in Git while using git-drs to hydrate large data files from the Gregor data hosted on the AnVIL platform.
 
The repository should let project users clone normal Git history, inspect which Gregor data objects are referenced, and download only the files they need through AnVIL DRS access controls.
 

Primary User Story

 
As a Gregor project analyst, I want a Git repository that contains git-drs pointer files for Gregor datasets on AnVIL so that I can version analyses and data references together without committing large controlled-access files to Git.
 

Supporting User Stories

 

Bootstrap repository owner

 
As a repository owner, I want a bootstrap utility that queries the AnVIL DRS server and creates git-drs pointer files so that I can initialize or refresh the repository from authoritative AnVIL metadata instead of hand-writing pointers.
 

Project analyst

 
As an analyst, I want to clone the repository and run git drs pull for selected files or directories so that I can hydrate only the Gregor files required for my current analysis.
 

Data steward

 
As a data steward, I want pointer files to be generated from AnVIL DRS records so that Git history records stable DRS identifiers, object sizes, checksums where available, and intended repository paths.
 

Reviewer or collaborator

 
As a reviewer, I want pull requests to show changes to manifests, analysis code, and DRS pointers so that I can evaluate which data inputs changed without downloading controlled-access payloads.
 

Scope

 
In scope:
 

• Create a new Git repository initialized for git-drs.
• Configure a git-drs remote for the AnVIL-hosted Gregor DRS service or compatible gateway.
• Write a bootstrap utility that interacts with the AnVIL DRS server.
• Generate repository-relative pointer files and local git-drs metadata from AnVIL DRS object metadata.
• Track generated data paths with .gitattributes so Git stores pointers instead of large payloads.
• Support selective hydration with git drs pull -I <pattern>.
• Document the bootstrap and refresh workflow.
   
  Out of scope for the first increment:
   
• Bypassing AnVIL authentication or authorization.
• Mirroring all Gregor payload bytes into Git or a separate unmanaged bucket.
• Editing AnVIL source metadata from the Git repository.
• Replacing Terra/AnVIL workspace data management.
   

Assumptions

 

• Gregor data objects are discoverable through an AnVIL DRS endpoint or an AnVIL-compatible service that returns GA4GH DRS object metadata.
• Users have valid AnVIL credentials and authorization for the Gregor objects they hydrate.
• The bootstrap utility can map AnVIL DRS objects to deterministic repository paths using a manifest, workspace table export, DRS object metadata, or a project-specific naming policy.
• git-drs can hydrate pointer files through configured DRS access URLs after authentication.
   

Proposed Repository Layout

 

gregor-drs-repo/
README.md
.gitattributes
manifests/
gregor-drs-manifest.tsv
data/
<study-or-cohort>/
<sample-or-family>/
<file>.cram
<file>.crai
workflows/
notebooks/
scripts/
bootstrap-gregor-anvil-drs

 

Bootstrap Utility Requirement

 
A bootstrap utility needs to be written because the initial repository state should be derived from the AnVIL DRS server rather than manually assembled.
 
The utility should:
 

1. Authenticate to AnVIL using the supported local credential mechanism.
2. Read an input manifest, workspace table export, or query parameters that identify the Gregor DRS objects to include.
3. Call the AnVIL DRS server for each object.
4. Validate required metadata, including DRS ID, name or suggested path, size, checksum when available, and access method metadata.
5. Create or update git-drs pointer files at deterministic repository-relative paths.
6. Create or update any local metadata files required by git-drs for pointer hydration.
7. Write a manifest that records the DRS ID, repository path, size, checksum, source AnVIL workspace or table row, and bootstrap timestamp.
8. Leave payload bytes unhydrated by default.
9. Be idempotent so it can refresh pointer metadata when AnVIL records change.
10. Report additions, changes, deletions, and metadata conflicts before the user commits.
     

Acceptance Criteria

 

• A maintainer can create an empty Git repository and configure it with git-drs for the Gregor AnVIL DRS endpoint.
• The bootstrap utility can generate pointer files for a representative Gregor manifest without downloading payload bytes.
• Generated files are tracked by .gitattributes with the git-drs filter.
• git status shows reviewable pointer and manifest changes after bootstrap.
• A collaborator can clone the repository, authenticate to AnVIL, and run git drs pull -I <path-or-pattern> to hydrate selected files.
• Pointer refreshes are deterministic: rerunning the bootstrap utility with unchanged AnVIL metadata produces no Git diff.
• Metadata conflicts, missing checksums, inaccessible DRS IDs, and path collisions are reported clearly.
• Documentation explains required credentials, bootstrap inputs, expected outputs, and common troubleshooting steps.
   

Example Workflow

 

mkdir gregor-drs-repo
cd gregor-drs-repo
git init
 
git drs install
git drs init
git drs remote add gen3 anvil GREGOR/AnVIL --cred ~/.config/anvil/credentials.json
 
git drs track "data/**"
git add .gitattributes
 
scripts/bootstrap-gregor-anvil-drs \
--drs-endpoint https://drs.anvilproject.org \
--manifest manifests/gregor-source.tsv \
--output-manifest manifests/gregor-drs-manifest.tsv \
--data-root data
 
git add README.md .gitattributes manifests/ data/ scripts/
git commit -m "Bootstrap Gregor AnVIL DRS pointers"

 
A collaborator can then hydrate a subset:
 

git clone <gregor-drs-repo-url>
cd gregor-drs-repo
git drs remote add gen3 anvil GREGOR/AnVIL --cred ~/.config/anvil/credentials.json
git drs pull -I "data/cohort-a/**"

 

Open Design Questions

 

• Which AnVIL endpoint and authentication flow should be considered canonical for Gregor DRS discovery?
• Should the bootstrap utility be a new git-drs subcommand or a project-local script in the Gregor repository?
• What manifest format should be the source of truth for DRS IDs and desired repository paths?
• How should the utility represent DRS records that lack checksums?
• Should deleted or withdrawn AnVIL records remove pointers automatically or require an explicit tombstone workflow?
• What validation is required before publishing a refreshed pointer commit?

[bwalsh]

ADR 0004 / GitHub Issue: Support Terra/AnVIL DRS downloads in git-drs

Status

Proposed

Issue type

Architecture Decision Record / feature gap analysis

Download

Download this ADR as Markdown

Summary

git-drs currently supports reviewable pointer files and selective hydration, but it is not yet a Terra/AnVIL-native DRS downloader. The current implementation is oriented around Syfon/Gen3-style remotes, scope-aware checksum lookup, and Git LFS-shaped SHA256 pointer identities. Terra/AnVIL workflows need DRS-URI-first pointer identity, Terra-compatible authentication, Terra/Martha or GA4GH DRS access URL resolution, and a bootstrap/import workflow that can generate deterministic pointer files from AnVIL manifests or workspace table exports without downloading payload bytes.

Background

Gregor project users want a Git repository that stores project structure, manifests, analysis code, and git-drs pointer files while leaving controlled-access payload bytes on AnVIL. Users should be able to clone normal Git history, review which data objects are referenced, authenticate through AnVIL/Terra access controls, and hydrate only selected paths with commands such as:

git drs pull -I "data/cohort-a/**"

Terra/AnVIL DRS downloads typically start from stable DRS URIs and use Terra-compatible authentication and resolution tooling to obtain authorized access URLs. A repository bootstrap workflow also needs to query AnVIL DRS metadata and generate pointer files and manifests deterministically.

Current state

git-drs already provides several useful building blocks:

• Git-compatible pointer workflows.
• git drs track for tracked data path patterns.
• git drs pull -I <pattern> for selective hydration.
• git drs add-ref <drs_uri> <dst path> as an initial single-object reference workflow.
• Gen3/Syfon remote configuration.
• Local metadata and cache plumbing for DRS-backed objects.

However, these pieces do not yet compose into a complete Terra/AnVIL workflow.

Decision

Add first-class Terra/AnVIL DRS support to git-drs as a read-oriented, DRS-URI-first workflow. The implementation should treat the DRS URI as canonical remote identity, use a deterministic derived local OID only as pointer/cache identity, and resolve downloads through a provider-specific Terra/AnVIL resolver rather than requiring checksum-first lookup through Syfon/Gen3 extensions.

Goals

• Allow repositories to store stable Terra/AnVIL DRS references in Git without committing payload bytes.
• Support selective hydration using existing include-filter semantics.
• Support Terra/AnVIL authentication mechanisms such as Google Application Default Credentials or bearer tokens.
• Resolve DRS URIs to authorized access URLs using GA4GH DRS and/or Terra/Martha-compatible resolution.
• Generate deterministic pointer files from manifests or workspace table exports.
• Preserve reviewability of data-reference changes in pull requests.
• Report metadata conflicts, missing checksums, inaccessible DRS IDs, and path collisions clearly.

Non-goals

• Bypass AnVIL authentication or authorization.
• Mirror Gregor payload bytes into Git or unmanaged buckets.
• Replace Terra workspace data management.
• Edit AnVIL source metadata from a Git repository.
• Require all Terra/AnVIL DRS records to have SHA256 checksums.

Gap analysis

Gap 1: Missing Terra/AnVIL remote type

Current remote configuration is centered on Gen3/Syfon remotes with an organization/project scope. Terra/AnVIL workflows need a remote type that can be configured with a DRS endpoint, optional Martha/resolution endpoint, and Terra-compatible authentication.

Example target CLI:

git drs remote add terra anvil \
  --drs-endpoint https://drs.anvilproject.org \
  --auth google-adc \
  --mode read-only

Gap 2: Hydration is checksum-first instead of DRS-URI-first

Current hydration resolves pointer OIDs through checksum lookup and then fetches an access URL for the matched DRS object. Terra/AnVIL pointer hydration should start from the canonical DRS URI stored in pointer metadata.

Required behavior:

1. Read the canonical DRS URI from local metadata.
2. Resolve DRS object metadata from the Terra/AnVIL remote.
3. Select an access method.
4. Resolve an authorized access URL.
5. Download payload bytes into the cache/worktree.

Gap 3: DRS URI is not yet first-class pointer metadata

The existing design still has code paths where local pointer identity is treated as oid sha256:<64hex>. Terra/AnVIL requires a split identity model:

• drs_uri: canonical remote identity.
• local_oid: deterministic compact pointer/cache identity.
• checksums: optional content metadata.

Recommended local OID derivation:

sha256("git-drs-object:v1\ndrs_uri=<normalized-drs-uri>\n")

Gap 4: SHA256 is too central

Terra/AnVIL records may lack SHA256 checksums or provide different checksum types. Missing SHA256 should not prevent pointer creation if a stable DRS URI and size metadata are available.

Required behavior:

• DRS URI is required.
• Size should be captured when available.
• SHA256 should be captured when available.
• Missing checksum should be reported as a warning or policy-controlled condition, not always a hard failure.

Gap 5: Missing Terra/Martha-compatible resolver

Terra downloads often require a resolver layer that can turn a DRS URI into an authorized signed URL. git-drs needs a provider abstraction with Terra/AnVIL implementation.

Suggested interface:

type DRSResolver interface {
    GetObject(ctx context.Context, drsURI string) (*DRSObject, error)
    GetAccessURL(ctx context.Context, drsURI string, accessIDOrType string) (*AccessURL, error)
}

Expected implementations:

• syfonResolver
• gen3Resolver
• ga4ghResolver
• terraResolver

Gap 6: Missing manifest bootstrap/import workflow

git-drs needs a batch utility to initialize or refresh repositories from authoritative AnVIL data sources.

Target command shape:

git drs import-drs-manifest \
  --remote anvil \
  --input manifests/gregor-source.tsv \
  --drs-uri-column drs_uri \
  --path-column repo_path \
  --output-manifest manifests/gregor-drs-manifest.tsv \
  --data-root data

Required behavior:

• Read TSV/CSV manifests or workspace table exports.
• Resolve each DRS URI.
• Validate metadata.
• Write deterministic pointer files.
• Write or update local git-drs metadata.
• Write a deterministic output manifest.
• Support dry-run mode.
• Report additions, changes, deletions, collisions, inaccessible IDs, and metadata conflicts.

Gap 7: Missing steward/reviewer manifest output

Bootstrap should create a stable, reviewable manifest recording at least:

• repository path;
• DRS URI;
• DRS ID;
• object name;
• size;
• checksum type and value when available;
• access method types;
• source workspace/table/row metadata when available;
• metadata status.

Run-specific details such as timestamps should either be omitted from the committed manifest or isolated in an optional report artifact to preserve idempotency.

Gap 8: Current scope model does not match Terra discovery

Terra/AnVIL source metadata may come from workspace tables, Data Repository Service snapshots, manifests, or other project-specific exports. The canonical scope for hydration should be the DRS URI stored in metadata, not a Gen3-style organization/project plus bucket mapping.

Gap 9: Access method selection needs policy

Terra/AnVIL DRS objects may expose multiple access methods. git-drs should support explicit and default access preferences.

Example target flags:

git drs pull --access-method auto
git drs pull --access-method https
git drs pull --access-method gs

Gap 10: Authentication UX is not Terra-native

Terra users should be able to configure auth using expected local mechanisms.

Target options:

--auth google-adc
--auth gcloud
--auth bearer-token

Error messages should distinguish missing credentials, expired credentials, 401 unauthorized, 403 forbidden, inaccessible controlled-access records, and expired signed URLs.

Gap 11: Need read-only remote mode

Gregor/AnVIL repositories reference existing controlled-access data. They should not accidentally trigger payload upload or DRS registration.

Target behavior:

git drs remote add terra anvil ... --mode read-only

In read-only mode:

• git drs pull works.
• manifest import/bootstrap works.
• upload/register push behavior is disabled or fails with a clear message.

Gap 12: Missing Terra/AnVIL documentation

Add documentation covering:

• required credentials;
• remote setup;
• manifest input format;
• bootstrap workflow;
• refresh workflow;
• selective hydration;
• missing checksum policy;
• controlled-access troubleshooting;
• common 401/403 causes.

Proposed implementation plan

Phase 1: Resolver and identity foundation

• Add explicit identity model with canonical DRS URI, derived local OID, and optional checksums.
• Persist DRS URI in local metadata for pointer hydration.
• Add DRS-URI-first hydration path.
• Keep existing pointer format compatible where possible.

Phase 2: Terra/AnVIL remote and auth

• Add terra or generic ga4gh remote type.
• Support Google ADC and bearer token authentication.
• Add Terra/Martha-compatible access URL resolution.
• Add provider-specific diagnostics for authorization failures.

Phase 3: Manifest bootstrap/import

• Add git drs import-drs-manifest or equivalent.
• Generate pointer files without downloading payload bytes.
• Generate deterministic output manifests.
• Add dry-run, conflict reporting, and deletion policy options.

Phase 4: Documentation and acceptance tests

• Add docs/terra-anvil.md.
• Add unit tests for DRS-URI-derived local OIDs.
• Add resolver tests for mocked GA4GH/Terra DRS responses.
• Add integration-style tests for manifest import and selective hydration.

Acceptance criteria

• A maintainer can configure a Terra/AnVIL read-only remote without Gen3/Syfon bucket mapping.
• A bootstrap/import command can generate pointer files from a representative AnVIL manifest without downloading payload bytes.
• Generated pointer metadata includes canonical DRS URIs.
• Missing SHA256 checksums are handled according to documented policy.
• Rerunning bootstrap with unchanged AnVIL metadata produces no Git diff.
• git drs pull -I <path-or-pattern> hydrates selected Terra/AnVIL pointer files after authentication.
• 401/403 and inaccessible DRS records produce actionable error messages.
• Documentation explains credentials, bootstrap inputs, outputs, refresh workflow, and troubleshooting.

Open questions

• Should the canonical command be git drs remote add terra or a generic git drs remote add ga4gh with Terra-specific auth/resolution options?
• Which AnVIL/Terra endpoint should be treated as canonical for Gregor DRS discovery?
• Should Martha/Terra resolution be implemented directly or invoked through Terra Notebook Utils behavior?
• What should the default policy be for records that lack checksums?
• Should deleted or withdrawn AnVIL records remove pointers automatically, create tombstones, or require explicit maintainer action?
• Should bootstrap/import live in core git-drs or as a project-local Gregor script first?

Priority

High for Terra/AnVIL adoption.

Labels

• adr
• enhancement
• terra
• anvil
• drs
• bootstrap
• controlled-access

[matthewpeterkort]

Alot of this stuff can be solved by making add remote mode "terra" and assuming that auth/provider can be wired in at that step. The remaining most pressing issue would be the sha256 compat and drs URL compat since the first is non-negotiable in git-drs and the 2nd is not well defined in git-drs. T

hat said similar code already exists for "add-url" and a new CLI function might be needed for "add-ref" and then you specify the remote type, in this case "terra" and some type of drs resolver would need to resolve given that the client knows it's a terra ref I think this can be donw quite easily, reaslistically.

overall I don't see this as as big of a lift as this doc makes it seem. But it is not trivial either.

[kellrott]

Can we enable the system to properly respond to git-drs specific pointer files. Possible example of what it would look like

version https://calypr.github.io/spec/v1
oid drs://cgc-ga4gh-api.sbgenomics.com/4c33ae65e4b08832ce3d94e9c
size 11305017366

Note, this record comes from https://docs.cancergenomicscloud.org/reference/get-info-about-a-drs-object

[bwalsh]

ADR 0004 / GitHub Issue: Support Terra/AnVIL DRS downloads in git-drs

Status

Proposed

Issue type

Architecture Decision Record / feature gap analysis

Summary

git-drs already has most of the repository and hydration machinery needed for Terra/AnVIL, so the remaining work should be scoped as an incremental provider/identity extension rather than a wholesale redesign. The most direct path is to add a terra remote mode, wire Terra authentication and provider resolution at remote setup time, and then close the two compatibility gaps that are currently not well-defined for Terra references: SHA256-shaped local identity and direct DRS URL pointer identity. A manifest bootstrap workflow remains useful, but it can build on the same lower-level add-ref/resolver work instead of being treated as a separate large subsystem.

Background

Gregor project users want a Git repository that stores project structure, manifests, analysis code, and git-drs pointer files while leaving controlled-access payload bytes on AnVIL. Users should be able to clone normal Git history, review which data objects are referenced, authenticate through AnVIL/Terra access controls, and hydrate only selected paths with commands such as:

git drs pull -I "data/cohort-a/**"

Terra/AnVIL DRS downloads typically start from stable DRS URIs and use Terra-compatible authentication and resolution tooling to obtain authorized access URLs. A repository bootstrap workflow also needs to query AnVIL DRS metadata and generate pointer files and manifests deterministically.

Current state

git-drs already provides several useful building blocks:

• Git-compatible pointer workflows.
• git drs track for tracked data path patterns.
• git drs pull -I <pattern> for selective hydration.
• git drs add-ref <drs_uri> <dst path> as an initial single-object reference workflow.
• Gen3/Syfon remote configuration.
• Local metadata and cache plumbing for DRS-backed objects.

However, these pieces do not yet compose into a complete Terra/AnVIL workflow.

Decision

Add first-class Terra/AnVIL DRS support as an incremental extension of the existing remote and reference workflows. The primary decision is to introduce git drs remote add terra ... so authentication and provider behavior are known at remote configuration time. Once a pointer or reference is associated with a Terra remote, git-drs can use a Terra-aware resolver to translate the DRS URI into downloadable access URLs. The main unresolved design decision is how to represent DRS URL identity in pointer files while preserving the existing SHA256-oriented cache and compatibility assumptions.

Goals

• Allow repositories to store stable Terra/AnVIL DRS references in Git without committing payload bytes.
• Support selective hydration using existing include-filter semantics.
• Support Terra/AnVIL authentication mechanisms such as Google Application Default Credentials or bearer tokens.
• Resolve DRS URIs to authorized access URLs using GA4GH DRS and/or Terra/Martha-compatible resolution.
• Generate deterministic pointer files from manifests or workspace table exports.
• Preserve reviewability of data-reference changes in pull requests.
• Report metadata conflicts, missing checksums, inaccessible DRS IDs, and path collisions clearly.

Non-goals

• Bypass AnVIL authentication or authorization.
• Mirror Gregor payload bytes into Git or unmanaged buckets.
• Replace Terra workspace data management.
• Edit AnVIL source metadata from a Git repository.
• Require all Terra/AnVIL DRS records to have SHA256 checksums.

Team feedback incorporated

This ADR is revised to reflect team feedback that Terra support should be smaller than a broad re-architecture. The preferred shape is:

• add a terra remote mode and wire auth/provider behavior there;
• extend add-ref or add a related reference command so callers can create Terra-backed DRS references explicitly;
• resolve Terra references through a remote-aware DRS resolver;
• focus design discussion on SHA256 compatibility and DRS URL pointer compatibility;
• keep manifest/bootstrap support as a batch layer over the reference primitive.

The ADR also captures an open pointer-format question raised in review: whether git-drs should recognize a git-drs-specific pointer file whose oid is a DRS URI rather than a Git LFS-style SHA256 value, for example:

version https://calypr.github.io/spec/v1
oid drs://cgc-ga4gh-api.sbgenomics.com/4c33ae65e4b08832ce3d94e9c
size 11305017366

That pointer shape is attractive for reviewability and direct DRS compatibility, but it needs an explicit cache-key strategy because existing git-drs code relies on SHA256-shaped OIDs in several places.

Gap analysis

Gap 1: Missing Terra/AnVIL remote type

Current remote configuration is centered on Gen3/Syfon remotes with an organization/project scope. Team feedback suggests that a terra remote mode is the cleanest and smallest enabling feature: auth, provider selection, and resolver behavior can all be wired when the remote is added.

Example target CLI:

git drs remote add terra anvil \
  --drs-endpoint https://drs.anvilproject.org \
  --auth google-adc \
  --mode read-only

Gap 2: Terra references need a remote-aware add-ref path

Similar reference-building behavior already exists in add-url, and add-ref already has the right conceptual role for existing DRS objects. The missing piece is a Terra-aware reference flow where the command knows which remote type owns the reference and can call the matching resolver.

Possible target command shapes:

git drs add-ref --remote anvil drs://example/object data/object.cram
git drs add-ref --remote-type terra drs://example/object data/object.cram

Required behavior:

1. Resolve the DRS URI using the selected remote.
2. Create a pointer and local metadata without downloading payload bytes.
3. Preserve the DRS URI as canonical remote identity.
4. Store or derive whatever SHA256-compatible local identity git-drs needs for cache compatibility.

Gap 3: DRS URI pointer compatibility is not well defined

The existing design still has code paths where local pointer identity is treated as oid sha256:<64hex>. Terra/AnVIL can work with that constraint if the DRS URI is stored as first-class metadata and a deterministic SHA256-shaped local cache key is derived from it. However, review feedback also proposes a git-drs-specific pointer format where the pointer oid itself is a DRS URI.

Option A: keep Git LFS-shaped pointers and store DRS URI in sidecar/local metadata.

version https://git-lfs.github.com/spec/v1
oid sha256:<derived-local-oid>
size <size>

Option B: add a git-drs pointer spec that accepts DRS URI OIDs directly.

version https://calypr.github.io/spec/v1
oid drs://example/object
size <size>

Option A is likely lower-risk because it preserves current SHA256 assumptions. Option B is more reviewable and direct, but requires parser, cache fanout, inventory, and hydration updates so DRS URI OIDs can be normalized into safe local cache keys.

Gap 4: SHA256 compatibility is still non-negotiable internally

Review feedback notes that SHA256 compatibility remains non-negotiable in current git-drs. Terra/AnVIL records may still lack SHA256 checksums or provide different checksum types, so the design should preserve a SHA256-shaped local/cache identity even when that identity is derived from the DRS URI rather than from payload content.

Required behavior:

• DRS URI is required.
• Size should be captured when available.
• SHA256 should be captured when available.
• Missing checksum should be reported as a warning or policy-controlled condition, not always a hard failure.

Gap 5: Missing Terra/Martha-compatible resolver

Terra downloads often require a resolver layer that can turn a DRS URI into an authorized signed URL. git-drs needs a provider abstraction with Terra/AnVIL implementation.

Suggested interface:

type DRSResolver interface {
    GetObject(ctx context.Context, drsURI string) (*DRSObject, error)
    GetAccessURL(ctx context.Context, drsURI string, accessIDOrType string) (*AccessURL, error)
}

Expected implementations:

• syfonResolver
• gen3Resolver
• ga4ghResolver
• terraResolver

Gap 6: Manifest bootstrap/import should be a batch wrapper over reference creation

git-drs still needs a batch utility to initialize or refresh repositories from authoritative AnVIL data sources, but this should be implemented as a thin wrapper around the Terra-aware reference primitive rather than a separate resolver path.

Target command shape:

git drs import-drs-manifest \
  --remote anvil \
  --input manifests/gregor-source.tsv \
  --drs-uri-column drs_uri \
  --path-column repo_path \
  --output-manifest manifests/gregor-drs-manifest.tsv \
  --data-root data

Required behavior:

• Read TSV/CSV manifests or workspace table exports.
• Resolve each DRS URI.
• Validate metadata.
• Write deterministic pointer files.
• Write or update local git-drs metadata.
• Write a deterministic output manifest.
• Support dry-run mode.
• Report additions, changes, deletions, collisions, inaccessible IDs, and metadata conflicts.

Gap 7: Missing steward/reviewer manifest output

Bootstrap should create a stable, reviewable manifest recording at least:

• repository path;
• DRS URI;
• DRS ID;
• object name;
• size;
• checksum type and value when available;
• access method types;
• source workspace/table/row metadata when available;
• metadata status.

Run-specific details such as timestamps should either be omitted from the committed manifest or isolated in an optional report artifact to preserve idempotency.

Gap 8: Current scope model does not match Terra discovery

Terra/AnVIL source metadata may come from workspace tables, Data Repository Service snapshots, manifests, or other project-specific exports. The canonical scope for hydration should be the DRS URI stored in metadata, not a Gen3-style organization/project plus bucket mapping.

Gap 9: Access method selection needs policy

Terra/AnVIL DRS objects may expose multiple access methods. git-drs should support explicit and default access preferences.

Example target flags:

git drs pull --access-method auto
git drs pull --access-method https
git drs pull --access-method gs

Gap 10: Authentication UX is not Terra-native

Terra users should be able to configure auth using expected local mechanisms.

Target options:

--auth google-adc
--auth gcloud
--auth bearer-token

Error messages should distinguish missing credentials, expired credentials, 401 unauthorized, 403 forbidden, inaccessible controlled-access records, and expired signed URLs.

Gap 11: Need read-only remote mode

Gregor/AnVIL repositories reference existing controlled-access data. They should not accidentally trigger payload upload or DRS registration.

Target behavior:

git drs remote add terra anvil ... --mode read-only

In read-only mode:

• git drs pull works.
• manifest import/bootstrap works.
• upload/register push behavior is disabled or fails with a clear message.

Gap 12: Missing Terra/AnVIL documentation

Add documentation covering:

• required credentials;
• remote setup;
• manifest input format;
• bootstrap workflow;
• refresh workflow;
• selective hydration;
• missing checksum policy;
• controlled-access troubleshooting;
• common 401/403 causes.

Proposed implementation plan

Phase 1: Terra remote and resolver

• Add git drs remote add terra ....
• Wire Terra auth/provider configuration at remote setup time.
• Add a Terra-aware resolver used by reference creation and hydration.
• Add provider-specific diagnostics for authorization failures.

Phase 2: Reference creation and pointer identity

• Extend add-ref or add a related command to create Terra-backed references.
• Persist the DRS URI as canonical remote identity.
• Preserve a SHA256-shaped internal/local cache identity, either from a real checksum or a deterministic DRS URI-derived key.
• Decide whether to support direct oid drs://... git-drs pointer files in addition to Git LFS-shaped pointers.

Phase 3: Manifest bootstrap/import

• Add git drs import-drs-manifest or equivalent as a batch wrapper over Terra-aware reference creation.
• Generate pointer files without downloading payload bytes.
• Generate deterministic output manifests.
• Add dry-run, conflict reporting, and deletion policy options.

Phase 4: Documentation and acceptance tests

• Add docs/terra-anvil.md.
• Add unit tests for DRS-URI-derived local OIDs.
• Add resolver tests for mocked GA4GH/Terra DRS responses.
• Add integration-style tests for manifest import and selective hydration.

Acceptance criteria

• A maintainer can configure a Terra/AnVIL read-only remote without Gen3/Syfon bucket mapping.
• A bootstrap/import command can generate pointer files from a representative AnVIL manifest without downloading payload bytes.
• Generated pointer metadata includes canonical DRS URIs.
• Missing SHA256 checksums are handled according to documented policy.
• Rerunning bootstrap with unchanged AnVIL metadata produces no Git diff.
• git drs pull -I <path-or-pattern> hydrates selected Terra/AnVIL pointer files after authentication.
• 401/403 and inaccessible DRS records produce actionable error messages.
• Documentation explains credentials, bootstrap inputs, outputs, refresh workflow, and troubleshooting.

Open questions

• Should the canonical command be git drs remote add terra or a generic git drs remote add ga4gh with Terra-specific auth/resolution options?
• Which AnVIL/Terra endpoint should be treated as canonical for Gregor DRS discovery?
• Should Martha/Terra resolution be implemented directly or invoked through Terra Notebook Utils behavior?
• What should the default policy be for records that lack checksums?
• Should deleted or withdrawn AnVIL records remove pointers automatically, create tombstones, or require explicit maintainer action?
• Should bootstrap/import live in core git-drs or as a project-local Gregor script first?

Priority

High for Terra/AnVIL adoption.