Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 28 additions & 18 deletions Documentation/config/receive.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -50,18 +50,31 @@ receive.fsck.skipList::
details.

receive.keepAlive::
After receiving the pack from the client, `receive-pack` may
produce no output (if `--quiet` was specified) while processing
the pack, causing some networks to drop the TCP connection.
With this option set, if `receive-pack` does not transmit
any data in this phase for `receive.keepAlive` seconds, it will
send a short keepalive packet. The default is 5 seconds; set
to 0 to disable keepalives entirely.
After receiving the pack from the client, `receive-pack` may
produce no output (if `--quiet` was specified) while processing
the pack, causing some networks to drop the TCP connection.
With this option set, if `receive-pack` does not transmit
any data in this phase for `receive.keepAlive` seconds, it will
send a short keepalive packet. The default is 5 seconds; set
to 0 to disable keepalives entirely.

receive.lop.enable::
Enable server-side offloading of blobs to Large-Object Promisor
(LOP) repositories. When set to true, `receive-pack` examines the
promisor remotes configured in the repository (for example via
`remote.<name>.promisor = true`) and uses their
`remote.<name>.partialCloneFilter` values to determine which
pushed blobs should be written to each LOP. Filters such as
`blob:none` and `blob:limit=<size>` follow the same semantics as
partial-clone negotiation: a `blob:none` promisor stores every
blob, while `blob:limit=<size>` captures blobs larger than the
advertised limit. Blobs that are not claimed by any promisor
remain in the primary repository. Defaults to false.

receive.unpackLimit::
If the number of objects received in a push is below this
limit then the objects will be unpacked into loose object
files. However if the number of received objects equals or
If the number of objects received in a push is below this
limit then the objects will be unpacked into loose object
files. However if the number of received objects equals or
exceeds this limit then the received pack will be stored as
a pack, after adding any missing delta bases. Storing the
pack from a push can make the push operation complete faster,
Expand Down Expand Up @@ -89,15 +102,15 @@ receive.denyCurrentBranch::
out of sync with the index and working tree. If set to "warn",
print a warning of such a push to stderr, but allow the push to
proceed. If set to false or "ignore", allow such pushes with no
message. Defaults to "refuse".
+
message. Defaults to "refuse".

Another option is "updateInstead" which will update the working
tree if pushing into the current branch. This option is
intended for synchronizing working directories when one side is not easily
accessible via interactive ssh (e.g. a live web site, hence the requirement
that the working directory be clean). This mode also comes in handy when
developing inside a VM to test and fix code on different Operating Systems.
+

By default, "updateInstead" will refuse the push if the working tree or
the index have any difference from the HEAD, but the `push-to-checkout`
hook can be used to customize this. See linkgit:githooks[5].
Expand All @@ -122,19 +135,16 @@ receive.procReceiveRefs::
variable is not defined, the "proc-receive" hook will never be
used, and all commands will be executed by the internal
`execute_commands` function.
+
For example, if this variable is set to "refs/for", pushing to reference
such as "refs/for/master" will not create or update a reference named
"refs/for/master", but may create or update a pull request directly by
running the hook "proc-receive".
+
Optional modifiers can be provided in the beginning of the value to filter
commands for specific actions: create (a), modify (m), delete (d).
A `!` can be included in the modifiers to negate the reference prefix entry.
E.g.:
+
git config --system --add receive.procReceiveRefs ad:refs/heads
git config --system --add receive.procReceiveRefs !:refs/heads
git config --system --add receive.procReceiveRefs ad:refs/heads
git config --system --add receive.procReceiveRefs !:refs/heads

receive.updateServerInfo::
If set to true, git-receive-pack will run git-update-server-info
Expand Down
164 changes: 164 additions & 0 deletions Documentation/technical/lop-ux-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# LOP UX: Seamless clone/fetch, push offload, and trace-verified data paths

## Problem

Using Large-Object Promisors (LOPs) is clunky for end users. Today you must juggle partial-clone filters, promisor remotes, and server quirks. Push is worse: servers keep all blobs locally even when policy says "large blobs live in a LOP."

## Goal

Make LOPs "just work":

1. Clone/Fetch UX: a single, simple clone that sets up one or more promisor remotes and fetch filters correctly. Subsequent fetches continue to respect those filters without user babysitting.
2. Commit/Push Offload (Server-side): when a client pushes, the server automatically offloads blobs that match policy to the configured LOP(s) instead of storing them locally.
3. Verification: tests and trace instrumentation prove that only the expected objects cross the wire and that routing/offload happens as intended.

## Scope (v1)

- Client: quality-of-life around `--filter`, multi-promisor routing metadata, and predictable fetch behavior.
- Server: an offload path that routes matching blobs to LOP storage using the new ODB interface.
- Tests: end-to-end `sh` tests that assert both behavior and packet/trace content.

**Non-goals (v1):** protocol changes, on-disk format changes, re-implementing LFS, or Gerrit/JGit integration.

## Design

### 1. Client UX: "one-line" partial clone with LOPs

A single clone sets up multiple promisor remotes and accepts server-provided objects from any of them once the user has opted in via `promisor.acceptFromServer=all` (for example in a global config).

```
git clone \
--branch=main \
file:///.../server.git client
```

**Behavioral guarantees**

- Subsequent `git fetch` continues to honor the server-advertised `blob:none`
filter and pulls missing blobs on demand from any configured promisor
remote.
- If a promisor is unreachable, fetch falls back gracefully (trees/commits always, blobs on-demand later).
- When multiple promisors advertise different blob limits, the client adopts
the strictest (smallest) limit so that no blob which should live in a LOP is
downloaded eagerly during the initial clone.

### 2. Server-side offload on push

When a client pushes a pack that contains blobs matching a promisor remote’s
partial-clone filter, the server:

1. Uses the advertised `remote.<name>.partialCloneFilter` values (for example
`blob:none` or `blob:limit=<size>`) to decide whether an incoming blob should
live in that LOP repository.
2. Writes matching blobs into the LOP ODB via the new ODB interface
(plugin/aux ODB).
3. Keeps trees/commits (and any non-matching blobs) locally.
4. Records promisor metadata so downstream partial clones can retrieve those
blobs from the LOP.

**Policy sources (v1)**

- Promisor remote filters: `remote.<name>.partialCloneFilter = blob:none`,
`blob:limit=<bytes>`, or other supported list-objects filters. The server
interprets the same filters it advertises to clients, so no bespoke policy
syntax is required.

### 3. Trace-verified data paths

We rely on packet and Trace2 to prove behavior in tests and in the field.

- `GIT_TRACE_PACKET=1` to assert filter negotiation, e.g. filter `blob:none`.
- `GIT_TRACE2_EVENT=/path/trace.json` with custom categories:
- `lop/router`: chosen promisor for on-demand blob
- `lop/offload`: server offloaded N blobs to `<lopName>`
- `lop/match`: policy matched `<blob>` by size/path/type

## Configuration (new/used keys)

**Client**

- `promisor.acceptFromServer = all|listed|none` (we use `all`)
- No manual `remote.<lop>.promisor` configuration; the client accepts the
server-advertised LOP inventory and filter metadata.

**Server**

- `receive.lop.enable = true`
- `promisor.sendFields = partialCloneFilter`
- `remote.<lop>.promisor = true`
- `remote.<lop>.partialCloneFilter = blob:none|blob:limit=<size>`

These are existing knobs: enabling LOPs simply causes `receive-pack` to obey
the same promisor filters it already advertises to clients.

## Components (proposed file/map)

- ODB integration: `promisor-odb.c`, `promisor-odb.h` (adapter to the new ODB interface for storing/serving offloaded blobs)
- Server receive path hook-in: `receive-pack.c` (calls into lop/offload when `receive.lop.enable`)
- Docs: `Documentation/technical/lop.txt`, `Documentation/config/lop.txt`
- Tests: `t/t571x-lop-offload.sh`, `t/t571x-lop-multipromisor.sh`

## Tests

We add end-to-end `sh` tests that set up:

- `server.git` (primary), `lop-small.git`, `lop-large.git` (promisors)
- Client clones with `promisor.acceptFromServer=all` and relies on the
server-advertised filter metadata.

**Test cases**

1. Clone advertises filter automatically. Assert `GIT_TRACE_PACKET` contains
`filter blob:none`.
2. On-demand blob fetch. Checkout forces missing blobs; trace shows `lop/router` picking correct promisor.
3. Push offload by size. Push pack with >1 MiB blobs → server emits `lop/offload` and those blobs land in `lop-large.git`.
4. Mixed thresholds. Configure `remote.lopSmall.partialCloneFilter=blob:limit=1k` and
`remote.lopLarge.partialCloneFilter=blob:limit=1m`; verify mid-sized blobs
land in the small promisor while very large payloads route to the large one.
5. Negative coverage. Offload disabled → server keeps everything local; traces
confirm no offload.
6. Negative paths. Offload disabled → server keeps everything local; traces confirm no offload.
7. Resilience. LOP temporarily unavailable → push is rejected with actionable error in server logs; no partial writes.

**Assertions**

- Packet filter negotiation: `test_i18ngrep -e 'packet:.*(clone|fetch)>\s*filter blob:none' trace.filt`
- Trace2 JSON has `lop/offload` events with counts and target remote.
- `git cat-file -t` against promisor repos confirms object placement.

## User Experience

**End user**

- Clones with one command. Doesn’t learn internals. Things “just fetch” on demand.

**Server operator**

- Enables offload and declares policy once. No new protocol. Uses existing ODB plug-points.

**Failure modes**

- Misrouted or missing promisor: clear error; local store remains consistent; retries are safe.

## Compatibility & Migration

- 100% compatible with existing partial clone and promisor remotes.
- If `receive.lop.enable=false`, we behave like today: the server stores everything locally.
- No repo reformat needed. Offload metadata leverages promisor semantics already present.

## Performance & Safety

- Offload runs in the receive path after validation to avoid accepting bad packs.
- Batching minimizes round-trips to LOP backends.
- Integrity is identical to normal Git: hashes don’t change; only storage location does.

## Future Work (post-v1)

- Smarter heuristics (delta-aware decisions, content-type sniffing via file headers).
- Push-time hints from client (optional) to pre-seed offload routing.
- Admin visibility: `git lop status` and `git lop verify` commands.
- Native multi-promisor advertisement to reduce cold misses.

## TL;DR

We make LOPs boring: one-line filtered clone, fetch that keeps honoring filters, and server-side push that offloads matching blobs to the right LOP automatically. Tests and traces prove the data path is correct.
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -1229,6 +1229,8 @@ LIB_OBJS += preload-index.o
LIB_OBJS += pretty.o
LIB_OBJS += prio-queue.o
LIB_OBJS += progress.o
LIB_OBJS += promisor-odb.o
LIB_OBJS += lop-offload.o
LIB_OBJS += promisor-remote.o
LIB_OBJS += prompt.o
LIB_OBJS += protocol.o
Expand Down
Loading
Loading