Refactoring: merge diff computation into saver for v2 katas
Background
differ is a standalone microservice that computes diffs between two events
of a kata. Its current call chain for any diff request is:
client --> differ --> saver (x2, one per event index) --> differ --> client
For each request, differ:
- Calls saver.kata_event(id, was_index) -- HTTP call
- Calls saver.kata_event(id, now_index) -- HTTP call
- Creates a temporary git repo in /tmp
- Writes the "was" files, runs: git add . && git commit && git tag 0
- Removes the working tree (preserving .git), writes the "now" files
- Runs: git add . && git commit && git tag 1
- Runs: git diff --unified=99999999999 ... 0 1
- Parses the output and discards the temp repo
Saver versioning
Saver stores kata data in one of three formats, determined at kata creation
time and recorded in the manifest:
v0 - plain JSON files, one directory per event index
v1 - plain JSON files, one flat file per event index
v2 - a persistent git repo; each event is a commit tagged with its index
All new katas are v2. v0 and v1 are legacy (read-only in practice).
differ works correctly for all three versions: all three return kata_event
responses with the same structure (files as { filename => { 'content' => string } }).
Why v2 opens a shortcut
For v2 katas, saver already holds a git repo with every event tagged
sequentially. A diff between was_index and now_index can be computed
with a single command on the existing repo:
git diff --unified=99999999999 ... <was_index> <now_index> -- files/
No temp repo needs to be constructed. The estimated saving is 100-280ms
per diff (dominated by eliminating two git commits, not by network overhead).
The incoming request to differ carries only (id, was_index, now_index) --
no version information -- so differ cannot take the shortcut without first
discovering the kata version.
Final design
Route all diff HTTP requests to saver. Saver becomes the single entry point
for diff requests and handles version dispatch internally:
v2: saver runs git diff directly on the kata's existing repo and returns
the result. No call to differ. No temp git repo.
v0/v1: saver fetches both events in-process (it already knows the version
from the manifest), then delegates to differ by passing the raw file
contents -- not (id, was_index, now_index). Differ does the temp-git
computation and returns the diff to saver, which returns it to the
client.
Call chains after the refactoring:
v2 (all current traffic):
client --> saver --> client
(saver runs git diff in-process)
v0/v1 (legacy traffic only):
client --> saver --> differ(was_files, now_files) --> saver --> client
(saver fetches events, differ does the temp-git work)
Key consequence for differ
Differ's API changes: it no longer accepts (id, was_index, now_index) and
calls saver to fetch files. Instead it accepts two file maps directly:
(was_files, now_files). Differ loses its saver dependency entirely and
becomes a pure computation service: given two sets of file contents,
return a diff. This eliminates the previously circular dependency
(saver calling differ, differ calling saver back).
No extra network call is added for any version:
- v2: one fewer network call than today (client calls saver, not differ)
- v0/v1: same number of calls as today (client call + differ call),
but differ no longer calls saver back because saver passes
the file contents directly
Impact on saver write operations (e.g. kata_ran_tests)
Net effect on saver call volume:
The refactoring reduces the number of saver calls per diff request for v2
katas (all current traffic):
Before: client --> differ --> saver (kata_event x2) --> differ --> client
After: client --> saver (single git diff) --> client
Saver goes from receiving 2 HTTP calls per diff down to 1. The concern that
more reads might stress saver does not materialise for v2.
Could a concurrent git diff interfere with kata_ran_tests?
kata_ran_tests (kata_v2.rb) does:
- git worktree add /tmp/
- Write files, commit
- git merge --ff-only (updates HEAD on the main repo)
- git tag
A "git diff <was_index> <now_index>" is a pure read from the git object
database -- it resolves two tagged commits and compares their trees. It does
not touch the index, HEAD, or working tree. Git's internal ref-locking means
the diff safely sees either the pre- or post-merge state without corrupting
the repo or racing with the merge.
Existing design that protects writes:
app_base.rb gives every write operation (kata_ran_tests, etc.) a per-kata
Mutex held for the entire HTTP request. Read operations (kata_event,
kata_manifest, and the new diff path) carry no mutex and run freely
concurrently. The git diff path goes through External::Shell, not
External::Disk, so it also bypasses the flock(LOCK_EX) used by the
file-level read/write helpers -- no new contention point is introduced.
Conclusion: the refactoring has no adverse impact on kata_ran_tests or other
write calls. v2 diff load on saver decreases, git diff is read-only on the
object DB and does not conflict with git merge --ff-only, and the per-kata
mutex already serialises all writes independently of read concurrency.
Refactoring: merge diff computation into saver for v2 katas
Background
differ is a standalone microservice that computes diffs between two events
of a kata. Its current call chain for any diff request is:
client --> differ --> saver (x2, one per event index) --> differ --> client
For each request, differ:
Saver versioning
Saver stores kata data in one of three formats, determined at kata creation
time and recorded in the manifest:
v0 - plain JSON files, one directory per event index
v1 - plain JSON files, one flat file per event index
v2 - a persistent git repo; each event is a commit tagged with its index
All new katas are v2. v0 and v1 are legacy (read-only in practice).
differ works correctly for all three versions: all three return kata_event
responses with the same structure (files as { filename => { 'content' => string } }).
Why v2 opens a shortcut
For v2 katas, saver already holds a git repo with every event tagged
sequentially. A diff between was_index and now_index can be computed
with a single command on the existing repo:
git diff --unified=99999999999 ... <was_index> <now_index> -- files/
No temp repo needs to be constructed. The estimated saving is 100-280ms
per diff (dominated by eliminating two git commits, not by network overhead).
The incoming request to differ carries only (id, was_index, now_index) --
no version information -- so differ cannot take the shortcut without first
discovering the kata version.
Final design
Route all diff HTTP requests to saver. Saver becomes the single entry point
for diff requests and handles version dispatch internally:
v2: saver runs git diff directly on the kata's existing repo and returns
the result. No call to differ. No temp git repo.
v0/v1: saver fetches both events in-process (it already knows the version
from the manifest), then delegates to differ by passing the raw file
contents -- not (id, was_index, now_index). Differ does the temp-git
computation and returns the diff to saver, which returns it to the
client.
Call chains after the refactoring:
v2 (all current traffic):
client --> saver --> client
(saver runs git diff in-process)
v0/v1 (legacy traffic only):
client --> saver --> differ(was_files, now_files) --> saver --> client
(saver fetches events, differ does the temp-git work)
Key consequence for differ
Differ's API changes: it no longer accepts (id, was_index, now_index) and
calls saver to fetch files. Instead it accepts two file maps directly:
(was_files, now_files). Differ loses its saver dependency entirely and
becomes a pure computation service: given two sets of file contents,
return a diff. This eliminates the previously circular dependency
(saver calling differ, differ calling saver back).
No extra network call is added for any version:
but differ no longer calls saver back because saver passes
the file contents directly
Impact on saver write operations (e.g. kata_ran_tests)
Net effect on saver call volume:
The refactoring reduces the number of saver calls per diff request for v2
katas (all current traffic):
Before: client --> differ --> saver (kata_event x2) --> differ --> client
After: client --> saver (single git diff) --> client
Saver goes from receiving 2 HTTP calls per diff down to 1. The concern that
more reads might stress saver does not materialise for v2.
Could a concurrent git diff interfere with kata_ran_tests?
kata_ran_tests (kata_v2.rb) does:
A "git diff <was_index> <now_index>" is a pure read from the git object
database -- it resolves two tagged commits and compares their trees. It does
not touch the index, HEAD, or working tree. Git's internal ref-locking means
the diff safely sees either the pre- or post-merge state without corrupting
the repo or racing with the merge.
Existing design that protects writes:
app_base.rb gives every write operation (kata_ran_tests, etc.) a per-kata
Mutex held for the entire HTTP request. Read operations (kata_event,
kata_manifest, and the new diff path) carry no mutex and run freely
concurrently. The git diff path goes through External::Shell, not
External::Disk, so it also bypasses the flock(LOCK_EX) used by the
file-level read/write helpers -- no new contention point is introduced.
Conclusion: the refactoring has no adverse impact on kata_ran_tests or other
write calls. v2 diff load on saver decreases, git diff is read-only on the
object DB and does not conflict with git merge --ff-only, and the per-kata
mutex already serialises all writes independently of read concurrency.