feat(pelican): add GET /pelican/read for inline object contents - #265
Merged
Conversation
The integration could list objects and hand back a file to save, but there was no way to get an object's contents inline, so a caller had to write to disk and read back just to work with the data. /pelican/read returns the contents in the response body instead. Text comes back as text and non-UTF-8 payloads are base64-encoded rather than refused, with an encoding field saying which. The read is capped by PELICAN_MAX_READ_BYTES and the cap bounds the read itself, so an oversized object is never pulled into memory just to be rejected. Part of #262.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #262. The
subscribehalf of that issue follows separately and will close it.Why
The Pelican integration could list objects (
/browse,/info) and hand back a file to save (/download), but there was no way to get an object's contents inline. A caller wanting to work with the data had to write it to disk and read it back./pelican/readreturns the contents in the response body, so the object referenced by an event can be piped straight into the caller's own code.The route
{ "success": true, "path": "/ospool/uc-shared/public/osg-training/testing/nested/actual_dir/test.txt", "size": 29, "encoding": "utf-8", "content": "Testing symlinks and Pelican\n" }Three decisions worth flagging for review:
Binary is encoded, not refused. Pelican namespaces hold binary payloads, so a body that is not valid UTF-8 comes back base64-encoded and the
encodingfield says which of the two it is, rather than the request failing.The read is capped by
PELICAN_MAX_READ_BYTES, 10 MiB by default. The contents travel in the response body, so an uncapped read would put an arbitrarily large object into the API's memory. The cap bounds the read itself — the service reads through an open handle and asks for one byte past the limit — so an oversized object is never pulled in just to be rejected. A missing, non-numeric or non-positive value falls back to the default, which matters because settings are declared withextra: "allow"and a typo would otherwise pass silently.Failures are distinguishable from the status code: 404 when the object is not in the federation, 413 when it is past the cap, 502 when the federation cannot be reached.
Verification
Exercised against the live OSDF federation, not only against mocks:
encoding: "utf-8", contents returned.sif/downloadThe 413 returned in seconds on the 364 MB object, which confirms the cap bounds the read rather than rejecting after the fact.
17 new cases in
tests/test_pelican_read.pycover the service (text, binary, at the cap, past the cap, missing object, federation failure carrying the exception type, and that the read is capped while reading), thePELICAN_MAX_READ_BYTESresolution including the malformed-value fallbacks, and the route's status-code mapping plus its inherited authorization.Local gate:
blackandflake8clean, 1290 tests pass. The three failures intests/repositories/test_catalog_settings.pyandtests/test_publi_env.pypredate this branch — they reproduce on a cleanmainand come from a local.envbeing picked up by the settings tests, which does not exist in CI.Backwards compatibility
Purely additive.
PELICAN_MAX_READ_BYTESis optional and documented inexample.envanddocs/configuration.md, so existing deployments need no change. The new route sits behind the authorization added in #261 and is only mounted whenPELICAN_ENABLEDis set.