feat(pelican): add GET /pelican/read for inline object contents

CHANGELOG.md

@@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- **`GET /pelican/read` returns the contents of a Pelican object in the response body.** The Pelican integration could list objects (`/browse`, `/info`) and hand back a file to save (`/download`), but there was no way to get an object's contents inline — a caller wanting to work with the data had to write it to disk first and read it back. The new route returns the contents directly, so a subscriber can take the object referenced by an event and feed it straight into its own code. Text is returned as text; anything that is not valid UTF-8 is base64-encoded rather than refused, since Pelican namespaces hold binary payloads too, and the response says which of the two it is in an `encoding` field alongside `path`, `size` and `content`. The route is capped by `PELICAN_MAX_READ_BYTES` (10 MiB by default) because the contents travel in the response body; the cap is enforced while reading rather than after, so an oversized object is never pulled into the API's memory just to be rejected, and the caller is told to use `/download` instead. Failures are distinguishable from the status code: 404 when the object is not in the federation, 413 when it is past the limit, 502 when the federation cannot be reached.
+
+### Backwards compatibility
+- Purely additive. `PELICAN_MAX_READ_BYTES` is optional and defaults to 10 MiB, and a missing, non-numeric or non-positive value falls back to that default, so existing deployments need no change. The new route sits behind the same authorization as the rest of the Pelican routes, and is only mounted when `PELICAN_ENABLED` is set.
+
 ## [0.34.22] - 2026-08-30
 
 ### Fixed

api/routes/pelican_routes.py

@@ -15,6 +15,7 @@
     get_file_info,
 )
 from api.services.pelican_services.download_file import download_file, stream_file
+from api.services.pelican_services.read_file import read_object
 from api.services.pelican_services.import_metadata import import_file_as_resource
 from api.services.auth_services import (
     get_user_for_read_operation,
@@ -25,6 +26,11 @@
 
 logger = logging.getLogger(__name__)
 
+# Largest object /pelican/read will return inline. Anything bigger is a
+# download, not a read: the contents go into the response body, so an
+# unbounded read would put an arbitrary object into the endpoint's memory.
+DEFAULT_MAX_READ_BYTES = 10 * 1024 * 1024
+
 # The read gate is declared on the router rather than on each route: these
 # endpoints shipped completely unauthenticated (issue #261), and a
 # router-level dependency means a route added later cannot silently miss it.
@@ -245,6 +251,94 @@ async def download(
         raise HTTPException(status_code=500, detail=f"Error downloading file: {str(e)}")
 
 
+def _max_read_bytes() -> int:
+    """
+    Resolve the inline read limit from ``PELICAN_MAX_READ_BYTES``.
+
+    Settings are declared with ``extra: "allow"``, so a malformed value
+    reaches this point instead of failing at startup; fall back to the
+    default rather than letting a typo disable the limit.
+
+    Returns
+    -------
+    int
+        Limit in bytes.
+    """
+    raw = os.getenv("PELICAN_MAX_READ_BYTES", "")
+    if not raw:
+        return DEFAULT_MAX_READ_BYTES
+    try:
+        value = int(raw)
+    except ValueError:
+        logger.warning(
+            f"PELICAN_MAX_READ_BYTES is not an integer ({raw!r}); "
+            f"using {DEFAULT_MAX_READ_BYTES}."
+        )
+        return DEFAULT_MAX_READ_BYTES
+    if value <= 0:
+        logger.warning(
+            f"PELICAN_MAX_READ_BYTES must be positive (got {value}); "
+            f"using {DEFAULT_MAX_READ_BYTES}."
+        )
+        return DEFAULT_MAX_READ_BYTES
+    return value
+
+
+@router.get("/read")
+async def read_file_contents(
+    path: str = Query(..., description="Path of the object to read"),
+    federation: str = Query("osdf", description="Federation to query"),
+):
+    """
+    Read a Pelican object and return its contents in the response body.
+
+    ``/download`` hands back a file to save; this returns the contents
+    inline so they can be piped straight into the caller's own code.
+
+    Parameters
+    ----------
+    path : str
+        Path of the object to read
+    federation : str
+        Federation name (default "osdf")
+
+    Returns
+    -------
+    dict
+        ``path``, ``size``, ``encoding`` ("utf-8" or "base64") and
+        ``content``
+
+    Raises
+    ------
+    HTTPException
+        - 404: Object not found in the federation
+        - 413: Object larger than the inline read limit
+        - 502: The federation could not be reached
+    """
+    try:
+        pelican_repo = get_pelican_repo(federation)
+        result = read_object(pelican_repo, path, _max_read_bytes())
+
+        if not result["success"]:
+            status_by_reason = {
+                "not_found": 404,
+                "too_large": 413,
+                "unavailable": 502,
+            }
+            raise HTTPException(
+                status_code=status_by_reason.get(result.get("reason"), 502),
+                detail=result["error"],
+            )
+
+        return result
+
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Error reading Pelican object {path}: {e}")
+        raise HTTPException(status_code=500, detail=f"Error reading object: {str(e)}")
+
+
 @router.post("/import-metadata")
 async def import_metadata(
     request: ImportMetadataRequest,

api/services/pelican_services/read_file.py

@@ -0,0 +1,98 @@
+# api/services/pelican_services/read_file.py
+"""
+Service for reading Pelican objects into the response body.
+"""
+
+import base64
+import logging
+from typing import Any, Dict
+
+from api.repositories.pelican_repository import PelicanRepository
+
+logger = logging.getLogger(__name__)
+
+
+def read_object(
+    pelican_repo: PelicanRepository, path: str, max_bytes: int
+) -> Dict[str, Any]:
+    """
+    Read the contents of a Pelican object and return them inline.
+
+    Unlike :func:`api.services.pelican_services.download_file.download_file`,
+    which hands the caller a file to save, this returns the contents in the
+    response body so they can be fed straight into the caller's own code
+    without a temporary file.
+
+    The size cap is enforced *while* reading rather than after: an object
+    larger than the cap must not be pulled into the endpoint's memory just
+    to be rejected, which is why this reads through an open handle instead
+    of calling ``read_file``.
+
+    Parameters
+    ----------
+    pelican_repo : PelicanRepository
+        Initialized Pelican repository
+    path : str
+        Path of the object to read
+    max_bytes : int
+        Largest object this endpoint will return inline
+
+    Returns
+    -------
+    dict
+        On success, ``path``, ``size``, ``encoding`` and ``content``.
+        On failure, ``error`` plus a ``reason`` of ``not_found``,
+        ``too_large`` or ``unavailable`` for the caller to map to a
+        status code.
+    """
+    try:
+        with pelican_repo.open_file(path, mode="rb") as handle:
+            # One byte past the cap, so an oversized object is detected
+            # without being read in full.
+            payload = handle.read(max_bytes + 1)
+    except FileNotFoundError as exc:
+        logger.info(f"Pelican object not found: {path}")
+        return {
+            "success": False,
+            "path": path,
+            "error": f"Object not found in the federation: {str(exc) or path}",
+            "reason": "not_found",
+        }
+    except Exception as exc:
+        logger.error(f"Error reading Pelican object {path}: {exc}")
+        return {
+            "success": False,
+            "path": path,
+            "error": f"{type(exc).__name__}: {exc}",
+            "reason": "unavailable",
+        }
+
+    if len(payload) > max_bytes:
+        return {
+            "success": False,
+            "path": path,
+            "error": (
+                f"Object is larger than the {max_bytes} byte inline read "
+                "limit. Use /pelican/download to retrieve it as a file."
+            ),
+            "reason": "too_large",
+        }
+
+    # Text is returned as text so a caller can use it directly; anything
+    # that is not valid UTF-8 is base64-encoded rather than rejected, and
+    # says so, because Pelican namespaces hold binary payloads too.
+    try:
+        content = payload.decode("utf-8")
+        encoding = "utf-8"
+    except UnicodeDecodeError:
+        content = base64.b64encode(payload).decode("ascii")
+        encoding = "base64"
+
+    logger.info(f"Read {len(payload)} bytes from Pelican object {path}")
+    return {
+        "success": True,
+        "path": path,
+        "size": len(payload),
+        "encoding": encoding,
+        "content": content,
+    }

docs/configuration.md

@@ -340,6 +340,15 @@ Default Pelican federation, format `pelican://host` (e.g. `pelican://osg-htc.org
 Read straight from origin servers instead of caches. **Where:** keep `False` for
 better performance unless you have a reason.
 
+#### `PELICAN_MAX_READ_BYTES`
+*Optional · default: `10485760` (10 MiB).*
+Largest object `GET /pelican/read` returns inline. That endpoint puts the
+contents in the response body, so this caps what a single request can pull into
+the API's memory; a larger object is refused with 413 and must be fetched with
+`/pelican/download`. A non-numeric or non-positive value falls back to the
+default. **Where:** raise it only if your callers genuinely read larger objects
+inline.
+
 ---
 
 ## Remote execution (Rexec)

example.env

@@ -318,6 +318,13 @@ PELICAN_FEDERATION_URL=
 # Set to False to use caching infrastructure (recommended for better performance)
 PELICAN_DIRECT_READS=False
 
+# Largest object /pelican/read will return inline, in bytes
+# The read endpoint returns file contents in the response body, so this caps
+# how much can be pulled into the API's memory by a single request
+# Anything larger must be fetched with /pelican/download instead
+# Default: 10485760 (10 MiB)
+PELICAN_MAX_READ_BYTES=10485760
+
 # ==============================================
 # Rexec Deployment API Configuration
 # ==============================================