feat(proxy): reload the upstream allowlist from HEADROOM_ALLOWED_BASE_URLS_FILE

docs/content/docs/configuration.mdx

@@ -197,6 +197,12 @@ reachable via `x-headroom-base-url` through an outbound proxy, name it in
 `HEADROOM_ALLOWED_BASE_URLS` — an explicit operator decision to admit that
 destination by name, which also turns off the resolve-and-pin check for it.
 
+Entries are bare hosts (every port on that host) or URLs (that exact origin), separated by commas, tabs or line breaks (newline, carriage return, vertical tab, form feed, the file, group and record separators, NEL, and the Unicode line and paragraph separators). A scheme-less entry must be a hostname and nothing else: one with a port, even an empty one (`127.0.0.1:4100`, `127.0.0.1:`), a path (`gw.internal/v1`), userinfo (`user@gw.internal`), a query or a fragment is skipped as invalid, as is any entry containing whitespace. Write `http://127.0.0.1:4100` for that one origin, or the bare host for every port. Before this, such entries were read as a bare host and admitted every port, so a `HEADROOM_ALLOWED_BASE_URLS` (or `HEADROOM_PROXIED_BASE_URLS`) that relied on one now admits nothing for it. Any entry that cannot be parsed (an unbalanced `[`, a host that Unicode normalisation turns into URL syntax) is skipped too, and the rest of the list still applies; the proxy logs one warning per distinct variable value that held any.
+
+The variable is fixed for the life of the proxy process. When an allowed endpoint can move (a supervisor that lists its own loopback port, for example), put the entries in a file and pass it as `headroom proxy --allowed-base-urls-file PATH`, name it in `HEADROOM_ALLOWED_BASE_URLS_FILE`, or set the Upstream allowlist file in the dashboard's settings, instead of the variable or as well. The path is read at startup and a blank value means no file; a leading `~` is expanded (a `~name` for a user that does not exist is left as written, and so is resolved as a relative path), and a relative path resolves against the proxy's working directory when it is first used and keeps that meaning for the life of the process. Changing the path needs a restart, but the file's contents do not. Its entries are merged with the variable's, and the proxy re-reads the file on the next request after it changes, with no restart; a change is detected by the file's device, inode, size, modification time and status-change time, so an unchanged file is never read again: each request costs a `stat` of the file plus an `lstat` of every directory from `/` down that resolving its path passes through, and those directories are re-checked on every request, so one that becomes writable by others stops a cached version being used straight away, without waiting for the file to change. Which directories are checked is worked out when the file is read: repointing a symlink in the chain at a different file is picked up on the next request, but repointing it so that it still reaches the very same file is only re-evaluated once that file changes. Replace it atomically (write a sibling file, then rename it over the original). The file takes the variable's format, plus comments and an optional UTF-8 byte-order mark. A comment starts at a `#` at the beginning of the file or after any whitespace character and runs to the next line break, so `http://127.0.0.1:4100 # front door` lists that origin. A `#` straight after an origin in a URL (`http://127.0.0.1:4100#x`) begins the URL's fragment, which an origin ignores, so that entry is accepted; a `#` anywhere else makes its entry invalid. In `HEADROOM_ALLOWED_BASE_URLS` a `#` has no comment meaning at all: it is part of whichever entry holds it, and it does not hide anything after the next comma. Each load logs how many hosts and origins it found and how many invalid entries it skipped.
+
+Whoever can change the file, or the directory that holds it, could list any upstream they like, so both are checked. The file must be owned by the user running the proxy or by root and must not be writable by group or others. So must every directory that resolving the path passes through: every directory from `/` down to the one holding the file and, when the path meets a symlink anywhere along it (at its end or in the middle), every directory down to the one holding that link and on down its target, since whoever can write any of them could swap an entry and choose the file. Each must be owned by the user running the proxy or by root and must not be writable by group or others, whether or not the sticky bit is set. Anything under a shared directory such as `/tmp` is therefore refused, a private subdirectory made there with `mkdir -m 700` included: whoever can write the shared directory can rename that subdirectory away and put a symlink to a file of their choosing (or, where allowed, a hard link to a root-owned one) in its place, and the file it leads to cannot show who did it. This is stricter than OpenSSH's StrictModes (there is no exception for sticky directories, and every directory a symlink passes through counts), but like it, it judges owners and mode bits only: access control lists are not read, so a macOS ACL such as `everyone allow add_file,delete_child` on a directory or `allow write,append` on the file, or an NFSv4 ACL, can grant write access the check cannot see (a POSIX ACL is covered, because its mask shows as the group bits), and ownership is whatever the filesystem reports, which a FUSE mount with `allow_other` or NFS without `root_squash` may not make trustworthy. Root is accepted because it can change anything anyway; that is the one way this rule is looser than the one for the `--uds` socket's directory, which must belong to the user running the proxy, because the proxy only reads the allowlist. Supported places for the file are therefore a directory under the home directory of the user running the proxy (where every directory from `/` down is owned by that user or root and none is writable by group or others), or a root-controlled directory such as one under `/etc`, including a bind mount of such a directory into a container whose every directory above it meets the same rule. A Kubernetes ConfigMap volume does not qualify: its directory is writable by group or others, so it is refused. Anything under `/tmp`, `/var/tmp` or another sticky shared directory is refused, including a private subdirectory made there with `mkdir -m 700`, because the directory above it is writable by others. Windows, which has no POSIX ownership or modes, skips this check. Naming a file always means allowlist mode. If it is empty, missing, unreadable, not UTF-8, not a regular file (a FIFO or a device), larger than 1 MiB, or changeable by another user, only the variable's entries are admitted (none when the variable is unset), any other `x-headroom-base-url` is not used (`/v1/messages`, `/v1/alpha/search` and the catch-all passthrough answer 400; the OpenAI chat-completions and responses handlers log it and send the request to the configured upstream instead), and the proxy logs a warning naming the file and the reason once per distinct failure. An oversized file is refused whole, never read in part, and a file that cannot be used (including one whose path keeps resolving to a different file than the one the check inspected) is not tried again until it changes. A refusal because of a directory on the path, or a transient error such as running out of file descriptors, is retried on the next request instead.
+
 ## SmartCrusher Configuration
 
 Fine-tune JSON compression behavior:

headroom/cli/proxy.py

@@ -263,6 +263,20 @@ def dashboard(port: int, no_open: bool) -> None:
         "replaced, and the socket is removed on shutdown."
     ),
 )
+@click.option(
+    "--allowed-base-urls-file",
+    default=None,
+    type=click.Path(dir_okay=False, resolve_path=False),
+    envvar="HEADROOM_ALLOWED_BASE_URLS_FILE",
+    help=(
+        "File of upstream hosts and URLs that x-headroom-base-url may name, in "
+        "the HEADROOM_ALLOWED_BASE_URLS format (commas, tabs or line breaks between "
+        "entries) plus # comments, merged with that variable. Re-read whenever "
+        "it changes, with no restart; replace it atomically. Naming it means "
+        "allowlist mode even while it is empty, missing or unreadable. "
+        "A blank value means no file (env: HEADROOM_ALLOWED_BASE_URLS_FILE)."
+    ),
+)
 @click.option(
     "--workers",
     default=1,
@@ -1056,6 +1070,7 @@ def proxy(
     host: str,
     port: int,
     uds: str | None,
+    allowed_base_urls_file: str | None,
     workers: int,
     limit_concurrency: int,
     max_connections: int,
@@ -1356,6 +1371,28 @@ def proxy(
         except UnixSocketUnusableError as exc:
             raise click.ClickException(str(exc)) from None
 
+    if allowed_base_urls_file is not None:
+        from headroom.proxy.upstream_guard import (
+            ALLOWED_BASE_URLS_FILE_ENV,
+            resolve_allowlist_file_path,
+        )
+
+        allowed_base_urls_file = allowed_base_urls_file.strip()
+        if allowed_base_urls_file:
+            # The guard reads the environment on every check, in this process and in any worker it spawns, so the option is handed over there, resolved the way the guard resolves it so a worker started in another directory still names the same file.
+            try:
+                os.environ[ALLOWED_BASE_URLS_FILE_ENV] = resolve_allowlist_file_path(
+                    allowed_base_urls_file
+                )
+            except OSError as exc:
+                raise click.ClickException(
+                    f"cannot resolve --allowed-base-urls-file {allowed_base_urls_file!r} against "
+                    f"the working directory: {exc.strerror}"
+                ) from None
+        else:
+            # Blank means no file, as a blank HEADROOM_ALLOWED_BASE_URLS means no list. Set to "" rather than removed: create_app re-applies stored settings with setdefault (in this process and in every worker), which would put a stored path back into a missing variable but leaves a present one alone, so an explicit `--allowed-base-urls-file ""` overrides a stored or inherited path.
+            os.environ[ALLOWED_BASE_URLS_FILE_ENV] = ""
+
     config = ProxyConfig(
         host=host,
         port=port,

headroom/proxy/private_paths.py

@@ -0,0 +1,44 @@
+"""Whether another user could change a file or directory the proxy trusts.
+
+Shared by the ``--uds`` socket's parent-directory check and the upstream allowlist file, which both trust a path only if nobody but the user running the proxy can replace what it names. Standard library only, so the SSRF guard can import it without widening its own dependencies. POSIX only: ownership and mode bits are what is judged, so callers on a platform without ``os.geteuid`` do not call it.
+"""
+
+from __future__ import annotations
+
+import os
+import stat
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ForeignWriteAccess:
+    """Why someone other than the current effective user could change what a ``stat`` describes.
+
+    ``description`` completes "<subject> is ...", for example ``owned by uid 0, not by the user running the proxy (uid 501)``. ``other_owner`` tells the two causes apart for callers that advise differently on each.
+    """
+
+    description: str
+    other_owner: bool
+
+
+def foreign_write_access(
+    info: os.stat_result, *, root_may_own: bool = False
+) -> ForeignWriteAccess | None:
+    """Return why another user could change the file or directory ``info`` describes, else ``None``.
+
+    Its owner can always change it, so an owner other than the current effective user is refused first; otherwise group or other write permission is. For a directory, write permission is what allows unlinking or renaming its entries, whoever owns them, so either lets someone else replace what the directory holds.
+
+    The sticky bit is never a substitute for write permission: in a sticky shared directory anyone can still create a new entry, such as a symlink to a file of their choosing or (where hard links to others' files are allowed) a hard link to one, at a path nobody has claimed yet, and checking the file that entry leads to cannot tell who made it.
+
+    ``root_may_own`` relaxes the owner rule for paths the proxy only reads, never binds; it is off by default, which is the ``--uds`` rule. It also accepts uid 0, which can change anything anyway, so trusting it adds no one, and it lets a proxy running as another user read a root-owned file, such as one in a root-owned bind-mounted directory.
+    """
+    euid = os.geteuid()
+    if info.st_uid != euid and not (root_may_own and info.st_uid == 0):
+        owners = f"the user running the proxy (uid {euid})" + (" or root" if root_may_own else "")
+        return ForeignWriteAccess(f"owned by uid {info.st_uid}, not by {owners}", other_owner=True)
+    if info.st_mode & (stat.S_IWGRP | stat.S_IWOTH):
+        return ForeignWriteAccess(
+            f"writable by group or others (mode {stat.S_IMODE(info.st_mode):04o})",
+            other_owner=False,
+        )
+    return None

headroom/proxy/unix_socket.py

@@ -31,6 +31,8 @@
 from dataclasses import dataclass
 from types import FrameType
 
+from headroom.proxy.private_paths import foreign_write_access
+
 #: Owner read and write. Connecting to a unix socket needs write permission on it, so this admits only the proxy's own user.
 SOCKET_MODE = stat.S_IRUSR | stat.S_IWUSR
 
@@ -126,16 +128,16 @@ def _require_safe_parent(parent: str, socket_path: str) -> None:
         raise UnixSocketUnusableError(
             f"{parent} holds the socket {socket_path} but is not a directory"
         )
-    if info.st_uid != os.geteuid():
-        raise UnixSocketUnusableError(
-            f"directory {parent} is owned by uid {info.st_uid}, not by the user running the proxy "
-            f"(uid {os.geteuid()}); another owner could replace the socket"
-        )
-    if info.st_mode & (stat.S_IWGRP | stat.S_IWOTH):
-        raise UnixSocketUnusableError(
-            f"directory {parent} is writable by group or others (mode {stat.S_IMODE(info.st_mode):04o}); "
+    problem = foreign_write_access(info)
+    if problem is None:
+        return
+    if problem.other_owner:
+        consequence = "another owner could replace the socket"
+    else:
+        consequence = (
             "they could replace the socket. Run chmod go-w on it or use a private directory"
         )
+    raise UnixSocketUnusableError(f"directory {parent} is {problem.description}; {consequence}")
 
 
 def checked_unix_socket_path(path: str) -> str: