Repository navigation
Expand file tree
/
Copy pathengineering.html
More file actions
395 lines (393 loc) · 113 KB
/
Copy pathengineering.html
File metadata and controls
395 lines (393 loc) · 113 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Shifter Web Proxy — Engineering Guide</title><style>*{box-sizing:border-box}html{scroll-behavior:smooth;color-scheme:light}body{margin:0;background:#ffffff;color:#1f2328;font:16px/1.65 system-ui,-apple-system,sans-serif}header,main,footer{max-width:1040px;margin:auto;padding:26px 40px}header{color:#59636e;font-size:12px;letter-spacing:1.5px;font-weight:600;border-bottom:1px solid #d1d9e0}main{padding-top:20px}h1,h2,h3{color:#1f2328;line-height:1.3}h1{font-size:34px;letter-spacing:-.9px;margin:28px 0 16px}h2{margin-top:42px;padding-bottom:10px;border-bottom:1px solid #d1d9e0;font-size:24px}h3{margin-top:28px;font-size:19px}p{margin:16px 0}p[align=center]{text-align:center}p[align=center] img{display:block;width:100%;max-width:960px;height:auto;margin:0 auto}h1[align=center]{text-align:center}a{color:#0969da;text-underline-offset:3px;text-decoration:none}a:hover{text-decoration:underline}code{font:13px/1.7 ui-monospace,SFMono-Regular,Consolas,monospace;overflow-wrap:anywhere;background:#eff1f3;border-radius:4px;padding:2px 5px}pre{padding:20px;overflow:auto;background:#f6f8fa;border:1px solid #d1d9e0;border-radius:8px}pre code{white-space:pre;overflow-wrap:normal;padding:0;background:none}.table{overflow-x:auto}table{width:100%;border-collapse:collapse;font-size:14px}th,td{padding:12px 14px;border:1px solid #d1d9e0;text-align:left;vertical-align:top}th{background:#f6f8fa;color:#1f2328}tr:nth-child(even){background:#f6f8fa}li{margin:6px 0}footer{border-top:1px solid #d1d9e0;color:#59636e;font-size:13px}h2,h3{scroll-margin-top:18px}@media(max-width:650px){header,main,footer{padding:18px}h1{font-size:27px}table{min-width:540px}}@media print{html{color-scheme:light}body{background:white;color:#172235}header,main,footer{max-width:none;padding:12px}h1,h2,h3,th{color:#172235}h2,h3{break-after:avoid}pre,th,tr:nth-child(even){background:#f1f4f8}a{color:#1756a5}}</style></head><body><main><h1 id="shifter-web-proxy--engineering-guide">Shifter Web Proxy — Engineering Guide</h1>
<p>A web proxy that combines <strong>Scramjet in the browser</strong>, <strong>HAProxy (local TCP or HTTPS/WSS termination)</strong>, <strong>Rust Wisp gateways</strong>, <strong>Redis session enforcement</strong>, and <strong>Shifter residential proxies</strong>.</p>
<p>Visitors enter a website, choose an exit country, and browse inside the page without configuring their browser’s proxy settings. Each visitor receives a server-generated sticky session ID. The gateway adds the upstream credentials and country targeting on the server.</p>
<p><strong>Status: shared browser SDK and server-verified reCAPTCHA v2 implemented.</strong> The hosted entry point is <code>/sdk/v1/shifter-web-proxy.js</code>, exposing <code>ShifterWebProxy.create()</code>. Production API access uses site-bound anonymous bearer credentials; the legacy cookie API exists only in synthetic test mode. Production activation still requires real CAPTCHA credentials and final acceptance on both websites. This does not establish a unique-person identity or universal destination compatibility.</p>
<p>For the product overview, see <a href="../README.md">README.md</a>. An offline HTML version of this guide is included at <a href="../docs/engineering.html">docs/engineering.html</a>.</p>
<h2 id="contents">Contents</h2>
<ul><li><a href="#what-is-included">What is included</a></li><li><a href="#shared-sdk-integration">Shared SDK integration</a></li><li><a href="#architecture">Architecture</a></li><li><a href="#repository-and-secret-policy">Repository and secret policy</a></li><li><a href="#requirements">Requirements</a></li><li><a href="#quick-start-with-synthetic-traffic">Quick start with synthetic traffic</a></li><li><a href="#using-a-real-shifter-account">Using a real Shifter account</a></li><li><a href="#httpswss-deployment">HTTPS/WSS deployment</a></li><li><a href="#configuration-reference">Configuration reference</a></li><li><a href="#redis-security-rules">Redis security rules</a></li><li><a href="#session-lifecycle-and-limits">Session lifecycle and limits</a></li><li><a href="#browser-integration-and-isolation">Browser integration and isolation</a></li><li><a href="#api-reference">API reference</a></li><li><a href="#gateway-forwarding-and-destination-policy">Gateway forwarding and destination policy</a></li><li><a href="#haproxy-regions-and-recovery">HAProxy, regions, and recovery</a></li><li><a href="#testing-and-validation">Testing and validation</a></li><li><a href="#operations-and-troubleshooting">Operations and troubleshooting</a></li><li><a href="#project-layout">Project layout</a></li><li><a href="#dependencies-and-reproducibility">Dependencies and reproducibility</a></li><li><a href="#production-work-still-required">Production work still required</a></li></ul>
<h2 id="what-is-included">What is included</h2>
<ul><li>Trusted browsing controls: destination input, country selector, Browse, Back, Forward, Reload, Stop, remaining time, remaining bytes, and status messages.</li><li>An embedded Scramjet runtime using bare-mux and libcurl transport.</li><li>Two Rust gateway replicas, a Rust session API, Redis, and an HAProxy connection balancer.</li><li>Country selection and a cryptographically random 128-bit SID per browsing assignment.</li><li>Shared time/byte allowances, single-use transport tickets, one active connection per visitor, stream admission limits, revocation, and expiry.</li><li>An authenticated SOCKS5 connector that sends every destination stream through a configured Shifter regional endpoint.</li><li>Synthetic destination/login fixtures, integration checks, outage tests, and a 1,000-session load test.</li><li>Explicit, bounded live smoke tests for country targeting, sampled sticky IP behavior, and TLS forwarding.</li><li>Example configuration files and a bootstrap script that creates ignored local copies.</li></ul>
<p>The SDK includes reCAPTCHA v2; live use requires configured credentials. Fingerprint identity is not part of this integration.</p>
<h2 id="web-proxy-page-example">Web proxy page example</h2>
<p><code>web/control/index.html</code> retains the Shifter landing page and browser toolbar. Its <code>app.js</code> is a UI adapter: it consumes SDK state and invokes methods; API calls, CAPTCHA, runtime messages, polling, expiry, and cleanup live in the SDK. <code>web/control/minimal.html</code> demonstrates a different interface using the same library.</p>
<p>Search validates the address and opens the library-owned reCAPTCHA v2 modal. Closing it or pressing Escape cancels. A successful challenge is verified by the API before browsing starts. Navigation, reload, and location changes within an active session do not require another challenge. Every landing-page Search does. Failed or unavailable CAPTCHA never grants access.</p>
<p>The browser retains Back, Forward, Reload, country selection, the URL field, time/data allowance, End session, return-to-home, and resume controls. The website owns these visuals and the loading overlay. The SDK owns the destination iframe and emits state for the website to render.</p>
<h2 id="shared-sdk-integration">Shared SDK integration</h2>
<p>The examples use reserved origins <code>https://example.com</code>, <code>https://second.example.com</code>, and <code>https://staging.example.com</code>. Set your actual origins only in the ignored production environment file; production startup requires an explicit <code>INTEGRATIONS</code> value. No <code>www</code>, HTTP, wildcard, or other subdomain is implicitly included; add an explicit <code>https://*.example.com</code> entry to allow its subdomains. Loading the public JavaScript does not grant authorization; API and runtime origins are checked separately.</p>
<pre><code><div id="proxy-view" style="height:70vh"></div>
<script src="https://proxy.example.net/sdk/v1/shifter-web-proxy.js"></script>
<script>
const proxy = ShifterWebProxy.create({
container: document.querySelector('#proxy-view')
});
proxy.subscribe(state => {
// Render your controls using state.active, busy, loading, countries,
// country, url, session, remainingSeconds, persistent and error.
});
proxy.init().then(() => {
// Enable your Search control. Call from its submit handler:
// proxy.search({url: address.value, country: country.value});
}).catch(error => console.error(error.message));
</script></code></pre>
<p><code>create()</code> returns immediately; <code>init()</code> waits for the selected release and configuration. Methods return promises. <code>subscribe(callback)</code> invokes the callback immediately and returns an unsubscribe function. <code>getState()</code> returns a defensive copy. One instance may own a container at a time.</p>
<div class="table"><table><tr><th>Method</th><th>Contract</th></tr><tr><td><code>init()</code></td><td>Fetch allowed-site configuration and restore available allowance; does not start browsing</td></tr><tr><td><code>search({url, country})</code></td><td>Validate, display CAPTCHA, authorize, and browse; rejects with <code>CANCELLED</code> on dismissal</td></tr><tr><td><code>navigate(url)</code></td><td>Navigate an active session without another CAPTCHA</td></tr><tr><td><code>back()</code>, <code>forward()</code></td><td>Traverse destination history</td></tr><tr><td><code>reload()</code></td><td>Reconnect with the same assignment/allowance after lease release</td></tr><tr><td><code>changeCountry(country)</code></td><td>Revoke old assignment, clear website state, preserve allowance, and navigate</td></tr><tr><td><code>stop()</code></td><td>Revoke access and await runtime cleanup; retain anonymous identity</td></tr><tr><td><code>destroy()</code></td><td>Abort SDK work, detach iframe/listeners/timers, and release the container; does not reset server allowance</td></tr><tr><td><code>subscribe(callback)</code>, <code>getState()</code></td><td>Read UI state without exposing credentials or tickets</td></tr></table></div>
<p>State status is <code>idle</code>, <code>initializing</code>, <code>ready</code>, <code>verifying</code>, <code>connecting</code>, <code>browsing</code>, <code>interrupted</code>, <code>stopped</code>, <code>expired</code>, <code>exhausted</code>, or <code>destroyed</code>. Errors contain <code>code</code> and <code>message</code>. Handle rejected promises; only <code>CANCELLED</code> normally needs no error message. Concurrent operations reject with <code>BUSY</code>. A storage-disabled browser uses memory and reports <code>persistent: false</code>.</p>
<p>The default API origin is the loader's origin. <code>apiOrigin</code> is an optional local-integration override. <code>nonce</code> can be supplied for CSP-authorized injected scripts/styles; the loader inherits its script nonce by default. Neither option changes server authorization.</p>
<h3 id="older-browser-api-compatibility">Older browser API compatibility</h3>
<p>SDK 1.0.6 uses a request AbortController with a 15-second timer and forwards instance destruction to that controller. The deadline covers both the HTTP request and its response body; timers and abort listeners are released after each attempt. Safe API reads and one-use ticket requests retain their existing two retries. Ambiguous mutations are never automatically repeated.</p>
<p>Brave 1.45.133 uses Chromium 107.0.5304.141 (<a href="https://github.com/brave/brave-browser/releases/tag/v1.45.133">official release</a>). Chromium added <code>AbortSignal.any()</code> in version 116 (<a href="https://github.com/mdn/browser-compat-data/blob/main/api/AbortSignal.json">compatibility data</a>). The earlier SDK called that helper before sending a request, and its catch block misclassified the resulting TypeError as <code>NETWORK_UNAVAILABLE</code>, displaying “Connecting to the web proxy. Please try again shortly.” Removing the helper in regression tests reproduces that exact error with the previous code and succeeds with 1.0.6. This explains a deterministic API failure on an unmodified Chromium 107 engine; it does not prove every destination works in that older browser. The full browser fixture can disable both <code>AbortSignal.any()</code> and <code>AbortSignal.timeout()</code> with <code>--legacy-api-compat</code>.</p>
<h3 id="mobile-browser-setup">Mobile browser setup</h3>
<p>The runtime uses native SharedWorkers when available. On engines without SharedWorker support, a dedicated worker owned by the trusted outer runtime hosts the same BareMux transport. Fresh MessagePorts connect the runtime, rewritten documents and service worker to that transport. The vendor build patches both BareMux copies, including the one bundled in Scramjet, and checks the pinned channel-setup code before applying the patch. Destination websites retain their native SharedWorker feature detection. Authorization, one-use tickets, private Redis enforcement and quotas use the same paths.</p>
<p>Cleanup always clears the pinned <code>$scramjet</code> database and also clears other databases when enumeration is available. Missing <code>indexedDB.databases()</code> no longer fails cleanup. Opening a nonexistent database aborts its creation so Scramjet can install its own schema. Aborted transactions fail cleanup. Cleanup has a six-second deadline; runtime configuration reads have a five-second abort deadline, and engine initialization, suffix data, worker registration/activation and transport setup each have a twelve-second deadline. The SDK retains its twenty-second readiness deadline. A setup failure updates the preparation screen and SDK error state. Reload repeats a failed reset before starting; stale messages cannot authorize a failed runtime.</p>
<p>Browsers or embedded webviews that block service workers or required site storage receive an actionable setup error. A setup/storage failure is not evidence of a bad residential exit, so it does not rotate the SID. Reported browser/device combinations still need acceptance on those physical devices; desktop feature-disable tests establish fallback behavior, not complete Android compatibility.</p>
<h3 id="automatic-connection-recovery">Automatic connection recovery</h3>
<p>SDK 1.0.6 negotiates a runtime heartbeat over the existing origin/source/channel/engine-bound bridge. A visible, active embed probes every five seconds and replaces the outer runtime iframe after 15 seconds without a matching reply (up to 20 seconds from a crash). Hidden tabs and host suspension reset the probe deadline. Replacement uses the bounded recovery endpoint, retires the old lease, then loads <code>index.html</code> without running storage reset. Each outer runtime generation installs a fresh service worker in the same scope and waits for it to control the frame; this replaces stale engine/transport state while retaining the IndexedDB cookie jar. A narrow compatibility shim serializes v1’s persisted cookie objects before loading them because the pinned <code>CookieStore.load</code> implementation otherwise ignores objects. The session keeps its SID, country, expiry, consumed bytes and persisted website storage; the address is opened as GET, never replaying a form body. In-memory page state and navigation history cannot survive renderer loss. Stop, expiry, destruction and explicit navigation cancel pending recovery; Reload or navigation can rebuild an interrupted runtime. The watchdog requires the updated runtime’s heartbeat capability, so older runtimes continue to work without false alarms.</p>
<p>The neutral worker error document retains a script-blocking CSP but omits its own <code>frame-ancestors</code>; the enclosing runtime’s configured ancestor allowlist protects the embed. Browser-generated inaccessible error documents are reported as errors rather than successful loads.</p>
<p>SDK 1.0.6 silently retries a failed GET document navigation at most twice, keeping <code>loading: true</code> and the website's branded overlay visible. Backoffs are 300 and 600 milliseconds; each reconnect waits 6.5 seconds for the previous gateway lease to retire before requesting a fresh one-use ticket. Stop, a new SDK operation, destruction, or expiry cancels queued recovery. After the retry budget is exhausted, the SDK sets <code>status: interrupted</code>, releases the overlay, and leaves browser controls available without displaying a connection-error banner.</p>
<p>A thrown WebSocket connection failure reconnects with the existing SID. A transient SOCKS connection failure can rotate the SID only when the active gateway has recorded that failure for the requested hostname and session revision. Rotation preserves country, region, expiry, used bytes, and byte limit. The atomic Redis operation rejects stale/concurrent recovery, inactive sessions, and more than two recoveries in a 30-second window. The old lease is never cleared early. Redis failures deny recovery.</p>
<p>Website HTTP errors, CAPTCHAs, and login pages are normal responses and pass through unchanged. Failed POST/PUT submissions, engine errors, destination policy/DNS rejection, explicit upstream authentication rejection, and exhausted/expired authorization are not automatically replayed. Recovery applies to failed document loads, not arbitrary background requests or embedded widgets. Changing a residential SID may affect a destination's IP-bound login or session.</p>
<p>The runtime replaces failed document responses with a neutral placeholder before they reach the screen. A short-lived, one-use capability held by the service worker lets the trusted runtime retrieve the actual request method and failure classification; destination HTML and untrusted window messages cannot manufacture that classification. Existing SDK releases receive a friendly error, while 1.0.2 negotiates automatic recovery over the existing v1 protocol. Reloading a website using the stable hosted SDK loader picks up the update after deployment; site adapters render <code>state.loading</code> directly, without a second loading timeout that could reveal a recovery in progress.</p>
<p>SDK 1.0.3 retries API reads and ticket issuance up to twice (500/1000 ms backoff) on network failures or unstructured gateway 502/503/504 responses. Failed background session checks stay silent, preserve the destination and SID, and back off up to 30 seconds; the local expiry clock keeps running. Explicit authorization, quota, and storage refusals are not treated as transient gateway failures. Mutating session requests and destination form submissions are not blindly replayed. Slow-page timeouts release the overlay without a warning banner.</p>
<p>The runtime routes Scramjet v1's injected WASM script through the engine's JavaScript wrapper; raw WASM fetches and reset assets still bypass engine configuration. On browsers without transferable request streams, the service worker buffers request bodies up to 8 MiB before the first BareMux send, preserving method, headers and bytes. Larger buffered uploads fail before sending. Capable browsers keep streaming. This addresses a reproducible request-stream compatibility gap; it does not guarantee that Google will accept a proxy IP or that every CAPTCHA works. A destination CAPTCHA remains on its existing SID and is never used as a reason to rotate an IP automatically.</p>
<p>The runtime seeds Scramjet v1's Public Suffix List cache from a bundled snapshot before starting destination requests. Without that cache, concurrent cross-origin requests download the list through the residential connection; a failed list download can abort an unrelated script or image. Initialization waits for the IndexedDB transaction to commit and fails if the bundled data is missing or invalid. The one-hour cache covers the maximum 30-minute session and is refreshed whenever the outer runtime is recreated. No domain rules are removed and no destination checks are bypassed.</p>
<p>An incomplete successful GET script or stylesheet response is retried once on the same transport before Scramjet's empty 500 reaches the page. This only applies when reading a 2xx response body fails; website HTTP errors, connection refusals, rewriting errors, other resource types, and non-GET requests are not retried by this path. Both attempts remain subject to the existing session authorization and byte accounting. This does not establish the cause of every blank page or guarantee YouTube compatibility.</p>
<p>The browser transport is pinned to libcurl-transport 1.5.2, which integrates with the existing bare-mux worker and authenticated Wisp gateway. In local US comparisons, Epoxy returned failed response-body reads for YouTube application bundles and left the page on placeholders, even when video playback worked. Changing only the transport to libcurl loaded those bundles and the full interface. The exact defect inside Epoxy remains unconfirmed; this change does not require a Scramjet v2 migration or a new residential session.</p>
<p>The vendoring step patches libcurl.js to propagate transfer failures to the response stream after headers have arrived. Without that patch, an interrupted script can look like a successful truncated response. The synthetic browser fixture deliberately interrupts a 200 script body and verifies the single retry executes the complete script. TLS verification stays enabled; local US checks accept a valid certificate and reject expired, wrong-hostname, and self-signed certificates.</p>
<p>The rewritten-document bootstrap disables the native <code>Document.startViewTransition</code> and <code>Element.startViewTransition</code> APIs before destination scripts run. Sites select their normal DOM-update fallback. This avoids a reproduced Chrome 155 compositor crash in a proxied YouTube frame; the embedding website and trusted runtime retain the native APIs. Local crash dumps identify a compositor <code>map::at</code> failure, rather than establishing an out-of-memory cause. Dependency builds fail if either patched upstream location changes, so upgrades require explicit review.</p>
<p>The gateway explicitly flushes queued Wisp CONTINUE packets after reading upload data. The pinned Wisp library otherwise leaves them buffered until another outgoing write, which can stall an upload waiting for renewed credits. A 64 KiB binary POST regression crosses the four-packet receive window and checks the complete response byte-for-byte. TLS certificate verification, stream admission, destination policy, authorization, expiry and byte accounting remain enabled.</p>
<p>Gateway DNS lookups for one hostname share an in-flight result, with at most 512 cached entries lasting 30 seconds. This includes the shared Shifter upstream hostname, avoiding a separate lookup for every destination stream. One failed lookup is retried within the existing connection timeout. Every resolved destination address still passes the destination policy, and the validated IP is pinned in the SOCKS request. <code>RUST_LOG=shifter_web=debug</code> enables connection-stage diagnostics without logging destination URLs, session IDs, or upstream credentials; the default remains <code>shifter_web=info</code>.</p>
<h3 id="captcha-configuration">CAPTCHA configuration</h3>
<p>Register both production website domains and the explicitly enabled staging hostname <code>staging.example.com</code> on the existing standard v2 checkbox key. The <strong>site key is public</strong>, including in GitHub. The <strong>secret is private</strong> and must never enter SDK code, HTML, URLs, repository files, or logs. Set <code>RECAPTCHA_SITE_KEY</code> and mount <code>RECAPTCHA_SECRET_FILE</code> only in the API container. The API needs outbound HTTPS to Google's fixed SiteVerify endpoint. Google responses must be successful, free of error codes, and match the exact caller hostname. Google enforces the response token's two-minute lifetime; the returned challenge timestamp is validated as a load timestamp, not mistaken for when the user solved it. A SHA-256 token reservation in Redis rejects concurrent replay; raw CAPTCHA responses are not stored. See <a href="https://developers.google.com/recaptcha/docs/verify">Google verification</a>.</p>
<p>For local real-CAPTCHA development, add <code>compose.captcha.example.yaml</code> as an overlay, set the public key and local secret-file path in ignored configuration, and register localhost on a development key. The overlay mounts the secret only in the API. Synthetic browser tests instead use the disposable test stack; Google test tokens are not automatically accepted by its Redis-backed mock verifier.</p>
<p>If Google reports a widget error, the SDK keeps its frame and modal open so Google's own message remains visible, and offers Try again or Close. It does not reset/remove the frame automatically on that error: doing so hides domain/key errors and produces misleading canceled network requests. Only a completed challenge can call the session API. Script-load failures and expired challenges also offer an explicit retry. For a localhost rejection, verify that the exact site key returned by the local API is the key whose domain settings you saved in Google; changing another key does not affect the running demo.</p>
<p>Production startup rejects missing credentials and Google's public test credentials. Synthetic tests seed random one-use challenge hashes in private Redis; those records are read only in <code>APP_ENV=test</code>. There is no configurable production verifier URL or bypass switch.</p>
<h3 id="download-restrictions">Download restrictions</h3>
<p>SDK 1.0.7 and the accompanying hosted runtime disable downloads in both iframe sandbox layers. The service worker checks upstream response headers before Scramjet reads or rewrites the body: attachment/unknown dispositions, executable/archive MIME types and dangerous filenames are denied. Navigation responses must have an explicitly allowlisted display MIME type; missing, unknown and unsupported media types (including TIFF) are denied. Media subresource requests are not restricted to this navigation allowlist. Redirect destinations are checked on their subsequent request. The same policy covers fetch, script/style retries, POST and partial responses. A blocked navigation renders a responsive, dark 403 “Downloads are disabled” notice explaining that malware-related abuse violates the Terms of Service and the restriction applies to all files. It directs visitors to the existing proxy Back button or address controls. The notice uses only inline HTML/CSS/SVG, with no scripts, external assets, or destination details; its CSP permits inline styling only. Blocked subresources return an empty 403. The worker cancels rejected bodies without waiting for transport cancellation and does not trigger connection recovery for a policy denial.</p>
<p>A CSP sandbox is appended after rewriting every proxied navigation response. It omits download, popup and top-navigation permissions and remains effective if a page removes its iframe sandbox attribute. Normal HTML, scripts, styles, images, JSON, WebAssembly and binary API fetches remain supported, subject to the response policy. Fetch/XHR requests for <code>.js</code> assets require a JavaScript MIME type and no attachment disposition; navigating to the same script filename remains blocked. This intentionally blocks legitimate attachment downloads too. Browser formats and website compatibility still require testing; a supported MIME type is not a malware verdict.</p>
<p>These are browser-side browsing restrictions, not a guarantee that malware bytes never pass through the relay or that a user cannot save already-received data. The Wisp backend still relays opaque HTTPS connections. Deliberate custom-client abuse and server-side malware inspection require a separate architecture change.</p>
<p>Deploy the updated runtime assets to <strong>every serving region</strong>, along with immutable SDK <code>1.0.7</code> assets and the stable <code>/sdk/v1/shifter-web-proxy.js</code> loader. The Docker image includes both, so rebuild and redeploy it through the normal release process. Existing stable-URL website integrations need no adapter changes; reload open tabs to pick up the new SDK and runtime worker. Integrations pinned to an immutable older release must update their pin. Publishing only the SDK does not install response filtering. Historical releases remain unchanged for rollback; rolling back these components can restore download capability.</p>
<h3 id="release-and-rollback">Release and rollback</h3>
<p>Maintain source only under <code>web/sdk/src/</code>. <code>node scripts/build-sdk.mjs 1.0.4</code> creates/verifies immutable files and a SHA-256 manifest in <code>web/sdk/releases/1.0.4/</code>. With no argument, the script verifies the release selected by the loader. It rejects changed content for an existing version. To release an update, pass a new semantic version and change <code>RELEASE</code> in <code>web/sdk/v1/shifter-web-proxy.js</code> after validating that release.</p>
<p>The loader uses <code>Cache-Control: public, no-cache</code>; release assets use a one-year immutable cache. Each page imports one fixed release. Rollback changes the loader's release target; existing page instances keep their release until reloaded. Retain old release directories. v1 releases must preserve protocol 1 so the runtime supports the current and previous SDK release; introduce <code>/sdk/v2/</code> for a breaking contract. Release 1.0.4 adds runtime heartbeat detection and replacement without a new CAPTCHA, plus an embeddable worker failure document. Release 1.0.3 adds silent status-check retries and removes connection/slow-page notices. The accompanying runtime fixes injected WASM routing and unsupported request-stream transfers. Release 1.0.2 adds bounded proxy recovery. Releases 1.0.1 (CAPTCHA error-frame handling) and 1.0.0 remain available for rollback.</p>
<p>Deploy the API and assets before changing website adapters, and disable the old production session creation at that cutover. Old cookie sessions are not imported: visitors complete a fresh CAPTCHA for a new origin-scoped identity. Update Redis ACLs before deploying; missing permissions fail closed.</p>
<h2 id="architecture">Architecture</h2>
<pre><code>Visitor browser
|
+-- http://localhost:8080
| HAProxy TCP listener
| -> Rust API / trusted browsing controls
| -> Redis: identity-linked daily allowance and tickets
|
+-- http://localhost:8081
HAProxy TCP listener / leastconn
-> Rust gateway A or B
|-- serves Scramjet, bare-mux, libcurl, and service worker
|-- redeems Wisp ticket and claims connection ownership
|-- validates destination and reserves destination bytes
|
+-- authenticated SOCKS5 -> blr.p.shifter.io:443
-> residential exit
-> destination website</code></pre>
<p>Scramjet rewrites proxied website behavior in the visitor’s browser. Wisp carries multiplexed destination TCP streams over a WebSocket. The Rust gateway provides the session-specific upstream connector; Scramjet does not receive the residential proxy credentials.</p>
<p>The same Rust binary has two roles:</p>
<div class="table"><table><tr><th>Role</th><th>Responsibility</th><th>Secret access</th></tr><tr><td><code>api</code></td><td>Trusted controls, country list, session lifecycle, ticket issuance</td><td>Does not load the upstream credential file</td></tr><tr><td><code>gateway</code></td><td>Runtime assets, authenticated Wisp, DNS/destination checks, SOCKS forwarding</td><td>Reads the mounted upstream credential file</td></tr></table></div>
<p>Redis is shared across the API and gateway replicas. No process-global proxy URL is changed when a visitor selects a country. Upstream authenticated connections are not pooled across visitors.</p>
<h2 id="repository-and-secret-policy">Repository and secret policy</h2>
<p><strong>Only portable examples of deployment and credential configuration belong in this repository.</strong> Real accounts, passwords, machine paths, resolved configuration, operational logs, session records, live IP evidence, and local validation reports remain outside Git.</p>
<div class="table"><table><tr><th>Tracked example</th><th>Generated local file</th><th>Purpose</th></tr><tr><td><code>compose.example.yaml</code></td><td><code>compose.yaml</code></td><td>Local development topology</td></tr><tr><td><code>compose.test.example.yaml</code></td><td><code>compose.test.yaml</code></td><td>Synthetic upstream and isolated test database</td></tr><tr><td><code>compose.production.example.yaml</code></td><td><code>compose.production.yaml</code></td><td>HTTPS ingress and applications connected to existing private Redis</td></tr><tr><td><code>deploy/haproxy.production.example.cfg</code></td><td><code>deploy/haproxy.production.cfg</code></td><td>TLS certificates, HTTPS routing, redirects and WSS</td></tr><tr><td><code>deploy/production.example.env</code></td><td><code>.env.production</code></td><td>Hostnames and private certificate/secret paths</td></tr><tr><td><code>deploy/haproxy.example.cfg</code></td><td><code>deploy/haproxy.cfg</code></td><td>TCP listeners, balancing, DNS resolution, health checks</td></tr><tr><td><code>.env.example</code></td><td><code>.env</code></td><td>Non-secret mock defaults; local credential-file path for live mode</td></tr><tr><td><code>tests/fixtures/credentials.example.toml</code></td><td><code>tests/fixtures/credentials.toml</code></td><td>Public synthetic values accepted only by the mock fixture</td></tr><tr><td><code>examples/shifter-credentials.example.toml</code></td><td>A file under <code>.secrets/</code> or outside the checkout</td><td>Placeholder schema for a real upstream account</td></tr></table></div>
<p>The synthetic fixture values are deliberately non-secret test data; they are not a working Shifter account.</p>
<p>Run <code>sh scripts/configure.sh</code> to generate missing local files. It never overwrites an existing local configuration. Generated files are created with restrictive permissions. Before adopting updated templates, compare them with your local copies and apply changes deliberately.</p>
<p><code>.gitignore</code> excludes local YAML/CFG/TOML files, environment files, credentials directories, certificates/keys, build products, logs, artifacts, and local-only research/report files. Explicit <code>*.example.*</code> templates, dependency manifests, source, tests, CI definitions under <code>.github/</code>, and this documentation are tracked. CI fails when a tracked file matches <code>.gitignore</code> or tracked text contains a private key block or an absolute home-directory path. <code>.dockerignore</code> uses a build-context allowlist so local configuration and evidence do not enter Docker builds.</p>
<p>The HTTP server never serves the repository root. Both the API's demo and the gateway serve only explicitly allowed browser assets from their respective <code>web/control</code>, <code>web/runtime</code>, and published <code>web/sdk</code> directories. Unknown files, dotfiles, configuration, backups, SDK source directories, source maps, encoded aliases, traversal paths, and symlinked assets return 404 with <code>Cache-Control: no-store</code>. Adding a new public asset requires updating <code>src/public_assets.rs</code>; SDK releases accept numeric <code>major.minor.patch</code> directories and only the release's client, CAPTCHA module, and manifest. Runtime dependencies remain at their pinned versions.</p>
<p>Keep <code>.env.captcha</code> and <code>.secrets/recaptcha-secret</code> outside <code>web/</code>, ignored by Git, with host permissions 600 (the <code>.secrets</code> directory should be 700). The public site key is intentionally returned by <code>/api/v1/config</code>; the private secret is mounted read-only at <code>/run/secrets/recaptcha_secret</code> only in the API container, never in gateway containers, container environment variables, or images. Docker build exclusions also cover nested secret directories, environment/configuration files, keys, backups, and database files accidentally placed beneath an allowed source directory. Production serves a read-only container filesystem; do not mount the checkout, secret directories, or writable content into public asset roots. Git ignore rules alone do not provide HTTP protection.</p>
<p><code>node tests/public-files.mjs</code> probes the running local demo and runtime using raw GET/HEAD paths (including traversal and percent encoding) and checks that required public assets still load. The same denial probes run against the disposable production HTTPS ingress in <code>tests/https-deployment.mjs</code>. Rust tests plant synthetic sensitive files and symlinks inside a temporary public tree to verify that the guard denies them even when present. <code>node tests/build-context.mjs</code> builds a disposable image with synthetic secret files to verify Docker's actual exclusion rules. Local validation does not verify an existing public deployment; deploy these changes before relying on them at <code>proxy.example.net</code>.</p>
<p>Never put a real password in a template, a command-line argument, a Docker build argument, a frontend bundle, an issue, or a test-output file. Git ignore rules do not remove previously committed data: inspect the staged diff before every push.</p>
<h2 id="requirements">Requirements</h2>
<ul><li>Docker with the Compose plugin, a running local Docker engine, support for BuildKit cache mounts and bridge gateway mode <code>isolated</code> (validated locally with Engine 29.4.0).</li><li>A browser with service workers, WebAssembly, and either SharedWorkers or dedicated workers. Browser compatibility beyond the observed local fixture flows is not established.</li><li>Node.js 24 when running the host-side test scripts or vendoring assets outside Docker.</li><li>Rust/Cargo compatible with Rust 1.94.1 when running host-side Rust checks. The Docker build includes the pinned toolchain.</li><li>Python 3.11 or newer for <code>tests/secret-scan.py</code>; Python 3 for the offline README renderer.</li><li>Available loopback ports 8080 and 8081 for the normal stack, or 8180 and 8181 for the isolated SDK test runner.</li></ul>
<p>The build fetches pinned dependencies and base images. After the images and assets exist locally, the mock destination does not require a real upstream account or destination Internet traffic.</p>
<h2 id="quick-start-with-synthetic-traffic">Quick start with synthetic traffic</h2>
<pre><code>git clone https://github.com/leftclick-io/web-proxy.git
cd web-proxy
# Optional explicit bootstrap; stack.sh also runs it automatically.
sh scripts/configure.sh
# Build and start the API, two gateways, Redis, HAProxy, and mock upstream.
sh scripts/stack.sh mock</code></pre>
<p>Open <strong>http://localhost:8080</strong> to inspect the UI. Synthetic backend tests use <code>http://fixture.test/</code> and never consume paid upstream traffic. Browser Search now requires verification: the mock backend accepts only one-use challenge records seeded directly by tests, not arbitrary checkbox state or Google's public test responses. Use the disposable SDK runner below for automated checks; use the local CAPTCHA overlay for real verification.</p>
<p>The fixture displays the configured exit country and a synthetic assignment marker. It provides a JavaScript fetch, navigation links, redirects, a synthetic username form, and a second virtual origin for cookie-isolation checks. Use only synthetic identities with the fixture.</p>
<p>Only the HAProxy listeners are published to the host:</p>
<div class="table"><table><tr><th>URL</th><th>Purpose</th></tr><tr><td><code>http://localhost:8080</code></td><td>Trusted controls and session API</td></tr><tr><td><code>http://localhost:8081</code></td><td>Isolated runtime origin and Wisp endpoint</td></tr></table></div>
<p>Use <code>localhost</code> consistently. Origin checks expect these exact origins; replacing the host with <code>127.0.0.1</code> changes the origin even though both reach loopback.</p>
<p>Localhost HTTP supports the service-worker secure-context exception. It does not validate production HTTPS certificates, WSS termination, or a production origin layout.</p>
<p>To stop the project while retaining its Redis volume:</p>
<pre><code>sh scripts/stack.sh down</code></pre>
<h2 id="using-a-real-shifter-account">Using a real Shifter account</h2>
<p>The repository does not include a real credential file or a path to anyone’s existing secrets.</p>
<p>Create a private local file from the placeholder schema:</p>
<pre><code>mkdir -p .secrets
chmod 700 .secrets
cp examples/shifter-credentials.example.toml .secrets/shifter.toml
chmod 600 .secrets/shifter.toml</code></pre>
<p>Edit that ignored file locally and replace the placeholders with your account and password. The expected TOML structure is:</p>
<pre><code>[shifter]
account = "YOUR_ACCOUNT"
password = "REPLACE_LOCALLY"</code></pre>
<p>The account may include the <code>customer-</code> prefix; the loader removes it before constructing the full username. Supply the base account, not a username already decorated with country/SID flags.</p>
<p>Edit the ignored <code>.env</code> file:</p>
<pre><code>SHIFTER_CREDENTIALS_FILE=./.secrets/shifter.toml
SHIFTER_REGION=blr
SESSION_SECONDS=86400
BYTE_LIMIT=10000000000</code></pre>
<p>Then start live development mode:</p>
<pre><code>sh scripts/stack.sh live</code></pre>
<p>The credential file is mounted read-only at <code>/run/secrets/shifter_credentials</code> in each gateway. The non-root container user must be able to read the mount under your Docker engine’s host-file permission mapping. If access fails, provision a suitably owned/group-readable secret for UID 10001; do not solve it by committing the file or making the secret world-readable.</p>
<p>The default regional hostname is <code>blr.p.shifter.io:443</code>. The selected exit country remains independent of this regional ingress setting.</p>
<p>Switching between mock and live recreates the application containers and interrupts current requests. The profiles use separate Redis databases: mock uses DB 1; live development uses DB 0. The live command removes the orphaned mock fixture container.</p>
<h2 id="httpswss-deployment">HTTPS/WSS deployment</h2>
<p>Use <code>compose.production.example.yaml</code> as a <strong>standalone</strong> stack. Do not merge it with <code>compose.example.yaml</code>, which supplies local HTTP and unauthenticated Redis. The production example uses your existing Redis and does not create, publish, or modify it.</p>
<pre><code>sh scripts/configure.sh
mkdir -p .secrets/https .secrets/redis-tls</code></pre>
<p>Edit the ignored <code>.env.production</code> generated from <code>deploy/production.example.env</code>:</p>
<ul><li><code>RUNTIME_HOST=proxy.example.net</code> is the public gateway hostname. Point its DNS record at the ingress host. Only HAProxy publishes ports 80 and 443. The API and gateway listeners stay on the internal ingress network; gateways have a separate outbound network for Shifter.</li><li><code>INTEGRATIONS</code> maps exact HTTPS origins to stable website IDs. The production environment example keeps <code>https://example.com</code> (<code>shifter</code>) and <code>https://second.example.com</code> (<code>ip-info</code>) and adds <code>https://staging.example.com</code> (<code>shifter</code>) for staging. Origins must be unique; explicitly configured aliases may share a site ID and its bearer-identity/quota namespace. Browser storage remains separate per origin, and CAPTCHA verification still checks the exact requesting hostname. Set the same mapping in API and gateways. Website IDs namespace quotas; do not rename them during an ordinary release. There is no built-in production allowlist; production startup requires this explicit setting. No implicit <code>www</code>, other subdomains, or localhost are added. An origin of the form <code>https://*.example.com</code> explicitly allows every subdomain at any depth (<code>https://abc.example.com</code>, <code>https://a.b.example.com</code>) but not the apex, so list <code>https://example.com</code> separately when it is needed. Wildcards must be the whole leftmost label and cover a domain, not a bare top-level domain. An exact origin always wins over a wildcard, and CAPTCHA verification checks the concrete requesting hostname. CORS, <code>frame-ancestors</code>, and the runtime parent check apply the same rule.</li><li>Set <code>TLS_CERT_DIR</code> to a directory containing one or more <code>.pem</code> files, each with the full certificate chain followed by its private key. The certificate must cover <code>proxy.example.net</code> (or the configured <code>RUNTIME_HOST</code>). Provision certificates through your existing certificate manager or ACME DNS challenge workflow. The example does not issue or renew certificates automatically. Mount only the needed server PEM files, not an entire CA/ACME account directory.</li><li>Set <code>REDIS_NETWORK</code> to the existing Redis Docker network. It must be an internal bridge with IPv4/IPv6 gateway mode <code>isolated</code>, no Redis published ports, and only Redis/API/gateways as members. HAProxy never joins it. For Redis on separate private infrastructure, adapt the application state-network attachment and firewall policy to that deployment; the supplied Compose topology assumes a shared Docker network.</li><li>Set <code>REDIS_URL_SECRET_FILE</code> to an ignored file containing <code>rediss://USER:URL_ENCODED_PASSWORD@PRIVATE_REDIS_HOST:6379/0</code> for production Redis with verified TLS. The hostname must resolve only to private addresses. Use a dedicated named ACL user, with unauthenticated default access disabled. Production requires TLS with certificate and hostname verification for client and replication traffic, as specified by the Redis security rule.</li><li>Use <code>rediss://</code>. For a private CA place its PEM bundle at <code>.secrets/redis-tls/ca.crt</code> and set <code>REDIS_CA_FILE=/run/redis-tls/ca.crt</code>. For a system-trusted CA, leave this variable empty. TLS certificate and hostname verification cannot be disabled. Mutual TLS client certificates are not configured by this example.</li><li>Set <code>SHIFTER_CREDENTIALS_FILE</code> to the existing account TOML. Ensure application secret files and the CA are readable by container UID 10001, while restricting host access. HAProxy must be able to read its private keys. Never bake these files into images.</li></ul>
<p>The least-privilege Redis ACL tested for application traffic permits <code>~daily:* ~ticket:* ~identity:* ~captcha:*</code> keys and commands <code>+get +set +ping +hello +select +client|setinfo +evalsha +script|load +exists +hset +pexpireat +hget +hincrby +hgetall +hdel +setex +getdel</code>. Apply this to a dedicated application user through your existing Redis administration process; do not replace operational or replication ACLs with this list.</p>
<p>Validate configuration, build, and launch <strong>on the intended deployment host</strong>:</p>
<pre><code>docker compose --env-file .env.production -f compose.production.yaml config --quiet
docker compose --env-file .env.production -f compose.production.yaml run --rm --no-deps haproxy haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
docker compose --env-file .env.production -f compose.production.yaml up -d --build --wait</code></pre>
<p>HAProxy rejects unknown hostnames, redirects HTTP to the gateway HTTPS hostname, and terminates TLS 1.2 or later. On <code>proxy.example.net</code>, <code>/api/*</code> routes to the session API; <code>/wisp/*</code> and runtime assets route to the gateways. The control-page example is not published by this production ingress. WebSockets use HTTP/1.1 with an eleven-minute idle tunnel timeout. The browser automatically chooses WSS. The SDK uses cookie-free bearer authentication and exact-origin CORS. HSTS is set on backend responses. Public <code>/health</code> and <code>/metrics</code> requests are denied; backend health checks remain internal. Access logs are disabled to avoid storing Wisp tickets.</p>
<p>Certificate renewal: have your certificate manager atomically replace the fullchain-plus-key <code>.pem</code> files in the mounted directory, validate with the same HAProxy command, then recreate HAProxy to load them:</p>
<pre><code>docker compose --env-file .env.production -f compose.production.yaml up -d --no-deps --force-recreate haproxy</code></pre>
<p>Recreating HAProxy interrupts active Wisp connections. Schedule renewal deployment accordingly. Production requires server-verified CAPTCHA and site-bound credentials; only synthetic test mode enables the legacy cookie API.</p>
<p>Use the shared SDK integration above on either allowed website. The production ingress serves the library and runtime on <code>proxy.example.net</code>; <code>/api/v1/*</code> routes to the API service. The API has a separate outbound network for Google verification and remains attached to private Redis.</p>
<p>Validate HTTPS locally with a disposable production-mode stack:</p>
<pre><code>docker build -t shifter-web:https-check .
HTTPS_TEST_REDIS_TLS=1 node tests/https-deployment.mjs</code></pre>
<p>The test uses disposable certificates and ACL secrets, the real production HAProxy configuration and application mode, and a private isolated Redis network with ACL/password authentication. The <code>HTTPS_TEST_REDIS_TLS=1</code> setting checks the encrypted Redis path required by production policy. It checks redirects, certificate trust, cookie-free bearer access, exact CORS, production legacy-API rejection, and rejection of foreign Origins, authenticated WSS, ticket reuse rejection, invalid Redis passwords/hostnames/CAs, unauthenticated Redis rejection, application connectivity, HAProxy isolation, and failure when quota storage stops. Only loopback ingress ports are published; Redis has none. Test containers and fixture credentials are removed afterward. Passing these checks does not establish browser compatibility or validate an existing production Redis server.</p>
<p>Configuration references: <a href="https://www.haproxy.com/documentation/haproxy-configuration-tutorials/protocol-support/websocket/">HAProxy WebSocket support</a> and <a href="https://docs.rs/crate/redis/0.32.7">Redis Rust 0.32.7 TLS features</a>.</p>
<h2 id="configuration-reference">Configuration reference</h2>
<p>The example Compose environment supplies the following defaults. Customize the ignored <code>.env</code> and local Compose files as needed.</p>
<div class="table"><table><tr><th>Variable</th><th>Default</th><th>Meaning</th></tr><tr><td><code>SHIFTER_CREDENTIALS_FILE</code></td><td>Mock fixture path in the example</td><td>Host-side path mounted as the live gateway secret</td></tr><tr><td><code>SHIFTER_REGION</code></td><td><code>blr</code></td><td>Explicit Shifter ingress region</td></tr><tr><td><code>SESSION_SECONDS</code></td><td><code>86400</code></td><td>Automatically renewing allowance period, 1–86400 seconds; use 86400 in production</td></tr><tr><td><code>BYTE_LIMIT</code></td><td><code>10000000000</code></td><td>10 GB (decimal) combined destination payload per allowance period</td></tr><tr><td><code>MAX_STREAMS</code></td><td><code>64</code></td><td>Concurrent destination streams per Wisp connection; configuration maximum 128</td></tr><tr><td><code>STREAM_RATE</code></td><td><code>32</code></td><td>New destination streams per second</td></tr><tr><td><code>STREAM_BURST</code></td><td><code>64</code></td><td>Redis token-bucket burst size</td></tr><tr><td><code>CONNECT_TIMEOUT_SECONDS</code></td><td><code>15</code></td><td>DNS plus authenticated upstream establishment timeout</td></tr><tr><td><code>IDLE_TIMEOUT_SECONDS</code></td><td><code>15</code></td><td>Destination-stream idle timeout</td></tr></table></div>
<p>Gateway admission waits up to 20 seconds for a concurrent stream slot and a Redis rate token. Active and waiting tasks together are bounded at twice <code>MAX_STREAMS</code>; overflow or timeout refuses only that stream. Rate-token checks serialize during bursts and deny access on Redis failure. The 15-second default idle timeout frees unused connections from earlier pages. A separate protocol-flood guard terminates a Wisp connection after more than 256 CONNECT packets within its one-second window, or more than 256 unconsumed CONNECTs in the mux admission backlog. Authorization, destination policy, expiry and byte accounting still apply.</p>
<p>These are new defaults and portable examples, not changes to an existing installation’s ignored <code>.env</code> or deployment files. Deployments that explicitly set 32/8/16/60 retain those values until operators update them. Validate capacity before adopting the new 64-stream, 32/s, 64-burst, 15-second idle settings.</p>
<p>These advanced settings are set in the local Compose environment or supplied when running the binary directly:</p>
<div class="table"><table><tr><th>Variable</th><th>Example/default</th><th>Meaning</th></tr><tr><td><code>APP_ENV</code></td><td><code>development</code>, <code>test</code>, or <code>production</code></td><td>Production policy requires distinct HTTPS origins and private Redis with a named ACL user/password and verified TLS. The API requires real reCAPTCHA credentials; SDK identity is a site-bound bearer credential.</td></tr><tr><td><code>ROLE</code></td><td><code>api</code> or <code>gateway</code></td><td>Binary role</td></tr><tr><td><code>REPLICA</code></td><td><code>api</code>, <code>gateway-a</code>, <code>gateway-b</code></td><td>Operational identity for health/metrics</td></tr><tr><td><code>LISTEN</code></td><td><code>0.0.0.0:3000</code></td><td>Listener inside the container</td></tr><tr><td><code>REDIS_URL</code></td><td><code>redis://redis:6379/</code></td><td>Local shared session store; production policy requires private <code>rediss://</code> with named ACL credentials and verified TLS</td></tr><tr><td><code>REDIS_URL_FILE</code></td><td><code>/run/secrets/redis_url</code> in production</td><td>Read the connection URL from a secret file; takes precedence over <code>REDIS_URL</code></td></tr><tr><td><code>REDIS_CA_FILE</code></td><td>Empty (system trust)</td><td>Optional PEM CA bundle for private Redis TLS; hostname verification always remains enabled</td></tr><tr><td><code>INTEGRATIONS</code></td><td>Two production websites; localhost in development</td><td>JSON array of <code>{origin, site}</code> mappings (exact origins or <code>https://*.domain</code> wildcards)</td></tr><tr><td><code>CONTROL_ORIGIN</code></td><td><code>http://localhost:8080</code></td><td>Legacy synthetic-test origin and local default only</td></tr><tr><td><code>RECAPTCHA_SITE_KEY</code></td><td>Empty outside tests</td><td>Public standard v2 checkbox key; required in production</td></tr><tr><td><code>RECAPTCHA_SECRET_FILE</code></td><td>None</td><td>API-only mounted secret file; required in production</td></tr><tr><td><code>RUNTIME_ORIGIN</code></td><td><code>http://localhost:8081</code></td><td>Exact runtime/WebSocket origin</td></tr><tr><td><code>CREDENTIALS_FILE</code></td><td><code>/run/secrets/shifter_credentials</code></td><td>In-container credential path</td></tr><tr><td><code>WEB_DIR</code></td><td><code>/app/web</code> in Docker</td><td>Static asset root</td></tr></table></div>
<p>Country availability is generated from the <a href="https://github.com/leftclick-io/session-manager/blob/master/config/weights.json">Shifter session-manager weights</a>. Sum only <code>shifter</code> leaf counts within each country in <code>locations</code>; include countries with a total <strong>strictly greater than 250</strong>. The current snapshot yields <strong>54 countries</strong>, which form <code>SHIFTER_POOL_COUNTRIES</code>. This is a weights-based snapshot, not a live availability check. A separate, hand-maintained <code>GLOBAL_POOL_COUNTRIES</code> list in <code>src/config.rs</code> adds 66 countries routed through the global pool, so 120 countries are selectable. The two lists must not overlap; the importer refuses to run when a newly eligible country is still listed in the global pool. Both the country API and gateway validation use the combined allowlist. Missing or unlisted countries are rejected; country targeting is never silently omitted.</p>
<p>The searchable country picker groups eligible locations into Tier 1, Tier 2, and Tier 3, with the US, UK, Canada, Germany, Italy, and France first. These are configurable product display priorities in <code>web/control/country-order.js</code>, independent of the IP-count eligibility filter. Filtering search results preserves that order, and future weights refreshes cannot replace it with alphabetical order. Countries not explicitly prioritized fall into Tier 3 alphabetically. The picker defaults to the United States and uses the Astro site’s existing circular SVG flags and country names. Flags appear in the selected value and every menu option, in both the landing page and browser toolbar. Keyboard users can search, move with arrow keys, select with Enter, and dismiss with Escape.</p>
<p>To refresh, obtain the latest weights JSON through authenticated GitHub access into an ignored location or outside the repository, then run:</p>
<pre><code>node scripts/sync-countries.mjs /path/to/weights.json /path/to/Shifter-Astro
node --test tests/countries.test.mjs</code></pre>
<p>The importer updates <code>SHIFTER_POOL_COUNTRIES</code> in <code>src/config.rs</code> and copies eligible flags from the Astro checkout. It does not copy provider weights, counts, or operational metadata into this repository. Rebuild the API and gateways together after refreshing the list. The boundary test excludes a country at exactly 250 and ensures other providers cannot make a country eligible.</p>
<p>The regional allowlist is <code>fra</code>, <code>ams</code>, <code>lon</code>, <code>nyc</code>, <code>tor</code>, <code>sgp</code>, <code>blr</code>, and <code>syd</code>. Only the BLR upstream was exercised in the initial live validation.</p>
<h2 id="redis-security-rules">Redis security rules</h2>
<p><strong>Redis must run exclusively on private network addresses in every region. Public Redis endpoints and published Redis ports are prohibited.</strong> This applies to primary/replica instances, Sentinel, and cluster management ports.</p>
<p>The permanent project rule is <a href="../.cursor/rules/redis-security.mdc">Redis security rule</a>, referenced by <a href="../AGENTS.md">AGENTS.md</a>. It applies to application, infrastructure, example configuration, and operational changes.</p>
<p>The local Compose example attaches Redis only to <code>session_state</code>, a bridge network with <code>internal: true</code> and <code>isolated</code> gateway mode for IPv4 and IPv6. The API and both gateways join that network and the separate default network. HAProxy and the synthetic destination stay off the state network. Redis publishes no host port, including loopback port mappings. Its <code>6379/tcp</code> image metadata in <code>docker compose ps</code> is not a published host listener; any <code>HOST:PORT->6379</code> mapping would violate the rule. Docker host administrators retain privileged access to container networks; this isolation does not defend against a compromised host or Docker administrator.</p>
<p>The explicit bridge mode is required: on the development device, <code>internal: true</code> alone still allowed a direct connection from HAProxy to Redis. With <code>isolated</code> mode, the deployed-container check confirms that the API and both gateways can use Redis while HAProxy cannot connect to its private port. Run this check on each deployment engine; do not remove the isolation options to work around an unsupported engine.</p>
<p>For an existing checkout, <code>scripts/configure.sh</code> preserves local files. Merge the network changes from <code>compose.example.yaml</code> into your ignored <code>compose.yaml</code>, then recreate this project’s containers and networks with <code>docker compose down</code> followed by your usual stack start command. Do not pass <code>--volumes</code>: the Redis data volume must survive. This interrupts active local browsing connections.</p>
<p>Production requirements:</p>
<ul><li>Private IPs and private DNS only; no public IP, public load balancer, public DNS endpoint, NAT/port forwarding, or host-network shortcut for Redis.</li><li>Bind Redis only to private interfaces and necessary loopback addresses. Firewall/security groups deny access by default, allowing the regional API/gateways and explicitly approved private administration, monitoring, and replication peers.</li><li>Require named ACL users with least-privilege key and command permissions; disable unauthenticated default access. Keep protected mode enabled.</li><li>Require TLS with certificate and hostname verification for client and replication connections. Private networking and named ACL authorization remain mandatory.</li><li>Keep credentials, ACL material, certificates, private keys, and actual regional configuration in secret storage or ignored local files. Do not log authenticated Redis URLs.</li><li>Continue failing closed when authorization or quota storage is unavailable.</li></ul>
<p>The local configuration uses unauthenticated Redis <strong>only inside its isolated Docker state network in development/test mode</strong>. Production policy requires a named ACL user and password on private Redis with verified TLS. The application also supports plaintext private connections, but that support does not satisfy the production TLS requirement. Redis TLS supports a private CA bundle. Startup rejects insecure TLS verification, default/missing ACL users, and Redis DNS resolving outside RFC1918/ULA space. Network isolation alone must not be described as complete production hardening.</p>
<p>Run the policy rejection checks and the deployed-container checks after networking changes:</p>
<pre><code>node --test tests/redis-network-policy.test.mjs
node tests/configuration.mjs</code></pre>
<p>The configuration check rejects Redis host port mappings, host networking, non-internal networks, missing isolated bridge modes, non-private assigned IPs, and unexpected state-network members. It also confirms application Redis access and requires a direct connection attempt from HAProxy to fail. These are local Docker checks; actual cloud firewall rules, private endpoint settings, ACLs, certificates, and regional replication access require deployment-specific verification.</p>
<p>These requirements follow the <a href="https://redis.io/docs/latest/operate/oss_and_stack/management/security/">Redis security guidance</a> and <a href="https://docs.docker.com/reference/compose-file/networks/#internal">Docker Compose internal network configuration</a>, and <a href="https://docs.docker.com/engine/network/port-publishing/#gateway-modes">Docker isolated gateway mode</a>.</p>
<h2 id="session-lifecycle-and-limits">Session lifecycle and limits</h2>
<h3 id="identity-and-daily-allowance">Identity and daily allowance</h3>
<p>The SDK persists a server-generated 256-bit opaque credential in the embedding website's first-party local storage, scoped to API origin and website ID. Redis stores only its SHA-256 hash and an anonymous visitor/site mapping, with a 30-day inactivity expiry refreshed by verified session creation, authenticated session reads and the active gateway lease. A renewing SDK session binds that identity to gateway authorization: missing or expired identities revoke forwarding rather than being recreated by a stale bearer token. An API 401 also tears down the browser runtime, including when Stop was requested. Requests omit cookies. The runtime receives only one-use tickets, never the persistent credential. Allowance session keys include both website ID and visitor ID.</p>
<p>Deleting storage or changing browser profiles can obtain another anonymous identity after CAPTCHA. This is not an unresettable per-person quota. Scripts trusted by the embedding website can access its storage, so that website's XSS protections remain important. A stopped session retains its credential and allowance. Legacy <code>shifter_dev</code> cookies are accepted only in <code>APP_ENV=test</code>.</p>
<p>On first session creation, Redis records the visitor’s country, SID, region, expiry, byte usage, limit, revision, and stopped state. The browsing window starts immediately, including time spent idle or stopped. The default period is a full 24 hours from creation, independent of UTC midnight. Redis advances the period boundary by whole periods and resets consumed bytes atomically when a read, lease check, ticket claim, recovery, stream admission or byte reservation reaches that boundary. Country, SID, revision, connection ownership and explicit Stop are preserved. Multiple elapsed periods produce one fresh allowance, without accumulating unused traffic.</p>
<p>The daily allowance is shared across streams and replicas. Reconnects, country changes, Stop, and resume do not reset time or byte usage. Session records remain for 30 days beyond the current boundary rather than disappearing when browsing stops. Exhaustion denies traffic until the next period; stopping does not disable or advance the period clock. Both directions share the same 10,000,000,000-byte budget.</p>
<p>Session keys use <code>daily:<site>:<visitor>:rolling</code>. On first access after upgrading, an existing record for the current UTC day migrates once into a fresh 24-hour period with the new limit, preserving already-consumed bytes and explicit Stop. Old authorization is revoked in the same transaction. Deploy all API/gateway instances together and refresh open tabs; the migration may reconnect old transports. New periods thereafter preserve healthy transports. Existing environment/Compose overrides must explicitly change to <code>SESSION_SECONDS=86400</code> and <code>BYTE_LIMIT=10000000000</code>; editing examples alone does not change an active deployment.</p>
<p>SDK 1.0.10 uses the additive <code>autoRenew</code> session field. At a zero countdown it keeps the runtime and polls for the authoritative next boundary; it never grants traffic locally. Legacy sessions without this field retain hard expiry. The UI displays HH:MM:SS and decimal GB. Stable-loader integrations update on page reload; pinned/copied releases need updating. The upstream <code>ttl-1800</code> sticky-IP setting is independent and unchanged: continuous browsing does not promise one residential IP for 24 hours.</p>
<h3 id="connection-authorization">Connection authorization</h3>
<ol><li>The trusted UI requests a short-lived ticket from the session API.</li><li>The API creates a random ticket containing an immutable session snapshot and a 30-second TTL.</li><li>The trusted page sends the ticket to the runtime using exact-origin <code>postMessage</code> with source-window validation.</li><li>The runtime opens the authenticated Wisp WebSocket.</li><li>The gateway atomically consumes the ticket using Redis GETDEL, then atomically validates the session revision and claims its connection lease.</li><li>An active owner blocks another simultaneous connection for the same daily visitor record.</li><li>Backend checks renew the six-second lease approximately every second and verify expiry, revision, ownership, and available bytes.</li></ol>
<p>A ticket cannot be reused. Country changes, revocation, and explicit reconnect revision changes invalidate previously issued tickets. The browser cannot supply an upstream SID, account, password, or host through the API.</p>
<h3 id="forwarding-quota">Forwarding quota</h3>
<p>Before forwarding each chunk, a Redis script validates authorization and reserves up to 16 KiB from the shared allowance. Both directions count, including destination TLS traffic. Wisp/control framing is not destination payload. A failed write may conservatively consume its reservation; quotas do not depend on a delayed after-the-fact counter.</p>
<p>Storage errors fail authorization/forwarding closed. Existing connections are also monitored while idle. An interface countdown alone does not grant or revoke access.</p>
<h3 id="country-changes-stop-and-reconnect">Country changes, Stop, and reconnect</h3>
<p>Changing country revokes the old connection, generates a fresh SID, preserves the original allowance, and starts a clean proxied website context. Sites may require login again.</p>
<p>Stop revokes active access and requests website-state cleanup. Resuming uses the same daily time/byte budget and creates a new assignment after a stopped session.</p>
<p>Reload performs an explicit reconnect, retains country/SID/region/allowance, waits for the old connection lease to clear, and reloads the displayed address. Interrupted form submissions are not automatically replayed. Browser cookie retention across this reconnect still needs final acceptance testing.</p>
<p>Different SIDs represent independent assignment requests. They do not guarantee globally unique residential IPs. Even a sticky SID can receive a new IP when its residential peer disconnects or its upstream TTL expires.</p>
<h2 id="browser-integration-and-isolation">Browser integration and isolation</h2>
<p>The embedding website supplies its UI. The runtime on a separate origin owns Scramjet, its service worker, bare-mux SharedWorker, libcurl, and a sandboxed destination iframe. No service worker is installed on either website by the SDK. Runtime and reset documents validate the configured parent origin, actual parent window, protocol version, and per-instance channel identifier. Unknown or stale messages are ignored. <code>frame-ancestors</code> limits browser embedding to self and configured websites.</p>
<p>The integration does not require site-wide COOP/COEP headers. The pinned Scramjet version runs with synchronous XHR disabled. Destination features requiring cross-origin isolation or SharedArrayBuffer are not supported in this mode. Browser service-worker, SharedWorker, WASM, and third-party storage restrictions can still affect compatibility; the SDK reports initialization errors/timeouts. It cannot override a website's restrictive CSP. Allow the SDK script/module origin, API connect origin, runtime frame origin, and Google's reCAPTCHA script/frame/connect resources as appropriate to that website's policy. Use CSP nonces for injected scripts/styles when required.</p>
<p>The runtime clears its local/session storage, service-worker registrations, caches, and IndexedDB contents on reset. Stop removes the engine frame before running reset in an independent offscreen iframe, so a website hiding its viewport cannot interrupt cleanup. Runtime/reset assets bypass the service worker’s engine configuration read, allowing reset to finish after storage is cleared. Stop waits for acknowledgement; a cleanup failure is surfaced. Cleanup is not forensic erasure. Server revocation and quotas remain effective when browser cleanup fails. Runtime cookies and emulated destination origins still need adversarial review: rewriting does not provide browser-enforced origin isolation between every proxied destination. No persistent SDK credential is stored at the runtime origin.</p>
<h2 id="api-reference">API reference</h2>
<p>All SDK API calls use <code>/api/v1</code>. Send the embedding site's exact <code>Origin</code>; authenticated routes additionally require <code>Authorization: Bearer <credential></code>. Browsers omit Origin on same-origin GETs: these are accepted only with same-origin Fetch Metadata and a matching configured Host. Credentialed CORS is not used. JSON requests have a 16 KiB body limit; CAPTCHA response tokens have an 8 KiB field limit.</p>
<div class="table"><table><tr><th>Route</th><th>Body</th><th>Result</th></tr><tr><td><code>GET /api/v1/config</code></td><td>None</td><td>Countries, site ID, runtime origin, public CAPTCHA site key, protocol version</td></tr><tr><td><code>POST /api/v1/sessions</code></td><td><code>{country, captchaToken}</code></td><td><code>{session, credential}</code>; credential returned only for a new identity</td></tr><tr><td><code>GET /api/v1/session</code></td><td>None</td><td>Current allowance and status</td></tr><tr><td><code>POST /api/v1/session/tickets</code></td><td>None</td><td>One-use ticket, session revision, 30-second TTL, runtime origin</td></tr><tr><td><code>POST /api/v1/session/country</code></td><td><code>{country}</code></td><td>Updated assignment; allowance retained</td></tr><tr><td><code>POST /api/v1/session/reconnect</code></td><td>None</td><td>Old connection revision revoked</td></tr><tr><td><code>POST /api/v1/session/recover</code></td><td><code>{revision, url, reason}</code></td><td>Bounded recovery; reason is <code>websocket</code> or <code>upstream</code>; returns <code>{status, retryAfterMs}</code></td></tr><tr><td><code>DELETE /api/v1/session</code></td><td>None</td><td>Stopped access</td></tr></table></div>
<p>Public session fields are <code>country</code>, <code>status</code>, <code>serverTime</code>, <code>expiresAt</code>, <code>remainingBytes</code>, <code>byteLimit</code>, and <code>connected</code>. Times are Unix milliseconds; upstream credentials/SID are private. Session status is <code>active</code>, <code>stopped</code>, <code>expired</code>, or <code>exhausted</code>.</p>
<p>Error responses use <code>{code, error}</code>. Codes include <code>ORIGIN_DENIED</code>, <code>AUTH_REQUIRED</code>, <code>NO_SESSION</code>, <code>INVALID_INPUT</code>, <code>INVALID_COUNTRY</code>, <code>CAPTCHA_REQUIRED</code>, <code>CAPTCHA_REJECTED</code>, <code>CAPTCHA_UNAVAILABLE</code>, <code>SESSION_ENDED</code>, <code>RECOVERY_UNAVAILABLE</code>, and <code>STORAGE_UNAVAILABLE</code>. Verification failures never silently mint another credential. An invalid saved credential returns 401; the SDK discards it and the visitor retries Search with a new challenge.</p>
<p>The runtime keeps <code>/wisp/?ticket=...</code> and <code>/wisp/<ticket>/</code> WebSocket routes. Only its exact Origin can redeem tickets. Forged/reused/expired tickets, unavailable sessions, ownership conflicts, regional mismatches, and Redis failures deny access. <code>/settings</code> exposes only allowed parent origins and protocol versions. <code>/health</code> and <code>/metrics</code> stay internal at the production ingress.</p>
<p>The old <code>/api/*</code> cookie endpoints are available only for isolated synthetic transport regression tests. They are absent in production and development.</p>
<h2 id="gateway-forwarding-and-destination-policy">Gateway forwarding and destination policy</h2>
<p>The gateway uses Tokio and pinned <code>wisp-mux</code> components for framing and multiplexing. It does not implement a second browser or replay website requests on the server.</p>
<p>For each destination stream, the gateway:</p>
<ol><li>Checks stream type, concurrent-stream admission, and the shared rate bucket.</li><li>Allows only TCP destination ports 80 and 443; UDP is rejected.</li><li>Resolves the requested hostname locally, validates the returned addresses, and rejects blocked destinations.</li><li>Constructs the Shifter username using the immutable authorized session context.</li><li>Authenticates with SOCKS5 to the explicit regional Shifter endpoint.</li><li>Requests the validated destination IP through that proxy.</li><li>Forwards bytes while enforcing quota, idle timeout, expiry, and revocation.</li></ol>
<p>The username format is:</p>
<pre><code>customer-<account>-country-<country>-strict-true-sid-<128-bit-random-hex>-ttl-1800[-pool-shifter]</code></pre>
<p>The final <code>pool-shifter</code> flag restricts upstream assignments to the Shifter pool. The gateway adds it server-side for <code>SHIFTER_POOL_COUNTRIES</code> and omits it for <code>GLOBAL_POOL_COUNTRIES</code>, which use the global pool. Visitors cannot choose or override the pool; it follows from the country.</p>
<p>Account syntax and SOCKS5 credential lengths are validated. Authentication failures do not trigger a direct-connect fallback. Authenticated upstream sockets are not shared between sessions.</p>
<p>The destination policy rejects private/reserved IPv4/IPv6 ranges, loopback, link-local, multicast, mapped/translation ranges, local/internal names, Shifter infrastructure hostnames, and the configured upstream’s resolved addresses. DNS answers containing a blocked address are rejected rather than selecting a different answer. Connecting to the validated IP prevents a second, unchecked destination DNS lookup by the upstream proxy.</p>
<p>The destination’s original TLS hostname remains available to the browser transport. The live Node TLS smoke test separately checked opaque TLS forwarding with hostname/certificate verification enabled; it does not validate production frontend certificates or browser-wide compatibility.</p>
<p>The reserved <code>fixture.test</code>, <code>second.test</code>, and <code>failed-upstream.test</code> mappings exist only in test mode. The synthetic SOCKS server serves fixture responses or a deliberate SOCKS failure without opening destination Internet connections.</p>
<h2 id="haproxy-regions-and-recovery">HAProxy, regions, and recovery</h2>
<p>Local HAProxy runs in TCP mode; the production example terminates HTTPS in HTTP mode. Both use <code>leastconn</code>, backend health checks, Docker DNS resolution, and an eleven-minute connection/tunnel idle timeout. A gateway sends periodic WebSocket pings. Backend enforcement renews the allowance period without disconnecting healthy transports and denies access on traffic exhaustion, Stop, identity loss or invalid authorization.</p>
<p><strong>HAProxy balances connections, not individual requests inside a Wisp connection.</strong> An established WebSocket remains attached to one replica. Independent visitors and later reconnects may select different replicas.</p>
<p>If a replica stops, in-flight requests can fail. SDK 1.0.2 automatically reconnects eligible failed GET document loads within its retry budget; unsafe submissions and exhausted retries require user action. The surviving replica reads shared Redis state and preserves the session’s country, SID, region, and remaining allowance. A crashed replica’s lease may take up to six seconds to expire, in addition to health-check/routing recovery time.</p>
<p>The production design should place a regional HAProxy in front of healthy replicas in that same region. Each replica should use the corresponding explicit Shifter regional hostname. Geographic/latency-aware DNS can select a nearby healthy ingress region; it cannot guarantee the nearest server for every visitor.</p>
<p>The gateway binds sessions to their assigned region and rejects a region mismatch. Cross-region recovery that preserves allowance while creating a fresh residential assignment is not implemented. It must not silently promise IP continuity across regions.</p>
<h2 id="testing-and-validation">Testing and validation</h2>
<h3 id="continuous-integration">Continuous integration</h3>
<p><code>.github/workflows/ci.yml</code> is the merge gate. It runs on the shared self-hosted CI pool (CI-SelfHosted servers; GitHub-hosted runners are never used) as one job, <code>tier1</code>, on one fresh server per run: <code>[self-hosted, ci-8cpu]</code>, or the size a manual run picks. Its step "Checks and tests in parallel" starts every part at the same time through <code>.github/scripts/ci-run.sh</code> (parts runner: <code>.github/scripts/ci-parts.sh</code>). Each part has its own log; the job summary shows the parts table (result and seconds per part), and the log of every failed part is printed in its own group. A part that is killed fails the run as "no result". Each run checks the pull request's own head commit.</p>
<div class="table"><table><tr><th>Part</th><th>Command</th><th>Runs when</th></tr><tr><td><code>policy</code></td><td><code>bash .github/scripts/repo-policy.sh</code></td><td>always</td></tr><tr><td><code>docs</code></td><td><code>python3 scripts/render-readme.py</code>, then <code>git diff --exit-code -- docs/readme.html docs/engineering.html</code>, in its own worktree</td><td>always</td></tr><tr><td><code>actionlint</code></td><td>actionlint 1.7.12 (pinned, checksum-verified download), all workflows</td><td>always</td></tr><tr><td><code>fmt</code></td><td><code>cargo fmt --check</code></td><td>Rust inputs changed</td></tr><tr><td><code>clippy</code></td><td><code>cargo clippy --all-targets --locked --target-dir target/clippy -- -D warnings</code></td><td>Rust inputs changed</td></tr><tr><td><code>rust-test</code></td><td>the Redis image (<code>.github/ciod/prefetch.sh</code>, then <code>docker load</code>), then <code>sh tests/allowance-redis.sh cargo nextest run --locked --profile ci</code> (or <code>full</code>) <code>--target-dir target/test --run-ignored all</code></td><td>Rust inputs changed</td></tr><tr><td><code>npm-ci</code></td><td><code>npm ci --ignore-scripts --no-audit --no-fund --prefer-offline</code></td><td>not documentation only</td></tr><tr><td><code>node-test</code></td><td><code>node --test --test-timeout=90000 tests/*.test.mjs</code> (plus <code>tests/slow/*.test.mjs</code> in full mode), after <code>npm-ci</code></td><td>not documentation only</td></tr><tr><td><code>vendor</code></td><td><code>node scripts/vendor.mjs</code>, after <code>node-test</code> (it rewrites <code>web/runtime/vendor/</code>, which the tests read)</td><td>not documentation only</td></tr><tr><td><code>npm-audit</code></td><td><code>npm audit --audit-level=high --package-lock-only</code></td><td>pull requests that change package files or <code>.github/</code>, and manual runs</td></tr></table></div>
<p>Before the parts, the job checks that <code>rust-toolchain.toml</code> (Rust 1.94.1 with rustfmt and clippy) names the version of the Dockerfile's <code>FROM rust:<version>-</code> build stage, so change both together, and that Node is major 24, as in the Dockerfile's web stage. The CI server image pre-installs the toolchain <code>rust-toolchain.toml</code> names. Rust builds share the pool's sccache; build folders are set with <code>--target-dir</code>, never with a <code>CARGO_*</code> variable, which sccache would put into its cache key.</p>
<p>Triggers and modes:</p>
<ul><li><strong>Pull requests</strong> (opened, synchronized, reopened, marked ready) run in fast mode. A draft gets no <code>tier1</code> result (its job is named <code>tier1 (skipped on drafts)</code>); marking it ready runs <code>tier1</code>.</li><li><strong>Merge queue</strong> groups (<code>merge_group</code>) run in full mode: GitHub tests each pull request together with <code>main</code> before it merges.</li><li><strong>Every 6 hours</strong> (<code>0 */6 * * *</code>) a scheduled run tests <code>main</code> in full mode. When it is not green, the <code>report</code> job opens one issue, "Scheduled full CI run failed on main" (<code>.github/scripts/report-red.sh</code>; it adds nothing when that issue is already open). No other event runs <code>report</code>.</li><li><strong>Manual runs</strong> (<code>workflow_dispatch</code>) run in full mode, with the server size (<code>ci-8cpu</code>, <code>ci-16cpu</code> or <code>ci-32cpu</code>) as input.</li><li>Pushes to <code>main</code> run nothing: <code>main</code> only ever receives what the merge queue tested.</li></ul>
<p>A pull request runs only the parts its changed files need (<code>.github/scripts/changed-scope.sh</code>). One that changes only documentation (<code>docs/</code>, <code>.cursor/</code>, README.md, ENGINEERING.md, AGENTS.md, LICENSE and the documentation scripts) still reports <code>tier1</code>; it runs <code>policy</code>, <code>docs</code> and <code>actionlint</code>, and the summary names the rule that skipped each other part. The Node parts run for any other change, the Rust parts when <code>src/</code>, <code>.cargo/</code>, the Cargo files, the toolchain file or the Dockerfile change, and everything runs when <code>.github/</code> changes. The merge queue, the 6-hour run and manual runs run every part except as noted for <code>npm-audit</code>.</p>
<p><code>npm-audit</code> runs on pull requests that change <code>package.json</code>, <code>package-lock.json</code> or <code>.github/</code>, and on manual runs. The merge queue and the 6-hour run leave it out, so a newly published advisory never stops merges; <code>maestro-nightly</code> audits the dependencies every night instead.</p>
<p>Slow tests and the 90-second rule: pull requests leave out Rust tests whose name starts with <code>slow_</code> and JavaScript test files under <code>tests/slow/</code>; the merge queue, the 6-hour run and manual runs include them. In every mode a test that runs 90 seconds fails the run (nextest <code>slow-timeout</code> in <code>.config/nextest.toml</code>, <code>node --test-timeout</code> and the "CI time" step, <code>.github/scripts/ci-timing.py</code>), and a test over 60 seconds gets a warning to mark it slow. Tests are never retried (<code>retries = 0</code>); a flaky test gets its own issue. The "CI time" step adds the test counts and the slowest tests to the job summary, and the run keeps them with the JUnit files as the <code>ci-timing</code> artifact for 90 days.</p>
<p>Merging: pull requests merge only through the merge queue, as squash commits. The required checks are <code>tier1</code>, <code>maestro-policy</code>, <code>maestro/review</code> and <code>maestro/qa</code>, enforced for administrators too.</p>
<p>Server size and times: <code>ci-8cpu</code> is the smallest size whose measured runs all keep the parts step under 90 seconds and the job under 4 minutes. Measured with a warm cache (three manual runs per size, November 2026): on <code>ci-8cpu</code> the parts step takes 22–24 seconds and the job 33–34 seconds, with sccache at 100% and about $0.002 per run; <code>ci-16cpu</code> takes 13–15 seconds and <code>ci-32cpu</code> 9–11 seconds for the parts step, at 1.3 to 2.4 times the cost. Getting a fresh server takes about 90–115 seconds before the job starts, whatever the size; the slowest test takes under 3 seconds.</p>
<p>CI runs the 11 real-Redis store tests (<code>src/store_tests.rs</code>) in <code>rust-test</code>: <code>tests/allowance-redis.sh</code> starts Redis on the server's rootless Docker with <code>--pull never</code>, <code>--network none</code>, a Unix socket in a temporary folder and no published port, runs nextest with <code>REDIS_TEST_URL</code> set and removes the container afterwards, also when tests fail. The image is the <code>redis:7.4-alpine</code> index digest pinned in <code>tests/redis-image.sh</code>, and a test run never reaches a registry for it. The nightly CI server image build runs the repository's hook <code>.github/ciod/prefetch.sh</code>, which downloads the linux/amd64 image through the registry API (Docker Hub, or <code>public.ecr.aws</code> with the same digests when Docker Hub fails; one attempt each), checks every part against its sha256 digest and stores it as an OCI archive in the runner's cache. <code>rust-test</code> runs the same hook first: on a prefetched image it does nothing, and on a server image built without the hook it downloads the image once, with a warning. It then loads the archive with <code>docker load</code>. The job summary says <code>redis image: prefetched</code> or <code>redis image: installed now</code>. A failed download, a digest mismatch, a failed load or a Redis that does not start fails the part; nothing is retried. Mark a Rust test <code>#[ignore]</code> only when it needs <code>REDIS_TEST_URL</code>; CI runs every ignored test (<code>--run-ignored all</code>), so <code>#[ignore]</code> never hides a test from CI.</p>
<p>CI does not run the Docker stack tests (<code>integration.mjs</code>, <code>limits.mjs</code>, <code>load.mjs</code>, <code>failover.mjs</code>, <code>configuration.mjs</code> and the other stack scripts), the browser tests, the credential scan or the live smoke tests: they need a running stack with Chrome, or private credentials. Run them locally as described below. The CI server refuses pull requests from forks or from anyone but the repository owner, before any step runs.</p>
<p>Local equivalents of the CI commands:</p>
<pre><code>bash .github/scripts/repo-policy.sh
python3 scripts/render-readme.py && git diff --exit-code -- docs/readme.html docs/engineering.html
cargo fmt --check
cargo clippy --all-targets --locked -- -D warnings
sh tests/allowance-redis.sh cargo nextest run --locked --profile ci --run-ignored all # Linux Docker
npm ci --ignore-scripts
node --test tests/*.test.mjs
node scripts/vendor.mjs
npm audit --audit-level=high --package-lock-only</code></pre>
<h3 id="install-host-side-test-dependencies">Install host-side test dependencies</h3>
<pre><code>npm ci --ignore-scripts
sh scripts/stack.sh mock</code></pre>
<p>The Docker asset build runs the vendoring script itself. To populate local generated assets for inspection:</p>
<pre><code>npm run vendor</code></pre>
<p>Generated vendor files are ignored by Git and excluded from the input Docker build context.</p>
<h3 id="mock-and-code-checks">Mock and code checks</h3>
<pre><code>node tests/integration.mjs
node tests/limits.mjs
node tests/load.mjs
node tests/failover.mjs
node --test tests/redis-network-policy.test.mjs
node tests/configuration.mjs
sh tests/allowance-redis.sh # Linux Docker: private Redis Unix socket; no published ports
cargo test --locked
cargo clippy --all-targets -- -D warnings
npm audit --audit-level=high</code></pre>
<div class="table"><table><tr><th>Check</th><th>Coverage</th></tr><tr><td><code>integration.mjs</code></td><td>Country validation, Origin checks, forged/reused tickets, per-visitor country/SID isolation, one active connection, malformed Wisp, forbidden destinations/ports/UDP, reconnect context, country revocation, synthetic form response, shared byte cap, expiry, Stop/resume allowance</td></tr><tr><td><code>limits.mjs</code></td><td>Ticket TTL/expiry, outstanding ticket revocation, unauthorized country changes, paced connection bursts, queued navigation, configured concurrency cap, bounded overflow, idle retirement and CONNECT flood protection</td></tr><tr><td><code>load.mjs</code></td><td>10, 100, 500, and 1,000 simultaneous mock sessions; two 128 KiB requests per session; errors, latency percentiles, throughput, replica counts, sampled Docker CPU/memory</td></tr><tr><td><code>failover.mjs</code></td><td>Different countries on one replica, replica failure/recovery with preserved SID/allowance, Redis outage and fail-closed behavior</td></tr><tr><td><code>configuration.mjs</code></td><td>Production-mode rejection, read-only gateway secret mounts, loopback-only application listeners, private/internal Redis with no published ports, isolated bridge modes, restricted network membership, working application access, and blocked HAProxy access</td></tr><tr><td><code>redis-network-policy.test.mjs</code></td><td>Rejection of published Redis ports, host networking, non-internal networks, missing isolated bridge modes, public addresses, and unauthorized network members</td></tr><tr><td>Rust unit tests</td><td>Destination policy, configuration and public assets; <code>allowance-redis.sh</code> adds real-Redis renewal, concurrent charging, migration, lease/revision/Stop and 64-bit quota checks</td></tr></table></div>
<p>Load tests refuse live mode. Failover tests deliberately stop/start gateways and pause/unpause Redis. Use a dedicated local test stack and do not run those tests while using the browsing UI. The scripts restore stopped services in cleanup paths, but inspect the stack after any externally interrupted test.</p>
<p>Results are written under ignored <code>artifacts/</code>. They can contain local resource information, synthetic identifiers, or live IP addresses; do not commit the raw results.</p>
<p>To change mock load stages:</p>
<pre><code>LOAD_STAGES=10,100,500,1000 node tests/load.mjs</code></pre>
<p>These are short synthetic bursts, not production capacity measurements or a sustained saturation benchmark. Docker CPU samples may include setup or idle periods, and browser JavaScript execution is outside this server load test.</p>
<h3 id="reset-synthetic-browsing-state">Reset synthetic browsing state</h3>
<pre><code>node tests/reset-mock.mjs</code></pre>
<p>This command refuses live mode and clears only this project’s mock Redis DB 1. It allows a new fixture run after the daily window ends. Normal user flows do not reset the daily allowance.</p>
<h3 id="explicit-live-smoke-checks">Explicit live smoke checks</h3>
<p>After configuring a real private credential file:</p>
<pre><code>sh scripts/stack.sh live
node tests/live-smoke.mjs
node tests/live-tls.mjs</code></pre>
<p>Set <code>IP_CHECK_HOST</code> to your real IP-information endpoint before running live smoke checks (and <code>IP_CHECK_PATH</code> if its path differs from <code>/json</code>). The country test checks US and DE using that endpoint, with three samples per country including a Wisp reconnect. The TLS test fetches <code>https://example.com</code> through the Wisp/Shifter path with Node certificate verification enabled.</p>
<p>The scripts inspect existing usage in the dedicated live-development Redis DB 0 and reserve bounded per-visitor quotas within a cumulative <strong>20 MiB</strong> test ceiling. Run them serially on a dedicated stack. This is test accounting, not a global production billing limiter; unrelated concurrent visitors or simultaneous test runners are outside that cumulative-test calculation. Do not clear DB 0 to bypass the smoke-test cap.</p>
<p>Live smoke scripts print observed IP addresses and store them in ignored artifacts. No upstream credentials should appear in their output.</p>
<h3 id="credential-scan">Credential scan</h3>
<pre><code># Requires .env to reference the real private credential file and Python 3.11+.
python3 tests/secret-scan.py</code></pre>
<p>The local scan compares exact upstream account/password and configured local CAPTCHA secret values against source, generated assets, test evidence, application/HAProxy logs, image metadata/history, and the final image filesystem without printing those values. It supplements review; it does not prove the absence of every possible secret or replace a staged-file audit.</p>
<h3 id="shared-sdk-validation">Shared SDK validation</h3>
<pre><code>npm test
cargo test
node scripts/build-sdk.mjs
docker build -t shifter-web:sdk-check .
node tests/run-sdk-tests.mjs --browser --downloads --mobile-compat --legacy-api-compat # Requires local Chrome; omit browser flags for backend checks only
HTTPS_TEST_IMAGE=shifter-web:sdk-check HTTPS_TEST_REDIS_TLS=1 node tests/https-deployment.mjs</code></pre>
<p>The SDK runner creates a separate disposable local project on loopback ports 8180/8181, runs the origin/CAPTCHA/identity/cache checks, transport/quota integration tests, stream admission checks, and Redis network-policy checks, then removes that project. It never rewrites the developer's active configuration. <code>--browser</code> additionally exercises a forced Chrome renderer crash and cross-origin navigation recovery before cleanup. <code>--keep</code> retains it for further browser checks and writes its cleanup context under ignored <code>artifacts/</code>.</p>
<p><code>--downloads</code> also runs real Chrome download checks against harmless local fixtures. An unsandboxed localhost control must save every format before the proxy checks run; the harness allows downloads and records Chrome download events. Checks cover attachments, mislabelled executable/archive URLs, extensionless files, redirects, POST, Range, download attributes, Blob/data URLs, nested srcdoc frames, direct proxied URLs and removal of both iframe sandbox attributes. After starting a retained local stack (<code>node tests/run-sdk-tests.mjs --keep</code>), run <code>node tests/browser-recovery.mjs --same-site-downloads</code> to require direct top-level navigation with an active worker and transport, including unsupported media MIME denials. Both browser modes test JavaScript Fetch/XHR loaders and attachment rejection. Bodies are inert text regardless of their filename or MIME label, not real executable or media samples. Download files, fixture servers and the disposable browser profile are removed after the run. Results are recorded under ignored <code>artifacts/</code>. Run <code>node tests/browser-download-notice.mjs</code> to preview the actual 403 notice and its CSP on localhost without a proxy account or Docker stack. It saves desktop/mobile screenshots under <code>artifacts/</code> and checks narrow layouts, script blocking, and the absence of external requests.</p>
<p>Recovery tests cover hidden error documents, worker capability validation, website HTTP-error passthrough, POST safety, stale frames, retry limits, Stop/destruction cancellation, gateway-attested SID rotation, concurrent recovery, and unchanged quota/expiry. The TLS fixture also exercises recovery under the production Redis ACL and verifies that storage loss denies recovery. Unit runtime tests use a simulated DOM/service-worker bridge. <code>node tests/browser-recovery.mjs</code> also runs local system Chrome in a disposable profile against the stack kept by <code>tests/run-sdk-tests.mjs --keep</code>: it crashes the cross-site runtime renderer with CDP, verifies automatic recovery and persisted state, and exercises the worker error document under a cross-origin embed. It never opens the person’s normal browser profile.</p>
<p>Local live US investigation used an explicitly approved 10 GB cumulative ceiling on a separate, quota-bounded stack; this does not change the default 20 MiB smoke-test ceiling. Epoxy response-body failures and a Chrome compositor crash were reproduced separately. Production gateway throttling was not established as their cause. Live acceptance results are recorded below; synthetic tests alone do not establish destination compatibility.</p>
<p>Live acceptance on 2026-10-08 used system Chrome 155 in disposable visible profiles, a cross-origin local embed, the final runtime assets, and real US residential forwarding. No browser feature flags or injected test workaround were used for acceptance.</p>
<div class="table"><table><tr><th>Scenario</th><th>Observed result</th></tr><tr><td>Example → Wikipedia → BBC.com → YouTube</td><td>Full YouTube interface; the entire 19-second video played; stable through the 90-second observation</td></tr><tr><td>BBC.co.uk → wait 75 seconds → YouTube</td><td>BBC images rendered; full YouTube interface and the entire 19-second video played</td></tr><tr><td>YouTube as the first page</td><td>Full interface and buffered video; a normal Play click produced 11 seconds of playback</td></tr><tr><td>Return to Example.com</td><td>Loaded after each run; repeated sequence, delayed sequence and fresh-page checks retained country/expiry and the declining byte allowance</td></tr></table></div>
<p>There were no renderer crashes in these acceptance runs and no failed YouTube application scripts in the successful playback runs. Some media fetches returned errors while YouTube successfully used another media response. Separate exits returned a Google traffic challenge or YouTube sign-in prompt; those are recorded as destination restrictions, not playback passes. The changes were validated locally; deployment and acceptance on the actual embedding websites remain separate steps.</p>
<p>Unit tests cover modal cancellation, concurrent clicks, errors, bearer requests, stale messages, storage denial, cleanup, and Google challenge focus handling. The TLS fixture seeds a verified synthetic identity directly in its private Redis for WSS testing; it does not claim to solve a real CAPTCHA or expose a production bypass. Real credentials are never used by synthetic tests. Local browser checks exercised the branded and minimal UIs on different local site origins, mobile and desktop browsing, cancellation, and Stop/cleanup. Google’s live widget script was unavailable in that browser session; synthetic verification was used for successful session flows. Real Google challenge acceptance on the two production websites remains outstanding.</p>
<h3 id="validation-status-at-the-initial-implementation">Validation status at the initial implementation</h3>
<ul><li>Twenty-two backend integration, limit, and outage checks passed.</li><li>The 1,000-session mock stage completed 2,000 requests with zero errors and distributed connections across both gateways.</li><li>Live US/DE targeting and sampled sticky behavior passed through the BLR endpoint.</li><li>Node TLS forwarding with certificate verification passed.</li><li>Total recorded live validation traffic was 19,314 destination bytes, including an initial incomplete test-client attempt.</li><li>The destination-policy Rust unit test, Clippy, dependency audit, configuration checks, and exact-value credential scan passed at the time of validation.</li><li>Embedded fixture HTML/JavaScript, a synthetic login, and subsequent link navigation were observed in the browser.</li><li>Final browser idle stability, Back/Forward, full redirect/cookie separation, reconnect cookie retention, and Stop/country/expiry cleanup remain pending.</li></ul>
<p>Raw local reports, live IPs, and workstation-specific evidence are intentionally not published in this repository. Rerun the included checks to produce evidence for your checkout and device. The summary above is a historical observation, not a promise that future dependency or environment changes will pass.</p>
<h3 id="remaining-manual-browser-acceptance">Remaining manual browser acceptance</h3>
<ol><li>Start the mock stack, open the trusted page, and browse <code>http://fixture.test/</code>.</li><li>Confirm the JavaScript country message and submit the synthetic username form.</li><li>Follow Next page, then use Back and Forward; confirm navigation and login state.</li><li>Leave the page idle for at least one minute, then navigate again and check connection stability.</li><li>Verify Redirect, visit Second origin, and confirm that the first origin’s login does not appear there.</li><li>Return to the first origin and confirm its login remains; use Reload and check reconnect cookie retention.</li><li>Change country and confirm a fresh website context with the original allowance retained.</li><li>Stop and resume; confirm website state is cleared while time and byte usage remain shared.</li><li>Expire a session while traffic is active; verify both backend closure and the VPN informational state.</li><li>Interrupt/reload the tab and check next-load cleanup. Perform any additional public-site browser tests under a separately bounded live test budget.</li></ol>
<h2 id="operations-and-troubleshooting">Operations and troubleshooting</h2>
<h3 id="inspect-the-local-services">Inspect the local services</h3>
<pre><code>docker compose ps
docker compose logs --tail 50 api gateway-a gateway-b haproxy
curl -fsS http://localhost:8080/health
curl -fsS http://localhost:8081/health</code></pre>
<p>The runtime listener balances independent HTTP connections as well as WebSocket connections, so repeated health requests can report different replicas. Metrics contain counters, not destination URLs or visitor labels. There is intentionally no request URL/access logging; adding it could expose tickets and browsing history.</p>
<h3 id="common-issues">Common issues</h3>
<div class="table"><table><tr><th>Symptom</th><th>Checks and action</th></tr><tr><td>Compose cannot find local configuration</td><td>Run <code>sh scripts/configure.sh</code>. It copies examples only when local files are missing.</td></tr><tr><td>Live secret cannot be read</td><td>Check the ignored <code>.env</code> path, TOML schema, and UID 10001 mount readability. Do not put the secret in an image or template.</td></tr><tr><td>API denies a request Origin</td><td>Use <code>http://localhost:8080</code> for controls and preserve the configured runtime origin. Update both settings together if changing origins.</td></tr><tr><td>A second tab cannot connect</td><td>One connection is allowed per visitor. Stop the other session or use explicit reconnect and wait for lease cleanup.</td></tr><tr><td>Reconnect is briefly denied after a crash</td><td>Allow the six-second lease and HAProxy health checks to converge. The same country/SID/allowance should be retained.</td></tr><tr><td>Daily allowance ended</td><td>This is expected after expiry/exhaustion. Reconnect and country changes must not reset it. Reset DB 1 only for synthetic tests.</td></tr><tr><td>Fixture hostname fails in live mode</td><td>Fixture mappings are test-only. Start the mock profile for fixture checks.</td></tr><tr><td>Redis is paused after interrupted outage testing</td><td>Inspect the project, then use <code>docker compose unpause redis</code> and <code>docker compose start gateway-a gateway-b</code>.</td></tr><tr><td>Runtime cannot initialize</td><td>Check service worker, SharedWorker, WebAssembly support, exact origins, and generated asset availability.</td></tr><tr><td>Browser loses the connection or a website fails</td><td>This remains an open compatibility/regression area. Use explicit Reload; do not automatically replay form submissions.</td></tr><tr><td>Template changes do not appear locally</td><td>Bootstrap preserves local copies. Compare the example with the ignored local file and apply the change yourself.</td></tr></table></div>
<p>Redis uses an AOF-backed named volume and a no-eviction memory policy in the example. <code>stack.sh down</code> preserves that volume. Deleting it removes the local allowance ledger. Redis high availability, backup/restore policy, crash-durability guarantees, and production observability remain deployment work.</p>
<p>Production uses server-verified CAPTCHA and anonymous bearer identity. Clearing first-party storage can still reset anonymous identity; verified per-person identity and stronger abuse controls are separate work.</p>
<h2 id="project-layout">Project layout</h2>
<pre><code>.github/
workflows/ci.yml Merge-gate CI: one self-hosted server per run, parts in parallel
scripts/ CI change scope, repository policy, parts runner, test timing, red-run issue, pinned tools
src/
main.rs Role selection, routing, headers, process setup
config.rs Validated settings and server-side credential loading
api.rs Session API, trusted controls, health and metrics
store.rs Redis session state, tickets, leases and quotas
relay.rs Wisp adapter, admission, destination policy and SOCKS5
web/
control/ Shifter UI adapter and minimal integration example
sdk/ SDK source, stable loader, immutable release assets
runtime/ Scramjet setup, service worker and state cleanup
scripts/
configure.sh Create missing ignored local files from examples
stack.sh Local mock/live/down lifecycle
vendor.mjs Copy locked browser dependencies into generated assets
render-readme.py Produce self-contained HTML documentation
examples/
shifter-credentials.example.toml Placeholder live-account schema
deploy/
haproxy.example.cfg Portable TCP balancer template
tests/
fixtures/ Synthetic SOCKS5/destination and mock credential example
client.mjs Wisp protocol client and TLS tunnel helper
integration.mjs Core authorization, isolation, quota and expiry checks
limits.mjs Ticket and stream-admission checks
load.mjs Increasing-concurrency mock benchmark
failover.mjs Gateway/Redis outage checks
configuration.mjs Deployment-default assertions
live-smoke.mjs Bounded real country/sticky-IP checks
live-tls.mjs Bounded real TLS-forwarding check
reset-mock.mjs Reset only isolated mock data
secret-scan.py Local exact-value credential scan
compose.example.yaml Portable development stack example
compose.test.example.yaml Synthetic test-profile example
Dockerfile Multi-stage source/asset build, non-root runtime
Cargo.toml / Cargo.lock Rust dependencies
package.json / package-lock.json Browser and test dependencies
README.md Product overview and quickstart
ENGINEERING.md Detailed engineering documentation
docs/readme.html Offline README
docs/engineering.html Offline engineering guide</code></pre>
<p>Local-only configuration, artifacts, generated dependencies, and pre-existing research documents are omitted from this layout because they are not repository deliverables.</p>
<h2 id="dependencies-and-reproducibility">Dependencies and reproducibility</h2>
<div class="table"><table><tr><th>Component</th><th>Pin</th></tr><tr><td>Scramjet</td><td>Release tarball <code>1.1.0</code>, locked in <code>package-lock.json</code></td></tr><tr><td>bare-mux</td><td><code>2.1.9</code></td></tr><tr><td>libcurl transport</td><td><code>1.5.2</code> (bundled libcurl.js <code>0.7.4</code>)</td></tr><tr><td>Public Suffix List</td><td><code>2026-10-07_07-28-19_UTC</code>, commit <code>3929462652695bad04f0a27afb600974014a3c8b</code></td></tr><tr><td>wisp-mux</td><td>epoxy-tls Git revision <code>0c11678d72a636c3a4bc723db87e03e7b888eaf9</code></td></tr><tr><td>Node build stage</td><td>Node 24 image pinned by digest</td></tr><tr><td>Rust build stage</td><td>Rust 1.94.1 image pinned by digest</td></tr><tr><td>HAProxy / Redis / fixture Python</td><td>Example image tags plus immutable digests</td></tr></table></div>
<p>Use <code>npm ci --ignore-scripts</code> and Cargo’s <code>--locked</code> mode. The Docker build follows those locked inputs and generates browser assets during the build. Container digests and lockfiles improve repeatability; this is not a claim of fully bit-for-bit reproducible images, since operating-system packages are installed during the build.</p>
<p>Review transport/runtime compatibility when updating dependencies. Their contracts, browser isolation behavior, service-worker caching, and Wisp protocol behavior matter as much as version numbers. The pinned versions and known browser gaps need review before any public rollout.</p>
<p>Refresh <code>web/runtime/public-suffix-list.dat</code> from the <a href="https://publicsuffix.org/list/public_suffix_list.dat">official Public Suffix List</a> when maintaining runtime dependencies. Keep its version, commit, and MPL-2.0 license header, update the pin above, and run the runtime-data and browser tests. This snapshot uses the pinned Scramjet v1 IndexedDB schema; review that contract when upgrading the engine.</p>
<p>Regenerate the standalone README after editing Markdown:</p>
<pre><code>python3 scripts/render-readme.py</code></pre>
<p>The HTML file embeds its styling, has no CDN dependencies, and opens without a development server.</p>
<p>This repository does not grant a blanket license for third-party components. Upstream licenses, including AGPL terms for relevant proxy components, remain applicable. Review their obligations before distributing or hosting a modified public service.</p>
<h2 id="production-work-still-required">Production work still required</h2>
<ul><li>Complete the pending browser acceptance procedure and investigate any remaining idle/navigation failures.</li><li>Validate representative public websites and supported authentication flows; define a compatibility policy.</li><li>Provision the real v2 CAPTCHA key/secret and register both domains; verify real Google challenge acceptance on both websites. Consider stronger identity and abuse monitoring if anonymous quotas are insufficient.</li><li>Add broader service-level admission controls, operational resource limits, and sustained load testing.</li><li>Review engine isolation and escape risks for the requested original-site integration: controls at the configured Shifter origin and the runtime at <code>proxy.example.net</code> are distinct origins; evaluate the actual deployment domains.</li><li>Provision real-domain certificates and renewal on the deployment host using the HTTPS/WSS example below; complete browser acceptance against those domains.</li><li>Verify the existing production Redis deployment against the rules above: private endpoints, named ACL users, and network access restrictions; verify TLS certificates and hostnames for client and replication connections. Client ACL/TLS support is implemented and covered by local integration checks.</li><li>Establish regional Redis/state availability and an explicit cross-region recovery protocol that preserves allowance but rotates residential assignment.</li><li>Deploy regional HAProxy/gateway groups and geographic/latency-aware DNS, with health-aware routing.</li><li>Add production telemetry that protects browsing privacy and excludes credentials, connection tickets, and destination URLs.</li><li>Configure a real VPN download destination and final expired-session UX.</li></ul>
<p>The HTTPS implementation and tests run locally. Production DNS changes, certificate issuance, and execution on a deployment host are separate operational steps; no remote deployment was performed.</p>
<p>References: <a href="https://github.com/MercuryWorkshop/scramjet">Scramjet</a>, <a href="https://github.com/MercuryWorkshop/libcurl-transport">libcurl transport</a>, <a href="https://github.com/MercuryWorkshop/epoxy-tls">Wisp gateway library</a>, <a href="https://www.haproxy.com/documentation/haproxy-configuration-tutorials/protocol-support/tcp/">HAProxy TCP configuration</a>, <a href="https://shifter.io/docs/products/residential-proxies/gateway-and-auth/">Shifter gateway and regions</a>, <a href="https://shifter.io/docs/products/residential-proxies/geo-targeting/">Shifter geo-targeting</a>, and <a href="https://shifter.io/docs/products/residential-proxies/sessions/">Shifter sessions</a>.</p>
<h2 id="license">License</h2>
<p>The project code is available under the <a href="../LICENSE">MIT License</a>. Third-party dependencies, fonts, and assets retain their own licenses.</p></main></body></html>