Document and address 32MB request body size limit

[jodeleeuw]

Problem

Firebase Cloud Functions v2 (Cloud Run) enforces a hard 32MB limit on HTTP request body size (HTTP/1.x). This limit cannot be increased — it's a platform constraint, not a configuration option.

Most jsPsych experiment data is well under this limit (typical datasets are 50KB–5MB), but edge cases exist:

• Base64 endpoint: webcam recordings, screen captures, or high-resolution images can easily exceed 32MB
• Data endpoint: experiments with very high-frequency sampling (e.g., mouse tracking at 60Hz, eye tracking) over long sessions could approach the limit
• No user-facing error: requests exceeding 32MB are rejected at the infrastructure layer before any function code runs, so no DataPipe error message or logging occurs — the researcher just sees a network error

Current behavior

When a payload exceeds 32MB:

1. Cloud Run gateway rejects the request
2. No function code executes (no logging, no persist, no queue)
3. The client receives a connection error with no explanation
4. Data is silently lost

Potential solutions (roughly ordered by effort)

1. Document the limit

Add a note to the FAQ and getting-started pages about the 32MB limit. Low effort but poor UX.

2. Client-side size check in jsPsych plugin

Add a pre-flight size check in @jspsych-contrib/plugin-pipe that warns or errors with a clear message if the payload exceeds ~30MB. Doesn't solve the problem but prevents silent data loss.

3. Client-side chunking

Split large payloads into <32MB chunks in the plugin, reassemble server-side. Moderate effort, backward-compatible for existing experiments.

4. Signed URL upload

Client requests a signed URL from a lightweight Cloud Function, uploads directly to Cloud Storage (no size limit), then a storage trigger or scheduled function processes it. Most robust solution but largest architectural change.

References

• Firebase Cloud Functions Quotas — "Max uncompressed HTTP request size: 32MB (cannot be increased)"
• Cloud Run Quotas
• Discovered during testing of PR Merge test into main #147

[jodeleeuw]

Compression as a workaround

After investigation, gzip compression can effectively bypass the 32MB limit:

• The 32MB limit applies to the on-the-wire (compressed) size, not the decompressed size
• Cloud Run does not decompress request bodies — it just measures the bytes it receives
• JSON/CSV data compresses very well (typically 10-20x), so a 100MB dataset would compress to ~5-15MB, well under the limit
• This would effectively raise the practical limit to ~300-600MB for typical experiment data

Implementation would require changes in two places:

Client side (in @jspsych-contrib/plugin-pipe or the calling code):

• Gzip the data string before sending
• Set Content-Encoding: gzip header on the request

Server side (api-data.ts / api-base64.ts):

• Detect Content-Encoding: gzip on incoming requests
• Decompress the raw body before processing
• Note: Firebase Functions' built-in Express body parser may not handle gzipped bodies automatically — we'd likely need to intercept the raw request stream

Caveats

• The official Google Cloud docs don't explicitly state whether the 32MB limit measures compressed or uncompressed bytes. The conclusion that it measures wire size is inferred from the fact that Cloud Run doesn't decompress requests (so it can only measure what it receives). Should be verified with a test before committing to this approach.
• HTTP/2 connections bypass the 32MB limit entirely, but Express (used by Firebase Functions) doesn't natively support HTTP/2 end-to-end.

[nikbpetrov]

I like chunking. It's so much more natural and robust. I've always been irked by setups that only allow saving at the end: a failure mid-experiment loses all the data (and there is no way to confirm with paid participants that they actually reached 50% or w/e in the experiment to pay them proportionally). For experiments I create, solutions that only solve data at the end are unusable. (not to mention the inability ot implement resumable sessions)

I am solving this with DataPipe by still saving chunked data but then recombining it at the end of the data collection.