Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-inbox-upload-media.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Drop audio and video onto the Inbox (mp3, m4a, aac, wav, ogg, opus, webm, mp4, mov, checked by container signature): the file streams straight to the hive's disk under the recording size cap (`inbox.recording.max_bytes`, 200 MiB), plays back from disk through the same-origin stream, and is transcribed as soon as it is added when transcription is enabled (otherwise 🎧 Transcribe shows why it is unavailable); the machine transcript lands in the item's Knowledge fact. Originals are deleted after the recordings retention (5 days, `inbox.uploads.media_retention_days` overrides) while the transcript and item stay; video keeps only its audio track. Uploads now get a 20-minute read window instead of the dashboard's 30-second read timeout.
34 changes: 25 additions & 9 deletions dashboard/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -40173,7 +40173,7 @@
"Knowledge"
],
"summary": "Add dropped files to the Inbox",
"description": "Owner or read-write. Each multipart `file` field is typed by its content (magic bytes, UTF-8 validation and a parse check), never by its name or client MIME type alone. Markdown, text, JSON, JSON Lines, YAML, CSV, TSV, HTML, Word (.docx, read with zip-bomb caps and no DTDs or external entities) and PDF (parsed out of process in the hive-pdftext sandbox; a PDF without a text layer is refused) become `upload` Inbox items with extracted markdown, stored under `connector-state/uploads/<id>/` (0700 directory, 0600 files) and written as a Knowledge fact with the text inside an `<untrusted-inbox-data>` envelope. Limits come from `inbox.uploads` (`max_file_bytes`, default 25 MiB; `max_files_per_drop`, default 20; `quota_bytes`, default 2 GiB). Requests marked `Sec-Fetch-Site: cross-site` are refused. Every file is audited as `inbox_upload` with its outcome.",
"description": "Owner or read-write. Each multipart `file` field is typed by its content (magic bytes, UTF-8 validation and a parse check), never by its name or client MIME type alone. Markdown, text, JSON, JSON Lines, YAML, CSV, TSV, HTML, Word (.docx, read with zip-bomb caps and no DTDs or external entities) and PDF (parsed out of process in the hive-pdftext sandbox; a PDF without a text layer is refused) become; audio and video (mp3, m4a, aac, wav, ogg, opus, webm, mp4, mov, checked by container signature) are streamed to disk under `inbox.recording.max_bytes`, played back from disk through the same-origin stream and transcribed when transcription is on, and their originals are deleted after the recordings retention. Upload items otherwise become `upload` Inbox items with extracted markdown, stored under `connector-state/uploads/<id>/` (0700 directory, 0600 files) and written as a Knowledge fact with the text inside an `<untrusted-inbox-data>` envelope. Limits come from `inbox.uploads` (`max_file_bytes`, default 25 MiB; `max_files_per_drop`, default 20; `quota_bytes`, default 2 GiB). Requests marked `Sec-Fetch-Site: cross-site` are refused. Every file is audited as `inbox_upload` with its outcome.",
"requestBody": {
"required": true,
"content": {
Expand Down Expand Up @@ -40236,7 +40236,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40304,7 +40306,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40372,7 +40376,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40446,7 +40452,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40514,7 +40522,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40582,7 +40592,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -40650,7 +40662,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down Expand Up @@ -41421,7 +41435,9 @@
"tsv",
"html",
"docx",
"pdf"
"pdf",
"audio",
"video"
]
},
"code": {
Expand Down
10 changes: 10 additions & 0 deletions dashboard/test/inbox-item.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -184,3 +184,13 @@ test('share history pills sit with the other pills and read users get no link',
assert.match(html, /<span class="kc-pill inbox-share-pill"[^>]*>→ RFC #18412/);
assert.doesNotMatch(html, /href="https:\/\/github/);
});

test('an uploaded audio file is pending with its own note and keeps the player', () => {
const ctx = makeContext('read-write');
const audio = { source: 'upload', id: '20261116t093000-0a1b2c3e', title: 'standup', origin: 'upload · standup.m4a', added_by: 'ana', upload_kind: 'audio', pending: true, transcribable: true, fact_slug: 'upload-x', fact_state: 'pending' };
const html = vm.runInContext('inboxItemHtml(' + JSON.stringify(audio) + ')', ctx);
assert.match(html, /this audio was added to the Inbox without text/);
assert.doesNotMatch(html, /Plaud/);
assert.match(html, /data-action="inboxPlay"/);
assert.match(html, /▸ Transcript<\/button>/);
});
4 changes: 2 additions & 2 deletions src/docs/api-reference/knowledge.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,8 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f
| `GET` | `/api/inbox/transcriptions` | Dashboard auth | Live progress of queued and running transcriptions in queue order (`{active: [{source, id, state}]}` with `phase`, `progress`, `estimated`, `started_at`, `audio_seconds`, `processed_seconds`, `eta_seconds`, `queue_position`); reads no source, so the Inbox polls it every 3 s while a job runs; `Cache-Control: no-store`; see [inbox-transcription.md](../inbox-transcription.md) | `pkg/dashboard/api_inbox.go:34` |
| `GET` | `/api/inbox/{source}/{id}/audio` | Owner or read-write | Mint a fresh short-lived audio/video link for one item (`{url, expires_at}`; Plaud: presigned, 24h; Zoom: audio-only or video `download_url` with a one-hour download token; Webex: the recording's temporary direct audio (else MP4) download link, valid 3 hours; Teams and Google Meet: 404, no browser-playable link; Slack huddles: 404, Slack keeps no huddle audio; Otter: the conversation's MP3 download link (`GET /conversations/{id}/audio`), minted per click; Granola: 404, Granola keeps no meeting audio; Limitless: 404, Pendant audio is served only behind the API key; Fathom: the signed url of a recording download (`POST /recordings/{id}/download`, audio-only recordings complete at once, video is polled), valid about 24 hours; Fireflies: the transcript's hashed `audio_url` (else `video_url`), valid 24 hours, Pro plan or higher); never stored by the hive, replied with `Cache-Control: no-store`, audited as `inbox_audio` (403 role, 404 unknown item or no audio, 502 source unreachable) | `pkg/dashboard/api_inbox.go:30` |
| `GET` | `/api/inbox/{source}/{id}/audio/stream` | Owner or read-write | Stream one item's audio from the dashboard origin, for the Inbox player (the CSP refuses cross-origin media). The hive mints the link as above, fetches it server-side and copies the bytes with the media `Content-Type` (from storage, else the link's extension); a single `Range: bytes=…` is passed through (206 + `Content-Range`; 200 whole file when storage ignores it; 416 passed through). The link stays in memory for one playback's Range requests (10 min, never past expiry), is never logged, returned or stored, and is re-minted once when storage refuses it; nothing is written to disk. `Cache-Control: no-store`; audited as `inbox_audio_stream` once per minted link (403 role, 404 unknown item or no audio, 413 over the 2 GiB cap, 502 source or storage unreachable, or link refused/expired) | `pkg/dashboard/api_inbox.go:31` |
| `POST` | `/api/inbox/uploads` | Owner or read-write | Add dropped files to the Inbox: `multipart/form-data` with one or more `file` fields (at most `inbox.uploads.max_files_per_drop`, default 20, each up to `inbox.uploads.max_file_bytes`, default 25 MiB). Each file is typed by content (magic bytes, UTF-8, a parse check), never by name or client MIME alone; md, txt, json, jsonl, yaml, csv, tsv, html, docx and pdf (parsed in the `hive-pdftext` sandbox) become `upload` items with extracted markdown and a Knowledge fact. One file: 201 `{results: [{filename, ok, source, item_id, title, kind}]}` or its refusal (400 `empty`/`too_many_files`, 413 `too_large`, 415 `unsupported_type`, 422 `parse_failed`, 507 `quota_exceeded`, each with `code` and `error`); several files: 200 with per-file results unless all failed. Cross-site requests are refused; every file is audited as `inbox_upload` (403 role, 404 no upload store); see [inbox-uploads.md](../inbox-uploads.md) | `pkg/dashboard/api_inbox_uploads.go:27` |
| `POST` | `/api/inbox/uploads/url` | Owner or read-write | Add a link: `{url}`. The hive fetches it under the SSRF guard (http/https only, no credentials, its own DNS lookup with the connection pinned to the checked address, private/loopback/link-local/CGNAT/multicast/ULA/metadata ranges refused, every redirect re-checked, at most 3, body capped at `inbox.uploads.max_file_bytes`, 60 s) and runs the body through the upload type pipeline (an HTML page becomes readable markdown titled by its `<title>`). 201 `{results: [{filename: host, ok, source, item_id, title, kind}]}`; 400 `link_refused`, 413, 415, 422, 502 `fetch_failed`, 507; audited as `inbox_upload` with `origin=link` and the host (403 role, 404 no upload store) | `pkg/dashboard/api_inbox_uploads.go:28` |
| `POST` | `/api/inbox/uploads` | Owner or read-write | Add dropped files to the Inbox: `multipart/form-data` with one or more `file` fields (at most `inbox.uploads.max_files_per_drop`, default 20, each up to `inbox.uploads.max_file_bytes`, default 25 MiB). Each file is typed by content (magic bytes, UTF-8, a parse check), never by name or client MIME alone; md, txt, json, jsonl, yaml, csv, tsv, html, docx and pdf (parsed in the `hive-pdftext` sandbox) become `upload` items; audio and video (mp3, m4a, aac, wav, ogg, opus, webm, mp4, mov) stream to disk under `inbox.recording.max_bytes`, become pending audio items played from disk and are transcribed when transcription is on with extracted markdown and a Knowledge fact. One file: 201 `{results: [{filename, ok, source, item_id, title, kind}]}` or its refusal (400 `empty`/`too_many_files`, 413 `too_large`, 415 `unsupported_type`, 422 `parse_failed`, 507 `quota_exceeded`, each with `code` and `error`); several files: 200 with per-file results unless all failed. Cross-site requests are refused; every file is audited as `inbox_upload` (403 role, 404 no upload store); see [inbox-uploads.md](../inbox-uploads.md) | `pkg/dashboard/api_inbox_uploads.go:34` |
| `POST` | `/api/inbox/uploads/url` | Owner or read-write | Add a link: `{url}`. The hive fetches it under the SSRF guard (http/https only, no credentials, its own DNS lookup with the connection pinned to the checked address, private/loopback/link-local/CGNAT/multicast/ULA/metadata ranges refused, every redirect re-checked, at most 3, body capped at `inbox.uploads.max_file_bytes`, 60 s) and runs the body through the upload type pipeline (an HTML page becomes readable markdown titled by its `<title>`). 201 `{results: [{filename: host, ok, source, item_id, title, kind}]}`; 400 `link_refused`, 413, 415, 422, 502 `fetch_failed`, 507; audited as `inbox_upload` with `origin=link` and the host (403 role, 404 no upload store) | `pkg/dashboard/api_inbox_uploads.go:35` |
| `POST` | `/api/inbox/recordings` | Owner or read-write | Start a dashboard recording (● Record): `{title, mime: audio/webm\|audio/mp4, consent_version, includes_tab_audio}`. `consent_version` must name the current consent notice; it is stored with the user in the recording's metadata. One recording per user at a time. 201 `{id, max_chunk_bytes, max_seconds, retention_days, transcribes}`; audited as `inbox_recording_start` (400 consent missing or stale, 403 role, 409 `{active_id}` already recording, 412 `inbox.recording.enabled` off, 415 other media types, 507 `inbox.recording.quota_bytes` used up); see [inbox-recording.md](../inbox-recording.md) | `pkg/dashboard/api_inbox_recordings.go:20` |
| `POST` | `/api/inbox/recordings/{id}/chunks` | The recorder (owner or read-write) | Store one MediaRecorder chunk `?seq=N`, the raw bytes as the body (at most `max_chunk_bytes`, 512 KiB). In order only; each part is fsynced before the 200 `{id, seq, next_seq, bytes, duplicate?}`. The first chunk must start with a WebM or MP4 signature; the hive never decodes audio. A re-send with the same bytes is acknowledged again (`duplicate: true`) (400 bad seq, 403 someone else's, 409 `{expected_seq}` gap, other bytes for a stored seq, or already ended, 413 chunk over 512 KiB or past `max_bytes` / `max_seconds`, 415 not WebM/MP4, 507 quota) | `pkg/dashboard/api_inbox_recordings.go:21` |
| `POST` | `/api/inbox/recordings/{id}/finalize` | The recorder | End a recording: `{duration_ms?}` (the recorder's count, pauses excluded). The chunks are concatenated in order into `audio.webm` or `audio.mp4`, a pending Knowledge fact (personal layer) is written and the item appears in the Inbox. 200 `{id, state, title, duration_ms, bytes, chunks, transcribes}`; audited as `inbox_recording_finalize` (400 no chunk received, 403, 404, 409 already ended) | `pkg/dashboard/api_inbox_recordings.go:22` |
Expand Down
Loading
Loading