Skip to content

Commit f3bf76c

Browse files
Guide ScreenPipe capture-node setup in wizard (#339)
* feat: guide ScreenPipe capture-node setup * fix: refresh capture timeline and tolerate recorder startup * fix: group Timeline audio chunks into sessions * fix: unblock CI — test env defaults + workers-test startup deadlock - tests/conftest.py: provide AUTH_SECRET_KEY/ADMIN_PASSWORD defaults so collection works without a local .env (CI has none; auth.py raises at import since the readiness/fleet work landed). - docker-compose-test.yml: drop workers-test dependency on backend health. /readiness now requires the worker-fleet heartbeat, so gating workers on backend health deadlocked startup (backend unhealthy -> workers never start). Matches prod compose: workers need mongo+redis. * fix: land SessionStore conversation-assignment API tested by #336 test_session_store.py was committed in #336 but the SessionStore implementation it exercises (set/get/clear/expire current conversation, conversation_create_lock) stayed in a local stash. Bring the implementation in so the committed tests pass. * feat: add Linux Chronicle desktop tray * feat: add ScreenPipe context thumbnails to Timeline * ci: install libopus0 for advanced backend unit tests services/device_audio.py imports opuslib at module level, which needs the native Opus library; the GitHub runner doesn't ship it. Was masked until now by the earlier AUTH_SECRET_KEY collection error. * fix: preserve meaningful ScreenPipe transitions * fix: normalize ScreenPipe audio timestamps * fix: ignore opuslib SyntaxWarning under pytest error filter opuslib 3.x contains `is not 0`, a SyntaxWarning on py3.12. With filterwarnings=error it becomes a SyntaxError on first import in a fresh venv (CI), killing collection. Cached .pyc masked it locally. * fix: make memory.processed fire under the no-api mock config Two layers were breaking the plugin-event tests: 1. mock-services.yml had no defaults.fallback_llm, so the memory agent's note-guarantee recovery pass (force_fallback=True) raised 'No fallback LLM is configured' and the memory job failed with nothing recorded. Add mock-llm-fallback (same mock server, distinct name — a fallback identical to defaults.llm is rejected as pointless). 2. chronicle.py treated a recovery-pass exception as fatal, skipping the source-preserving fallback note entirely. The note guarantee now degrades: log the recovery failure, write the fallback note, return its path — so memory.processed still fires and the vault keeps the conversation. Also lands the speaker-rename guard in _speaker_rename_guidance: conversation-scoped relabels between real named people no longer trigger a vault-wide rename_person (which merged the wrong person's history); only placeholder labels (Speaker N / Unknown Speaker N) may be globally renamed. * ci: real redis/mongo services for backend unit tests; upload robot service logs - test_vault_rename_person needs Redis (vault locks are Redis-backed and fail closed) and test_leading_silence_trim_db needs Mongo; both passed locally only because a dev stack was listening. Give the CI job real service containers on the ports the code defaults to. - robot-tests saved backend/worker logs to files but never uploaded them, leaving container-side failures undiagnosable. Upload them. * fix: resolve host.docker.internal in test containers on Docker/CI Every real HTTP caller in the test stack (memory-agent LLM, streaming STT consumer) addresses the mock services as host.docker.internal, but Docker on Linux does not resolve that name without an explicit host-gateway mapping — CI workers logged 'Name or service not known' for every LLM and streaming-STT call. This broke memory.processed events and streaming speech detection (open_conversation) in the no-api suite, while passing locally under podman, which resolves the name natively. Prod compose already carries the same extra_hosts. * fix: runtime config overrides survive reloads save_config_section() patched the in-memory config cache once, but any subsequent reload_config() (frequent in normal operation) rebuilt the cache from disk and silently reverted the change whenever CONFIG_FILE points away from config.yml (test environments). The always-persist robot test toggles always_persist_enabled via /api/misc-settings and raced exactly this: on slow CI a reload landed between the toggle and the stream open, so a placeholder conversation appeared despite always_persist=false. Register overrides and re-apply them on every load instead. * fix: first runtime settings save no longer fails without config.yml save_config_section() passed a plain {} to OmegaConf.update() when config.yml didn't exist yet, raising 'Unexpected type: {}' — so the very first runtime settings save on a fresh install failed (and the API reported it only in the response body, still HTTP 200).
1 parent fe259a5 commit f3bf76c

27 files changed

Lines changed: 1361 additions & 114 deletions

File tree

‎.github/workflows/python-tests.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,18 @@ jobs:
7575
defaults:
7676
run:
7777
working-directory: backends/advanced
78+
services:
79+
# Some backend tests hit real local services (vault locks are Redis-backed
80+
# and fail closed; test_leading_silence_trim_db uses Mongo). Match the
81+
# defaults the code assumes: redis://localhost:6379, mongodb://localhost:27018.
82+
redis:
83+
image: redis:7-alpine
84+
ports:
85+
- 6379:6379
86+
mongo:
87+
image: mongo:8
88+
ports:
89+
- 27018:27017
7890

7991
steps:
8092
- name: Checkout code
@@ -90,6 +102,11 @@ jobs:
90102
with:
91103
version: "latest"
92104

105+
- name: Install system dependencies
106+
# libopus0: native library behind opuslib (services/device_audio.py),
107+
# imported transitively during test collection.
108+
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends libopus0
109+
93110
- name: Install test dependencies
94111
run: uv sync --locked --group test
95112

‎.github/workflows/robot-tests.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,15 @@ jobs:
100100
echo "✓ Logs saved to backends/advanced/logs/"
101101
ls -lh logs/
102102
103+
- name: Upload service logs
104+
if: always()
105+
uses: actions/upload-artifact@v4
106+
with:
107+
name: robot-service-logs
108+
path: backends/advanced/logs/
109+
if-no-files-found: warn
110+
retention-days: 14
111+
103112
- name: Check if test results exist
104113
if: always()
105114
id: check_results

‎backends/advanced/docker-compose-test.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,11 @@ services:
6262
- LANGFUSE_BASE_URL=http://langfuse-web-test:3000
6363
- LANGFUSE_PUBLIC_KEY=pk-lf-test-public-key
6464
- LANGFUSE_SECRET_KEY=sk-lf-test-secret-key
65+
extra_hosts:
66+
# Mock services (LLM/STT) publish ports on the host and are addressed as
67+
# host.docker.internal. Docker on Linux (CI) does not resolve that name
68+
# without an explicit host-gateway mapping; podman resolves it natively.
69+
- "host.docker.internal:host-gateway"
6570
depends_on:
6671
mongo-test:
6772
condition: service_healthy
@@ -383,9 +388,16 @@ services:
383388
- LANGFUSE_BASE_URL=http://langfuse-web-test:3000
384389
- LANGFUSE_PUBLIC_KEY=pk-lf-test-public-key
385390
- LANGFUSE_SECRET_KEY=sk-lf-test-secret-key
391+
extra_hosts:
392+
# See chronicle-backend-test: required for Docker-on-Linux (CI) to reach
393+
# the mock LLM/STT services published on the host.
394+
- "host.docker.internal:host-gateway"
386395
depends_on:
387-
chronicle-backend-test:
388-
condition: service_healthy
396+
# NOTE: no dependency on chronicle-backend-test. The backend's /readiness
397+
# healthcheck requires the worker fleet heartbeat in Redis, so gating the
398+
# workers on backend health deadlocks startup (backend waits for workers,
399+
# workers wait for backend). Matches prod compose: workers only need
400+
# mongo + redis.
389401
mongo-test:
390402
condition: service_healthy
391403
redis-test:

‎backends/advanced/pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,9 @@ filterwarnings = [
104104
"ignore::UserWarning",
105105
"ignore::DeprecationWarning",
106106
"ignore::PendingDeprecationWarning",
107+
# opuslib 3.x uses `is not 0`, a SyntaxWarning on py3.12 that "error" would
108+
# promote to SyntaxError on first (uncached) import. Third-party; not ours.
109+
"ignore::SyntaxWarning:.*opuslib.*",
107110
]
108111

109112
[tool.coverage.run]

‎backends/advanced/src/advanced_omi_backend/config_loader.py‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,11 @@
1616
# Global config cache
1717
_config_cache: Optional[DictConfig] = None
1818

19+
# Runtime overrides registered by save_config_section(), keyed by section path.
20+
# Re-applied on every load_config() so they survive cache reloads even when
21+
# CONFIG_FILE points away from config.yml (test environments).
22+
_runtime_overrides: dict = {}
23+
1924

2025
def get_config_dir() -> Path:
2126
"""Get config directory path (single source of truth)."""
@@ -117,6 +122,14 @@ def load_config(force_reload: bool = False) -> DictConfig:
117122
f"{[m.get('name') for m in extra_defaults]}"
118123
)
119124

125+
# Re-apply runtime overrides saved via save_config_section(). When
126+
# CONFIG_FILE points away from config.yml (test environments), the saved
127+
# values exist only in memory — without this they silently revert on the
128+
# next reload, so a runtime toggle only "took" until some unrelated code
129+
# path refreshed the config.
130+
for section_path, values in _runtime_overrides.items():
131+
OmegaConf.update(merged, section_path, values, merge=True)
132+
120133
# Cache result
121134
_config_cache = merged
122135

@@ -182,8 +195,11 @@ def save_config_section(section_path: str, values: dict) -> bool:
182195
try:
183196
config_path = get_config_dir() / "config.yml"
184197

185-
# Load existing config
186-
existing_config = {}
198+
# Load existing config. Must be a DictConfig even when the file doesn't
199+
# exist yet — OmegaConf.update() raises "Unexpected type" on a plain
200+
# dict, which made the very first runtime settings save fail silently
201+
# on installs without a config.yml.
202+
existing_config = OmegaConf.create({})
187203
if config_path.exists():
188204
existing_config = OmegaConf.load(config_path)
189205

@@ -193,13 +209,16 @@ def save_config_section(section_path: str, values: dict) -> bool:
193209
# Save back to file
194210
OmegaConf.save(existing_config, config_path)
195211

196-
# Reload config from the primary config file (CONFIG_FILE env var)
197-
merged = reload_config()
212+
# Register a runtime override BEFORE reloading: when CONFIG_FILE points
213+
# to a different file than config.yml (test configs), the value we just
214+
# saved is not in the file load_config() reads, so it must be re-applied
215+
# on every load — a one-shot in-memory patch would silently revert on
216+
# the next reload_config() from any code path.
217+
_runtime_overrides[section_path] = values
198218

199-
# Also apply the values to the in-memory cache directly.
200-
# This is needed when CONFIG_FILE points to a different file than config.yml
201-
# (e.g., test configs), so the saved values still take effect at runtime.
202-
OmegaConf.update(merged, section_path, values, merge=True)
219+
# Reload config from the primary config file (CONFIG_FILE env var);
220+
# load_config() re-applies _runtime_overrides on top.
221+
reload_config()
203222

204223
logger.info(f"Saved config section '{section_path}' to {config_path}")
205224
return True

‎backends/advanced/src/advanced_omi_backend/routers/modules/device_input_routes.py‎

Lines changed: 99 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,9 @@
3232
utcnow,
3333
)
3434
from advanced_omi_backend.models.user import User
35-
from advanced_omi_backend.services.device_context import request_conversation_context_jobs
35+
from advanced_omi_backend.services.device_context import (
36+
request_conversation_context_jobs,
37+
)
3638
from advanced_omi_backend.services.memory.vault_manager import ConvDocVaultManager
3739

3840
router = APIRouter(prefix="/device-input", tags=["device-input"])
@@ -180,6 +182,24 @@ async def ingest_activity(
180182
try:
181183
await item.insert()
182184
accepted += 1
185+
frame_id = incoming.metadata.get("representative_frame_id")
186+
if (
187+
frame_id is not None
188+
and "screen_context" in source.capabilities
189+
and any(
190+
incoming.metadata.get(key)
191+
for key in ("app_name", "window_name", "text")
192+
)
193+
):
194+
await DeviceInputJob(
195+
user_id=source.user_id,
196+
source_id=source.source_id,
197+
kind="thumbnail",
198+
start_at=incoming.captured_at,
199+
end_at=incoming.ended_at,
200+
purpose="timeline_thumbnail",
201+
payload={"item_id": str(item.id), "frame_id": frame_id},
202+
).insert()
183203
except DuplicateKeyError:
184204
duplicates += 1
185205
existing = await DeviceInputItem.find_one(
@@ -307,6 +327,37 @@ async def complete_job(
307327
return {"ok": True}
308328

309329

330+
@router.post("/jobs/{job_id}/thumbnail")
331+
async def complete_thumbnail_job(
332+
job_id: str,
333+
file: UploadFile = File(...),
334+
source: CaptureSource = Depends(_device_source),
335+
):
336+
job = await DeviceInputJob.get(job_id)
337+
if job is None or job.source_id != source.source_id or job.kind != "thumbnail":
338+
raise HTTPException(status_code=404, detail="Thumbnail job not found")
339+
item_id = job.payload.get("item_id")
340+
item = await DeviceInputItem.get(item_id) if item_id else None
341+
if item is None or item.source_id != source.source_id:
342+
raise HTTPException(status_code=404, detail="Timeline item not found")
343+
content_type = (file.content_type or "").split(";", 1)[0]
344+
if not content_type.startswith("image/"):
345+
raise HTTPException(status_code=415, detail="Thumbnail must be an image")
346+
data = await file.read(_MAX_IMAGE_BYTES + 1)
347+
if len(data) > _MAX_IMAGE_BYTES:
348+
raise HTTPException(status_code=413, detail="Thumbnail exceeds the media limit")
349+
item.media_data = data
350+
item.media_filename = file.filename or "screenpipe-thumbnail.jpg"
351+
item.media_content_type = content_type
352+
item.content_hash = hashlib.sha256(data).hexdigest()
353+
item.metadata = {**item.metadata, "thumbnail_available": True}
354+
await item.save()
355+
job.status = "complete"
356+
job.completed_at = utcnow()
357+
await job.save()
358+
return {"ok": True, "item_id": str(item.id)}
359+
360+
310361
@router.get("/sources")
311362
async def list_sources(user: User = Depends(current_active_user)):
312363
rows = (
@@ -476,6 +527,16 @@ async def _immich_bytes(asset_id: str, endpoint: str) -> tuple[bytes, str]:
476527
@router.get("/items/{item_id}/thumbnail")
477528
async def context_thumbnail(item_id: str, user: User = Depends(current_active_user)):
478529
item = await _owned_item(item_id, user)
530+
if (
531+
item.media_data
532+
and item.media_content_type
533+
and item.media_content_type.startswith("image/")
534+
):
535+
return Response(
536+
content=item.media_data,
537+
media_type=item.media_content_type,
538+
headers={"Cache-Control": "private, max-age=3600"},
539+
)
479540
asset_id = item.metadata.get("asset_id")
480541
if item.kind != "immich_memory" or not asset_id:
481542
raise HTTPException(
@@ -489,6 +550,43 @@ async def context_thumbnail(item_id: str, user: User = Depends(current_active_us
489550
)
490551

491552

553+
@router.post("/items/{item_id}/request-thumbnail")
554+
async def request_item_thumbnail(
555+
item_id: str, user: User = Depends(current_active_user)
556+
):
557+
item = await _owned_item(item_id, user)
558+
if item.kind != "activity":
559+
raise HTTPException(status_code=409, detail="Only activity items have frames")
560+
if item.media_data:
561+
return {"status": "complete"}
562+
frame_id = (
563+
item.metadata.get("representative_frame_id")
564+
or item.metadata.get("last_frame_id")
565+
or item.metadata.get("first_frame_id")
566+
)
567+
if frame_id is None:
568+
raise HTTPException(status_code=409, detail="Activity has no source frame")
569+
existing = await DeviceInputJob.find_one(
570+
DeviceInputJob.source_id == item.source_id,
571+
DeviceInputJob.kind == "thumbnail",
572+
{"payload.item_id": item_id},
573+
{"status": {"$in": ["pending", "claimed"]}},
574+
)
575+
if existing:
576+
return {"status": existing.status, "job_id": str(existing.id)}
577+
job = DeviceInputJob(
578+
user_id=item.user_id,
579+
source_id=item.source_id,
580+
kind="thumbnail",
581+
start_at=item.captured_at,
582+
end_at=item.ended_at,
583+
purpose="timeline_thumbnail",
584+
payload={"item_id": item_id, "frame_id": frame_id},
585+
)
586+
await job.insert()
587+
return {"status": "pending", "job_id": str(job.id)}
588+
589+
492590
@router.post("/items/{item_id}/promote")
493591
async def promote_context_item(item_id: str, user: User = Depends(current_active_user)):
494592
item = await _owned_item(item_id, user)

0 commit comments

Comments
 (0)