Skip to content

feat(windows): local chat in the dashboard for the Windows Desktop profile - #8009

Open
IronicRayquaza wants to merge 8 commits into
Osmantic:mainfrom
IronicRayquaza:feat/windows-desktop-chat-ui
Open

IronicRayquaza wants to merge 8 commits into
Osmantic:mainfrom
IronicRayquaza:feat/windows-desktop-chat-ui

Conversation

@IronicRayquaza

Copy link
Copy Markdown
Contributor

Summary

Sixth step of #7968: a chat in the dashboard itself for the Windows Desktop profile, the screen an owner actually uses.

Builds on #7975 (and #7970). Please review only the last commit. I'll rebase as the earlier PRs merge.

In this profile /api/pixel/status reports "Portal is not enabled". Pixel's existing branch for that state now renders LocalChat instead of a link to Open WebUI, which Desktop doesn't have.

components/LocalChat.jsx:

  • Streaming: answers stream from feat(windows): stream local chat from the Windows Desktop llama-server #7975's /api/local-chat/stream through Portal's own PortalStreamingText (Markdown, code highlighting, reveal), so local chat looks like the rest of ODS. The owner's Enter or Ctrl+Enter send setting is respected.
  • Stable context: each answer reports contextStart and the next message sends it back, so the model's prompt prefix stays cached (long-chat follow-ups take ~0.8 s instead of ~28 s on a laptop CPU; measured in feat(windows): stream local chat from the Windows Desktop llama-server #7975). When earlier messages fall outside the model's view, the chat says so and keeps them saved.
  • Errors: a refused message (too long, model loading or stopped) goes back into the box with the server's plain sentence, never lost. Stop keeps the partial answer.
  • Status polling: /api/local-chat/status is polled one request at a time with a 5 s timeout and stops once the model is ready, so requests can't pile up.

lib/localChatStore.js: history lives in this browser, bounded to the newest 50 chats within 4 MB; the open chat is never dropped. A storage failure (private window, full quota) shows a notice but never blocks chatting. Chats can be switched, started, deleted, and saved as Markdown or JSON; the file is named after the chat in any script (Vietnamese, Hindi, Japanese), and a character split across network chunks still arrives whole (tested).

What a non-technical owner no longer runs into:

  • No password prompt: /api/auth/dashboard-session now reports remoteAccess, which is false in the Desktop profile (loopback only, so there's no other device to sign in from). The sign-in gate no longer asks that owner to "Choose your password" before the first chat. Compose is unchanged (remoteAccess: true).
  • Only working pages in the sidebar: it hides Extensions, Models, GPU Monitor and Portal's own conversation list in this profile. Those pages can't work there, and a second, empty chat list next to Local chat would only confuse.

AI Assistance

Claude Code drafted the code, tests and this description, and ran the validation below on my Windows 11 laptop. I chose the scope and design (see #7968), and I'm accountable for the diff and for responding to review.

Release Lane

  • Stable hotfix targeting release/2.6.x
  • Mainline change targeting main
  • Next-minor work targeting the next feature/minor release
  • Not sure; reviewer should help classify

Changed Surface

  • Docs only
  • Tests only
  • Dashboard UI
  • Dashboard API / host agent
  • Installer / bootstrap / lifecycle
  • Docker Compose / service manifests
  • Model routing / Hermes / capabilities
  • Network exposure / auth / proxy
  • Dependencies / runtime wiring

Risk And Validation

  • Risk level: Low for Compose. Every UI change keys off runtimeProfile === 'windows-desktop'; the API change only adds remoteAccess: true to an existing response there.
  • Validation run:
    • git diff --check
    • Dashboard lint/test/build
    • Focused tests listed below
    • mypy 2.4.0 as CI runs it plus the ratchet: 0 errors added
    • Scoped hardware validation (the real UI in a real browser engine on a consumer laptop)
vitest (Windows, Node 22):
  localChatStore (9), localChatStream (8), core.desktop sidebar (2), LocalChat (6), Sidebar (+1),
  DashboardSignInGate (+1), Pixel (+1 Desktop case) -> 192 passed in the touched suites
  full suite: 3,031 passed, 0 failed (tokenSpySettings.test.js cannot load without `python` on PATH,
  identically on the base branch)
eslint src: 0 errors, nothing in touched files
vite build: ok
pytest test_dashboard_session.py: 33 passed (incl. remoteAccess for Compose and Desktop)

Live, end to end. On Windows 11 Home 26200 (i5-10500H, 16 GB RAM, CPU only), with no Docker and no WSL, I ran:

The real UI was then driven in headless Edge over the DevTools protocol (typing into the box and pressing Send):

1. Local chat ready: Llama-3.2-1B-Instruct-Q4_K_M.gguf · 4,096 tokens
   password prompt shown: False      Portal "New task" list in sidebar: False
2. "In one short sentence: what is the capital of France?"  first text 0.88s, done 1.2s
   -> "Paris is the capital of France."
3. "And of Japan?" (same chat)                              first text 0.56s, done 1.0s
   -> "Tokyo is the capital of Japan."
4. Reload -> the chat is still there: [['In one short sentence: what is the capital of France?', 4 messages]]

The first render found the two issues fixed here: the "Choose your password" prompt, and the duplicate Portal chat list in the sidebar.

Operational Change Check

  • This is not an operational change.
  • This is an operational change and validation is recorded above.
  • This is an operational change and validation is intentionally deferred for:

Notes For Reviewers

  • Real browser vs. headless: I couldn't drive an interactive desktop browser here, so the live run used headless Edge (the same engine PR6's app window will use) through the DevTools protocol.
  • Still to come: model switching in Desktop waits for the switchboard "PR 2B" adapters. The installer and launcher are PR6.
  • Running uvicorn under pythonw.exe (for PR6, from live runs):
    • it needs stdout and stderr redirected to files;
    • a .cmd launcher must use 8.3 short paths for non-ASCII folders (cmd.exe mangles José even after chcp 65001);
    • Start-Process -Redirect* lets the child inherit the caller's pipes.

🤖 Generated with Claude Code

IronicRayquaza and others added 3 commits October 23, 2026 01:52
Add an opt-in ODS_RUNTIME_PROFILE=windows-desktop so dashboard-api can run
as a native Windows process with no Docker, no WSL and no host agent.

- Refuse to start without an absolute ODS_DATA_DIR and DASHBOARD_API_KEY;
  the generated-key fallback no longer writes to /data (C:\data on Windows).
- Never dial a host agent: AGENT_URL is empty and the shared client raises
  AgentUnavailable before any socket, so a Docker or WSL ODS on the same PC
  cannot answer.
- Start no child processes: GPU probes, the curl cluster refresh, Token Spy
  polling and the Docker preflight are skipped. A windowless process would
  otherwise flash a console every five seconds.
- Load only loopback services, since Compose service names resolve to
  nothing on a native PC and each failed lookup stalls /api/status.
- Report Windows CPU and RAM through kernel32 instead of None.
- Add a path-filtered windows-latest workflow that installs the hashed lock
  wheels-only and runs the profile tests.

Refs Osmantic#7968

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mypy runs on Linux, where ctypes has no windll or WinError, so assert the
Windows platform inside the two kernel32 readers. Count lock hashes in a
typed dict instead of a dict mixing names and counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mypy 2.4 does not treat an assert on sys.platform as unreachable, so define
the two readers under a module-level sys.platform == "win32" branch, with
stubs that raise OSError elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IronicRayquaza and others added 3 commits October 23, 2026 13:00
Add the chat route for the Windows Desktop profile (Osmantic#7968): dashboard-api
talks to the loopback llama-server that the native runtime started, with its
key file. No LiteLLM, no host agent, no Portal stack.

- native_llm.py reads the launcher's ODS_NATIVE_LLM_URL (loopback http only)
  and ODS_NATIVE_LLM_RUNTIME_DIR: the api-key file plus the runtime's own
  intent.json and ready.json. The key is cached by mtime and never returned
  or logged.
- context_budget.py fits history into the model's real context before every
  request: measured with llama-server's /apply-template and /tokenize (the
  prompt it will actually evaluate), answer tokens reserved, whole turns
  dropped from the oldest end by binary search, a kept slice always starting
  with a user turn. A message that cannot fit on its own is refused before
  inference with its token numbers. Saved history is never shortened.
- /api/local-chat/stream passes llama-server's SSE through with bounded lines
  and a first "ods" event reporting how many messages were left out of the
  model's view. /api/local-chat/status reports ready, loading, stopped or
  failed from /health, intent.json and ready.json. Every failure has a fixed
  code and sentence; exception text never reaches the browser.
- The setup wizard's /api/chat uses the same native llama-server in this
  profile instead of a Docker hostname.
- A Desktop launcher without these settings fails at startup, naming the
  variable.

Refs Osmantic#7968

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-server reuses its cached prompt only while the start of the prompt is
unchanged. Re-cutting history to the full budget on every message moved the
first kept turn almost every time, so a long chat on a laptop CPU re-read
about 3,000 tokens (28 s with a 1B model) before each answer.

The client now sends back the contextStart the previous answer reported.
While the conversation still fits from that turn it is reused unchanged;
when it no longer fits, history is cut to 75% of the budget so the next
several messages keep the new start. Measured on an i5-10500H with
Llama 3.2 1B: one 24.7 s re-read, then 0.8 s first tokens for every
following message (previously about 28 s whenever the window moved).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ofile

Give the Windows Desktop profile (Osmantic#7968) a chat in the dashboard itself. In
that profile /api/pixel/status reports "Portal is not enabled", and Pixel's
existing branch for that state now renders LocalChat instead of a link to
Open WebUI, which Desktop does not have.

LocalChat streams from /api/local-chat/stream and reuses Portal's
PortalStreamingText (Markdown, highlighting, reveal) and the owner's Enter or
Ctrl+Enter send setting. It sends back the contextStart each answer reports,
so the model's prompt prefix stays cached between messages, and says when
earlier messages are outside the model's view. A refused message (too long,
model loading or stopped) goes back into the box with the server's sentence.
Stop keeps the partial answer. Status is polled one request at a time with a
timeout, and polling stops once the model is ready.

History lives in this browser, bounded to the newest 50 chats within 4 MB;
the open chat is never dropped, and a storage failure (private window, full
quota) is shown but never blocks chatting. Chats can be saved as Markdown or
JSON, switched, started and deleted.

The sidebar hides Extensions, Models, GPU Monitor and Portal's own
conversation list in this profile: those pages cannot work there, and a
second, empty chat list beside Local chat would only confuse.

The dashboard session status now reports remoteAccess: a loopback-only
Desktop install has no other device to sign in from, so the sign-in gate
no longer asks its owner to choose a password before the first chat.

Refs Osmantic#7968

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IronicRayquaza and others added 2 commits October 24, 2026 19:52
The app and the model's task both start at sign-in, and the model's
launcher checks its files before llama-server listens. In that window
the chat status said "not answering, restart ODS" while nothing was
wrong. A standard-account test caught it 17 s after sign-in.

For the app's first 3 minutes, a silent model with no recorded stop or
failure is now "model_starting". The UI already shows a server message
for a state it does not know.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant