Skip to content

ChaitanyaParate/Deskai

BranchesTags

Repository files navigation

DeskAI

DeskAI is a local-first desktop intelligence daemon that continuously observes your active desktop context and allows you to interact with it using natural language via a lightweight CLI. It runs fully offline, powered by local LLMs through Ollama, and is designed to behave like a background system service rather than a typical app.

This project intentionally focuses on systems-level engineering: IPC, background services, streaming inference, and OS integration.

Table of Contents

  • Overview
  • Why DeskAI
  • Architecture
  • Tech Stack
  • Core Concepts
  • Installation
  • Usage
  • Services & Startup
  • Error Handling Philosophy
  • Project Structure
  • Development Notes
  • Future Work
  • License

Overview

DeskAI runs as a user-level daemon that:

  • Tracks desktop context (window under cursor, screen state, text signals)
  • Routes user intents from a CLI
  • Streams responses from a locally running LLM
  • Automatically starts on login

The user interacts with DeskAI through the deskai command without worrying about starting servers or managing processes.

Why DeskAI?

Most AI assistants:

  • Depend on cloud APIs
  • Have no awareness of your desktop
  • Are UI-heavy and slow for developers

DeskAI is different:

  • can run 100% locally
  • CLI-first
  • Streaming responses
  • Deep OS integration

It is meant to feel like a system utility, not an app.

What this project demonstrates

  • Design of long-running user-level Linux daemons
  • systemd user service configuration and lifecycle management
  • Client–daemon IPC using UNIX domain sockets
  • Token-level streaming pipelines with fault-tolerant client handling
  • Local-first system design with zero cloud dependencies

Architecture

DeskAI uses a daemon–client architecture with UNIX domain sockets to decouple long-running system state from short-lived CLI invocations.

User Command
   |
   v
deskai CLI
   |
   | UNIX socket
   v
DeskAI Daemon
   |
   | subprocess ───────────────|
   v                           v
Local LLM Runtime (Ollama)  Online API(OpenAI/Gemini)

Runtime Flow

  1. systemd starts DeskAI on login
  2. Daemon initializes context loop
  3. Daemon initializes internet check loop
  4. CLI connects via socket
  5. Intent is inferred and routed
  6. LLM streams tokens
  7. Tokens are forwarded to CLI
  8. Client exits, daemon stays alive

Tech Stack

Layer Technology
Language Python 3.10+ (tested on 3.10–3.12)
OS Linux
Display Server X11
Runtime Python venv
OCR Tesseract
ML / Embeddings PyTorch, SentenceTransformers
LLM (Offline) Ollama (phi3:mini)
LLM (Online) OpenAI API / Gemini API (optional)
IPC UNIX domain sockets
Service Manager systemd (user service)

Core Concepts

Daemon State

A persistent global state tracks:

  • Screen context
  • Window Under Cursor
  • Text streaks
  • Last interaction metadata
  • Internet status

Streaming First

Responses are streamed token-by-token.
No buffering. No waiting for full output.

Privacy (Local-First)

By default, DeskAI runs fully locally:

  • No telemetry
  • No background analytics
  • No external HTTP calls (offline mode)

If you enable an online LLM provider (OpenAI / Gemini), DeskAI will send only the captured OCR text + context needed for the requested task.

Installation

DeskAI is installed using install.sh, which configures everything correctly.

Requirements

  • Linux
  • Python 3.10+
  • systemd user sessions
  • curl

Install

git clone https://github.com/ChaitanyaParate/Deskai.git
cd Deskai
chmod +x install.sh
./install.sh

What install.sh Does

  • Creates a Python virtual environment (./venv/)
  • Installs all Python dependencies (requirements.txt)
  • Installs Ollama (only if missing)
  • Starts the Ollama server (ollama serve) in the background
  • Waits until the Ollama API is reachable
  • Pulls the default local model (phi3:mini)
  • Installs deskai.service and ollama.service into ~/.config/systemd/user/
  • Reloads systemd, enables the services, and starts DeskAI automatically on login
  • Installs the deskai CLI wrapper into ~/.local/bin/deskai
  • Prompts the user for an API key (for optional online LLM mode) and stores it for runtime usage

Commands & CLI Usage

deskai read

Reads the current ScreenContext.

Run a Task

Commands demonstrate intent routing and streaming execution rather than end-user features.

deskai stream summarize
deskai stream explain_error

Note: deskai stream search is currently under development and not yet functional.

Check Service

systemctl --user status deskai

Restart and Stop

systemctl --user restart deskai
systemctl --user stop deskai

Logs

journalctl --user -u deskai -f

Services & Startup

DeskAI installs two user services:

  • deskai.service
  • ollama.service

They live in:

~/.config/systemd/user/

They are enabled automatically and start on login.

Error Handling Philosophy

  • BrokenPipeError is expected when clients exit early
  • Daemon never crashes on client failure
  • Restart handled by systemd
  • Streaming errors are logged, not fatal
  • Client lifecycle is fully decoupled from daemon state

Project Structure

deskai/
├── main.py
├── client.py
├── state.py
├── llm_providers.csv    # API key config (genai / openai)
│
├── capture/
│   ├── backend.py
│   ├── screen.py
│   └── windows.py
│
├── context_model/
│   ├── context_classifier.pt  # trained model weights
│   ├── dataset.py
│   ├── infer.py
│   ├── model.py
│   ├── train.py
│   └── type.py
│
├── daemon/
│   ├── loop.py
│   └── internet_check.py
│
├── dataset/
│   └── data.json
│
├── intent/
│   ├── router/
│   │   └── router.py
│   └── executor/
│       ├── handlers/
│       │   ├── explain_error.py
│       │   ├── summarize.py
│       │   └── search.py
│       └── executor.py
│
├── llm/
│   ├── factory.py
│   ├── genai_clint.py
│   ├── local.py
│   └── openai_clint.py
│
├── ocr/
│   ├── engine.py
│   ├── postprocess.py
│   └── types.py
│
├── tools/               # development / debug utilities
│   ├── data_generator.py
│   ├── dataset_test.py
│   ├── debug_capture.py
│   ├── infer_test.py
│   └── req.py
│
├── deskai.sh            # CLI wrapper
├── install.sh           # Installer
├── deskai.service       # systemd service
├── ollama.service
└── requirements.txt

Development Notes

This project intentionally avoids:

  • Framework-heavy abstractions
  • Cloud dependencies
  • GUI complexity

It focuses on:

  • OS-level correctness
  • Robust streaming
  • Process lifecycle management

Known Limitations

  • Linux systems using X11 only
  • Requires systemd user sessions
  • LLM latency depends on model size
  • No sandboxing between intents
  • No GUI feedback

Future Work

  • Lightweight GUI overlay as an alternate client to the existing daemon
  • Plugin-based intent system with explicit isolation boundaries
  • Multi-model routing within the daemon for heterogeneous workloads
  • Persistent state storage decoupled from daemon process lifetime

License

MIT License

Author

Chaitanya Parate
Built as a systems-level project focused on:

  • systemd
  • Linux services
  • Streaming LLM inference

About

Local-first desktop AI daemon that runs fully offline. Tracks active desktop context, exposes a CLI, streams responses from local LLMs via Ollama, and runs as a systemd user service. Built for systems-level learning: IPC, daemons, streaming inference, OS integration.

Topics

linux streaming daemon ipc x11 python3 unix-socket systemd-service command-line-interface desktop-assistant ocr-python llm ollama local-ai-llm

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages