Skip to content

feat: AI Assistant with natural language app control, LiteLLM capabilities, and multimodal chat - #58

Merged
spelech merged 18 commits into
mainfrom
feature/ai-assistant
Sep 18, 2026
Merged

spelech merged 18 commits into
mainfrom
feature/ai-assistant

Conversation

@spelech

@spelech spelech commented Sep 18, 2026

Copy link
Copy Markdown
Owner

Summary

This PR introduces the full In-App AI Assistant & App Copilot module with natural language app control, living prompt orchestration, dynamic LiteLLM capability discovery, local model filtering, and multimodal vision support.

Key Capabilities

  1. LiteLLM Model Discovery & Local Exclusion:

    • Queries LiteLLM /model/info\ (with automatic fallbacks to /v1/model_info\ and /v1/models).
    • Extracts rich metadata (\SupportsVision, \SupportsFunctionCalling, \MaxInputTokens, \MaxOutputTokens, \SummaryBadge).
    • Strictly excludes local runtimes (\ollama/, \local/, \llama.cpp) while preserving network-hosted LiteLLM cloud models across LAN and localhost IPs.
  2. Composer Model Switcher & Badges:

    • Modern capability-aware dropdown placed directly in the composer bar.
    • Shows provider tag and capability icons (👁️ Vision, ⚡ Tools, context limit pill like \1M).
  3. Multimodal Vision & Clipboard Attachments:

    • File attachment button (📎) and clipboard paste (\Ctrl+V\ / \Cmd+V) for screenshots.
    • Staged attachment preview tray with thumbnail chips and \✕\ dismissal.
    • Serialized as \ImageContent\ into \Microsoft.Extensions.AI\ for full tool loop support.
  4. Natural Language App Control & Living Prompts:

    • 12 native app tools via \IAiAppTools\ (VRAM telemetry, model fit, engine control, settings, and workflows).
    • Living prompts system loading markdown prompts dynamically from \Prompts/.
  5. Testing & Documentation:

    • 681 unit tests passing (100% pass rate).
    • Live LiteLLM integration and E2E tests in \AiAssistantLiveLlmTests.cs.
    • Documentation in \docs/ai_assistant/\ updated per ASD-STE100 (Simplified Technical English) and formatted for VitePress.

@spelech
spelech merged commit dfa12e0 into main Sep 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant