Vision Assistant Pro is an advanced, multi-modal AI assistant for NVDA. It leverages world-class AI engines to provide intelligent screen reading, translation, voice dictation, and document analysis.
This add-on was released to the community in honor of the International Day of Persons with Disabilities.
Go to NVDA Menu > Preferences > Settings > Vision Assistant Pro.
- Provider: Select your preferred AI service. Supported providers include Google Gemini, OpenAI, Mistral, Groq, MiniMax, and Custom (OpenAI-compatible servers like Ollama, LM Studio, Jan.ai, or KoboldCPP).
- Important Note: We strongly recommend using Google Gemini for the best performance and accuracy (especially for image/file analysis).
- API Key: Required. You can enter multiple keys (separated by commas or new lines) for automatic rotation.
- Fetch Models: After entering your API key, press this button to download the latest list of available models from the provider.
- AI Model: Select the main model used for general chat and analysis.
Available for all providers including Gemini, OpenAI, Groq, Mistral, and Custom.
⚠️ Warning: These settings are intended for advanced users only. If you are unsure what a specific model does, please leave this unchecked. Selecting an incompatible model for a task (e.g., a text-only model for Vision) will cause errors and stop the add-on from working.
Check "Advanced Model Routing (Task-specific)" to unlock detailed control. This allows you to select specific models from the dropdown list for different tasks:
- OCR / Vision Model: Choose a specialized model for analyzing images.
- Speech-to-Text (STT): Choose a specific model for dictation.
- Text-to-Speech (TTS): Choose a model for generating audio.
- AI Operator Model: Select a specific model for autonomous computer operation tasks.
- Video Model: Select a specific model for video analysis and audio description generation. Note: Unsupported features (e.g., TTS for Groq) will be automatically hidden.
Available only when "Custom" is selected.
⚠️ Warning: This section allows for manual API configuration and is designed for power users running local servers or proxies. Incorrect URLs or model names will break connectivity. If you don't know exactly what these endpoints are, keep this unchecked.
Check "Advanced Endpoint Configuration" to manually input server details. Unlike native providers, here you must type the specific URLs and Model Names:
- Models List URL: The endpoint to fetch available models.
- OCR/STT/TTS Endpoint URL: Full URLs for specific services (e.g.,
http://localhost:11434/v1/audio/speech). - Custom Models: Manually type the model name (e.g.,
llama3:8b) for each task.
To make local, completely offline AI integration extremely simple, a dedicated "Setup Local AI" button is available inside the Custom Provider Settings.
If you are running a local AI model server on your computer:
- Select Custom as your Provider.
- Press the Setup Local AI button.
- Choose your local AI engine from the accessible dialog:
- Ollama (defaulting to
http://127.0.0.1:11434) - LM Studio (defaulting to
http://127.0.0.1:1234) - Jan.ai (defaulting to
http://127.0.0.1:1337) - KoboldCPP (defaulting to
http://127.0.0.1:5001)
- Ollama (defaulting to
- The add-on will instantly configure the correct local URL, API type, and automatically fetch your active offline models to populate the AI Model selection box.
Note on Network & Proxies: This local connection engine features an advanced proxy bypass mechanism. Even if you are running an active system VPN or TUN-mode proxy, your local AI requests will bypass it completely, ensuring stable offline connections without 502 Bad Gateway errors.
- OCR Engine: Choose between Chrome (Fast) for quick results or AI (Advanced) for superior layout preservation.
- TTS Voice: Select your preferred voice style. This list updates dynamically based on your active provider.
- Creativity (Temperature): Controls the randomness of the AI. Lower values are better for accurate translation/OCR.
- Proxy URL: Configure this if AI services are restricted in your region (supports local proxies like
127.0.0.1or bridge URLs). - Direct Output (No Chat Window): Check this if you want the AI to simply read the result aloud without opening an interactive chat window.
- Copy AI responses to clipboard: Automatically copies every AI answer to your system clipboard for easy pasting.
- Clean Markdown in Chat: Uncheck this if you prefer to see raw formatting symbols instead of a clean, formatted text view.
To prevent keyboard conflicts, this add-on uses a Command Layer.
- Press NVDA + Shift + V (Master Key) to activate the layer (you will hear a beep).
- Release keys, then press one of the following single keys:
| Key | Function | Description |
|---|---|---|
| Shift + A | AI Operator | Autonomous Operation: Tell the AI to perform a task on your screen. Pressing it again instantly aborts active operations. |
| E | UI Explorer | Interactive Click: Identifies and clicks UI elements in any app. |
| T | Smart Translator | Translates text under navigator cursor or selection. |
| Shift + T | Clipboard Translator | Translates content currently in the clipboard. |
| R | Text Refiner | Summarize, Fix Grammar, Explain, or run Custom Prompts. |
| V | Object Vision | Describes the current navigator object. |
| O | Full Screen Vision | Analyzes the entire screen layout and content. |
| Shift + V | Video Analysis | Analyze local video files or online YouTube, Instagram, TikTok, or Twitter (X) videos. |
| Control + V | Local Video Recording | Records a silent video of your screen and analyzes the actions and layout. |
| D | Document Reader | Advanced reader for PDF and images with page range selection. |
| F | Smart File Action | Context-aware recognition from selected image, PDF, or TIFF files. |
| A | Audio Transcription | Transcribe MP3, WAV, or OGG files into text. |
| C | CAPTCHA Solver | Captures and solves CAPTCHAs (Supports Gov portals). |
| S | Smart Dictation | Converts speech to text. Press to start recording, again to stop/type. |
| Control+L | Live Assistant | Real-time Copilot (Gemini only): Starts or ends a live voice and screen conversation with the AI assistant. |
| I | Status Reporting | Announces current progress (e.g., "Scanning...", "Idle"). |
| L | Label Object | Semantic AI Labeling: Permanently labels the current focused element/icon. |
| Shift + L | Manage/Scan Labels | Opens Label Manager (if labels exist) or scans the app for unnamed elements. |
| U | Update Check | Manually check GitHub for the latest version of the add-on. |
| Space | Recall Last Result | Shows the last AI response in a chat dialog for review or follow-up. |
| H | Commands Help | Displays a list of all available shortcuts. |
| Alt + S | Settings | Opens the Vision Assistant Pro settings dialog. |
| Alt + Q | Quota Exhausted Keys Report | Reports the number of Gemini API keys that have exceeded their daily quota and their reset time. |
| Alt + M | Routing Audit | Reports the AI models currently selected in advanced routing. |
The AI Operator turns Vision Assistant Pro from a passive reader into an active assistant that can interact with your computer on your behalf. You can ask it to describe the screen, answer questions about what it sees, or even take control—clicking buttons, dragging items, typing text, and navigating through applications using natural language commands.
The biggest advantage? It works perfectly in completely inaccessible software. If you are stuck in a custom app, a remote desktop, or a website where your screen reader goes totally silent, the operator doesn't mind. Because it "sees" the screen visually, it can find, read, and interact with elements that have zero accessibility labels.
- Press NVDA + Shift + V, then press Shift + A (or use the direct shortcut) to open the AI Operator dialog.
- Type what you want to do in plain language (e.g., "Click the Save button", "What does the error message say?", or "Rename the file to final.pdf").
- The AI will analyze your screen, identify the relevant elements, and carry out the action or provide the answer. If a task requires multiple steps, the operator will continue working until it's complete.
- Press Shift + A again at any time to instantly abort an ongoing operation.
The operator understands a wide range of commands:
- Describe & Answer: "Describe the screen layout" or "What does the error message say?"
- Click: "Click the Save button"
- Right Click: "Right-click the file"
- Double Click: "Double-click the document"
- Drag & Drop: "Drag the document to the Archive folder"
- Type: "Type 'Hello World' in the search box"
- Scroll: "Scroll down three times"
- Keypress: "Press Enter", "Press Tab", "Press Escape"
- Multi-step Tasks: "Open File Explorer, find the report, and rename it to final.pdf"
⚠️ API Usage Warning: Because the operator needs to "see" exactly what's happening on screen, it sends a high-resolution screenshot with every step. Frequent use will consume your API quota much faster than standard text-based features.- Administrator Applications: If NVDA is not running with Administrator privileges, the operator may not be able to interact with windows that require elevated permissions. This is a Windows security limitation, not a bug in the add-on.
- Best Practices: For best results, give clear and specific commands. "Click the blue Submit button at the bottom of the form" will almost always work better than just "Click the button".
Note: The Video Analysis and Audio Description features are strictly powered by the Google Gemini provider. Ensure that your active provider in the add-on settings is set to Google Gemini.
Vision Assistant Pro introduces powerful video processing capabilities designed specifically for blind users. It can analyze both online videos and local screen recordings to provide highly detailed visual descriptions and generate professional Audio Description scripts (SRT).
If you encounter a silent video, an animation, or a tutorial on your screen, you can capture it directly:
- Press NVDA + Shift + V to enter the Command Layer, then press Control + V.
- The add-on will silently record your screen in the background.
- Press Control + V again to stop recording.
- The AI will then analyze the recorded video segment and provide a highly detailed description of the scene, characters, and actions.
You can analyze both local video files and online videos. Simply select a local video file in Windows Explorer, or copy an online video link to your clipboard. You can also press Shift + V anywhere (like inside a media player) to open a dialog where you can browse for a video file or paste a URL manually.
- Supported Online Platforms: YouTube, Instagram, TikTok, and Twitter (X).
- The AI will automatically detect the local file or the URL, process the video, and provide a comprehensive visual description and audio summary.
For a more structured experience, the add-on can generate professional Audio Description scripts in standard SubRip (SRT) format.
- Smart Gap-Timing: The AI listens to the audio track and specifically anchors its visual descriptions to natural pauses and silent gaps to intelligently minimize dialogue overlap.
- Character Tracking: The engine performs a pre-pass to extract distinct characters based on immutable facial features. It builds a global dictionary to accurately track and label characters across different scenes without confusion.
- Verbatim Text OCR: Any text appearing on the screen (signs, phones, credits) is strictly quoted verbatim.
- How to Use: To listen to the generated subtitle, simply place the
.srtfile in the same folder as your video file and give it the exact same name. Then, configure your media player (e.g., VLC or PotPlayer) to route the subtitle text directly to your screen reader or TTS engine during playback.
Beyond just creating text-based SRT files, the add-on functions as a complete Audio Description production tool by synthesizing the descriptions into speech and mixing them with the video. When generating an MP3 for local video files, you have multiple mixing modes:
- Standard AD (Mix Voice): The narration is overlaid directly on top of the video's audio. You will be prompted if you want to apply Audio Ducking (lowering the background volume during descriptions) to ensure the narration is clear.
- Extended AD (Pause Audio): The engine pauses the original video audio during descriptions, ensuring you never miss a single word of the original dialogue or the AI narration.
- YouTube Videos: For YouTube sources (which are not downloaded locally), the MP3 export will strictly contain the synchronized AI voice track without the background video audio.
Vision Assistant Pro includes a highly optimized Document Reader designed for multi-page PDFs, complex images, and even iPhone HEIC formats.
You don't need to read a massive document all at once. Enter a page range (e.g., 1-20), and the AI will process all pages in the background. If NVDA crashes or you interrupt the scan, the add-on will remember your progress and offer to Resume exactly where it left off!
You don't always need to open the document first. In Windows File Explorer, simply highlight a PDF or image and press D (Document Reader) or F (Smart File Action) inside the Command Layer. The add-on will instantly bypass the file dialog and begin processing the highlighted file.
When the Document Reader window is open, you can use the following shortcuts:
- Ctrl + PageDown: Move to the next page.
- Ctrl + PageUp: Move to the previous page.
- Alt + A: Open a chat dialog to ask questions about the document.
- Alt + R: Force a Re-scan with AI using your active provider.
- Alt + G: Generate and save a high-quality audio file (WAV/MP3). (Hidden if provider doesn't support TTS).
- Alt + S / Ctrl + S: Save the extracted text as a TXT or HTML file.
Stuck in an application with "unlabeled button" everywhere? The Semantic AI Labeling engine solves this permanently.
Focus your screen reader on an unlabeled graphic or button and press L in the Command Layer. The AI will look at the button visually, determine its function, and apply a permanent label. Unlike older screen reader labeling tools, this add-on uses an advanced hybrid "Object Signature" system (AutomationId/ControlID). Your custom labels will survive window resizing, monitor switching, and application updates!
Press Shift + L to scan the entire active window at once. The AI will find all unlabeled elements and intelligently name them in one go. You can later manage, rename, or batch-delete these labels from the built-in Label Manager.
Need to interact with an element without navigating to it manually? Press E to activate the UI Explorer. The AI will scan the screen and generate an accessible list of every clickable element (ignoring system noise like taskbars). Pick an item from the list, and the add-on will instantly click it for you.
The Live Assistant turns Vision Assistant Pro into a real-time, interactive copilot. (Note: This feature is exclusive to Google Gemini and Gemini-compatible Custom providers).
- Activation: Press Control + L in the Command Layer to open the Live Assistant dialog.
- Real-time Interaction: Talk naturally through your microphone. The AI will simultaneously listen to your voice and look at your active screen. You can ask questions like "What am I looking at?" or "Read the third paragraph to me."
- Customization: Inside the dialog, you can change the AI's Voice Style (e.g., Professional, Friendly, Upbeat) and adjust its "Thinking Depth" to control how deeply it reasons before answering.
You can manage prompts in Settings > Prompts > Manage Prompts....
[selection]: Currently selected text.[clipboard]: Clipboard content.[clipboard_image]: Image currently in clipboard.[screen_obj]: Screenshot of the navigator object.[screen_fg_obj]: Screenshot of the active foreground window.[screen_full]: Full screen screenshot.[file_ocr]: Select image/PDF file for text extraction.[file_read]: Select document for reading (TXT, Code, PDF).[file_audio]: Select audio file for analysis (MP3, WAV, OGG).{target_lang}: Current target language.{source_lang}: Current source language.{response_lang}: Current AI response language.{swap_target}: Fallback language for smart swap translation.{swap_instruction}: Smart swap translation instruction block.
Vision Assistant Pro is packed with advanced tools. Here are some common scenarios to help you choose the right one:
-
Scenario: You want to understand the complete layout of a complicated window or inaccessible app. Solution: Press O (Full Screen Vision). The AI will analyze the entire screen and describe exactly where elements, texts, and buttons are positioned.
-
Scenario: You found an image on a webpage or an unlabeled graphic in a document. Solution: Move your navigator object to the graphic and press V (Object Vision). The AI will describe specifically what that image contains.
-
Scenario: You want to watch a movie or video clip with audio descriptions. Solution: Press Shift + V on your video and choose "Generate Audio Description (SRT File)". When it finishes, click "Generate Synced Narration (MP3)" and select "Extended AD". The add-on will create an audio track that intelligently pauses the movie's dialogue to describe the visual scenes.
-
Scenario: You encountered an app full of "unlabeled buttons". Solution: Press L to permanently label the specific button using AI. Or, press Shift + L to scan and label the entire window at once. If you just want to click something quickly, press E (UI Explorer) to get a list of all clickable items.
-
Scenario: You need to bypass an inaccessible CAPTCHA. Solution: Press C (CAPTCHA Solver). The AI will automatically capture the CAPTCHA, solve it, and inject the answer into the correct field.
-
Scenario: You want to read a long, 50-page PDF document. Solution: Press D (Document Reader), set your provider to Google Gemini, and enter the page range
1-50. The add-on will extract the text accurately in the background. -
Scenario: You are watching a silent video tutorial or animation on your screen. Solution: Press Control + V to start recording the screen. Let the tutorial play, then press Control + V again. The AI will explain exactly what was demonstrated.
Note: An active internet connection is required for all AI features. Multi-page documents are processed automatically.
Stay updated with the latest news, features, and releases:
- Telegram Channel: t.me/VisionAssistantPro
- GitHub Issues: For bug reports and feature requests.
A heartfelt thank you to our community members who support the continuous development and maintenance of this project through their generous financial contributions:
- @Alyabani94
- Ali Alamri
- Ilya
- Arne Siebert
- leonardo0216
- Sergei Fleytin
- Anonymous Supporter
If you wish to support the project financially and see your name here, you can find the Donate option in the NVDA Tools menu (Vision Assistant submenu) or during the setup process after installation.
- Intelligent API Model Filtering: Complete overhaul of the model filtering system to use a pure blacklist approach instead of whitelists. Added stronger filtering keywords (
embedding,bison,gecko,audio,realtime,babbage,moderation,deep,antigravity,computer) to ensure the main chat model dropdown remains perfectly clean and future-proof, while keeping all specialized models accessible in the Advanced Routing section. - Advanced Routing Search: All Advanced Model Routing dropdowns (OCR, STT, TTS, Operator, Video, Live) and the eSpeak Variant selector are now fully searchable. You can quickly type to filter and find your desired model or variant.
- New Command Layer Shortcuts:
- Settings (
Alt + S): Instantly opens the Vision Assistant Pro settings dialog. - Quota Exhausted Keys Report (
Alt + Q): Reports the exact number of Gemini API keys that have exceeded their daily quota, identifying which specific model they are exhausted on, and announces their exact reset time. - Routing Audit (
Alt + M): Audits and announces your current Advanced Routing configuration, reading out which models are actively selected for specialized tasks (skipping default settings).
- Settings (
- Video Analyzer Complete Overhaul: The Video Analyzer has been completely transformed! Previously, it only provided a basic description of online videos. Now, it is a comprehensive video processing suite tailored for blind users:
- Local Screen Recording (
Control+V): You can now record silent videos directly from your screen. The AI will analyze the recorded segment and provide a highly detailed description of the scene, layout, and actions. - Audio Description Generation (SRT): The add-on can now generate highly detailed Audio Description scripts (in standard SRT format) for videos, complete with smart gap-timing to intelligently anchor descriptions to natural pauses in the audio track, and verbatim OCR for any on-screen text.
- Synchronized Audio Narration (MP3 Export): Beyond text-based subtitles, the add-on can synthesize the Audio Description into speech, automatically mix it with the video's original audio track, apply audio ducking (lowering background volume during descriptions), and export the final synchronized result as an MP3 file!
- Smart Video File Action: If you focus on a local video file and press the video shortcut, the add-on will automatically detect it and process the file directly.
- Advanced Character Tracking: The AI now performs a character extraction pre-pass. It builds a global character dictionary and tracks characters accurately segment-by-segment without confusing identities.
- Video Analysis Configuration: Added new settings to control SRT chunk sizes, character subtitling, and disclaimers.
- Extended Model Routing: You can now explicitly select specialized video models (
gemini_video_model,custom_video_model) in the Advanced Model Routing settings.
- Local Screen Recording (
- Smart API Quota Management: Enhanced handling of 429 (Daily Limit) errors by tracking quotas per-model. If a key hits its daily limit on one model, it is intelligently quarantined for that specific model only, leaving the key available for use with other models.
- Resuming Unfinished Scans: Added a resume feature for both the Document Reader and Smart File Actions. If a scan gets interrupted, you can now continue from where it stopped instead of starting over from scratch.
- New
[screen_fg_obj]Variable: Added a custom prompt variable to capture a screenshot of only the active foreground window rather than the entire screen. - Smart Retries & Key Rotation: The addon now silently retries up to 5 times on the same key when hitting temporary server overloads (like "high demand" or malformed responses). If the retries fail, it automatically switches to the next API key in your list.
- Screen Curtain Detection: Added a check to prevent taking screenshots when the Screen Curtain is active (whether permanently enabled or toggled temporarily with the hotkey). It will warn you and stop, preventing you from sending black images and wasting API tokens.
- Document Reader Tweaks: The PDF range dialog now automatically pre-selects the default target language from your addon settings. Also improved thread handling to make sure background tasks stop cleanly when the reader is closed.
- Native Mistral OCR Integration: Integrated Mistral's native Document OCR API. Multi-page documents are automatically merged, uploaded, and processed in batches using Mistral's specialized
/v1/ocrendpoint, while single-page images are processed directly without unnecessary PDF conversions [1]. - Dynamic Custom URL Handlers: Modifying the Custom API URL now instantly clears the cached model list and restores the manual model entry text box. This ensures full compatibility with custom endpoints (such as Cloudflare AI Gateway) that do not support the standard
/v1/modelslisting endpoint. - Overhauled AI Operator Input Engine: Completely rewritten the underlying mouse and keyboard simulation system for the AI Operator. Replaced the legacy
mouse_eventAPI with the modern WindowsSendInputAPI, bringing significantly higher compatibility with modern applications, UAC-protected windows, and high-DPI displays. - Fixed Drag & Drop Operations: Drag and drop actions in the AI Operator are now fully stable and reliable. The new engine uses natural "easing" curves, precise cursor positioning, optimized timing, and a smart "nudge" technique to ensure that Windows and applications correctly recognize and execute drag-and-drop gestures without failing mid-way.
- Multi-Monitor Support: The AI Operator now fully supports multi-monitor setups. Mouse movements and clicks work correctly across all monitors using the
MOUSEEVENTF_VIRTUALDESKflag, ensuring accurate positioning regardless of which monitor the target application is on. - Enhanced Keyboard Simulation: Improved keystroke injection to fully support "Extended Keys" (such as Arrow keys, Home, End, Page Up/Down, Insert, Delete, and F1-F12). This ensures that navigation and shortcut commands sent by the AI Operator work flawlessly across all applications.
- HEIC/HEIF Image Support: Added native support for iPhone photo formats. You can now directly select
.heicand.heiffiles for AI description, OCR, or Document Reading without prior conversion.
- Live Assistant: Added a real-time voice and screen assistant feature, available exclusively for the Google Gemini provider (or Gemini-compatible custom providers). Includes interactive voice and thinking depth customization directly inside the dialog, with automatic reconnection upon changing settings.
- MiniMax AI Provider: Integrated MiniMax as a peer provider with full multimodal support (chat, vision, OCR), custom TTS using over 300+ dynamic voices, and automatic stripping of reasoning blocks (e.g.,
<think>...</think>) from outputs. - Document Viewer Translation: Corrected a silent translation failure for non-English NVDA users by ensuring the standard 2-letter language code is sent to Google Translate instead of the localized language name.
- PDF Batch Scan Retry: Implemented a highly optimized, separate, and silent retry logic for PDF document batch scanning to prevent redundant uploads and avoid disruptive error popups during retries.
- Document Viewer Status: Fixed a bug where the plugin's overall status (checked via
I) remained stuck on "Batch Processing Started" during long document scans. - Resolved Threading Crash: Fixed a severe
IsMain() failed in wxTimerImplthread assertion crash when opening documents from a background thread by transitioning the GUI callback queue towx.CallAfter.
- Duplicate Label Pre-Check: Fixed an issue in single labeling where the duplicate check used old coordinate keys, causing NVDA to make duplicate AI requests for already labeled objects instead of announcing the existing label.
- Document Chat for Non-Gemini Providers: Fixed a strict API key check in Document Chat (
on_ask) to ensure that users on OpenAI, Groq, or local Custom providers (like Ollama) can successfully chat with documents without being blocked. - Fast Chrome OCR Translation: Restored the free, keyless translation API for Chrome OCR. Translating extracted text now bypasses Gemini AI, saving API quotas and speeding up the translation process.
- CAPTCHA Alphanumeric Filter: Corrected the filtering logic in the CAPTCHA solver to ensure non-alphanumeric characters are properly cleaned in all situations.
- Command Layer Help Update: Corrected the status announcement shortcut in the help menu from
LtoI, and added both labeling commands (LandShift+L) to the list.
- Gemma 4 Thinking Output Fix: Fixed an issue with Gemma 4 models where the entire internal thought process was displayed as the final response, or where disabling thinking resulted in empty responses. The add-on now correctly isolates and extracts only the final clean text response.
- Batch OCR from File Explorer: You can now select multiple photos or PDFs directly in Windows File Explorer and extract text or analyze them in batch. The add-on will automatically filter and process only the supported file formats.
- Universal Local AI Integration (Setup Local AI): Added a new "Setup Local AI" button in Custom Provider Settings. Users can now automatically configure local AI engines, including Ollama, LM Studio, Jan.ai, and KoboldCPP instantly.
- Intelligent Local Proxy Bypass: Rebuilt the connection logic with an advanced proxy bypass mechanism. The add-on is now smart enough to completely bypass Windows system proxies for local loopback connections, ensuring stable local AI connections even when your VPN/TUN-mode is active.
- Ultra-Stable AI Labeling (v2): Replaced absolute screen coordinate keys with an advanced, hybrid Object Signature system. Labels now rely on programmatic identifiers (UIA AutomationId or Win32 ControlID) and window-relative coordinates, making your custom labels completely resistant to window resizing, moving, monitor switching, or scaling.
- Seamless Automatic Label Migration: Upgrading is completely transparent. The add-on will automatically migrate your older legacy coordinate-based labels to the new stable fingerprint format in the background upon first focus, with zero data loss.
- Introducing Semantic AI Labeling: Users can now permanently label unnamed buttons and icons using AI. Press L to label the current navigator object (supporting both Tab focus and object navigation) or Shift+L to scan and label the entire application at once.
- Intelligent Label Management: Added a new, fully accessible Label Manager dialog (via Shift+L if labels exist) to view, rename, or batch-delete custom labels.
- Direct File Analysis (Bypass File Dialog): The add-on is now smart enough to detect if you are currently focusing on a PDF or image file in Windows File Explorer. Pressing F (Smart File Action) or D (Document Reader) on a highlighted file will immediately process it, bypassing the standard "Open" dialog entirely.
- Added "None (Extract Text Layer)" OCR Engine: Users can now extract text directly from searchable PDFs without using AI credits, significantly improving speed and privacy for text-based documents.
- Refined UI Explorer Accuracy: Improved the UI Explorer prompt to better identify element types (like List Items) and accurately report states such as "(Checked)", "(Selected)", or "(Expanded)" while ignoring Windows system components like the Taskbar and Clock.
- Installation Setup Reminder: Added a notification after installation to guide users to the settings menu for configuring their API keys and preferences.
- Fixed AI Operator Typing Issue: Resolved a bug where the letter 'v' was typed instead of pasting text on certain systems. This fix addresses timing conflicts that occurred during high system load.
- Enhanced Stability: Added robust error handling for clipboard operations to prevent addon crashes when the system clipboard is temporarily locked by other applications.
- Timing Optimization: Adjusted internal delays for keyboard events to ensure higher reliability across different system speeds and better compatibility with third-party Clipboard Managers.
- AI Operator (Autonomous Control - Shift+A): This is the crown jewel of v5.5. Vision Assistant Pro has graduated from being a passive assistant to becoming your personal AI Operator. It doesn't just describe the screen—it takes command.
- How it works: You can now give verbal instructions to operate your PC. For example, in a completely inaccessible application where your screen reader stays silent, you can press Shift+A and type: "Click on the Settings button" or "Find the search field, type 'Latest News' and press enter." The AI visually identifies the elements, moves the mouse, and executes the task for you.
- Performance Note: This feature is optimized for Gemini 3.0 Flash (Preview), delivering incredibly fast and intelligent responses that can handle even the most complex UI layouts.
⚠️ API Usage Warning: Because the AI Operator needs to "see" exactly what's happening to be accurate, it sends a high-resolution screenshot with every step. Please note that frequent use will consume your API quota much faster than standard text-based tasks.
- Visual UI Explorer (E): Tired of navigating through "unlabeled buttons"? Press E to activate the UI Explorer. The AI will scan the entire window and generate a list of every clickable element it sees—including icons, graphics, and menus. Simply pick an item from the list, and the AI Operator will click it for you. It’s like having an "accessible layer" on top of any app.
- Context-Aware Smart File Action (F): The "F" key has been completely overhauled. It no longer assumes you only want OCR. When you select a single image, it now intelligently asks for your intent: you can choose a Detailed Visual Description to understand the scene or a Structured Text Extraction (OCR) for reading. The menu adapts dynamically based on the file type and your active AI engine.
- Core Optimization: We've performed a deep cleanup of the add-on’s internal logic, removing unused legacy functions and redundant code. This results in a leaner, faster, and more reliable experience for all users.
- Multi-Provider Architecture: Added full support for OpenAI, Groq, and Mistral alongside Google Gemini. Users can now choose their preferred AI backend.
- Advanced Model Routing: Users of native providers (Gemini, OpenAI, etc.) can now select specific models from a dropdown list for different tasks (OCR, STT, TTS).
- Advanced Endpoint Configuration: Custom provider users can manually input specific URLs and model names for granular control over local or third-party servers.
- Smart Feature Visibility: The settings menu and Document Reader UI now automatically hide unsupported features (like TTS) based on the selected provider.
- Dynamic Model Fetching: The addon now fetches the available model list directly from the provider's API, ensuring compatibility with new models as soon as they are released.
- Hybrid OCR & Translation: Optimized the logic to use Google Translate for speed when using Chrome OCR, and AI-powered translation when using Gemini/Groq/OpenAI engines.
- Universal "Re-scan with AI": The Document Reader's re-scan feature is no longer limited to Gemini. It now utilizes whatever AI provider is currently active to re-process pages.
- Interactive Result Recall: Added the Space key to the command layer, allowing users to instantly reopen the last AI response in a chat window for follow-up questions, even when "Direct Output" mode is active.
- Telegram Community Hub: Added an "Official Telegram Channel" link to the NVDA Tools menu, providing a quick way to stay updated with the latest news, features, and releases.
- Enhanced Response Stability: Optimized the core logic for Translation, OCR, and Vision features to ensure more reliable performance and a smoother experience when using direct speech output.
- Improved Interface Guidance: Updated the settings descriptions and documentation to better explain the new recall system and how it works alongside the direct output settings.
- Advanced Prompt Manager: Introduced a dedicated management dialog in settings to customize default system prompts and manage user-defined prompts with full support for adding, editing, reordering, and previewing.
- Comprehensive Proxy Support: Resolved network connectivity issues by ensuring that user-configured proxy settings are strictly applied to all API requests, including translation, OCR, and speech generation.
- Automated Data Migration: Integrated a smart migration system to automatically upgrade legacy prompt configurations to a robust v2 JSON format upon the first run without data loss.
- Updated Compatibility (2025.1): Set the minimum required NVDA version to 2025.1 due to library dependencies in advanced features like the Document Reader to ensure stable performance.
- Optimized Settings Interface: Streamlined the settings interface by reorganizing prompt management into a separate dialog, providing a cleaner and more accessible user experience.
- Prompt Variables Guide: Added a built-in guide within the prompt dialogs to help users easily identify and use dynamic variables such as [selection], [clipboard], and [screen_obj].
- Enhanced Network Resilience: Added an automatic retry mechanism to better handle unstable internet connections and temporary server errors, ensuring more reliable AI responses.
- Visual Translation Dialog: Introduced a dedicated window for translation results. Users can now easily navigate and read long translations line-by-line, similar to OCR results.
- Aggregated Formatted View: The "View Formatted" feature in the Document Reader now displays all processed pages in a single, organized window with clear page headers.
- Optimized OCR Workflow: Automatically skips the page range selection for single-page documents, making the recognition process faster and more seamless.
- Improved API Stability: Switched to a more robust header-based authentication method, resolving potential "All API Keys failed" errors caused by key rotation conflicts.
- Bug Fixes: Resolved several potential crashes, including an issue during add-on termination and a focus error in the chat dialog.
- Advanced Document Reader: A powerful new viewer for PDF and images with page range selection, background processing, and seamless
Ctrl+PageUp/Downnavigation. - New Tools Submenu: Added a dedicated "Vision Assistant" submenu under NVDA's Tools menu for quicker access to core features, settings, and documentation.
- Flexible Customization: You can now choose your preferred OCR engine and TTS voice directly from the settings panel.
- Multiple API Key Support: Added support for multiple Gemini API keys. You can enter one key per line or separate them with commas in the settings.
- Alternative OCR Engine: Introduced a new OCR engine to ensure reliable text recognition even when hitting Gemini API quota limits.
- Smart API Key Rotation: Automatically switches to and remembers the fastest working API key to bypass quota limits.
- Document to MP3/WAV: Integrated capability to generate and save high-quality audio files in both MP3 (128kbps) and WAV formats directly within the reader.
- Instagram Stories Support: Added the ability to describe and analyze Instagram Stories using their URLs.
- TikTok Support: Introduced support for TikTok videos, allowing for full visual description and audio transcription of clips.
- Redesigned Update Dialog: Features a new accessible interface with a scrollable text box to clearly read version changes before installing.
- Unified Status & UX: Standardized file dialogs across the add-on and enhanced the 'L' command to report real-time progress.
- Help System: Added a help command (
H) within the Command Layer to provide an easy-to-access list of all shortcuts and their functions. - Online Video Analysis: Expanded support to include Twitter (X) videos. Also improved URL detection and stability for a more reliable experience.
- Project Contribution: Added an optional donation dialog for users who wish to support the project’s future updates and continuous growth.
* **Command Layer:** Introduced a Command Layer system (default: NVDA+Shift+V) to group shortcuts under a single master key. For example, instead of pressing NVDA+Control+Shift+T for translation, you now press NVDA+Shift+V followed by T.
* **Online Video Analysis:** Added a new feature to analyze YouTube and Instagram videos directly by providing a URL.
- Direct Output Mode: Added an option to skip the chat dialog and hear AI responses directly via speech for a faster and more seamless experience.
- Clipboard Integration: Added a new setting to automatically copy AI responses to the clipboard.
- New Languages: Added Persian and Vietnamese translations.
- Expanded AI Models: Reorganized the model selection list with clear prefixes (
[Free],[Pro],[Auto]) to help users distinguish between free and rate-limited (paid) models. Added support for Gemini 3.0 Pro and Gemini 2.0 Flash Lite. - Dictation Stability: Significantly improved Smart Dictation stability. Added a safety check to ignore audio clips shorter than 1 second, preventing AI hallucinations and empty errors.
- File Handling: Fixed an issue where uploading files with non-English names would fail.
- Prompt Optimization: Improved Translation logic and structured Vision results.
- Added French and Turkish translations.
- Formatted View: Added a "View Formatted" button in chat dialogs to view the conversation with proper styling (Headings, Bold, Code) in a standard browseable window.
- Markdown Setting: Added a new option "Clean Markdown in Chat" in Settings. Unchecking this allows users to see raw Markdown syntax (e.g.,
**,#) in the chat window. - Dialog Management: Fixed an issue where the "Refine Text" or chat windows would open multiple times or fail to focus correctly.
- UX Improvements: Standardized file dialog titles to "Open" and removed redundant speech announcements (e.g., "Opening menu...") for a smoother experience.
- Added Italian translation.
- Status Reporting: Added a new command (NVDA+Control+Shift+I) to announce the current status of the add-on (e.g., "Uploading...", "Analyzing...").
- HTML Export: The "Save Content" button in result dialogs now saves output as a formatted HTML file, preserving styles like headings and bold text.
- Settings UI: Improved the Settings panel layout with accessible grouping.
- New Models: Added support for gemini-flash-latest and gemini-flash-lite-latest.
- Languages: Added Nepali to supported languages.
- Refine Menu Logic: Fixed a critical bug where "Refine Text" commands would fail if the NVDA interface language was not English.
- Dictation: Improved silence detection to prevent incorrect text output when no speech is input.
- Update Settings: "Check for updates on startup" is now disabled by default to comply with Add-on Store policies.
- Code Cleanup.
- Migrated project structure to the official NV Access Add-on Template for better standards compliance.
- Implemented automatic retry logic for HTTP 429 (Rate Limit) errors to ensure reliability during high traffic.
- Optimized translation prompts for higher accuracy and better "Smart Swap" logic handling.
- Updated Russian translation.
- Added Russian translation support (Thanks to nvda-ru).
- Updated error messages to provide more descriptive feedback regarding connectivity.
- Changed default target language to English.
- Added Native File OCR Command (NVDA+Control+Shift+F).
- Added "Save Chat" button to result dialogs.
- Implemented full localization support (i18n).
- Migrated audio feedback to NVDA's native tones module.
- Switched to Gemini File API for better handling of PDF and audio files.
- Fixed crash when translating text containing curly braces.
- Fixed an issue where the [file_ocr] variable was not functioning correctly within Custom Prompts.
- Standardized all shortcuts to use NVDA+Control+Shift to eliminate conflicts with NVDA's Laptop layout and system hotkeys.
- Implemented built-in Auto-Update system.
- Added Smart Translation Cache for instant retrieval of previously translated text.
- Added Conversation Memory to contextually refine results in chat dialogs.
- Added Dedicated Clipboard Translation command (NVDA+Control+Shift+Y).
- Optimized AI prompts to strictly enforce target language output.
- Fixed crash caused by special characters in input text.
- Added support for over 20 new languages.
- Implemented Interactive Refine Dialog for follow-up questions.
- Added Native Smart Dictation feature.
- Added "Vision Assistant" category to NVDA's Input Gestures dialog.
- Fixed COMError crashes in specific applications like Firefox and Word.
- Added automatic retry mechanism for server errors.
- Initial release.