Build Imagyx with virtualized browsing and local AI

[ExtraBinoss]

What changed

• scaffolded a Tauri 2 desktop application with Vue, Pinia, TypeScript 7.0.2 and Vite 8
• added a light-first Apple/SaaS interface with a neutral dark theme and no glow effects
• added reusable UI primitives under src/components/ui
• added followed-folder management, SQLite WAL storage and hybrid filename/semantic search
• added offline CLIP image/text embeddings through Transformers.js/WebGPU with WASM fallback
• added automatic model preparation with exact download progress in bytes, MB and percentage

Rust architecture cleanup

The Rust backend is now split into focused, small modules instead of large commands.rs, db.rs and indexer.rs files:

src-tauri/src/
├── commands/       # thin Tauri adapters grouped by domain
├── database/       # connection, migrations, assets, folders, embeddings and queries
├── fuzzy/          # query normalization and ranking helpers
├── indexer/        # scan, file metadata, progress and hybrid search
├── tracing/        # debug-only spans, events and p50/p95 helpers
├── vector_store/   # contiguous vectors, incremental upserts and bounded top-K
├── state.rs
└── lib.rs

Implementation files are kept below 200 lines; tests that would make a module too large live in dedicated test submodules.

Search performance

• added SQLite FTS5 with Unicode tokenization, diacritic removal and prefix indexes
• added a versioned FTS migration with a one-time rebuild for existing image rows
• returns lexical FTS results immediately while the text encoder prepares the semantic query
• replaces the first lexical result set with hybrid results when the semantic vector is ready
• retrieves only 200–500 lexical candidates and 200 semantic candidates before reranking
• uses Reciprocal Rank Fusion instead of sorting every image by a global weighted score
• caps non-empty searches at 60 results
• bounds empty-library browsing directly in SQL instead of loading every row and truncating afterward
• uses dedicated queries for requested IDs, pending embeddings and folder fingerprints

Vector search and indexing

• replaced RwLock<Vec<VectorEntry>> with an incremental VectorStore
• stores normalized vectors in one contiguous Vec<f32> for better cache locality
• upserts only the embeddings from the latest batch instead of reloading all vectors from SQLite
• removes deleted, invalidated or removed-folder vectors incrementally
• uses bounded heaps with Rayon for top-K search instead of allocating a score map and sorting all vectors
• loads vectors outside the critical startup path, so lexical search is available first
• queries images missing embeddings directly with SQL
• removes absent files in one transaction using a temporary path table and DELETE ... RETURNING

Development tracing

• tracing lives under src-tauri/src/tracing/
• debug builds record named spans and bounded p50/p95 samples
• release builds compile against no-op helpers
• tracing is enabled only through debug_assertions, without adding a runtime dependency or release logging cost
• startup vector loading, folder indexing and hybrid search have dedicated spans

Tests added

Unit tests cover:

• vector blob encoding/decoding
• FTS prefix search
• migration rebuild of historical rows
• pending-embedding queries
• bounded recent-image queries
• fuzzy normalization and FTS query construction
• reciprocal-rank helpers
• contiguous vector upsert, filtering, removal and top-K
• tracing percentile helpers and bounded samples
• supported image discovery helpers

Performance plan

The final architecture, budgets and next measurement-driven phases are documented in:

• docs/RUST_PERFORMANCE_PLAN.md

The next large steps are intentionally not mixed into this refactor: reproducible 10k/50k/100k benchmarks, a bounded multi-stage indexing pipeline, event-level watcher commands, persistent jobs, a persisted vector snapshot, HNSW only if required by p95 measurements, and native Rust ONNX inference as a separate packaging project.

Existing browsing and filesystem work

• virtualized the image grid so only visible rows plus overscan rows are mounted
• added on-demand thumbnail generation with bounded concurrency and memory/disk caches
• added native recursive filesystem watching with debouncing
• keeps original images untouched and ignores application-owned cache/model/database paths
• saves metadata before semantic analysis so images become visible immediately

Validation status

The refactor and unit tests are committed to the existing draft PR branch. No GitHub Actions workflow was triggered because this repository uses manual workflow_dispatch, and this environment could not obtain a local checkout to execute Cargo or npm commands. This PR therefore does not claim that the latest head has compiled or passed tests.

Please validate the latest head locally with:

npm install
npm run typecheck
npm test
npm run build
cargo fmt --manifest-path src-tauri/Cargo.toml --all -- --check
cargo test --manifest-path src-tauri/Cargo.toml
cargo clippy --manifest-path src-tauri/Cargo.toml --all-targets
cargo check --manifest-path src-tauri/Cargo.toml
npm run tauri dev

Recommended manual smoke tests:

1. open the application with an existing database and confirm the FTS migration completes once;
2. type a query before the text model is warm and confirm lexical results appear first;
3. confirm results rerank after semantic encoding completes;
4. index a small batch and verify vectors update without a full reload;
5. remove or modify images and confirm stale semantic matches disappear;
6. compare debug and release builds to confirm trace output exists only in development.

Notes

• original image files are never modified
• file auto-renaming is deliberately not enabled in this prototype
• actual WebGPU/WASM behavior still needs smoke testing on physical Windows and macOS hardware before release

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 752e425548

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Show onboarding on the first production launch

On a fresh production install, the main window starts hidden and lib.rs only shows it under debug_assertions; meanwhile onboarding.initialize() opens the first-run dialog inside that hidden window. The user therefore sees neither the required onboarding nor an obvious application window unless they discover the tray menu, so startup should reveal the main window when onboarding has not been completed.

AGENTS.md reference: AGENTS.md:L85-L85

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Load the Spotlight-selected image directly by ID

When Spotlight opens an older search result in Imagyx, this first refreshes the blank-query browse view, which contains only the 48 newest images, and then searches that page for the requested ID. Any selected result outside that first page produces null, so the main window opens without the requested preview; retrieve the image by ID or pass the selected asset instead of relying on the current browse page.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reconcile followed folders when the watcher starts

After files are added, changed, or deleted while Imagyx is not running, startup merely registers each existing folder with the native watcher. Filesystem watchers do not replay events that occurred while the process was stopped, so the SQLite library and search results remain stale until another event happens or the user manually reindexes; enqueue an initial index_folder pass after each watch is installed.

AGENTS.md reference: AGENTS.md:L64-L64

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Continue indexing after the 20,000-image pending page

For a library with more than 20,000 missing embeddings, pendingImages() returns only the Rust-side MAX_PENDING_IMAGES page, but this method processes that single array and marks the runtime ready without fetching another page. The remaining images stay semantically unindexed until the user manually presses Resume—potentially repeatedly—so completion should request subsequent pending pages until none remain.

AGENTS.md reference: AGENTS.md:L63-L64

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject overlapping followed-folder roots

When a user follows both a directory and one of its descendants, this check accepts both because it only detects exact path equality. Since the schema permits each image path to have only one folder_id, scans of the two roots repeatedly move descendant images between folders, making folder-filtered results unstable and allowing removal of one followed folder to delete images still covered by the other; reject ancestor/descendant overlaps or model folder membership separately.

Useful? React with 👍 / 👎.