Skip to content

perf: Optimize snapshot database queries (fix N+1 issue)#38

Closed
matthewro7263-hub wants to merge 3 commits into
mainfrom
jules-9755194764164504910-28be7c96
Closed

perf: Optimize snapshot database queries (fix N+1 issue)#38
matthewro7263-hub wants to merge 3 commits into
mainfrom
jules-9755194764164504910-28be7c96

Conversation

@matthewro7263-hub
Copy link
Copy Markdown
Owner

💡 What: Replaced the flatMap pattern for fetching storyboard panels with a single inArray query.
🎯 Why: The previous approach executed a separate database query for every single storyboard returned from the project (an N+1 query problem). Fetching all panel rows in a single batch using the IN clause dramatically reduces round-trips and SQLite overhead.
📊 Measured Improvement:
In a benchmark with 100 storyboards (10 panels each, totaling 1000 panels fetched):

  • N+1 (Baseline): ~10.0 - 13.0 ms
  • Optimized inArray: ~4.3 - 4.5 ms
  • Improvement: ~65% reduction in query execution time for this operation.

PR created automatically by Jules for task 9755194764164504910 started by @matthewro7263-hub

… for storyboard panels

This resolves the N+1 issue in snapshot creation, trash integrity scan, and trash item fetching.
@google-labs-jules
Copy link
Copy Markdown

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

This resolves the N+1 issue in snapshot creation, trash integrity scan, and trash item fetching.
@matthewro7263-hub
Copy link
Copy Markdown
Owner Author

Review notes before merge:

  1. Rebase onto current main to clear the merge conflict.
  2. PR description references SQLite + better-sqlite3 overhead, but the repo has since migrated to Postgres (Drizzle). Please re-verify the inArray query plan under Postgres and re-run the 1000-panel benchmark on the new backend — IN (...) semantics differ slightly and very large arrays may need chunking.
  3. Add/refresh a unit test that exercises the batched fetch path with >100 panels to lock in the perf improvement.
  4. Confirm no callers still depend on the old flatMap shape.

Aligned with keep-it-free goal: ✅ pure perf win, reduces server cost.

This resolves the N+1 issue in snapshot creation, trash integrity scan, and trash item fetching.
@matthewro7263-hub matthewro7263-hub temporarily deployed to jules-9755194764164504910-28be7c96 - cel-source PR #38 May 16, 2026 03:55 — with Render Destroyed
@matthewro7263-hub
Copy link
Copy Markdown
Owner Author

Going to close this one out — it's got merge conflicts against current main, and the same N+1 area was already addressed by the more recent #62 (snapshot panel loading) and the still-open #65 (project archive export). Rather than rebasing 3-week-old changes, suggest closing this and letting the newer PRs cover it. Nice benchmarks though.

@matthewro7263-hub
Copy link
Copy Markdown
Owner Author

Closing as duplicate — the same N+1 fix landed in #71.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant