diff --git a/.jules/bolt.md b/.jules/bolt.md index 341c7c91..96ca427f 100644 --- a/.jules/bolt.md +++ b/.jules/bolt.md @@ -67,3 +67,6 @@ ## 2025-02-12 - [Fast Path Execution in Directory Traversal and Log Parsing] **Learning:** Checking for string existence (`if "silence_" not in stderr`) before invoking regex matchers provides significant speed improvements when parsing large blocks of text. Similarly, moving expensive I/O operations like `os.path.realpath` inside conditional blocks prevents redundant disk access when configuration (like path exclusions) isn't utilized. **Action:** When working on large text processing or disk operations, verify if early exit conditions or conditional execution can bypass the expensive system or library calls. +## 2024-10-08 - [Optimize Search Set Intersection and Loops] +**Learning:** In tight loops involving `set`s (like when computing the intersection of matching documents for multi-word search terms), creating a new `set` with the `&` operator allocates memory and iterates needlessly. Modifying a target `set` in place using `.intersection_update()` avoids this allocation. In the same vein, using `sum()` with a generator expression introduces significant overhead due to object allocation of the generator. Replacing it with a simple `for` loop that accumulates in a local variable provides a substantial speedup. +**Action:** Always prefer `intersection_update` over `&` when working with sets you can modify safely. For performance-sensitive loops, replace `sum(a for b in c)` with a basic inline `for` loop to eliminate the generator overhead. diff --git a/transcript_search.py b/transcript_search.py index 1660c1f2..beb304c6 100644 --- a/transcript_search.py +++ b/transcript_search.py @@ -241,16 +241,19 @@ def search(self, query: str) -> list[Match]: postings = self._postings.get(term) if not postings: return [] - candidates = ( - set(postings) if candidates is None else candidates & postings - ) + if candidates is None: + candidates = set(postings) + else: + candidates.intersection_update(postings) if not candidates: return [] matches = [] for position in candidates or (): entry = self._entries[position] - score = sum(entry.counts[term] for term in unique_terms) + score = 0 + for term in unique_terms: + score += entry.counts[term] matches.append( Match( recording_id=entry.recording_id,