Skip to content

feat: show current file in ingest progress + slow file warnings #183 #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
four-bytes-robby merged 1 commit into from
Jun 23, 2026
Merged

feat: show current file in ingest progress + slow file warnings #183 #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions src/ingest/chunker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,8 @@ const MAX_TOKENS_PER_CHUNK = 1024;
const TOKEN_WINDOW = 512;
const TOKEN_OVERLAP = 77; // ~15 %

/** Maximum file size for tree-sitter symbol extraction (200KB).
* Files larger than this skip symbol extraction entirely and go straight
* to windowed chunking, preventing event-loop blocks from sync WASM parse. */
// Tree-sitter parses synchronously in WASM — large files block the event loop.
// Skip symbol extraction for files exceeding this threshold; use windowed chunking instead.
const SYMBOL_EXTRACTION_MAX_CHARS = 200 * 1024; // 200 KB

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -454,12 +453,16 @@ export async function chunkContent(input: ChunkInput): Promise<ChunkResult> {
},
];
} else {
// Code files: try symbol extraction
// Code files: try symbol extraction (only for files small enough that
// synchronous WASM parsing won't block the event loop for too long).
if (
language === "typescript" ||
language === "javascript" ||
language === "php" ||
language === "rust"
content.length <= SYMBOL_EXTRACTION_MAX_CHARS &&
(
language === "typescript" ||
language === "javascript" ||
language === "php" ||
language === "rust"
)
) {
// Skip tree-sitter for large files — sync WASM parse blocks the event loop
if (content.length > SYMBOL_EXTRACTION_MAX_CHARS) {
Expand All @@ -478,7 +481,7 @@ export async function chunkContent(input: ChunkInput): Promise<ChunkResult> {
}
}
} else {
// Large files without symbols: sliding windows
// Large files or non-symbol languages: sliding windows
chunks = await fallbackChunk(content, documentId, fileId, totalTokenCount);
}
}
Expand Down
43 changes: 34 additions & 9 deletions src/ingest/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,21 @@ export interface IngestResult {
durationMs: number;
}

export interface ProgressUpdate {
current: number;
total: number;
currentFile?: string;
currentFileSize?: number;
}

export interface IngestOptions {
recursive?: boolean;
reIndex?: boolean;
/** Project path for project_hash tagging on documents and symbols. */
project?: string;
/** Called after each file chunk+embed for progress reporting.
* Receives { current, total } — current is 0-based, total is filesFound. */
progressCallback?: (progress: { current: number; total: number }) => void;
/** Called after each file for progress reporting.
* Receives { current, total, currentFile?, currentFileSize? }. */
progressCallback?: (update: ProgressUpdate) => void;
}

// ---------------------------------------------------------------------------
Expand All @@ -66,7 +73,7 @@ const MAX_FILE_SIZE = 2 * 1024 * 1024; // 2 MB
const FILE_TIMEOUT_MS = 30_000; // 30 seconds per file

/** Files exceeding this duration get logged to app.log as a warning. */
const SLOW_FILE_WARN_MS = 30_000; // 30 seconds
const SLOW_FILE_WARN_MS = 10_000; // 10 seconds

// ---------------------------------------------------------------------------
// Progress event helpers (gated on BRAIN_DEBUG=true)
Expand Down Expand Up @@ -190,12 +197,20 @@ export async function ingestPath(
}

if (fileStats.size > MAX_FILE_SIZE) {
result.errors.push(
`Skipped ${filePath}: file size ${fileStats.size} exceeds 2MB cap`,
);
const msg = `Skipped (too large, ${(fileStats.size / 1024 / 1024).toFixed(1)}MB): ${filePath}`;
log("info", "ingest", msg);
result.errors.push(msg);
return;
}
Comment on lines +200 to 204

@coderabbitai coderabbitai Bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Missing progress callback on early return.

When a file is skipped due to size, the progress callback at lines 207-212 reports currentFile: filePath, but the early return at line 203 never calls the callback with currentFile: undefined. Consumers tracking "which file is currently being processed" will incorrectly believe this file is still in progress.

The same issue occurs for binary-content skips at line 228.

🔧 Recommended fix

Call the progress callback before returning:

         if (fileStats.size > MAX_FILE_SIZE) {
           const msg = `Skipped (too large, ${(fileStats.size / 1024 / 1024).toFixed(1)}MB): ${filePath}`;
           log("info", "ingest", msg);
           result.errors.push(msg);
+          options?.progressCallback?.({
+            current: i + 1,
+            total: walkedFiles.length,
+            currentFile: undefined,
+          });
           return;
         }

Apply the same fix at line 228 for binary-content skips.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const msg = `Skipped (too large, ${(fileStats.size / 1024 / 1024).toFixed(1)}MB): ${filePath}`;
log("info", "ingest", msg);
result.errors.push(msg);
return;
}
const msg = `Skipped (too large, ${(fileStats.size / 1024 / 1024).toFixed(1)}MB): ${filePath}`;
log("info", "ingest", msg);
result.errors.push(msg);
options?.progressCallback?.({
current: i + 1,
total: walkedFiles.length,
currentFile: undefined,
});
return;
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingest/index.ts` around lines 200 - 204, The early return statements that
skip files due to size constraints and binary content issues do not call the
progress callback to clear the current file context. Before the return statement
in the size check condition (where fileStats.size is compared), add a progress
callback invocation with currentFile set to undefined to signal that the file is
no longer being processed. Apply the same fix before the early return statement
for binary-content skips to ensure consumers correctly track that processing has
stopped for that file.


// Report which file is being processed
options?.progressCallback?.({
current: i + 1,
total: walkedFiles.length,
currentFile: filePath,
currentFileSize: fileStats.size,
});
Comment on lines +207 to +212

@coderabbitai coderabbitai Bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Progress counter inconsistency causes backwards progress reporting.

The current field uses different semantics across callback sites:

  • Line 208: current: i + 1 (file index)
  • Line 251: current: result.filesProcessed (processed count)
  • Line 441: current: result.filesProcessed (processed count)

When files are skipped (oversized, binary, failed), filesProcessed doesn't increment but i keeps advancing. This causes progress to jump forward then backwards.

Example: For 3 files where file 2 is oversized:

  1. File 1: {current: 1, currentFile: "f1"} → {current: 1, currentFile: undefined}
  2. File 2: {current: 2, currentFile: "f2"} → skipped → (no callback)
  3. File 3: {current: 3, currentFile: "f3"} → {current: 2, currentFile: undefined} ← progress goes 3→2!
🔧 Recommended fix

Use i + 1 consistently for the current field to ensure monotonic progress:

         options?.progressCallback?.({
-          current: i + 1,
+          current: i + 1,  // Keep as-is
           total: walkedFiles.length,
           currentFile: filePath,
           currentFileSize: fileStats.size,
         });

And update lines 251 and 441:

-        options?.progressCallback?.({ current: result.filesProcessed, total: walkedFiles.length });
+        options?.progressCallback?.({ current: i + 1, total: walkedFiles.length, currentFile: undefined });
         options?.progressCallback?.({
-          current: result.filesProcessed,
+          current: i + 1,
           total: walkedFiles.length,
           currentFile: undefined,
         });
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingest/index.ts` around lines 207 - 212, The progress callback's current
field uses inconsistent semantics across different callback sites: the
progressCallback at line 207-212 uses i + 1 (file index), while the
progressCallback calls at lines 251 and 441 use result.filesProcessed (processed
count). Since skipped files don't increment filesProcessed but i continues
incrementing, this causes non-monotonic progress values where current can jump
forward then backwards. Fix this by changing both the progressCallback
invocations at lines 251 and 441 to use i + 1 for the current field instead of
result.filesProcessed, ensuring consistent and monotonic progress reporting
across all callback sites.


// Read file as raw buffer — binary-safe hashing (avoids encoding issues)
let buf: ArrayBuffer;
try {
Expand All @@ -208,7 +223,7 @@ export async function ingestPath(

// Binary content guard: skip files with null bytes (safety net for misnamed binaries)
if (isBinaryContent(new Uint8Array(buf))) {
log("debug", "ingest", `Skipped binary content: ${filePath}`);
log("info", "ingest", `Skipped (binary content): ${filePath}`);
result.binarySkipped++;
return;
}
Expand Down Expand Up @@ -317,6 +332,12 @@ export async function ingestPath(
return;
}

// Slow-file warning after chunking completes
const fileElapsed = Date.now() - fileStart;
if (fileElapsed > SLOW_FILE_WARN_MS) {
log("warn", "ingest", `Slow file (${(fileElapsed / 1000).toFixed(1)}s): ${filePath} (${chunks ? chunks.length : 0} chunks)`);
}

// ── 9. Insert chunks (includes token_count) ───────────────────────
// Clean up old chunks for this file before inserting new ones
db.run("DELETE FROM chunks WHERE file_id = ?", [fileId]);
Expand Down Expand Up @@ -416,7 +437,11 @@ export async function ingestPath(

result.filesIndexed++;
result.filesProcessed++;
options?.progressCallback?.({ current: result.filesProcessed, total: walkedFiles.length });
options?.progressCallback?.({
current: result.filesProcessed,
total: walkedFiles.length,
currentFile: undefined,
});

// Per-file timing debug (BRAIN_DEBUG only)
if (process.env.BRAIN_DEBUG === "true") {
Expand Down
2 changes: 1 addition & 1 deletion test/ingest.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,7 +709,7 @@ describe("ingestPath — edge cases (E5.3)", () => {
expect(result.filesFound).toBe(1);
expect(result.filesIndexed).toBe(0);
expect(result.errors.length).toBeGreaterThanOrEqual(1);
expect(result.errors[0]).toContain("exceeds 2MB cap");
expect(result.errors[0]).toContain("Skipped (too large");
});

test("concurrent ingests don't corrupt database", async () => {
Expand Down
Loading