feat: enhanced binary detection with full-content scanning and per-chunk validation #178

[four-bytes-robby]

Summary

Enhanced binary detection during ingest to prevent binary content from being indexed. The previous 256-byte initial scan was insufficient for larger files.

Changes

1. Enhanced isBinaryContent() in src/ingest/loader.ts

• Small files (≤64 KB): scans the entire file content instead of just 256 bytes
• Large files (>64 KB): samples 3 regions — first 4KB, middle 4KB (at size/2), last 4KB — marks as binary if any region exceeds threshold
• Null-byte check scans across all sampled regions (or entire file for small files)
• Added isBinaryRegion() helper for reusable region scanning
• Renamed BINARY_SCAN_BYTES → FULL_SCAN_SIZE (64KB threshold)

2. Added per-chunk binary validation in src/ingest/chunker.ts

• New isBinaryChunk() helper that validates decoded text for null bytes or >30% non-printable chars
• Post-processing filter in chunkContent() catches binary chunks regardless of which sub-function produced them
• Binary chunks are skipped with a warning log, remaining chunks continue processing
• chunkContent() now returns ChunkResult { chunks, binarySkipped } for caller visibility

3. Added binarySkipped to IngestResult in src/ingest/index.ts

• New field tracks both file-level (whole file binary) and chunk-level (individual binary chunks) skips
• Incremented when isBinaryContent() returns true for a file
• Accumulated from chunkContent() result for per-chunk binary skips

4. Added tests in test/ingest.test.ts

• isBinaryContent test block: plain text, null byte at position 300, large file with binary middle, large file with binary end, >30% non-printable, empty file, UTF-8 BOM, control chars
• isBinaryChunk test block: normal text, embedded null byte, >30% non-printable, empty chunk, whitespace-only

Testing

• All 31 ingest tests pass
• All 139 related tests pass (across 5 test files)
• Lint clean on all modified files

---

Summary by cubic

Improved binary detection during ingest by scanning full small files, sampling large files, and validating each produced chunk. This prevents binaries from being indexed and records how many were skipped.

• New Features

  • isBinaryContent now fully scans files ≤64KB and samples first/middle/last 4KB for larger files; flags null bytes or >30% non‑printable chars.
  • Added per-chunk validation via isBinaryChunk; binary chunks are dropped with a warn log and processing continues.
  • chunkContent now returns ChunkResult { chunks, binarySkipped } to expose skipped chunk count.
  • IngestResult includes binarySkipped, covering both file-level and chunk-level skips; tests added for edge cases.

• Migration

  • Update callers of chunkContent(...) to handle const { chunks, binarySkipped } = await chunkContent(...);.
  • If you surface ingest metrics, include IngestResult.binarySkipped.

^{Written for commit 0eb0280. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes

  • Improved binary content detection during file ingestion with adaptive sampling algorithms for large files.
  • Binary files are now properly identified and skipped with detailed tracking in ingestion statistics.
  • Enhanced robustness for processing files of varying sizes.

• Tests

  • Added comprehensive test coverage for binary detection across various byte patterns and edge cases.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Binary content detection is enhanced at two levels: isBinaryContent in loader.ts switches from a fixed prefix scan to a size-aware strategy that fully scans small files and samples three regions of large files. chunkContent in chunker.ts gains a post-production binary filter via a new isBinaryChunk helper and now returns a ChunkResult carrying the filtered chunks plus a binarySkipped count. The ingest pipeline in index.ts propagates this count through IngestResult. Unit tests cover both detection functions.

Changes

Binary Detection Hardening and binarySkipped Reporting

Layer / File(s)	Summary
Size-aware isBinaryContent in loader src/ingest/loader.ts	Adds FULL_SCAN_SIZE (64KB) and SAMPLE_SIZE (4KB) constants. Introduces isBinaryRegion helper. Rewrites isBinaryContent to return false for empty buffers, fully scan buffers ≤64KB, and sample beginning/middle/end regions for larger buffers; null bytes trigger immediate binary classification.
ChunkResult type and per-chunk binary filtering src/ingest/chunker.ts	Exports ChunkResult interface (chunks, binarySkipped), adds isBinaryChunk using null-byte and non-printable-ratio detection, and refactors chunkContent to collect all chunks then filter through isBinaryChunk, emitting log warnings and returning the structured result.
binarySkipped counter wired through ingest pipeline src/ingest/index.ts	Adds binarySkipped: number to IngestResult, initializes it to 0, increments it on the pre-read binary guard, and accumulates chunkResult.binarySkipped from the updated chunkContent return.
Unit tests for isBinaryContent and isBinaryChunk test/ingest.test.ts	Adds E5.4 suite testing isBinaryContent across text, null bytes, large buffers, non-printable thresholds, empty input, UTF-8 BOM, and whitespace. Adds E5.5 suite with a local isBinaryChunk reimplementation and tests for the same edge cases.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• [FEAT] Enhanced binary detection: full-content scanning + per-chunk validation #178: This PR directly implements the solution described in that issue: size-aware binary scanning with FULL_SCAN_SIZE/SAMPLE_SIZE, per-chunk binary validation via isBinaryChunk, and binarySkipped reporting in IngestResult.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and accurately describes the main changes: enhanced binary detection with both full-content scanning and per-chunk validation.
Description check	✅ Passed	The PR description comprehensively covers all changes with detailed explanations of the implementation, includes testing results, but is missing the required template sections (Related Issue, Conventional commit format, and some checklist items).
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/178-binary-detection-enhancement

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch feat/178-binary-detection-enhancement

---

_{Comment @coderabbitai help to get the list of available commands.}