fix(fff-search): keep filename_offset on a char boundary for non-UTF-8 names

[kevin9327]

Fixes #799.

The crash

Scanning a directory that contains a filename with invalid UTF-8 bytes aborts the process from the Rayon scan thread:

thread 'fff-bg-0' panicked at crates/fff-core/src/file_picker.rs:2207:
end byte index N is not a char boundary; it is inside '\u{fffd}'
Rayon: detected unexpected panic; aborting

Root cause

FileItem::new_from_walk_bytes (the zlob walker fast path) is handed a basename_offset that indexes the raw path bytes, and stores it as the filename_offset into a String built with String::from_utf8_lossy(relative_path).

from_utf8_lossy replaces each ill-formed byte with a 3-byte U+FFFD, so on a non-UTF-8 name every index past the first bad byte shifts. The stored offset can then point inside a replacement char, and the later &rel[..filename_offset] slice (and the split_at at line 2207) panics on the non-boundary index. The SAFETY: filename_offset is always at a character boundary note a few lines up does not hold for this path.

The pure-Rust walker path (new_from_walk) is unaffected because it derives the offset from the lossy string itself via rfind(is_separator).

The fix

Re-derive the split from the lossy string when a byte was actually replaced (Cow::Owned), the same way the walker path does. Valid UTF-8 is the overwhelming common case and stays on the fast path: from_utf8_lossy borrows, so the raw basename_offset is used unchanged with no rfind.

let rel = String::from_utf8_lossy(relative_path);
let fname_offset = match &rel {
    std::borrow::Cow::Borrowed(_) => basename_offset,
    std::borrow::Cow::Owned(s) => s.rfind(std::path::is_separator).map_or(0, |i| i + 1) as u16,
};

Verification (Windows, default ripgrep features)

• New unit test walk_bytes_offset_stays_on_a_char_boundary_for_non_utf8_names: feeds b"\xff/.txt" with the raw basename_offset (2) that used to slice inside the U+FFFD, and asserts the split lands on a boundary ("\u{fffd}/" + ".txt"). It panics on main, passes here.
• cargo test -p fff-search --lib — 160 passed, 0 failed.
• cargo fmt --all -- --check — clean.

The test drives new_from_walk_bytes directly, so it exercises the fix under either glob backend.

Summary by CodeRabbit

• Bug Fixes
  • Fixed an issue where filenames containing invalid UTF-8 characters could be split incorrectly in the file picker.
  • Directory and file paths are now identified correctly even when filenames contain unusual characters.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 44c22527-906a-4e74-b4b8-bc3957fe373d

📥 Commits

Reviewing files that changed from the base of the PR and between d5b4abd and 5251fc4.

📒 Files selected for processing (1)

• crates/fff-core/src/file_picker.rs

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The file picker now recalculates basename offsets after lossy UTF-8 conversion. This prevents invalid string slicing for non-UTF-8 filenames. A regression test verifies correct directory and filename splitting.

Changes

Invalid UTF-8 path handling

Layer / File(s)	Summary
Safe path splitting crates/fff-core/src/file_picker.rs	new_from_walk_bytes preserves offsets for valid UTF-8 and recalculates character-safe offsets after replacement characters are inserted. A regression test validates directory and filename separation for non-UTF-8 paths.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 5251f

This localized change prevents crashes when scanning filenames with invalid UTF-8 bytes, with no actionable merge-blocking risk remaining after normal checks and review.

Suggested reviewers: dmtrkovalenko, gustav-fff

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the filename offset fix for invalid UTF-8 names.
Linked Issues check	✅ Passed	The code recalculates the lossy UTF-8 offset and adds a regression test, addressing issue #799.
Out of Scope Changes check	✅ Passed	The changes are limited to the crash fix and its regression test.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[dmtrKovalenko]

Thank you for your fix but I made a slightly more efficient one here #809