Optimize scancode ignore pattern size using wildcards

[JustinWonjaePark]

Description

Optimize the performance of Scancode's pre-scan tree traversal by reducing the size of the ignore pattern tuple.

Key Changes

• Group binaries by wildcards: Convert individual binary file paths with matching extensions (e.g. png, jpg, exe, mp3) into *.extension wildcard patterns.
• Suppress individual user exclude expansion: Prevent individual file paths inside directories excluded via the -e option from expanding into the ignore tuple by adding user-specified custom exclude patterns directly to coarse_patterns.
• Forward custom excludes: Pass the raw -e input from the CLI to run_scan as custom_exclude_patterns parameter.

[coderabbitai]

📝 Walkthrough

Walkthrough

The scanner now normalizes custom exclusions into path-aware ignore patterns, applies coarse ignores while walking directories, skips covered files, and passes exclusion configuration through the CLI scan invocation.

Changes

Scan exclusion flow

Layer / File(s)	Summary
Normalize and construct ignore patterns src/fosslight_source/run_scancode.py	Custom exclusions and binary paths now produce extension-based or path-based coarse and scancode ignore patterns.
Wire exclusions into scan traversal src/fosslight_source/cli.py, src/fosslight_source/run_scancode.py	The CLI passes path_to_exclude to run_scan, which prunes ignored directories and files before binary detection and logs excluded binaries.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant run_scan
  participant FileSystemWalk
  participant check_binary
  CLI->>run_scan: path_to_exclude and scan options
  run_scan->>FileSystemWalk: traverse with coarse patterns
  FileSystemWalk->>run_scan: uncovered files
  run_scan->>check_binary: inspect candidate files
  check_binary->>run_scan: binary result

Loading

Possibly related PRs

• fosslight/fosslight_source_scanner#265: Modifies the same binary-detection and exclusion flow in run_scan().
• fosslight/fosslight_source_scanner#275: Changes construction and application of binary exclusion lists and ignore patterns.
• fosslight/fosslight_source_scanner#276: Introduces related coarse-ignore and glob-pattern handling.

Suggested reviewers: soimkim

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: reducing Scancode ignore pattern size with wildcard-based patterns.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/optimize-ignore-patterns

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (3)

src/fosslight_source/run_scancode.py (3)

294-298: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Silently swallowed check_binary errors. Ruff flags the blind except Exception: continue (BLE001, S112). The safe fallback (treat as non-binary, keep scanning) is fine, but logging at debug helps diagnose why files were skipped.

♻️ Optional

-                        try:
-                            if not check_binary(full_path, True):
-                                continue
-                        except Exception:
-                            continue
+                        try:
+                            if not check_binary(full_path, True):
+                                continue
+                        except Exception as ex:
+                            logger.debug(f"check_binary failed for {rel_path}: {ex}")
+                            continue

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fosslight_source/run_scancode.py` around lines 294 - 298, Update the
exception handler around check_binary in the scanning loop to catch the error as
an exception variable and log it at debug level before continuing; preserve the
existing fallback of treating the file as non-binary and continuing the scan.

Source: Linters/SAST tools

---

279-286: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Directory-coverage probe via + "/a" is a bit cryptic. Appending a synthetic child so **/dir/** patterns match works, but it is non-obvious. A short inline comment explaining the intent would help future maintainers.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fosslight_source/run_scancode.py` around lines 279 - 286, Add a brief
inline comment in the directory filtering logic within the os.walk loop
explaining that the synthetic “/a” child probes directory contents so recursive
coarse-ignore patterns such as “**/dir/**” match correctly.

---

110-113: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid mutable default argument. Ruff flags path_to_exclude: list = [] (B006). Not mutated today, but use None and normalize inside to prevent shared-state surprises and satisfy the linter.

♻️ Proposed fix

 def _default_scancode_coarse_ignore_patterns(
-    path_to_exclude: list = [],
+    path_to_exclude: list = None,
     abs_path_to_scan: str = ""
 ) -> frozenset:

(the existing path_to_exclude or [] at Line 127 already handles None.)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/fosslight_source/run_scancode.py` around lines 110 - 113, Update
_default_scancode_coarse_ignore_patterns to use None instead of [] for the
path_to_exclude default, and retain or add normalization to an empty list inside
the function before iterating, leveraging the existing path_to_exclude or []
handling.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/fosslight_source/run_scancode.py`:
- Around line 173-183: Single-file custom exclusions are incorrectly converted
to extension-wide wildcard patterns, excluding unintended files. In the
exclusion-pattern construction logic, preserve path-specific patterns for
explicit user-provided files in both branches, and only generate *.<ext>
patterns for auto-discovered binary_paths exclusions; use the relevant
binary-path classification or source flag to distinguish these cases.
- Around line 94-97: Relativize absolute exclude paths before coarse ignore
matching so they align with the relative paths produced by os.walk(). Update the
logic around _default_scancode_coarse_ignore_patterns() and
_normalize_custom_pattern() to convert each absolute -e path relative to
abs_path_to_scan before generating directory patterns, while preserving existing
handling for relative excludes.

---

Nitpick comments:
In `@src/fosslight_source/run_scancode.py`:
- Around line 294-298: Update the exception handler around check_binary in the
scanning loop to catch the error as an exception variable and log it at debug
level before continuing; preserve the existing fallback of treating the file as
non-binary and continuing the scan.
- Around line 279-286: Add a brief inline comment in the directory filtering
logic within the os.walk loop explaining that the synthetic “/a” child probes
directory contents so recursive coarse-ignore patterns such as “**/dir/**” match
correctly.
- Around line 110-113: Update _default_scancode_coarse_ignore_patterns to use
None instead of [] for the path_to_exclude default, and retain or add
normalization to an empty list inside the function before iterating, leveraging
the existing path_to_exclude or [] handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a88b3031-1a7f-440b-ba11-cb768b83dd05

📥 Commits

Reviewing files that changed from the base of the PR and between fff35f2 and 78d419c.

📒 Files selected for processing (2)

• src/fosslight_source/cli.py
• src/fosslight_source/run_scancode.py

[coderabbitai]

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details

putComment timed out