Skip to content

perf(private_key): look up file size once per file instead of per line - #293

Open
Saarett wants to merge 1 commit into
masterfrom
perf/private-key-file-size-lookup-once-per-file
Open

Saarett wants to merge 1 commit into
masterfrom
perf/private-key-file-size-lookup-once-per-file

Conversation

@Saarett

@Saarett Saarett commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

PrivateKeyDetector.analyze_line runs for every line of every scanned file. The file-size guard was written as a single and expression:

if filename not in self._analyzed_files \
        and 0 < self.get_file_size(filename) < MAX_FILE_SIZE:
    self._analyzed_files.add(filename)

_analyzed_files was only populated when the size fell inside the scannable range. For any file at or above MAX_FILE_SIZE (8 KiB) the membership check kept failing, so get_file_size -> os.path.getsize fired again on every single line: a per-file operation executed per-line. Cost scaled with (files x lines) rather than (files).

Track the files whose size has already been measured in a separate _sized_files set, so the lookup happens at most once per file regardless of the result. The subsequent whole-file read is unchanged and still happens exactly once per file, so multi-line private keys are detected exactly as before.

Measured on a 21,359-file repository via
checkov --framework secrets --enable-secret-scan-all-files:

before:  real 2120s   user 2014s   sys 230s
after:   real 1679s   user 1638s   sys  60s

Wall time -21%; sys time (the syscall fingerprint of the redundant getsize calls) down 3.9x. Findings are identical before and after (14 findings, same files/lines/checks).

Adds two regression tests:

  • test_get_file_size_is_called_at_most_once_per_file - asserts the size lookup runs at most once for a 500-line, >8 KiB file (previously 500 times).
  • test_multiline_private_key_in_small_file_is_still_detected - guards the whole-file read path that multi-line key detection depends on.
  • Please check if the PR fulfills these requirements
  • Tests for the changes have been added
  • Docs have been added / updated
  • All CI checks are green
  • What kind of change does this PR introduce?
  • What is the current behavior?
  • What is the new behavior (if this is a feature change)?

  • Does this PR introduce a breaking change?

  • Other information:

`PrivateKeyDetector.analyze_line` runs for every line of every scanned
file. The file-size guard was written as a single `and` expression:

    if filename not in self._analyzed_files \
            and 0 < self.get_file_size(filename) < MAX_FILE_SIZE:
        self._analyzed_files.add(filename)

`_analyzed_files` was only populated when the size fell inside the
scannable range. For any file at or above MAX_FILE_SIZE (8 KiB) the
membership check kept failing, so `get_file_size` -> `os.path.getsize`
fired again on every single line: a per-file operation executed
per-line. Cost scaled with (files x lines) rather than (files).

Track the files whose size has already been measured in a separate
`_sized_files` set, so the lookup happens at most once per file
regardless of the result. The subsequent whole-file read is unchanged
and still happens exactly once per file, so multi-line private keys are
detected exactly as before.

Measured on a 21,359-file repository via
`checkov --framework secrets --enable-secret-scan-all-files`:

    before:  real 2120s   user 2014s   sys 230s
    after:   real 1679s   user 1638s   sys  60s

Wall time -21%; `sys` time (the syscall fingerprint of the redundant
getsize calls) down 3.9x. Findings are identical before and after
(14 findings, same files/lines/checks).

Adds two regression tests:
  - `test_get_file_size_is_called_at_most_once_per_file` - asserts the
    size lookup runs at most once for a 500-line, >8 KiB file
    (previously 500 times).
  - `test_multiline_private_key_in_small_file_is_still_detected` -
    guards the whole-file read path that multi-line key detection
    depends on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant