feat(secrets): detect dotenv runtime dependencies for the Keyverse migration - #1199
feat(secrets): detect dotenv runtime dependencies for the Keyverse migration#1199seonghobae wants to merge 7 commits into
Conversation
…e migration Add six declarative pattern-regex rules in the canonical scanner owner. Preserve advisory interpretation and negative controls; do not equate dotenv syntax with a proven credential leak or completed migration. Observed RED: 43 cases fail with rules absent. Local parser/compiler excerpt acceptance: 43 cases pass. Six complete-scanner integration cases are added but not locally executed; full CI, coverage, packaging, security and independent review remain required. No runtime scanner or vault implementation is copied into consumers.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review Please review exact head |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
expected='64305b18df06ed37f87d3c77ce8cf176567b1e43'
actual="$(git rev-parse HEAD)"
printf 'HEAD=%s\nEXPECTED=%s\n' "$actual" "$expected"
git diff --name-status "${expected}^" "$expected"
printf '\n--- STAT ---\n'
git diff --stat "${expected}^" "$expected"
printf '\n--- YAML FILES CHANGED: OUTLINES ---\n'
while IFS= read -r file; do
case "$file" in
*.yml|*.yaml) ast-grep outline "$file" --items all || true ;;
esac
done < <(git diff --name-only "${expected}^" "$expected")
printf '\n--- CHANGED FILE CONTENTS ---\n'
while IFS= read -r file; do
printf '\n===== %s =====\n' "$file"
sed -n '1,360p' "$file"
done < <(git diff --name-only "${expected}^" "$expected")
printf '\n--- RELEVANT DEFINITIONS AND REFERENCES ---\n'
rg -n -i -C 3 \
'pattern-regex|dotenv|dotenvy|env_file|package\.json|package discovery|complete.scanner|integration' \
. \
-g '!**/.git/**' -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' \
| head -n 1800Length of output: 2148 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- CHANGED FILE CONTENTS ---'
for file in \
CHANGELOG.d/keyverse-dotenv-dependency-rules.md \
docs/doctoring/dotenv_dependency_detection.md \
scanner/rules/dotenv_dependencies.yml \
tests/test_dotenv_dependency_rules.py \
tests/test_dotenv_scanner_integration.py
do
printf '\n===== %s =====\n' "$file"
cat -n "$file"
done
printf '%s\n' '--- CANDIDATE SCANNER/RULE ENGINE FILES ---'
fd -t f . scanner tests | sort | sed -n '1,700p'
printf '%s\n' '--- RULE/SCANNER DEFINITIONS AND TEST EXECUTION REFERENCES ---'
rg -n -i -C 4 \
'def (load|compile|scan|discover|parse)|class .*?(Rule|Scanner)|pattern_regex|pattern-regex|rules_dir|scanner/rules|yaml\.safe_load|dotenv_dependencies' \
scanner tests \
-g '*.py' -g '*.yml' -g '*.yaml' \
| head -n 1600Length of output: 68036 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- APPGUARDRAIL ENGINE: RULE PARSING, COMPILATION, AND DISCOVERY ---'
sed -n '880,1090p' scanner/cli/appguardrail.py
printf '%s\n' '--- APPGUARDRAIL ENGINE: FILE COLLECTION, LANGUAGE MAPPING, AND SCANNING ---'
rg -n -C 8 \
'def _scan_file\b|def _collect_files\b|def _language|LANGUAGE|EXTENSION|suffix|SCAN_RULES|pattern\.search|pattern\.finditer' \
scanner/cli/appguardrail.py
printf '%s\n' '--- PACKAGING METADATA CANDIDATES ---'
fd -t f -d 3 '^(pyproject\.toml|setup\.cfg|setup\.py|MANIFEST\.in|tox\.ini)$' . | sort
printf '%s\n' '--- PACKAGING RULE-INCLUSION REFERENCES ---'
rg -n -i -C 5 \
'package.data|include.package.data|scanner\.rules|\.yml|\.yaml|setuptools|wheel|sdist' \
pyproject.toml setup.cfg setup.py MANIFEST.in 2>/dev/null || true
printf '%s\n' '--- RULE LOAD CONTRACT TESTS ---'
rg -n -C 6 \
'_load_packaged_regex_rules|SCAN_RULES|resources\.files|scanner\.rules' \
tests -g '*.py' | head -n 1000Length of output: 51993 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- COMPLETE _scan_file IMPLEMENTATION ---'
sed -n '2919,3055p' scanner/cli/appguardrail.py
printf '%s\n' '--- SHARED LANGUAGE EXTENSION DEFINITIONS ---'
fd -t f '^language\.py$' . | while IFS= read -r file; do
printf '\n===== %s =====\n' "$file"
cat -n "$file"
done
printf '%s\n' '--- FINDING CONTEXT FOR DOCUMENTATION AND EXAMPLES ---'
sed -n '2260,2335p' scanner/cli/appguardrail.py
rg -n -C 5 'NON_BLOCKING_CONTEXTS|def _finding_context\b|scanner-fixture|example|doc' \
scanner/cli/appguardrail.py appguardrail_core \
-g '*.py' | head -n 900Length of output: 41183
The package metadata includes the new YAML file. The six integration tests validate rule registration and
|
Hosted verification update — unchanged head 64305b1Tests run SAST Semgrep, Security Scan, Security Process and existing coverage workflows also report success. CodeQL run Please use these exact coordinates when adopting this detector under ContextualWisdomLab/.github#2063. Do not publish matched source snippets or secret values into the migration inventory. |
|
@coderabbitai review Please review exact head |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
base='e71d37e7c58118e6764c96ab7c4492fe33eed6f8'
head='ca50e0fa8ebaa5a458bdb020c23e1734135e43fd'
printf '%s\n' '--- REVISION CHECK ---'
printf 'HEAD=%s\n' "$(git rev-parse HEAD)"
printf 'REQUESTED=%s\nBASE=%s\n' "$head" "$base"
git merge-base --is-ancestor "$base" "$head"
printf 'base_is_ancestor=yes\n'
printf '%s\n' '--- COMPLETE BASE-TO-HEAD CHANGESET ---'
git diff --name-status "$base" "$head"
git diff --stat "$base" "$head"
git diff --find-renames --unified=80 "$base" "$head" -- \
scanner/rules/dotenv_dependencies.yml \
tests/test_dotenv_dependency_rules.py \
tests/test_dotenv_scanner_integration.py \
docs/doctoring/dotenv_dependency_detection.md \
CHANGELOG.d/keyverse-dotenv-dependency-rules.md
printf '%s\n' '--- RULE AND TEST STRUCTURE ---'
for file in scanner/rules/dotenv_dependencies.yml tests/test_dotenv_dependency_rules.py tests/test_dotenv_scanner_integration.py; do
if test -f "$file"; then
printf '\n===== OUTLINE: %s =====\n' "$file"
ast-grep outline "$file" --items all || true
fi
done
printf '%s\n' '--- FULL RULES AND TARGETED TESTS ---'
for file in scanner/rules/dotenv_dependencies.yml tests/test_dotenv_dependency_rules.py tests/test_dotenv_scanner_integration.py; do
if test -f "$file"; then
printf '\n===== %s =====\n' "$file"
cat -n "$file"
fi
done
printf '%s\n' '--- RULE LOADING, APPLICABILITY, AND SCAN PATH ---'
rg -n -C 10 \
'def _load_packaged_regex_rules\b|def _get_applicable_rules\b|def _scan_file\b|LANGUAGE_EXTENSIONS|pattern_regex|pattern-regex|extensions' \
scanner appguardrail_core tests \
-g '*.py' -g '*.yml' -g '*.yaml' \
| head -n 2400Length of output: 175119
The typed Python assignment extension is bounded to one line and 128 characters. It preserves the tested comment, import, and string-literal negatives. The
You are interacting with an AI system. |
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
CWL migration checkpoint: current exact head 021a1f4c226e6d1bcb7575fe1d37a7140823611c has Tests, Security Scan, SAST Semgrep, OpenSSF Evidence, Retention Audit, Pinned HTTPS, Scan-path and Security Process GREEN. CodeQL PR is terminal failure only because the compatibility job is waiting for the central dispatched exact-head CodeQL verdict; its log does not identify a source vulnerability. Do not weaken or bypass that gate. The rule PR remains Draft until the central verdict and independent review are terminal.
seonghobae
added
enhancement
ChatGPT Codex Connector
1 participant
Scope and authority
Implements source-detection work for ContextualWisdomLab/.github#2063. Keyverse remains the secret-lifecycle authority; its bootstrap repair stays in the Keyverse owner path. This PR adds declarative migration findings to AppGuardrail; it does not become another secret store or copy scanner logic into consumers.
Protected base:
develop@e71d37e7c58118e6764c96ab7c4492fe33eed6f8.Current exact head:
021a1f4c226e6d1bcb7575fe1d37a7140823611c.Lifecycle: Draft / application-security checks GREEN except shared central CodeQL / independent post-last-push review required / not merge-authorized.
Changes
pattern-regexrules: Python direct loaders/settings, Rust dotenv/dotenvy, Node dotenv, shell source/dot, and literal Compose/container dotenv transport.let mutbindings.TDD repairs
0b23610142a93391774e41a5a9b34381b562896eadded the Python typed-assignment contract before its production fix. The prior regex did not matchvalues: dict[str, str] = dotenv_values(".env"); this is source-level deterministic RED, not a claimed hosted RED.0238485a0a9612aa395d999684eb12593b6a7dminimally extended only the Python loader assignment prefix with a bounded, same-line type-annotation allowance ([^=\r\n]{1,128}).ca50e0fa8ebaa5a458bdb020c23e1734135e43fdexercised the typed assignment through the packaged_scan_filepath as well as the parser/compiler contract.753b5c9e54a6267381ce3b7ce785394542f9feebadded a Rust positive control forlet mut loaded = dotenvy::dotenv();before changing the rule. The preceding regex acceptedlet name = ...but not the common mutable binding; direct regex evaluation is a deterministic source-level RED. No hosted RED is claimed because the production descendant followed before that generation settled.021a1f4c226e6d1bcb7575fe1d37a7140823611cminimally allows optionalmutafterletwhile preserving the same direct dotenv/dotenvy call boundary and the existing bounded-pattern controls.These remain WARNING-level migration findings, not assertions of an exploitable secret leak. The central CWL gate must explicitly adopt released rule IDs for new production regressions while preserving an owner backlog for existing dependencies. No blanket warning-to-error change, source rewrite, or production cutover is included.
Current exact-head evidence
At exact head
021a1f4c226e6d1bcb7575fe1d37a7140823611c, Tests34393095056, Security Scan34393095065, SAST Semgrep34393095096, Pinned HTTPS Coverage34393095120, OpenSSF Evidence Coverage34393095097, Retention Audit Coverage34393095107, Scan path context coverage34393095046, and Security Process34393095051are terminal SUCCESS. CodeQL PR34393095073is terminal FAILURE on the shared central CodeQL path. No qualifying review exists after the last production push.The application/security GREEN above is exact-head evidence; it does not waive the shared CodeQL gate. Draft remains intentional until central CodeQL reaches terminal GREEN for this generation and a qualifying independent post-last-push review is present.
Limits
Direct bounded syntax only: no general AST/alias/taint analysis, dynamic wrappers/filenames, arbitrary Python type expressions beyond the bounded assignment prefix, Rust destructuring/pattern bindings beyond the declared simple
let [mut] name =form, all complex YAML forms, implicit framework/Compose loading, or universal language support. No-finding is not proof of migration completion. The scanner does not follow the matched dotenv reference or read its values. Organization migration evidence should project rule ID/repository/revision/path/line/disposition rather than copy arbitrary matched source context.No workflow duplication, provider credential expansion, force-push/destructive rebase, self-approval, synthetic status, no-op retrigger, scanner suppression, or gate weakening.