fix(api): redact secrets in run debug data instead of storing them in clear text

[arminfauland]

Problem

The debug data collected for a run contains the resolved variables — and those include secrets. It is serialised and written to scrape_data verbatim, so a login password ends up in the database in clear text:

sqlite> SELECT value FROM scrape_data WHERE key='__debugData' LIMIT 1;
{"var_targetYear":"…","var_email":"user@example.com","var_password":"<clear text>", …}

GET /api/runs/:runId/debug then hands that row out unchanged. So does the artifacts endpoint, and so does every backup and database dump.

Found on my own instance: var_password was a 12-character plaintext string, straight out of the DB.

Why the fix is at the source

The project already ships SecretRedactionService, and ScrapeExecutionService already has it injected — the debug path simply never called it.

Redaction is applied before the row is written, not on the way out. Redacting only on read would leave the plaintext at rest, where backups and dumps still pick it up.

const debugData = this.secretRedaction.redactObject(
  this.buildDebugData(previousData),
);

Both read paths (/debug and /artifacts, which reads the same key) redact as well — a safety net for rows written before this change. That only works while the secrets are still registered in the running process, so it is explicitly not a substitute. Existing rows should be cleared once:

DELETE FROM scrape_data WHERE key = '__debugData';

Injection

SecretRedactionService is exported by EventsModule, which is @Global() — no module wiring needed for the controller.

Tests

test	proves
redacts the debug data before it reaches the database	the plaintext never reaches storeData
does not hand out credentials stored in the debug data	the endpoint does not serve it

nx lint api clean, full API suite green (84 files / 1418 tests).

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	Generic Password	51e7d18	apps/api/src/scrape/services/scrape-execution.service.spec.ts	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[arminfauland]

The GitGuardian finding is a false positive, and I have removed the cause.

What it flagged: the string in the new test fixture, sitting next to a var_password key in scrape-execution.service.spec.ts. It was an invented value, never a real credential — the test needs some value it can then assert is absent from the stored row.

Verified rather than assumed before replying: the string does not appear in any secret store on the machine this was written on (encrypted secrets table: 0 hits, scrape data: 0 hits, all env and compose files: 0 hits). I also scanned every added line across all five branches I opened today — 569 lines — for Anthropic keys, API keys, GitHub tokens, Telegram bot tokens, AWS keys, private keys, JWTs and long hex strings: zero matches. The only password-shaped assignment anywhere was that one fixture.

Fixed in 71eff1c: replaced with <<fixture-not-a-real-secret>>, which reads as a placeholder to a human and to a scanner, plus a comment saying why it is written that way. Both tests assert exactly the same thing as before and still pass; nx lint api is clean.

Worth noting the irony: this PR exists because real credentials were being written in clear text — into scrape_data, and served by /api/runs/:runId/debug. On my own instance 46 of 47 __debugData rows contained a plaintext password before the fix. Those rows need clearing once, as noted in the description.