[Product Gap] Build an auditable data-hygiene utility suite

[seonghobae]

Buyer problem

Naruon needs reusable text-hygiene tools for email and agent workflows, but a buyer must never mistake a narrow regular expression for comprehensive privacy protection or a legacy digest for a secure hash.

This issue replaces PR #1246 with a product-grade vertical slice that is explicit about supported scope, preserves evidence, and fails safely.

Product contract

Implement three standalone tools that also work as modular registry components.

1. url_evidence_extractor

Return bounded, source-grounded URL evidence rather than unvalidated regex strings.

Required output per match:

• raw_value
• normalized_value
• source_start
• source_end
• scheme_code
• host_value
• contains_userinfo
• validation_status
• warning_codes

Requirements:

• accept only explicitly supported http and https absolute URI references;
• parse components using a standard parser after candidate delimitation;
• preserve the exact source substring and Unicode offsets;
• handle parentheses, terminal punctuation, query strings, fragments, IPv6, IDNA and percent encoding deterministically;
• never fetch an extracted URL;
• mark embedded user information and other dangerous forms rather than silently treating them as safe;
• enforce input length, maximum matches and per-match length;
• produce deterministic order and deduplication without losing repeated-source locations.

2. contact_data_redactor

Do not call the first release a general PII redactor. The supported classes must be explicit and versioned.

P0 classes:

• email addresses, including documented internationalization limits;
• Korean and E.164-compatible telephone forms, including spaces, parentheses and country prefixes.

Required output:

• redacted text;
• immutable ordered match records with class, source span, replacement span and detector version;
• match counts by class;
• warning that unsupported PII classes are not removed;
• optional deterministic placeholders that preserve entity distinction without exposing values.

Requirements:

• no original personal-data value in logs, errors or metrics;
• configurable but bounded false-positive policy;
• golden corpus with Korean, English and mixed-script examples;
• hostile Unicode, overlapping match, malformed input and maximum-size tests;
• precision, recall, span F1 and calibration/reporting fixture against human-labelled data;
• no claim of anonymization or irreversible de-identification.

3. content_checksum_generator

Default and normal product surface:

• SHA-256;
• SHA-3-256;
• BLAKE2b-256 when interoperability permits.

MD5 and SHA-1 must not appear in the normal security-labelled surface. A legacy compatibility mode may be considered only when all of the following hold:

• tool and output are named legacy_checksum;
• disabled by default;
• explicit non_security_use=true input is required;
• response contains a machine-readable deprecation warning;
• doctoring explains collision risk and migration;
• no authentication, integrity, signature or security recommendation uses the legacy result.

Quality and security gates

• production statement coverage: 100%;
• production branch coverage: 100%;
• public module/class/function/docstring coverage: 100%;
• property tests for source-span round trips and deterministic output;
• fuzz tests for hostile Unicode and oversized candidate boundaries;
• exact current-head Application CI, Bandit, Semgrep, CodeQL/dependency/security scans;
• no test-only success substitutions and no skipped security tests;
• CHANGELOG.md, ARCHITECTURE.md or ADR update, and buyer/operator documentation;
• database/audit objects, if added, use two-or-more-word snake_case names;
• no LLM is required for deterministic detection. If an optional LLM verifier is introduced, use NVIDIA_NIM_API_KEY, never COPILOT_GITHUB_TOKEN, keep it outside the deterministic redaction path, and record model/prompt evidence without exposing source PII.

Realistic acceptance tests

• punctuation-balanced URLs in Korean and English prose;
• repeated same URL at different offsets;
• userinfo, IDNA, IPv6, fragments and percent-encoded paths;
• Korean mobile/landline/service numbers, E.164, spaces and parentheses;
• near-miss strings that must not be redacted;
• mixed email and phone overlaps;
• 1 MiB bounded input and excessive-match fail-closed behavior;
• deterministic checksum vectors and streaming/chunked equivalence;
• explicit proof that legacy digests cannot be selected without the compatibility acknowledgement.

Standards and doctoring — APA 7th

Doctoring must include at least the following current primary sources and state their status accurately:

Boeckl, K., & Lefkovitz, N. (2020). NIST Privacy Framework: A tool for improving privacy through enterprise risk management, version 1.0. National Institute of Standards and Technology. https://doi.org/10.6028/NIST.CSWP.01162020

Berners-Lee, T., Fielding, R., & Masinter, L. (2005). Uniform resource identifier (URI): Generic syntax (RFC 3986). RFC Editor. https://doi.org/10.17487/RFC3986

McCallister, E., Grance, T., & Scarfone, K. (2010). Guide to protecting the confidentiality of personally identifiable information (PII) (NIST SP 800-122). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-122

National Institute of Standards and Technology. (2022, December 15). NIST transitioning away from SHA-1 for all applications. https://www.nist.gov/news-events/news/2022/12/nist-transitioning-away-sha-1-all-applications

At implementation time, re-check whether NIST Privacy Framework 1.1 has moved beyond its initial public draft; as of April 1, 2026, NIST still labels it “coming soon.”

Delivery sequence

1. test-first contracts and gold corpus;
2. bounded pure detectors and checksum API;
3. API/registry integration;
4. audit and exact-value documentation;
5. full current-head verification;
6. one focused PR, independent approval, then merge.

Supersedes #1246.

[seonghobae]

2026-09-08 live handoff from generated #1615: the mixed URL/JSON/hash implementation was converted to zero-effective-delta Draft provenance at 956dbd33ce9cf2043d399a3a1495178dc88d429f rather than becoming a second utility source writer. The generated URL helper regressed this issue/#1418's evidence contract (no source spans/occurrences/warnings; weak DNS-label validation that can admit underscore and invalid dotted-IPv4-shaped hosts). The generated hash_generator exposed every hashlib.algorithms_available algorithm and explicitly tested MD5, contrary to this issue's normal security-surface contract. It also introduced unpinned pytest-cov>=7.1.0 without the lock/provenance path and duplicated an Unreleased CHANGELOG block. Keep URL ownership in #1418 (with #1590 subordinate) and checksum implementation in one future #1247-aligned canonical lane. json_formatter is the only unique proposal from #1615; it remains ancestry-only until a buyer/domain contract justifies production admission or it is explicitly rejected.

[seonghobae]

Checksum slice authority update — 2026-09-10

content_checksum_generator remains canonically owned by #1361, now repaired to exact dc480c3bc2294d351eef5fdaa61cbf80e0429b19 on current protected develop@042b0c70531b229af3acbd0421a2f23098d848b3 with behind_by=0 and exactly 10 effective files. The lane is Draft while exact-head checks/review re-run.

The prior unresolved invalid-Unicode finding was verified as real and fixed test-first. Direct registry and authenticated raw-JSON regressions now cover a decoded lone surrogate (\ud800), and the handler maps UnicodeEncodeError to stable content_checksum_invalid_utf8 rather than leaking a raw encoding exception. The existing modern algorithm allowlist, 1 MiB UTF-8 byte ceiling, exact-byte/no-normalization semantics, forged-session rejection, generic execute-envelope compatibility, ADR-0007, operator guidance and doctoring remain in scope.

#1418 continues to own URL/contact hygiene. #1247 remains open because protected integration is not complete and the broader suite/streaming-checksum acceptance is not yet shipped.

[seonghobae]

Correction to the checksum authority note above: #1361 advanced docs-only after the product fix to exact d6f8b31231709b90e464225ac7e79db155c52cef. The production/test causal fix remains dc480c3bc2294d351eef5fdaa61cbf80e0429b19; d6f8b312... only refreshes docs/doctoring/content-checksum-generator.md against 2026-09-10 NIST primary status material and adds invalid-Unicode rejection to acceptance evidence. Protected base remains develop@042b0c70531b229af3acbd0421a2f23098d848b3, effective scope remains 10 files and behind=0. Current-head checks/review remain pending, so this does not change shipped/merge-ready status.