-
Notifications
You must be signed in to change notification settings - Fork 1
feat: 데이터 비식별화(Data Anonymizer) 도구 추가 #1482
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/add-text-summarizer-and-pii-redactor-15112816368110641983
Are you sure you want to change the base?
Changes from all commits
7db3411
034d111
67433f3
a7cf64a
fa202dc
3f420e5
f633b23
cb6b718
9ca5329
ab8dfff
987afa6
052f79e
ae9b8a9
b535366
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -529,6 +529,8 @@ async def base64_decoder_handler(params: Dict[str, Any]) -> Dict[str, str]: | |
| "합니다", | ||
| } | ||
| ) | ||
|
|
||
|
|
||
| def _normalize_analysis_text(value: str) -> str: | ||
| """Normalize user text for deterministic, multilingual rule matching.""" | ||
| if len(value) > ANALYSIS_TEXT_MAX_CHARS: | ||
|
|
@@ -675,6 +677,49 @@ async def uuid_v4_generator_handler(params: Dict[str, Any]) -> Dict[str, str]: | |
| return {"uuid": str(uuid.uuid4())} | ||
|
|
||
|
|
||
| _INTERNATIONAL_EMAIL_PATTERN = re.compile( | ||
| rf"(?<![\w{_EMAIL_ATOM}.-])" | ||
| rf"[\w{_EMAIL_ATOM}-]+(?:\.[\w{_EMAIL_ATOM}-]+)*@" | ||
| r"(?:[^\W_](?:(?:[^\W_]|-){0,61}[^\W_])?\.)+" | ||
| r"[^\W_]{2,63}(?![\w-])" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an internationalized domain is supplied in its valid ASCII IDNA form, the address is returned unchanged because the final label rejects hyphens. For example, Useful? React with 👍 / 👎. |
||
| ) | ||
| _INTERNATIONAL_PHONE_PATTERN = re.compile( | ||
| r"(?<!\d)(?:01[016789][ .-]?\d{3,4}[ .-]?\d{4}" | ||
| r"|0[1-9](?:[ .-]?\d{2}){4})(?!\d)" | ||
|
Comment on lines
+686
to
+688
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a French number uses its standard international representation, such as Useful? React with 👍 / 👎. |
||
| ) | ||
| _KOREAN_RESIDENT_REGISTRATION_PATTERN = re.compile( | ||
| r"(?<!\d)\d{6}[ -]?[1-4]\d{6}(?!\d)" | ||
| ) | ||
|
|
||
|
|
||
| async def data_anonymizer_handler(params: Dict[str, Any]) -> Dict[str, str]: | ||
| """Mask bounded contact and Korean resident-registration identifiers.""" | ||
| text = params.get("text", "") | ||
| if text is None: | ||
| text = "" | ||
| if len(text) > ANALYSIS_TEXT_MAX_CHARS: | ||
| raise ValueError( | ||
| f"Analysis text must not exceed {ANALYSIS_TEXT_MAX_CHARS} characters" | ||
| ) | ||
| text = _EMAIL_PATTERN.sub("***@***", text) | ||
| text = _INTERNATIONAL_EMAIL_PATTERN.sub("***@***", text) | ||
| text = _PHONE_PATTERN.sub("***-****-****", text) | ||
| text = _INTERNATIONAL_PHONE_PATTERN.sub("***-****-****", text) | ||
| text = _KOREAN_RESIDENT_REGISTRATION_PATTERN.sub("******-*******", text) | ||
| return {"anonymized_text": text} | ||
|
|
||
|
|
||
| registry.register( | ||
| ToolInfo( | ||
| code="data_anonymizer", | ||
| name="데이터 비식별화 (Data Anonymizer)", | ||
| description="텍스트에서 이메일 주소, 일부 한국·북미·프랑스 전화번호, 한국 주민등록번호 형식을 단순 마스킹합니다. 완전한 개인정보 비식별화를 보장하지 않습니다.", | ||
| category="보안", | ||
| parameters={"text": "string"}, | ||
|
Comment on lines
+714
to
+718
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed at exact head
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Acknowledged. |
||
| ), | ||
| data_anonymizer_handler, | ||
| ) | ||
|
|
||
| registry.register( | ||
| ToolInfo( | ||
| code="uuid_v4_generator", | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Data anonymizer boundary | ||
|
|
||
| ## Decision and observed implementation | ||
|
|
||
| PR #1482 repair parent `034d111b6bef126929d6f0085c2fa15bbf9724be` | ||
| stacks on PR #1555 exact head | ||
| `03799bc157fa39a419cf6c3f77a29a2ca02cd7f4`. The handler reuses the stack's | ||
| canonical ASCII email and selected Korean/North American phone matchers, then | ||
| adds bounded Unicode-email, French phone, and Korean resident-registration | ||
| patterns. Every input is subject to `ANALYSIS_TEXT_MAX_CHARS` before scanning. | ||
|
|
||
| This tool performs deterministic format masking only. It does not measure | ||
| re-identification risk, detect names or organizations, transform free-form | ||
| quasi-identifiers, or certify that output is anonymous. Product copy must keep | ||
| that limitation visible. A downstream workflow that requires release-grade | ||
| de-identification needs a documented data model, threat model, risk metric, | ||
| review authority, and evidence that the transformed dataset meets its intended | ||
| use. It must not infer that assurance from this handler's successful response. | ||
|
|
||
| Endpoint regressions cover hyphenated and separator-free Korean identifiers, | ||
| an internationalized email address, a French phone number, punctuation | ||
| preservation, and the input-size boundary. The values are synthetic test data; | ||
| no real person's identifiers are committed. | ||
|
|
||
| ## Research grounding | ||
|
|
||
| NIST SP 800-188 treats de-identification as a managed process involving data | ||
| models, techniques, governance, and re-identification risk rather than a small | ||
| set of textual substitutions. That distinction supports the deliberately | ||
| narrow product claim above and rejects the earlier broad “data anonymization” | ||
| assurance. | ||
|
|
||
| Garfinkel, S., Guttman, B., Near, J., Dajani, A., & Singer, P. (2023). | ||
| *De-identifying government datasets: Techniques and governance* (NIST Special | ||
| Publication 800-188). National Institute of Standards and Technology. | ||
| https://doi.org/10.6028/NIST.SP.800-188 | ||
|
|
||
| The official publication page was available during verification, but its | ||
| linked PDF endpoint returned HTTP 404 on 2026-09-04. The PR therefore records | ||
| the DOI and bounded summary instead of committing an unverified or | ||
| redistribution-uncertain binary. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When an internationalized address is immediately followed by same-script prose—a normal construction in Korean and Japanese—the final
[^^\W_]{2,63}-style domain match consumes the prose as part of the TLD. For example,사용자@예시.한국으로 보내세요becomes***@*** 보내세요, deleting으로from the anonymized text. Bound the Unicode domain using validated IDN/public-suffix handling rather than treating every following Unicode word character as part of the address, and add this case to the existing data-anonymizer endpoint tests.Useful? React with 👍 / 👎.