Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions backend/api/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -529,6 +529,8 @@ async def base64_decoder_handler(params: Dict[str, Any]) -> Dict[str, str]:
"합니다",
}
)


def _normalize_analysis_text(value: str) -> str:
"""Normalize user text for deterministic, multilingual rule matching."""
if len(value) > ANALYSIS_TEXT_MAX_CHARS:
Expand Down Expand Up @@ -675,6 +677,49 @@ async def uuid_v4_generator_handler(params: Dict[str, Any]) -> Dict[str, str]:
return {"uuid": str(uuid.uuid4())}


_INTERNATIONAL_EMAIL_PATTERN = re.compile(
rf"(?<![\w{_EMAIL_ATOM}.-])"
rf"[\w{_EMAIL_ATOM}-]+(?:\.[\w{_EMAIL_ATOM}-]+)*@"
r"(?:[^\W_](?:(?:[^\W_]|-){0,61}[^\W_])?\.)+"
r"[^\W_]{2,63}(?![\w-])"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve prose after internationalized email domains

When an internationalized address is immediately followed by same-script prose—a normal construction in Korean and Japanese—the final [^^\W_]{2,63}-style domain match consumes the prose as part of the TLD. For example, 사용자@예시.한국으로 보내세요 becomes ***@*** 보내세요, deleting 으로 from the anonymized text. Bound the Unicode domain using validated IDN/public-suffix handling rather than treating every following Unicode word character as part of the address, and add this case to the existing data-anonymizer endpoint tests.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mask IDNA-encoded international email domains

When an internationalized domain is supplied in its valid ASCII IDNA form, the address is returned unchanged because the final label rejects hyphens. For example, user@xn--fsqu00a.xn--0zwm56d is not matched by either email pattern, even though the catalog advertises email-address masking and the boundary document specifically includes internationalized email. Accept and validate xn-- A-label TLDs, and add the ASCII IDNA equivalent of the existing Unicode-domain case to the endpoint regression test.

Useful? React with 👍 / 👎.

)
_INTERNATIONAL_PHONE_PATTERN = re.compile(
r"(?<!\d)(?:01[016789][ .-]?\d{3,4}[ .-]?\d{4}"
r"|0[1-9](?:[ .-]?\d{2}){4})(?!\d)"
Comment on lines +686 to +688

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mask French numbers in international form

When a French number uses its standard international representation, such as +33 1 42 68 53 00 or +33142685300, it is returned unchanged because this branch only accepts the domestic leading 0. This leaks the same identifier that the tested 01 42 68 53 00 form masks, and international/E.164 formatting is common in contact data. Add a bounded +33 alternative that accounts for the omitted domestic prefix and cover both spaced and separator-free forms in the endpoint test.

Useful? React with 👍 / 👎.

)
_KOREAN_RESIDENT_REGISTRATION_PATTERN = re.compile(
r"(?<!\d)\d{6}[ -]?[1-4]\d{6}(?!\d)"
)


async def data_anonymizer_handler(params: Dict[str, Any]) -> Dict[str, str]:
"""Mask bounded contact and Korean resident-registration identifiers."""
text = params.get("text", "")
if text is None:
text = ""
if len(text) > ANALYSIS_TEXT_MAX_CHARS:
raise ValueError(
f"Analysis text must not exceed {ANALYSIS_TEXT_MAX_CHARS} characters"
)
text = _EMAIL_PATTERN.sub("***@***", text)
text = _INTERNATIONAL_EMAIL_PATTERN.sub("***@***", text)
text = _PHONE_PATTERN.sub("***-****-****", text)
text = _INTERNATIONAL_PHONE_PATTERN.sub("***-****-****", text)
text = _KOREAN_RESIDENT_REGISTRATION_PATTERN.sub("******-*******", text)
return {"anonymized_text": text}


registry.register(
ToolInfo(
code="data_anonymizer",
name="데이터 비식별화 (Data Anonymizer)",
description="텍스트에서 이메일 주소, 일부 한국·북미·프랑스 전화번호, 한국 주민등록번호 형식을 단순 마스킹합니다. 완전한 개인정보 비식별화를 보장하지 않습니다.",
category="보안",
parameters={"text": "string"},
Comment on lines +714 to +718

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Masking scope overstates anonymization

The catalog presents three contact-pattern substitutions as broad personal-data anonymization. Repository guidance requires narrower claims for contact redaction.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at exact head 67433f375ea1b8eb38baf3bf512ac761d9325dbd. Catalog copy now names only the supported email, selected Korean/North American/French phone, and Korean resident-registration formats, and explicitly states that complete de-identification is not guaranteed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged.

),
data_anonymizer_handler,
)

registry.register(
ToolInfo(
code="uuid_v4_generator",
Expand Down
77 changes: 77 additions & 0 deletions backend/tests/test_tools_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -1111,6 +1111,83 @@ def test_execute_sentiment_analyzer():
assert "불만" in data["result"]["key_emotions"]


def test_execute_data_anonymizer():
with TestClient(app) as client:
# 정상적인 케이스
response = client.post(
"/api/tools/data_anonymizer/execute",
headers={"Authorization": f"Bearer {_signed_session_token()}"},
json={
"parameters": {
"text": "제 이메일은 test.user-1@gmail.com 이고, 폰 번호는 010-1234-5678, 주민번호는 900101-1234567 입니다. 011-123-4567도 됩니다."
}
},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
anonymized = data["result"]["anonymized_text"]
assert "***@***" in anonymized
assert "***-****-****" in anonymized
assert "******-*******" in anonymized
assert "test.user-1@gmail.com" not in anonymized
assert "010-1234-5678" not in anonymized
assert "900101-1234567" not in anonymized

# 빈 텍스트 케이스
response_empty = client.post(
"/api/tools/data_anonymizer/execute",
headers={"Authorization": f"Bearer {_signed_session_token()}"},
json={"parameters": {"text": ""}},
)
assert response_empty.status_code == 200
data_empty = response_empty.json()
assert data_empty["status"] == "success"
assert data_empty["result"]["anonymized_text"] == ""

# null 텍스트 케이스를 막는 동작은 _validate_parameters가 하지만,
# fallback 커버리지를 위해 직접 handler를 호출하는 비동기 테스트를 아래에 추가합니다.


def test_execute_data_anonymizer_masks_separator_free_and_international_formats():
source_values = (
"01012345678",
"9001011234567",
"01 42 68 53 00",
"사용자@예시.한국",
)
with TestClient(app) as client:
response = client.post(
"/api/tools/data_anonymizer/execute",
headers={"Authorization": f"Bearer {_signed_session_token()}"},
json={"parameters": {"text": " / ".join(source_values) + "."}},
)

assert response.status_code == 200
anonymized = response.json()["result"]["anonymized_text"]
assert all(source_value not in anonymized for source_value in source_values)
assert anonymized.endswith(".")
assert anonymized.count("***-****-****") == 2
assert "******-*******" in anonymized
assert "***@***" in anonymized


@pytest.mark.asyncio
async def test_data_anonymizer_handler_none():
from api.tools import data_anonymizer_handler

result = await data_anonymizer_handler({"text": None})
assert result["anonymized_text"] == ""

result_missing = await data_anonymizer_handler({})
assert result_missing["anonymized_text"] == ""

with pytest.raises(ValueError, match="Analysis text must not exceed"):
await data_anonymizer_handler(
{"text": "x" * (ANALYSIS_TEXT_MAX_CHARS + 1)}
)


def test_execute_grammar_checker():
with TestClient(app) as client:
response = client.post(
Expand Down
41 changes: 41 additions & 0 deletions docs/doctoring/data-anonymizer-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Data anonymizer boundary

## Decision and observed implementation

PR #1482 repair parent `034d111b6bef126929d6f0085c2fa15bbf9724be`
stacks on PR #1555 exact head
`03799bc157fa39a419cf6c3f77a29a2ca02cd7f4`. The handler reuses the stack's
canonical ASCII email and selected Korean/North American phone matchers, then
adds bounded Unicode-email, French phone, and Korean resident-registration
patterns. Every input is subject to `ANALYSIS_TEXT_MAX_CHARS` before scanning.

This tool performs deterministic format masking only. It does not measure
re-identification risk, detect names or organizations, transform free-form
quasi-identifiers, or certify that output is anonymous. Product copy must keep
that limitation visible. A downstream workflow that requires release-grade
de-identification needs a documented data model, threat model, risk metric,
review authority, and evidence that the transformed dataset meets its intended
use. It must not infer that assurance from this handler's successful response.

Endpoint regressions cover hyphenated and separator-free Korean identifiers,
an internationalized email address, a French phone number, punctuation
preservation, and the input-size boundary. The values are synthetic test data;
no real person's identifiers are committed.

## Research grounding

NIST SP 800-188 treats de-identification as a managed process involving data
models, techniques, governance, and re-identification risk rather than a small
set of textual substitutions. That distinction supports the deliberately
narrow product claim above and rejects the earlier broad “data anonymization”
assurance.

Garfinkel, S., Guttman, B., Near, J., Dajani, A., & Singer, P. (2023).
*De-identifying government datasets: Techniques and governance* (NIST Special
Publication 800-188). National Institute of Standards and Technology.
https://doi.org/10.6028/NIST.SP.800-188

The official publication page was available during verification, but its
linked PDF endpoint returned HTTP 404 on 2026-09-04. The PR therefore records
the DOI and bounded summary instead of committing an unverified or
redistribution-uncertain binary.