Explore PII-safe Request Logging

[adeutscher]

This issue is a reminder to explore an LLM-proposed item that I feel has merit.

LLM Suggestion Content

ILogger is everywhere, but there is no demonstrated rule for what must not be logged. Middleware or a destructor that redacts Authorization, emails, and upload checksums, plus a scoped request id, would show a production logging policy. Pair it with a test that fails if a known secret appears in log output.

LLM Suggestion Commentary

• Maybe it's just the late hour I'm filing this ToDo item at, but I'm not 100% understanding why upload checksums need to be updated.
  • It's not completely lost on me, though. It's not as much of a smoking gun as an e-mail, but I could see a log message of " uploaded ", therefore someone with ill intent could look for that file with that checksum. Or something. It's a bit of a stretch, and maybe I'm just tired.
• I'll need to clarify the scope of redaction. Would keeping broad details of an email or a partial checksum still be acceptable?

Documentation Action Item

Before closing, document how we apply our redaction in a docs/patterns/ document.

[adeutscher]

Sought the LLM's 2 cents on partially redacting sensitive values, though I'll probably be seeking a second opinion to make sure I'm not buying a hallucination.

Prompt: Elaborate on the scale of redaction. Would keeping broad details of an email or a partial checksum still be acceptable? For example, would it be acceptable to redact foo.bar@example.com down to f***@e***.com? For another example, would a partially-redacted checksum be too revealing?

Response Summary (Human Opinion)

Short answer: No, partial details are not acceptable for hiding PII.

I can't find a fault in the LLM's explanation. On reflection, removing personal information means doing so thoroughly and completely.

Some other key points:

• The response agreed with the caution of the example that I baked into the prompt: Redaction should not reveal the length of the unredacted content.
• Prefer app-specific internal and non-revealing fields such as the user ID (e.g. sub claim in JWT or the Customer ID that this template uses in places), but note that the LLM considers these to be pseudonymous identifiers and not full anonymization. If I am writing logs to a place that can't have any identifying information, it should be hashed (HMAC is recommended). See below.

Response Content

Partial masking is a support aid, not anonymization. Treat the two examples differently. Email still identifies a person. A checksum does not reconstruct a file, but a prefix can still be a lookup key.

Email

Example: foo.bar@example.com -> f***@e***.com

That form is better than logging the full address. It is not enough if the goal is "logs do not contain personal data."

What it still leaks:

• First letter of the local part
• First letter of the domain
• The TLD
• A rough shape of the address

In a large public domain (example.com, gmail.com) that is a weak hint. In this template it is stronger. Customer email is a real column, search supports exact Email and EmailContains, and requests already carry customer_id, roles, timestamps, and often an IP. j***@a***.com plus those fields can be unique in a small tenant.

Length-preserving stars (f******@e******.com) are worse. They leak local-part length.

Under GDPR-style tests, masked email is still personal data if the person is reasonably identifiable, including by combining fields. Do not describe this pattern as redaction that makes the log "clean."

What I would actually log:

• Prefer the customer Guid (and maybe a request id). That is the join key operators need.
• If humans must see a hint, keep the full domain and drop the local part (***@example.com), or HMAC the email with a log-only key and log 8-12 hex of that HMAC. HMAC lets you grep "same person, different requests" without reversing the address.
• f***@e***.com is acceptable only as a lossy display in a locked-down sink, with the understanding it is still PII. It is not acceptable as the privacy control, and it is a bad default for a template that people will copy.

Checksum

Partial checksum is the wrong fear if you are thinking "can someone rebuild the file?" They cannot, from 8 hex or from all 64. SHA-256 of object bytes is a content identifier, not a secret like a bearer token.

The real leak is joinability. This API indexes Sha256Checksum and lets authorized callers search by it. A full hash in a log line is a stable handle: log -> exact upload row. It also confirms "this request was file X" if the reader already has the file.

A prefix is still revealing in that sense, just weaker:

Prefix	Bits	Practical effect
8 hex	32 bits	Useful as a human fingerprint. Collides in larger sets. Often unique in a small table.
12-16 hex	48-64	Often unique in practice. Treat it as almost the full key.
Full 64 hex	256	A database lookup key. Fine in APIs and DB. Poor default in exported request logs.

So a partially redacted checksum is not "too revealing" as a path back to file bytes. It can be too revealing as a way to pick out a specific upload, or to confirm a candidate file (a unique resume, invoice, or dump). Confirmation only needs enough prefix to beat the other files in scope. For a handful of suspects, 8 hex is plenty.

If the log needs to answer "did we store what the client sent?", log the upload id and at most a short prefix (about 8 hex). Do not log a truncated hash and call the checksum redacted. Truncation is not a security property here. It only reduces how well the value works as an index.

Do not confuse this with the OAuth token-cache checksum. That hashes token URL, secret paths, and scopes, not the secret. It is a cache key. Still no need to put it in request logs.

Scale to demonstrate

Three buckets, not one masking function:

• Secrets: Authorization, access tokens, secret-manager values. Replace the whole value. Never eyJ***. A JWT fragment is still a fragment of a credential.
• Direct personal fields: customer Email, and DisplayName in the same bucket. Omit, or HMAC. Character masks like f***@e***.com are optional sugar, not the policy.
• Content hashes: omit from request logs, or 8 hex plus the upload Guid. Full hash stays in the upload APIs and DB, where it already belongs.

A test that greps log output for @ addresses and 64-char hex checksums is more honest than a test that accepts f***@e***.com as success.

Follow-up Question and Response

Prompt: Is the sub field of a JWT token (assume GUID format) safe enough to not be considered PII?

No. A GUID sub is still an identifier of a person. GUID format only makes it a better identifier to log than an email. It does not take it out of PII / personal-data territory.

Why the format does not save it

sub exists to name one subject at the issuer, stably and without reuse (OIDC). If Keycloak (or whoever) maps that GUID to a user row with an email and a name, anyone with the IdP, a support tool, or a leaked user export can join sub to a human. Randomness stops guessing and enumeration. It does not stop linkage.

GDPR talks about personal data, not "PII," and it includes pseudonyms. A value that singles someone out, or that is reasonably linkable to them, still counts. US "PII" lists are messier, but a user ID that distinguishes a person is usually treated as linked or linkable PII even when it is not an SSN or an email.

Same GUID in API logs, rate-limit keys, upload UploadedBy, and Keycloak is the join key. That is the point of sub. It is also why it is personal data.

What GUID format does buy you

• Not reversible to email or name from the string alone
• Not enumerable the way sub=42 is
• Appropriate as the request's "who" in logs, instead of email or preferred_username

That is pseudonymization. It is a safeguard. It is not anonymity.

UUID version still matters a little. v4 (or a random IdP-assigned GUID) is only an opaque key. v1 can leak timestamp and a MAC. Opacity of the string is not the legal test.

How to treat it in this template

This API already uses sub / NameIdentifier as the rate-limit subject and as the upload actor. Logging that same GUID next to a request id is the right operational default.

Do not document it as "not PII." Document it as:

• Direct identifiers (email, display name, raw JWT): omit or HMAC
• Pseudonymous identifiers (sub, customer Guid, upload id): OK in application logs that you already protect like other user-related operational data
• Anonymous: almost nothing in an authenticated request log, because sub plus time plus IP plus customer_id still points at one account

If logs go to a sink that is not allowed to hold personal data, hash sub with a log-only HMAC key as well. At that point you have traded support joinability for a stricter sink. The GUID-looking sub by itself does not get you there.

HMAC Hashing Function

Example of using HMAC hashing. It's like the hashing that I'm already familiar with, but the HMAC key value acts as a salt to make sure that internal processes yield consistent outputs for a given raw value that outsiders without the key cannot replicate with 1-factor brute-forcing:

private static string HashFieldForAudit(string rawValue)
{
    if (string.IsNullOrEmpty(rawValue)) return string.Empty;

    // Initialize HMAC-SHA256 with the secret logging key
    using (var hmac = new HMACSHA256(AuditHmacKeyConstant))
    {
        byte[] rawBytes = Encoding.UTF8.GetBytes(rawValue);
        byte[] hashBytes = hmac.ComputeHash(rawBytes);

        // Convert to Hexadecimal string (common format for log outputs)
        return Convert.ToHexString(hashBytes).ToLowerInvariant();
    }
}