Author(s): Marie Humbert-Droz, PhD

Originally published on Towards AI.

Every trace restates the patient’s note — in extractions, tool calls, and reasoning.

I added observability to my local clinical AI agent, opened the trace viewer, and stared at a patient’s full name, date of birth, and phone number sitting in plain text in a database I’d just spun up.

After setting up a local observability backend (Langfuse) and a masking layer (Presidio), the author finds that “redacting the note” is not enough because PHI can reappear in other trace payloads like extraction JSON, tool arguments, and the model’s reasoning. They show that PHI detection itself is brittle: NER misses templated fields and nonstandard date formats, detection behavior is context-dependent, and eponym surnames (disease names that also look like patient names) create difficult false positives and false negatives. The most serious “leak surface” is that each trace shape must be protected independently, since masking failures can happen after the model restates the PHI. For dates, simple placeholders break debugging for tools that do date arithmetic, so they adopt date-shift surrogates to hide absolute dates while preserving interval logic—turning a failing verification task into a passing one. The final policy reduces end-to-end leaks from 135 to 17 and leads to practical guidance: mask in-process pre-export, validate detection across exported trace components, exploit document structure before upgrading models, measure leaks against exported traces, and use date shifting instead of deleting information when trace fidelity matters.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI