The Problem
Every healthcare-adjacent SaaS is adding AI features right now. Summarize the chart note. Draft the prior-auth letter. Answer the patient’s question. The demos are compelling and the pressure to ship is real.
The compliance implications almost never get traced before the feature ships. An LLM feature isn’t a normal feature. It opens four new ways for protected health information to leave the boundary you spent a year building:
- The prompt. Whatever context you assemble to send to the model — the chart, the record, the message — is now leaving your systems for the vendor’s.
- The vendor. The model API is a new sub-processor, processing PHI on your behalf, usually without a signed agreement.
- The logs. Prompts and responses get logged for debugging, evals, and observability. Those logs are now a PHI store you didn’t architect.
- The output. A hallucinated dosage or a confidently wrong eligibility answer is a patient-safety event, not a bug.
None of these show up in a demo. All of them show up in an audit, a customer security review, or a breach investigation. The fix is to run the same pre-flight before every AI feature, the way you’d run a threat model before a new system.
The Approach
Treat every AI feature as a compliance change, not just a product change. Four gates, answered in order. If any gate fails, the feature doesn’t ship until it’s fixed.
Sign a BAA with the model vendor, or don’t send PHI
This is the gate that stops most teams cold, because it’s usually skipped. If protected health information is going into a prompt, the model vendor is processing PHI on your behalf. Under HIPAA that requires a Business Associate Agreement, the same as it does for AWS or your database host.
The major providers offer BAAs, but not on every tier and not by default. OpenAI signs one for API customers who request it and meet the terms; the free ChatGPT product does not qualify. Anthropic, Google, and AWS Bedrock have their own paths. The point is that the BAA is a specific artifact you have to go get, not something that comes with the API key.
If you can’t get a BAA, or the feature routes through a vendor who won’t sign one, you have two honest options: de-identify the data before it leaves your systems (Safe Harbor or Expert Determination), or don’t build the feature that way. “We’ll get the BAA later” is how PHI ends up in a training corpus.
Treat the model API as a sub-processor
Once the BAA is signed, the vendor belongs on your sub-processor list — the same public list your customers review, the same list your DPA references. Add it. A customer security review that finds an undisclosed AI sub-processor processing their data is a trust problem you don’t recover from quickly.
While you’re there, read the data-use terms. The question that matters: does the vendor train on your inputs by default, and can you turn it off? For most enterprise API tiers the answer is “no training, opt-out by default,” but the terms change and the default varies by product. Verify it in writing, and re-verify it when the vendor updates their terms.
Prompts and responses are a new PHI store
This is the leak point teams miss most often, because it’s invisible. Your app logs the prompt for debugging. Your observability stack captures the request and response in a trace. Your eval pipeline saves inputs and outputs to measure quality. Every one of those is now storing PHI, usually in a system that was never in your HIPAA scope.
The controls are the same ones from securing PHI in observability data, applied to a new source:
- Don’t log full prompts or responses for PHI-bearing features. Log the shape — token counts, latency, model version, a request ID — not the content.
- If you must retain prompts for evals, retain them in a scoped, encrypted store inside your HIPAA boundary, with the same access controls as your production database.
- Set the retention deliberately. A prompt log that keeps PHI for two years “just in case” is a two-year breach window.
Put a human in the loop for anything consequential
An LLM will produce a confident, fluent, wrong answer. In a consumer app that’s an annoyance. In a clinical or eligibility or financial context it’s a safety event and a liability.
The rule that holds up: the more consequential the output, the more the human has to be in the decision, not just notified of it. Summarizing a note for a clinician to review — the clinician is the gate, fine. Auto-sending a prior-auth denial with no human review — not fine. Draft, don’t decide. Surface, don’t act. The human gate is also your best defense against the hallucination you didn’t catch in testing.
The Template
Run this before every AI feature that could touch regulated data. Each item is yes / no / not-applicable. Any no on a required item blocks the ship until it’s resolved.
Legal + vendor
- BAA signed with the model vendor (or PHI is de-identified before it leaves our systems)
- Vendor added to the public sub-processor list
- Vendor data-use terms reviewed: confirmed no training on our inputs
- Vendor’s own retention of our data is documented and acceptable
Data flow
- Traced every field that goes into the prompt; confirmed the minimum necessary
- Confirmed no PHI in URL parameters, feature flags, or analytics events for this feature
- Response path checked: PHI in the output isn’t written anywhere out of scope
Logging + retention
- Prompts + responses are NOT logged in plaintext to general application logs
- Any retained prompts/responses live in a scoped, encrypted, in-boundary store
- Retention period for prompt/response data is set deliberately, not left at default
- Observability traces for this feature strip or exclude prompt/response content
Output safety
- Consequential outputs have a human in the decision, not just a notification
- Users can tell an AI generated the content
- There’s a documented plan for when the model is confidently wrong
Ongoing
- This feature is in scope for the next access review + evidence pipeline
- A re-review triggers when the vendor changes terms or we change the data flow
The checklist takes an hour to run for a new feature. The alternative — finding an undisclosed sub-processor and an out-of-scope PHI log during a customer’s security review — takes a lot longer and costs the deal.
Operating Notes
Run the pre-flight per feature, not per company. The BAA you signed for the summarization feature doesn’t cover the new vendor you’re using for embeddings. Each distinct model vendor and each distinct data flow gets its own pass.
Watch the vendor terms. AI providers update their data-use and retention terms far more often than your database host does. Put a quarterly reminder on the calendar to re-read the terms for every model vendor on your sub-processor list. A default that flips from “no training” to “training unless you opt out” is the kind of change that doesn’t send you an email.
The teams that get this wrong aren’t reckless. They’re moving fast on a feature that looks like every other feature and isn’t. The pre-flight is cheap precisely because it turns an invisible compliance change into a visible, answerable checklist — before the feature ships, not after the audit finds it.