AI Governance for Regulated Products: A Practical Framework

What is an AI governance framework, and what does a practical one look like for a regulated product? An AI governance framework is the set of controls, records, and decision gates that keep your AI features inside the compliance boundary you already built for SOC 2 or HIPAA — data-flow mapping, signed agreements with the model vendors, retention and training controls, human oversight on consequential output, and model/version tracking, each producing evidence an auditor or buyer will accept. It is not an ethics manifesto or a standing AI committee. It is the same operational discipline you apply to any system that touches regulated data, extended to a component that happens to be a probabilistic model behind someone else’s API.

This is the hub for the AI section of our library. Each part below is the map; the linked deep dives are the terrain. AI features don’t sit in a separate regime — they fall squarely under the SOC 2 and HIPAA umbrella you’re already carrying, so if you’re new to that machinery, start with the complete SOC 2 readiness guide and come back. Governance is what you add so the AI feature doesn’t quietly reopen everything that guide had you close.

Where does AI actually create compliance risk?

An AI feature creates risk in four specific places, and none of them show up in a demo. The demo shows a clean input and a useful output. The audit, the customer security review, and the breach investigation show the four things happening underneath.

• The LLM provider is a new sub-processor. The moment your app calls a model API with customer data in the payload, that vendor is processing regulated data on your behalf — the same relationship you have with AWS or your database host, and it carries the same obligations. Vet it the way you’d vet any vendor with standing access to sensitive data: run it through the vendor risk scoring rubric, score the data sensitivity and access depth honestly, and put it on your public sub-processor list. An undisclosed AI sub-processor discovered mid-review is one of the fastest ways to lose a deal.
• The PHI/PII boundary moves to wherever the prompt goes. Whatever context you assemble to send the model — the chart note, the record, the message — is now leaving the systems you spent a year putting inside your compliance boundary. If you’re on the healthcare path, the boundary you drew in minimum-viable HIPAA on AWS just grew an outbound edge, and the minimum-necessary principle applies to prompts as much as to database queries.
• Retention and training-on-your-data. Your data doesn’t just transit the vendor; it may be retained by them, and on some tiers used to improve their models. This is the risk that turns “we sent one prompt” into “our customer’s data is in a training corpus.”
• Prompt and output logging. Your app logs the prompt for debugging, your observability stack captures it in a trace, your eval pipeline saves inputs and outputs to measure quality — and every one of those is now a regulated-data store you never architected, usually with the wrong retention and the wrong access controls.

The discipline that catches all four before you ship is a pre-flight run per feature. The tactical version — four gates, answered in order, any failure blocks the ship — is adding AI features without breaking your compliance posture. Governance is what makes that pre-flight a standing control instead of a one-time heroics.

What goes into an AI governance framework?

Five controls, and each maps to a risk above. This is the whole framework; everything else is elaboration.

1. Data-flow mapping. For every AI feature, trace every field that enters the prompt, where it came from, where the response is written, and where any of it is logged. You can’t govern a data flow you haven’t drawn. The map is also the artifact that answers half of a buyer’s questions before they ask.
2. Signed BAAs / DPAs with model vendors. No PHI into a prompt without a Business Associate Agreement; no personal data into one without the DPA your privacy commitments require. This is the gate that stops most teams cold because it’s the one usually skipped. “We’ll get the BAA later” is how regulated data ends up somewhere it can never be recalled from.
3. Retention and no-training terms. Confirm in writing that the vendor does not train on your inputs, set the vendor’s retention of your data to something you can defend, and re-verify on a schedule because AI vendors change terms far more often than infrastructure vendors do.
4. Human-in-the-loop on consequential output. The more consequential the output, the more the human has to be in the decision, not merely notified of it. Draft, don’t decide; surface, don’t act. This is your best defense against the confidently-wrong answer you didn’t catch in testing, and it’s the control buyers probe hardest.
5. Model and prompt version tracking. Pin dated model snapshots rather than moving aliases, keep prompt templates in version control behind reviewed pull requests, and record which versions were live per request. This is what makes “what was running on March 14” answerable and folds AI behavior into your existing change-management control.

The framework earns its keep only when each control leaves a trace. Governance you can’t produce evidence for is a story, and stories fail Type II audits. Map every control to the artifact that proves it:

Read the evidence column and notice what it is: the same access, change-management, and logging evidence your SOC 2 or HIPAA program already produces, pointed at a new component. That’s the load-bearing idea. You are not standing up a parallel compliance regime for AI — you are extending the one you have.

How do you prove the framework works?

You prove it with provenance and process, never with stored content — and that distinction is the whole design. When an auditor is in fieldwork with your AI feature in scope, or a buyer’s security team runs the diligence call after the questionnaire, the questions converge: what data did the model see, who approved the output before it took effect, what model and prompt version were live on a given date, how do you know quality isn’t degrading, and what happened during the incident on X. Every one of those is answerable from metadata.

The mechanism is a content-free trace: one record per request carrying request and tenant ids, timestamp, the pinned model id, the prompt-template version, keyed hashes of each variable input (so you can prove provenance without keeping a second copy of the data), token counts and flags, and the downstream decision — auto-applied, sent to human review with the review id, or discarded. Pair that with human-review records treated as first-class data and a scheduled eval history, and you can answer the diligence call with a query instead of a scramble. The full trace schema, the retention traps, and the reason full prompt logs are the wrong answer are in proving your AI features work: auditability for LLM systems. The short version: the team that logged everything has eighteen months of PHI-soaked prompts they legally can’t hand over, and the team that logged nothing has claims with no artifacts. Content-free traces are the hand that beats both.

The reason the “log the shape, not the content” rule matters twice over is that it’s the same rule that keeps request bodies out of your logs generally — the discipline in securing PHI in observability data, extended one layer up to keep prompts out of your traces.

What will enterprise and healthcare buyers ask?

They’ll ask five clusters of questions, and they aren’t testing whether your AI is flawless — they’re testing whether you’ve thought about the AI-specific risks well enough to answer honestly. A CISO doesn’t expect your Series B to have a formal AI governance board. They expect you to know where the data goes, who the model vendor is, and what happens when the model is confidently wrong.

The clusters: data handling (do you train on our data, where do prompts go, what’s the retention), vendor and sub-processor (which provider, BAA or DPA in place, are they on your list), output safety (can the AI act irreversibly, is there human review, how do you handle hallucinations), isolation (is our data isolated at the model layer, could it surface in another tenant’s output), and governance (who approves AI features, what’s the incident plan for a harmful output). The pre-filled answer bank — fifteen of the most common questions with defensible answer patterns — is in what enterprise buyers actually ask about your AI features. Every good answer there is a claim your governance framework has already made true and can prove; the questionnaire is just where the framework becomes visible to the person writing the check.

Notice the loop the cluster closes: the buyer’s questions map one-to-one onto the five controls, and the auditability evidence is what makes each answer defensible. Build the framework and the questionnaire answers itself. Skip it and every deal re-litigates the same five themes from scratch.

The good-enough bar

The bar for a 10-80-engineer regulated SMB is legibility, not sophistication. You do not need an AI ethics board, a governance platform, or a research team red-teaming your prompts. You need to be able to, for any AI feature, name the model vendor and show the signed agreement, produce the data-flow map, demonstrate that prompts are logged as shape and not content, and prove a human reviews anything consequential. If you can do those four things on demand, you clear the bar that keeps you out of a breach notification and through a security review.

The teams that get this wrong aren’t reckless. They’re moving fast on a feature that looks like every other feature and isn’t — one that added a sub-processor, moved a data boundary, and created a log store, all in a single sprint. The framework is cheap precisely because it turns those invisible changes into a visible, answerable checklist, run before the feature ships instead of after the audit finds it.

The Template

An AI feature governance checklist. Run it before every AI feature that could touch regulated data, and re-run it per feature and per vendor — the BAA you signed for the summarization feature doesn’t cover the new vendor you’re using for embeddings. Score each item yes / no / not-applicable; any no on a required item blocks the ship until it’s resolved.

Data flow (map it first)

• Traced every field that enters the prompt; confirmed it’s the minimum necessary
• Confirmed no PHI/PII in URL parameters, feature flags, or analytics events for this feature
• Traced the response path; confirmed output isn’t written anywhere out of scope
• Data-flow map recorded and attached to the feature

Vendor + agreements

• Model vendor scored on the vendor-risk rubric before access was granted
• BAA signed with the vendor (or PHI is de-identified before it leaves our systems)
• DPA in place where personal data is involved
• Vendor added to the public sub-processor list
• Data-use terms reviewed: confirmed no training on our inputs, opt-out verified in writing
• Vendor’s retention of our data documented and acceptable

Logging + retention

• Prompts + responses are NOT logged as content to general application logs
• Traces are content-free (ids, versions, hashes, flags, decision)
• Anything retained lives in a scoped, encrypted, in-boundary store with production-grade access controls
• Retention period set deliberately, not left at the vendor default

Output safety + oversight

• Consequential outputs have a human in the decision, not just a notification
• Low-confidence / flagged outputs are surfaced and logged
• Users can tell the content was AI-generated
• A documented plan exists for when the model is confidently wrong

Versioning + change management

• Model version is a pinned dated snapshot, recorded per request (no moving aliases)
• Prompt templates live in version control and change via reviewed PR
• A model or prompt change triggers an eval run with a promotion threshold

Evidence + ongoing

• Every control above maps to a produced artifact (agreement, map, trace, review record, eval history)
• Human-review coverage is queryable: consequential outputs with no linked review = 0 rows
• This feature is in scope for the next access review + evidence pipeline
• A re-review triggers when the vendor changes terms or we change the data flow

The checklist takes about an hour per feature. The alternative — an undisclosed sub-processor, an out-of-scope PHI log, and an unanswerable “prove it” during a customer’s diligence call — takes a lot longer and usually costs the deal.