Do You Need an LLM Gateway? Self-Hosted vs. Managed for Regulated AI

Do you need an LLM gateway, and does it make your AI compliant? You need one the moment more than a feature or two sends regulated data to a model provider, and it does not make you compliant on its own — it is where you enforce the compliance boundary, not a compliance product you buy. An LLM gateway is a proxy that sits between your services and the model APIs, centralizing auth, routing, redaction, logging, and key management in one place instead of scattering them across every service that calls an LLM. For regulated AI the real question is not managed versus self-hosted in the abstract; it is whether the place you enforce that boundary sits inside your compliance perimeter or inside a vendor’s.

This is a deep dive under the AI governance framework hub. That framework names five controls that keep AI features inside your SOC 2 or HIPAA boundary; the gateway is the piece of infrastructure where four of them actually get enforced. If you are still deciding whether a given feature can ship at all, start with the pre-flight in adding AI features without breaking your compliance posture and come back here to decide where the controls live.

What is an LLM gateway?

An LLM gateway is a reverse proxy in front of your model providers that centralizes the cross-cutting concerns every AI feature needs: authentication and per-app internal keys, routing and failover across providers, rate limiting, cost and token tracking, request/response logging, PII/PHI redaction, and a single provider-agnostic API. Instead of each service holding its own OpenAI or Anthropic key and implementing its own logging, every service calls the gateway, and the gateway calls the provider.

It is the API-gateway pattern applied to model calls. The value is consolidation: one place that knows how to talk to every provider, one place where a prompt can be inspected before it egresses, one place where a model call is recorded, and one place where a leaked key can be rotated without a redeploy of six services. In a system with a single AI feature you may not feel the need. By the fifth feature — a summarizer, a classifier, a draft-generator, a support-bot, an embedding pipeline — the alternative to a gateway is the same four controls implemented five times, slightly differently, with one of them wrong.

Popular self-hostable options include LiteLLM and Portkey’s open-source proxy; managed and SaaS gateways, plus the model platforms’ own gateway features, cover the same surface as a service. Feature sets and pricing move fast in this category, so treat any specific capability as list-and-approximate and verify the current split before you commit.

What does an LLM gateway do for compliance?

For regulated AI, the gateway is the single chokepoint where you enforce the compliance boundary your governance framework defines — which is exactly why where it runs matters more than which product it is. Four jobs sit here, and each maps to one of the leak points an LLM feature opens:

• Scrub PII/PHI before it leaves your boundary. The gateway is the last hop you control before the prompt reaches a third party, so it is the natural place to apply minimum-necessary redaction — strip or tokenize the fields the model does not need to see. This only works if the scrub runs before egress, which is to say inside your perimeter.
• Restrict egress to BAA-covered providers only. A gateway can refuse any destination you have not covered with a Business Associate Agreement (or the DPA your privacy commitments require). That turns “we think everyone uses the approved provider” into an enforced allowlist — the difference between a policy and a control.
• Log inputs and outputs for audit — content-free. The gateway emits one record per call so you can answer “what model saw what, when, and who approved the output” without keeping a second copy of the data. Log the shape, not the payload; the trace schema and the reason full prompt logs are the wrong answer are in proving your AI features work: auditability for LLM systems.
• Rotate provider keys centrally. One custody point for OpenAI, Anthropic, and every other provider key means rotation is a single operation, not a scavenger hunt across services — the same discipline as killing long-lived credentials, applied to model providers. Downstream services hold an internal gateway key, not the provider’s, so a leaked internal key never reaches the provider directly.

The catch is that every one of those controls is only as good as its location. Redaction that runs after the data has already reached a third party is not redaction. A log store you do not control is an out-of-boundary regulated-data store. That is the whole self-hosted-versus-managed question in one sentence.

Self-hosted vs. managed vs. no gateway: how do they compare?

Self-hosted keeps data, logs, and keys inside your VPC at the cost of running the proxy; managed offloads the ops but makes the gateway vendor a new sub-processor that sees your prompts; no gateway avoids both but duplicates every control across every service and gives you no chokepoint. The table is the decision in one view.

Read the redaction and logging rows together, because they are the crux. A managed gateway that runs redaction on its side has already received the raw PHI by the time it scrubs — which is fine for de-identified traffic and disqualifying for protected data unless that vendor is itself under a BAA. The residency and sub-processor rows are the same point from the data-flow angle: a managed gateway moves your PHI/PII boundary outward to wherever the vendor runs, exactly the outbound edge the governance framework tells you to map before you ship.

Is a managed AI gateway a sub-processor?

Yes — a managed or SaaS gateway sees every prompt and response you route through it, which makes it a sub-processor processing your data on your behalf, exactly like the model provider behind it. It is the easy one to forget, because it feels like plumbing rather than a data processor. It is both.

Treat it like any vendor with standing access to sensitive data: disclose it on your public sub-processor list, sign the DPA, and run it through the vendor risk scoring rubric before access is granted — a gateway that can read prompts built from patient records scores high on data sensitivity and access depth, which is precisely the vendor that earns a real review. If PHI flows through it, you need a BAA with the gateway vendor too, not only with the model provider, because the gateway is handling PHI in its own right. An undisclosed AI gateway discovered mid-review reads the same way an undisclosed model vendor does: like you lost track of where the data goes.

There is a middle path that dodges the new-sub-processor problem: Amazon Bedrock. It is managed — AWS runs the models and the routing, so you skip the ops of a self-hosted proxy — but it sits inside AWS, under your existing AWS BAA, which covers its HIPAA-eligible services (verify current eligibility, as the list shifts). Because it is already in your AWS boundary, using Bedrock does not add a new external sub-processor the way a third-party SaaS gateway does. It is the managed-but-in-boundary option, and for many regulated SMBs already on AWS it is the shortest path to a compliant setup. You still put your own thin gateway or middleware in front of it for redaction, internal keys, and content-free logging — Bedrock is the model boundary, not the whole control set.

Which should you pick?

Pick a self-hosted in-VPC gateway (or Bedrock under a BAA) when regulated data hits the prompt, and a managed gateway when it does not. That is the verdict, and it follows directly from the residency and redaction rows above: for PHI or other regulated data, the redaction, the logs, and the provider keys have to stay inside your perimeter, which a self-hosted proxy or Bedrock gives you and an external SaaS gateway does not. For de-identified, synthetic, or genuinely low-stakes traffic — internal tooling, public-data summarization — a managed gateway is a fine trade: you get the convenience and you pay for it with one more disclosed, scored sub-processor.

Going without a gateway at all is defensible only at one or two AI features. Past that, the absence of a chokepoint is the risk: no single place to enforce the BAA allowlist, no single place to verify redaction ran, no single trace store to hand an auditor. The centralization is the point, and the point arrives fast.

The good-enough bar: a regulated SMB has this right when it can name one place where PII/PHI is scrubbed before egress, where provider keys are rotated, and where every model call is logged content-free — and can say honestly whether that place is inside its boundary or a vendor’s. If the answer is “inside ours” (a self-hosted proxy or Bedrock), you clear the bar for regulated data. If it is “a managed vendor’s,” you clear it for de-identified data provided the vendor is disclosed and scored. The failing answer is the one most teams actually have: redaction and key handling copy-pasted across five services, no chokepoint, and no one who can say where the March prompts went.

The Template

Two parts: decide whether and where the gateway lives, then confirm the controls actually run there. Score each item honestly. Any no in the control section blocks regulated traffic through that path until it is resolved.

Do you need one? (gateway trigger)

• More than one or two features send data to a model provider
• Any feature sends PHI, PII, or other regulated data into a prompt
• Provider keys currently live in more than one service or secret store
• You cannot today answer “what model saw what, when” from one place
• Two or more of the above true → stand up a gateway; regulated data true → it must be in-boundary

Where does it live? (build-vs-buy)

• Regulated data in prompts → self-host in-VPC, or use Bedrock under your BAA
• De-identified / low-stakes only → managed gateway acceptable if disclosed + scored
• Already on AWS and want managed → evaluate Bedrock before a third-party SaaS gateway
• If self-hosting: a named owner can deploy, patch, scale, and get paged on the proxy
• If managed: vendor scored on the vendor-risk rubric, on the sub-processor list, DPA signed (BAA if PHI)

Controls that must run at the gateway

• PII/PHI redaction runs before egress, inside your boundary (not on a vendor’s side for PHI)
• Egress allowlist enforces BAA/DPA-covered providers only — unapproved destinations refused
• Provider keys held once, in-boundary; rotated centrally; downstream services hold internal keys only
• Every call emits a content-free trace (ids, versions, hashes, flags, decision) to your own evidence store
• Prompts/responses are NOT logged as content to the gateway’s or a vendor’s default dashboard
• Model versions pinned (dated snapshots, not moving aliases) and recorded per request
• No silent fallback to a hosted tier that would route prompts back outside the perimeter

Prove it

• The one place data is scrubbed, keys rotate, and calls are logged is named and documented
• You can state, per feature, whether that place is inside your boundary or a vendor’s
• The gateway (self-hosted or managed) is in scope for the next access review + evidence pipeline
• A re-review triggers when a provider is added, a vendor changes terms, or the data flow changes

Most SMBs find the trigger section mostly true by their third or fourth AI feature and land on a self-hosted proxy in-VPC or Bedrock for anything touching regulated data, with a managed gateway reserved for the de-identified lane. All three shapes are defensible for the right data class. The one indefensible position is having no chokepoint at all once the AI features multiply — because that is the position where you cannot answer where the data went.