Proving Your AI Features Work: Auditability for LLM Systems

How do you prove an AI feature works to an auditor or enterprise buyer? Give them provenance and process, not payloads. Log a content-free trace of every request (ids, model versions, keyed hashes, flags), keep human-review records and scheduled eval scores as linked data, and flow it all into the evidence store you already run.

The Problem

The request arrives after the questionnaire, not with it. You answered the AI section, the deal moved to diligence, and now the buyer’s security team wants a working session: “You said consequential outputs get human review. Show us one. Show us the review record for an output from March. Show us which model version was live that week.” Or it arrives during SOC 2 fieldwork, once the auditor notices your product description mentions AI and asks how the change-management control applies to prompts.

The first losing hand is nothing. The team can produce the model vendor’s usage dashboard, a Datadog latency graph, and a Slack channel called #ai-weirdness where someone occasionally pastes a bad output. The claims made on the questionnaire (human in the loop, quality monitoring, controlled changes) have no artifacts behind them. The assessor writes it up as a gap; the buyer’s security team reads “we’ll get back to you” as “we made it up.”

The second losing hand is worse. A team that took observability seriously and logged everything now has eighteen months of full prompts and completions in their logging stack. The prompts were constructed from patient records. The log store is an unplanned PHI repository with the wrong retention, the wrong access controls, and no entry in the data inventory. Producing this evidence means handing over protected data: to a buyer’s security team it’s an impermissible disclosure outright, and even for your own auditor it means BAA and minimum-necessary gymnastics for what should have been a metadata exercise. Both hands are the same mistake from opposite directions: treating AI evidence as either optional or contentful. What an assessor actually needs is provenance and process. Payloads never enter into it. This evidence layer is how you prove the rest of your AI governance framework actually holds up — a control you can’t produce artifacts for is just a claim.

The Approach

Five artifact streams, one store, and a query interface. None of the artifacts contain customer content.

The questions you will actually get

Across SOC 2 fieldwork with AI features in scope, HIPAA risk assessments, and the diligence calls that follow the AI section of a security questionnaire, the asks converge on five:

1. What did the model see? Which data entered prompts, from which source systems, under what authorization. Inputs governance.
2. What did it produce, and who approved it? For any output that took effect, the reviewer, the decision, and the timestamp.
3. What was live on a given date? Model, model version, prompt template, sampling configuration, for any request in the period.
4. How do you know quality isn’t degrading? Not a vibe. A measurement, taken on a schedule, with history.
5. What happened during the incident on X? Reconstruct the AI activity in a time window: volumes, versions, flags, overrides.

Read the list again and notice what’s absent: none of the five requires stored content. Every one is answerable from metadata, version identifiers, and linked records. That observation is the whole design.

Trace without content

The core mechanism: log the shape of every AI interaction, never the payload. One record per request:

• request id and tenant id
• timestamp
• model provider, model id (a dated snapshot, not an alias), endpoint
• prompt-template id
• keyed hash (HMAC-SHA-256, key held outside the log pipeline) of each variable input, plus the name of the source system it came from
• token counts in and out, latency
• safety, guardrail, and confidence flags
• the downstream decision: auto-applied, sent to human review (with the review id), or discarded

The hashes are the clever part — provided they’re keyed. An unsalted SHA-256 of a DOB or an MRN is a dictionary attack away from recovery, which is why variable inputs get HMAC-SHA-256 with a key held outside the log pipeline. You already hold the patient record or the customer document in a system of record with its own access controls and retention. Hashing the exact bytes that entered the prompt lets anyone holding the key later prove input provenance: pull the source record, rehash it, match. The trace asserts “this input, from this system, entered this request” without becoming a second copy of the data. This is the same discipline as securing PHI in observability, extended one layer up: the rule that keeps request bodies out of your logs also keeps prompts out of your traces.

A content-free trace answers questions 1, 3, and 5 outright, and provides the spine that questions 2 and 4 link into.

Version everything that shapes output

An LLM output is a function of the prompt template, the variable inputs, the model version, and the sampling parameters. You have hashes for the inputs. The other three need versioning:

• Prompt templates live in git. A dedicated repo (or a directory with its own CODEOWNERS), one file per template, changes via reviewed PR. The template id in the trace is the short commit SHA of the version in production. The repo’s git log is now your change-management evidence for AI behavior: who changed the prompt, who approved it, when it shipped. This is the same move GitOps makes for infrastructure, and assessors who have seen that pattern accept this one immediately.
• Model versions are pinned, never aliased. Vendors move aliases silently; “latest” or an undated model name points somewhere new next quarter, and your answer to “what was live on March 14” becomes unknowable. Pin the dated snapshot id, record it per request, and treat a model upgrade as a change with an approval and an eval run (below).
• Sampling config rides in the trace. Temperature, max tokens, any tool configuration. Cheap to record, awkward to be unable to produce.

Human-review records as first-class data

Every “human in the loop” claim you made on the buyer questionnaire is a claim that a record exists. A Slack thread where a clinician said “looks fine” is not that record. Build a review table like any other audit table: review id, trace id, reviewer identity from your IdP (not a free-text name), decision as an enum (approve, edit, reject), a hash of the edited output if changed, timestamp, and a required reason field on any override of a safety flag.

The load-bearing property is linkage. The query an assessor actually wants is: consequential outputs in the period with no linked review record. The correct result is zero rows. If your product has a lane where outputs auto-apply, the trace’s decision field documents that honestly, and the review-coverage query scopes to the lanes where you claimed review. Aspirational claims die here, which is the point: better in your own Athena console than in the buyer’s diligence call.

Evals as recurring evidence

“How do you know quality isn’t degrading” has exactly one acceptable answer: a fixed eval set, run on a schedule and on every change, with retained results. The set is de-identified or synthetic (never raw production data; the eval store is not exempt from the no-content rule). Fifty to two hundred cases covering your feature’s real distribution is a legitimate start; a thousand-case benchmark you never run is not.

Run it weekly by cron and in CI on every prompt-template or model-version change. Store each run’s scores keyed by template SHA, model id, and eval-set version. Set regression thresholds that gate promotion: a prompt change that drops the pass rate below threshold fails the pipeline the way a failing unit test does. The retained history is the “AI QA program” an assessor will accept, because it has the three properties they check for: it’s defined, it’s recurring, and it produces artifacts. It also answers the model-upgrade question before anyone asks it, since every version bump has a before-and-after score attached.

Wire it into the evidence pipeline you already have

Do not build a parallel evidence system for AI. Trace records, review records, and eval results are events; they flow through the same collector as your identity and change events (EventBridge or Firehose into S3 with Object Lock, Parquet alongside for query speed, Athena on top) as described in evidence pipelines 101. The AI events become three more partitioned tables next to CloudTrail.

The payoff is the interaction model. When the auditor asks question 5, you don’t assemble screenshots; you run a query joining traces to review records for the incident window and export the result. When the buyer asks question 3, the answer is a one-line filter. Evidence pipelines convert audit requests from projects into queries, and AI evidence inherits that conversion for free once it lands in the same store.

The Template

The map ties each question you’ll be asked to the artifact that answers it. Anywhere you can’t fill in the source column for your own stack is the gap to close before the next audit cycle.

Two retention traps hide in that source column. GitHub’s native audit log holds events for months, not years, so it cannot carry a multi-year retention claim on its own: lean on merged-PR review metadata — which persists with the repository indefinitely — as the primary approval evidence, and stream audit-log events into the evidence store (GitHub audit log streaming) if you want them retained alongside it. Same trap with CI: pipeline artifacts expire in weeks, so CI is the producer of eval records, and the eval-results table in the evidence store is where they survive.

The trace record that makes the top rows answerable:

{
  "trace_id": "01J9F3T7Q8Z4",
  "tenant_id": "t_4821",
  "timestamp": "2026-09-02T14:31:07Z",
  "feature": "discharge-summary-draft",
  "model": {
    "provider": "anthropic",
    "id": "us.anthropic.claude-sonnet-4-20250514-v1:0",
    "snapshot": "claude-sonnet-4-20250514",
    "endpoint": "bedrock"
  },
  "prompt_template": {
    "id": "pt-a41f9c2",
    "repo": "acme/prompts",
    "committed_at": "2026-08-19T09:12:44Z"
  },
  "params": { "temperature": 0.2, "max_tokens": 1024 },
  "inputs": [
    {
      "name": "patient_record",
      "source": "records-api",
      "source_version": "v_20260902_1425",
      "hmac_sha256": "9f2b8c…",
      "tokens": 1893
    },
    {
      "name": "clinician_note",
      "source": "notes-api",
      "source_version": "v_20260902_1429",
      "hmac_sha256": "b71e04…",
      "tokens": 412
    }
  ],
  "output": {
    "sha256": "e4c1aa…",
    "tokens": 587,
    "finish_reason": "end_turn"
  },
  "latency_ms": 2140,
  "flags": {
    "safety_filter_triggered": false,
    "low_confidence": false,
    "guardrail_blocked": false
  },
  "decision": {
    "type": "human_review",
    "review_id": "rev-88f2",
    "outcome": "approved_with_edits"
  }
}

Every field is an identifier, a version, a count, a flag, or a hash. The model id is the one the platform actually invoked — here a Bedrock cross-region inference profile, which can route inference outside the calling region, which is why the endpoint field makes no region claim — so it joins directly against invocation logs and CloudTrail, exactly the cross-check an auditor runs; the snapshot field carries the normalized name for comparing across platforms. Nothing in the record is customer content, so it’s designed to sit outside the PHI boundary and flow to the evidence store in ordinary observability retention — confirm the hashed-trace determination with your privacy officer, and never hash a small-value-space field unkeyed.

Operating Notes

A hash is only evidence while the source is durable

The keyed hash proves provenance only if the bytes it was computed over can be produced again. If your source systems mutate records in place, the hash of March’s input won’t match September’s record and the proof evaporates. Record the source system’s own version id next to the hash (as above), and confirm the source’s retention meets your evidence retention. Where a source can’t version, snapshot the input into the source system’s audit trail at request time. That snapshot belongs to the system that already holds the data, not to your trace.

Don’t buy the observability platform before the discipline

The LLM-observability market (LangSmith, Langfuse, Braintrust, and a dozen others) defaults to capturing full prompts and completions, because that’s what makes their debugging UX good. Pointed at a PHI-bearing feature with defaults on, they recreate the second losing hand as a SaaS sub-processor you now have to disclose. The tools are fine for eval orchestration and for content-off tracing, and self-hosting inside your boundary changes the calculus. But the compliance artifact is the content-free trace flowing to your own store, and that’s 200 lines of middleware. Build that first; evaluate vendors after, with content capture treated as a data-flow decision rather than a default.

The evidence is also your operations substrate

Records built for the auditor answer operational questions all week. Token counts by tenant and feature, multiplied by the vendor’s price sheet, is your AI cost model. Latency and guardrail-flag rates by template SHA tell you whether Tuesday’s prompt change caused Wednesday’s pager noise. Review-decision ratios (edit rates creeping up, rejections clustering on one input source) catch quality drift between eval runs. Teams that treat AI auditability as pure compliance overhead build it grudgingly and thin. Treat it as the feature’s flight recorder and it earns its keep monthly, with the audit answer as a byproduct.