Auditable LLM Features in a Regulated .NET Backend:
What to Record for Every Answer

A reviewer picks one answer your AI feature gave three months ago, a claims summary or a draft reply to a customer, and asks: why did it say that? Your logs hold the prompt and the reply. They do not say which version of the system prompt was live that day, which model sat behind the deployment name, which version of the policy document the retrieval step read, or whether a person changed the text before it went out. An LLM audit trail exists to answer that question, and one built from ordinary application logs cannot.

You cannot count on answering by running the request again. The Azure OpenAI documentation treats chat output as nondeterministic by default, and its reproducible-output feature is best effort: determinism is not guaranteed even when the seed and the system fingerprint match. The inputs have also moved on, because deployments get upgraded and documents get replaced. The record has to be written when the answer is given. It cannot be rebuilt afterwards.

This article sets out how to build that record for a regulated .NET backend: five questions every answer record must settle (our own framing), a recorder written as Microsoft.Extensions.AI middleware that refuses to return an answer it could not record, and the places where a trail like this usually has silent holes. We compiled and ran the code on .NET 10 against a fake model provider, so the behaviours in the test results below are ones we watched; the claims about Azure, OpenTelemetry and the AI Act come from their documentation, linked where they appear. It is written for the CTO or engineering lead who has been told an AI feature must be "auditable" and needs to know what that means in code.

Does the Law Require This? Only Sometimes

Start with what is actually mandatory, because it is narrower than the marketing around AI compliance suggests. The EU AI Act's logging duty sits in Article 12 and applies to high-risk AI systems: they must technically allow automatic recording of events over the lifetime of the system, so that their functioning can be traced, risks identified, and the system monitored after it goes live. The article says what the logs are for. It lists no fields for most systems. For one category, remote biometric identification, it adds minimum logs, including the identity of the people who verified the results. Providers (Article 19) and deployers (Article 26) keep the logs they control for at least six months unless other law says otherwise, and those that are financial institutions keep them as part of the documentation they already hold under EU financial-services law.

Whether a feature is high-risk depends on what it is used for, not on the fact that it uses an LLM. Annex III lists, for example, systems that evaluate the creditworthiness of natural persons or set their credit score, with fraud detection carved out. A chatbot that answers policy questions is probably not on that list. A model that scores loan applicants probably is. That is a legal call and we are engineers, so take it to counsel. On timing, the AI Omnibus, published in the Official Journal on 24 July 2026, moved the date for these requirements from 2 August 2026 to 2 December 2027 for Annex III systems, and to 2 August 2028 for those embedded in regulated products (see the European Commission's announcement, the European Parliament's Legislative Train page and the Future of Privacy Forum's summary).

Outside the Act, regulators are not waiting for AI-specific rules. FINRA's Regulatory Notice 24-09 (June 2024) says its rules apply to generative AI, names supervision under Rule 3110 as one example, and states that it creates no new requirements. Our stance is a judgement, not a legal opinion: build the record for the strictest case you might land in. The expensive part is recovering version history you never captured, not the storage.

An LLM Audit Trail Answers Five Questions

The Act says what logs are for, not which fields to keep. The five questions below are our own framing, drawn from what a reviewer asks when handed one answer and told to explain it. Each has a field set, and each field set has a reason it cannot be rebuilt later.

The table turns "audit trail" from a feeling into a test: given an answer ID, can you produce all five rows? Row 4 is easy to skip, and it is the reason for the design rule in the next section.

Record, Don't Replay

The instinct is to make the answer reproducible: pin the temperature to zero, set a seed, and re-run the request when asked. Microsoft's reproducible-output page describes the seed as a best-effort feature and says determinism is not guaranteed even when the seed and the system_fingerprint are the same. The fingerprint is there to tell you when the backend configuration has changed. The model behind a deployment name can move too. In Azure's deployment settings a Standard deployment set to auto-update follows a new default version within two weeks. One pinned to a specific version still moves to the default at that model's retirement date, and one set to no auto-upgrade stops working at retirement. Inside your own code, Microsoft.Extensions.AI also ships experimental routing and failover clients that choose a different underlying client per request, so even a fixed configuration can end up answering from a different model.

So the rule is ours: store the output verbatim, and store the inputs that let a reviewer judge it. The test of the trail is not "can we regenerate the answer" but "can we show exactly what the model was given and exactly what it returned". Pin model versions anyway, because it narrows the surprises, but treat the pin as a control and not as the evidence. And record the model that answered, as the response reports it, next to the name you asked for.

Why Application Logs Fall Short

The tempting version wraps the model client and writes the prompt and the reply to the log:

// NAIVE: the prompt and the reply, in a log.
public sealed class NaiveRecordingClient(IChatClient inner, ILogger logger) : DelegatingChatClient(inner)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken ct = default)
    {
        var response = await base.GetResponseAsync(messages, options, ct);
        logger.LogInformation("Prompt: {Prompt} | Response: {Response}", messages.Last().Text, response.Text);
        return response;
    }
}

We ran it against the fake provider. A normal call wrote one line holding the prompt and the reply and nothing else: no template version, no model, no document versions. A streamed call wrote nothing at all, because DelegatingChatClient forwards every method you do not override to the inner client. Three further holes are worth naming:

• Telemetry is built for a different job. The OpenTelemetry chat client in Microsoft.Extensions.AI records metadata such as token counts by default and leaves out message content unless EnableSensitiveData is switched on. OpenTelemetry sampling exists to keep only part of the data: spans that are not sampled are not processed or exported. Both are right for observability and wrong for evidence.
• The provider's stores are not your record. Microsoft's data-privacy page describes an abuse-monitoring store that can hold a sample of prompts and completions for review of flagged abuse, which approved customers can opt out of, and a stored-completions feature, in preview when we checked, that customers can delete at any time. As we read it, neither is meant to be the audit record you control.
• Log lines are ordinary mutable data. They can be filtered, rotated, re-indexed or edited by anyone with access to the log store, and nothing in them would show it.

Build the Recorder as Chat-Client Middleware

Microsoft.Extensions.AI gives you the seam. IChatClient is the abstraction, and DelegatingChatClient is the base class for middleware that wraps another client (Microsoft's documentation builds a rate limiter this way). We use it to record each answer before the caller sees it. First the record, which holds the answers to the first four questions (review events, question five, get their own type below):

/// <summary>One retrieved document as the model saw it: identity, version and a hash of the exact text.</summary>
public sealed record SourceRef(string DocumentId, string Version, string ContentHash);

/// <summary>What was asked, what the model saw, what answered, what it said. Bulk content lives in the payload store.</summary>
public sealed record AnswerRecord(
    Guid AnswerId,
    DateTimeOffset At,
    string Feature,
    string Actor,
    string Tenant,
    string? CaseRef,                // the case or decision this answer feeds
    string PromptTemplate,
    string PromptTemplateHash,
    string RequestedModel,          // the deployment name we asked for
    string? ResolvedModel,          // the model the provider says answered
    string? ProviderResponseId,
    float? Temperature,
    long? Seed,
    IReadOnlyList<SourceRef> Sources,
    string InputHash,               // payload-store key for the messages sent
    string OutputHash,              // payload-store key for the messages returned
    string? FinishReason,
    long? InputTokens,
    long? OutputTokens);

Bulk content, meaning the messages sent and received, goes to a payload store under its own SHA-256, and the record carries only the keys. Three small types sit around it and are not listed here. AnswerContext, held in an AnswerContextAccessor, is set by the feature code before it calls the model (feature, actor, template ID and hash, retrieved sources). IAnswerLedger is an append-only, hash-chained log like the one in our immutable audit trail article. IPayloadStore is a write-once store. Now the recorder:

/// <summary>
/// Records one <see cref="AnswerRecord"/> per answer, before the caller sees it. If the record cannot
/// be written, the answer is not returned.
/// </summary>
public sealed class AuditedChatClient(
    IChatClient inner,
    IAnswerLedger ledger,
    IPayloadStore payloads,
    AnswerContextAccessor accessor,
    string requestedModel,
    TimeProvider clock,
    ILogger<AuditedChatClient> logger) : DelegatingChatClient(inner)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken ct = default)
    {
        var ctx = accessor.Current
            ?? throw new InvalidOperationException("No answer context. Refusing to call the model unaudited.");
        var input = messages.ToList();                         // enumerate once; this is what we record

        var response = await base.GetResponseAsync(input, options, ct);
        await RecordAsync(ctx, input, options, response, ct);  // throws => the caller gets no answer
        return response;
    }

    // DelegatingChatClient forwards streaming untouched, so without this override a streamed
    // answer would bypass the recorder. Buffer, record, then release.
    public override async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken ct = default)
    {
        var ctx = accessor.Current
            ?? throw new InvalidOperationException("No answer context. Refusing to call the model unaudited.");
        var input = messages.ToList();

        var updates = new List<ChatResponseUpdate>();
        await foreach (var update in base.GetStreamingResponseAsync(input, options, ct))
            updates.Add(update);

        await RecordAsync(ctx, input, options, updates.ToChatResponse(), ct);
        foreach (var update in updates) yield return update;
    }

    // RecordAsync follows in the next listing.
}

Two behaviours matter here. A call with no answer context throws before it reaches the model, so a new feature cannot use the model without declaring its template and its sources. And the streaming override exists because the base class forwards streaming calls untouched; without it every streamed answer skips the recorder. The price is that streaming is buffered: the user sees nothing until the whole answer is recorded. The recording itself:

// Inside AuditedChatClient:
    private async Task RecordAsync(
        AnswerContext ctx, List<ChatMessage> input, ChatOptions? options, ChatResponse response, CancellationToken ct)
    {
        var answerId = Guid.NewGuid();
        try
        {
            var inputKey  = await payloads.PutAsync(JsonSerializer.SerializeToUtf8Bytes(input, AIJsonUtilities.DefaultOptions), ct);
            var outputKey = await payloads.PutAsync(JsonSerializer.SerializeToUtf8Bytes(response.Messages, AIJsonUtilities.DefaultOptions), ct);

            await ledger.AppendAsync(new AnswerRecord(
                answerId, clock.GetUtcNow(), ctx.Feature, ctx.Actor, ctx.Tenant, ctx.CaseRef,
                ctx.PromptTemplate, ctx.PromptTemplateHash,
                RequestedModel: options?.ModelId ?? requestedModel,
                ResolvedModel: response.ModelId,               // what answered, not what we asked for
                ProviderResponseId: response.ResponseId,
                Temperature: options?.Temperature, Seed: options?.Seed,
                Sources: ctx.Sources, InputHash: inputKey, OutputHash: outputKey,
                FinishReason: response.FinishReason?.Value,
                InputTokens: response.Usage?.InputTokenCount,
                OutputTokens: response.Usage?.OutputTokenCount), ct);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            logger.LogError(ex, "Answer {AnswerId} could not be recorded; withholding it", answerId);
            throw new AuditUnavailableException("Answer not recorded; not returned.", ex);
        }

        (response.AdditionalProperties ??= new())["audit.answer_id"] = answerId;   // callers attach this to the decision or screen
    }

Three choices are visible in RecordAsync. The resolved model comes from response.ModelId and sits beside the name we asked for, so a silent version move shows up as two different values. The ledger write happens before the return, and any failure becomes an AuditUnavailableException, so an answer that cannot be recorded is not shown. And the answer ID is attached to the response, so the calling code can store it next to the decision or the screen the answer feeds.

What We Watched It Do

We ran the recorder against a fake provider whose backend model we could change, with an in-memory hash chain standing in for the ledger. Eight things we tried, and what happened:

These results come from a fake model that returns what we tell it to. They show what the recorder does with a model ID, not what any given provider's client library puts in ModelId. Check that against your own client before you rely on it.

Human Review Is an Event, Not a Column

Question five is where most designs fall back to a mutable Status column. Use events. Each of viewed, approved, edited, overridden and rejected is appended with the reviewer, the reason and a hash of the final text, and the current status is whatever the latest event says, taken in ledger order rather than clock order:

public enum ReviewAction { Viewed, Approved, Edited, Overridden, Rejected }

/// <summary>A human acting on an answer. Appended, never updated.</summary>
public sealed record ReviewEvent(
    Guid AnswerId, ReviewAction Action, string Reviewer, string Reason,
    string? FinalTextHash, DateTimeOffset At);

public static class ReviewStatus
{
    // Status is read, never stored: the latest event for the answer, in ledger order.
    public static ReviewAction? Of(IEnumerable<ReviewEvent> eventsInLedgerOrder, Guid answerId) =>
        eventsInLedgerOrder.LastOrDefault(e => e.AnswerId == answerId)?.Action;
}

In our run an answer went from Viewed to Overridden, and an answer nobody had touched had no status at all. That second case is the useful one: it is the list to query when a policy says every answer needs a review. The "viewed" event matters more than it looks, because a claim that a person reviewed an answer needs evidence that they saw it before acting. The Act points the same way for the biometric category mentioned earlier, where the minimum logs include the people who verified the results.

Where the Content Lives: Ledger, Payloads and Erasure

Prompts carry personal data: names, account numbers, free text. An immutable record and a right to erasure pull against each other, a tension we cover in the immutable audit trail article. The design above helps. The ledger holds identifiers, versions and hashes. The payload store holds the content, so it is where erasure happens: the content is erased or crypto-shredded (its encryption key destroyed). The actor reference in the ledger is an identifier, not a name, but counsel may still treat it as personal data. After an erasure the ledger still shows that an answer existed, when it was given, and which model and versions produced it. It can no longer prove what the text said.

Two cautions, both our own. A plain hash of a short or guessable input can sometimes be recovered by guessing, so if that matters, use a keyed hash with a key you are able to destroy. And in a multi-tenant product the tenant belongs on every record, with the payload store isolated like any other tenant data (see tenant data isolation in .NET).

Retention is policy, not code: the Act sets six months as the default minimum for high-risk logs and other law can change that period, while financial recordkeeping rules may require much longer. Ask compliance, then set the period on the storage.

What This Approach Doesn't Solve

• A record is not an explanation. It shows what went in and what came out. It does not show why the model chose those words, and it should not be presented to a regulator as interpretability.
• The ledger is now on the critical path. Failing closed means a ledger outage stops AI answers. For answers that feed a regulated decision, that is the point. For a low-stakes feature, recording after the fact and alerting on failures can be the better trade. Decide per feature.
• Streaming is buffered. A chat screen that streams tokens as they arrive cannot also guarantee that the record exists first.
• Retrieval is recorded, not judged. The hashes show what the model was given. They do not show whether retrieval found the right documents or missed the relevant one.
• Tool calls and agent loops were not tested. Where the recorder sits relative to function invocation decides whether you get one record per answer or one per model call. Decide that on purpose, then test it. The sample also does not record tool definitions, which the table says to keep; hash options.Tools into the record before you use it with tool calling.
• We tested the recorder, not a provider. We did not call Azure OpenAI, run under load, or check what any client library places in the model field.

One condition where we would take a different route: an internal drafting assistant whose output a person always rewrites before anything leaves the company. A lighter record (template version, resolved model, output hash, short retention) is usually enough there. The full five-question record is for answers that reach a customer or feed a regulated decision.

A Drill That Proves the Trail Works

A trail nobody has tested is a hope. This drill is our own suggestion, and an afternoon is enough:

1. Pick an answer from last quarter at random and give only its ID to someone who did not build the feature.
2. Ask them for the answers to the five questions from the ledger and the payload store, without asking the team anything.
3. Verify the chain and the payload hashes and write down any that fail.
4. In a test environment, change the prompt template, the model version and one retrieved document in turn, and check that each change shows up as a different value in the next record.
5. Switch the ledger off and confirm that no answer reaches the user.

The same discipline applies to deterministic software. The proxy voting case study describes recording every vote with its supporting data so that it can be reconstructed for compliance. An LLM answer cannot be reliably recomputed, which is why the trail above stores what the model saw and said instead. The broader properties are in our guide to architecting auditable backend systems.

What should an LLM audit trail record for each answer?

Five things. Who asked, and for which feature or decision. What the model saw: the prompt template version, the exact messages and any tool definitions. What it read: each retrieved document with its version and a hash of the text passed in. What answered: the model the provider reports, the parameters and the provider's response ID, plus the output verbatim. And what happened next: each human review as an appended event. Bulk content can sit in a write-once payload store, with the hash-chained ledger holding the keys.

Are application logs or telemetry enough as an AI audit trail?

Usually not. Plain request logging misses the template version, the model that answered and the document versions, and log lines can be filtered, rotated or edited. The OpenTelemetry chat client in Microsoft.Extensions.AI leaves message content out unless you opt in, and OpenTelemetry sampling keeps only part of the data by design. In our test, a client that overrode only GetResponseAsync wrote no log line for a streamed call.

Can we reproduce an LLM answer later by running the prompt again?

Not reliably. Microsoft documents the Azure OpenAI seed parameter as best effort and says determinism is not guaranteed even when the seed and the system fingerprint match. The model behind a deployment name can also change when the deployment's upgrade setting moves it to a newer version. Keep the output verbatim, with the inputs that produced it, rather than relying on a re-run.

Does the EU AI Act require logging of LLM outputs?

Only for systems that are high-risk under the Act. Article 12 requires those systems to record events automatically, and Articles 19 and 26 set a minimum of six months for the logs that providers and deployers control, unless other law says otherwise. Under the AI Omnibus the requirements for Annex III systems apply from 2 December 2027. Whether a feature is high-risk is a legal question; creditworthiness scoring of natural persons is one listed use, so take it to counsel.

How do you keep personal data out of an immutable AI audit record?

Keep the hash-chained ledger to identifiers, versions and hashes, and put prompts and outputs in a separate payload store that can be erased or crypto-shredded. After an erasure the ledger still shows that an answer existed and which model and versions produced it, but it can no longer prove what the text said. The actor reference in the ledger is an identifier, not a name, but counsel may still treat it as personal data. Whether erasure or retention wins for a given record is a question for counsel.

When to Bring in External Help

You can build the recorder and run the drill yourself; both are a few days for a senior engineer who knows the feature. Outside help pays off when an AI feature is already live and nobody can say which template and model version produced last month's answers, when compliance has asked for an evidence pack by a date, or when the feature touches a pipeline that must also pass an audit.

If your backend shows any of the patterns above, our Discovery Sprint is a two-week diagnostic that tells you exactly what to fix and in what order. Auditability is the third of the seven areas in the backend architecture review the sprint runs, and the audit trail is a first-class deliverable in our compliance reporting automation work.

Back to Insights