An agent that drafts a certificate makes twenty decisions before a CSR ever sees it. It picked an account. It picked a policy term. It read three endorsement PDFs and decided one of them answered the holder's additional insured requirement. It chose a form. Six weeks later a claim gets denied, the holder's counsel asks who said the coverage was there, and someone has to answer.
"The model decided" is not an answer. This tutorial covers the record we build next to every agent so that question has one: an append-only decision log, tied to the source documents, retained as long as the file is.
You need a place to write events (we use DynamoDB), object storage for the artifacts (S3), and whatever your agent already uses for LLM calls. The shape matters more than the services.
What has to be reconstructable
Start from the question, not from the log library. Two years out, given a certificate number or an Epic activity ID, you need to answer:
- What was sent, to whom, and when.
- Which licensed person approved it, and what they were shown at the time.
- What inputs the agent read: the request message, the AMS policy record as of that moment, each endorsement document by version.
- What the agent concluded, and which input supports each conclusion.
- What the agent was told to do: prompt version, model, retrieval index version, code version.
- Anything it declined to decide, and where it went instead.
Item 3 is the one teams miss. Logging "policy CPP-4417 checked" is useless if the policy was endorsed in the meantime. Log the values you read, not a pointer to a record that changes.
Item 5 is the one E&O counsel cares about second. A prompt edit changes behavior as surely as a code deploy. If you cannot say which prompt produced a September draft, you cannot explain the difference between September and January.
One event per decision
We write flat events, one per step, all keyed to a case_id that follows the work item from intake to send. No nesting, no updates.
import hashlib, json, time, uuid
from datetime import datetime, timezone
def log_event(table, case_id, kind, payload, *, actor, evidence=None):
"""Append one immutable decision event. Never updates an existing item."""
now = datetime.now(timezone.utc).isoformat()
item = {
"pk": f"case#{case_id}",
"sk": f"{now}#{uuid.uuid4().hex[:8]}",
"kind": kind, # extract | match | check | draft | approve | send | abstain
"actor": actor, # "agent:coi@3.4.1" or "user:jmoore@agency.com"
"at": now,
"payload": payload, # the decision, as data
"evidence": evidence or [], # s3 keys + sha256 of what was read
"build": {
"code": "coi-agent@3.4.1",
"prompt": "coi_check@2025-11-04",
"model": "claude-sonnet-4-5",
"index": "endorsements@2026-01-07",
},
"ttl": int(time.time()) + 86400 * 365 * 7,
}
table.put_item(
Item=item,
ConditionExpression="attribute_not_exists(pk) AND attribute_not_exists(sk)",
)
return item
Three things in there are deliberate.
The condition expression makes the write fail if that exact event already exists, so a Lambda retry cannot double-log. Combined with the claim-call-confirm layer in front of the AMS, a retried run produces one activity and one audit chain.
evidence is a list of content hashes, not filenames. A filename tells you a document was read. A hash tells you which document, which matters when the insured's broker sends a revised endorsement under the same name.
def evidence_ref(bucket, key, body: bytes, note=""):
return {
"s3": f"s3://{bucket}/{key}",
"sha256": hashlib.sha256(body).hexdigest(),
"bytes": len(body),
"note": note, # "CG 20 10 04 13, page 2, additional insured wording"
}
ttl is set to seven years here. Pick the number with the agency's E&O carrier and counsel, not with your storage bill; commercial file retention rules vary by state and by carrier, and some agencies keep certificate files indefinitely. If the retention answer is "forever," drop the attribute rather than guessing.
Write the table so it cannot be edited
An audit log a developer can UpdateItem is a set of notes, not a record. Deny the mutating actions on the table in IAM, for the agent role and yours:
{
"Effect": "Deny",
"Action": ["dynamodb:UpdateItem", "dynamodb:DeleteItem", "dynamodb:BatchWriteItem"],
"Resource": "arn:aws:dynamodb:us-east-1:*:table/agent-audit"
}
Turn on point-in-time recovery, and put the evidence bucket in S3 Object Lock in governance mode with the same retention as the events. Corrections then work the way they work on paper: you append a correction event that references the sk of the one it supersedes, and both stay.
Capture what the approver saw
The approval event is the one that carries legal weight, and it is usually logged as a boolean. That is not enough. If a CSR approved a draft, log the rendered draft, not the decision to render it.
log_event(
table, case_id, "approve",
payload={
"decision": "approved",
"queue_item": item_id,
"shown_sha256": rendered_hash, # hash of the exact PDF on screen
"edits": ["holder_address"], # fields the CSR changed before approving
"elapsed_seconds": 41,
},
actor="user:jmoore@agency.com",
evidence=[evidence_ref(bucket, f"{case_id}/acord25-v2.pdf", pdf_bytes)],
)
shown_sha256 closes the gap between what was approved and what went out. Before the send step runs, re-hash the artifact and compare. If they differ, stop and requeue: something regenerated the document after approval, which is exactly the failure you do not want to discover from a holder.
The edits list has a second use. Fields CSRs correct most often are your accuracy backlog, ranked by the people doing the work. On one workflow it is nearly always the holder address; that is a data problem in the request parser, not a model problem.
Trace the model calls separately
Decision events are for the file. They are the wrong grain for debugging why last Tuesday's extraction went sideways, and you do not want raw prompts and full document text in the record you hand to counsel.
Send model-level detail to your tracing backend instead: one span per LLM call, with the OpenTelemetry GenAI conventions (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) so you are not inventing attribute names that no dashboard understands. Keep the span ID on the decision event and the case_id on the span. Then a support question walks from the case to the spans, and a coverage question walks from the certificate to the decision events, without either view carrying the other's baggage.
Set the tracing retention to 30 days. Set the decision-log retention to the file's retention. Conflating the two is how agencies end up storing seven years of prompt text they never wanted.
Log the abstentions too
When the matcher cannot separate two accounts, or the endorsement check finds no supporting wording, the agent routes to a human. That is a decision. Log it with the same weight as a draft:
log_event(table, case_id, "abstain",
payload={"step": "policy_match", "reason": "margin_below_threshold",
"top": [{"policy": "CPP-4417", "score": 0.61},
{"policy": "CPP-9928", "score": 0.58}],
"routed_to": "queue:cl-service"},
actor="agent:coi@3.4.1")
Two months in, the abstention log is the most useful table you have. It tells you where the agent's coverage actually ends, in cases rather than in your estimate of cases, and it is the report we use to decide what to build next.
What this does not do
An audit trail does not make an agent safer. It makes an agent explainable. The controls that keep a bad draft from going out are the approval gate, the abstention thresholds, and the silent run before go-live; the log is what you have after one of those fails.
It also will not reconstruct the AMS side. If a CSR edits the account in Epic after the send, your log shows what was true at draft time and the AMS shows what is true now, and reconciling them is manual. We log the AMS record version where the API exposes one, which is not everywhere.
Budget a day or two to retrofit this onto a working agent, and expect the argument to be about retention rather than code. Have that argument with the agency's E&O carrier before go-live, not after the first denied claim.