Your extraction agent has been running against the agency's live book for four months. Field accuracy on ACORD 125s is where you measured it at go-live. Then the provider emails a deprecation notice: the model version you pinned retires in 90 days. Or a product manager ships a "quality improvement" to the model you call by an undated alias, and on Tuesday the FEIN field starts coming back with the dashes stripped.
Nothing in the agency changed. The reading layer under it did.
We scored the agent once, before go-live, against closed cases from the agency's own AMS. That run answers "is this safe to turn on". It does not answer "is this still the same agent it was in March". This tutorial covers the second question: how to pin a model, keep a small frozen case set in version control, and canary a model change against it before a single certificate draft reaches a CSR.
You need the agent already in production, an accuracy harness of some kind (a scored silent run gives you one), and write access to the repo and the CI pipeline. Nothing here is specific to one provider.
Pin the version, always
The single most common cause of unexplained agent drift is a floating model identifier. Aliases that track "latest" are convenient in a notebook and indefensible in a workflow that feeds documents to a licensed person for signature.
Pin it in config, not in code, and log the resolved value on every call:
# config/models.yaml
extraction:
provider: bedrock
model_id: anthropic.claude-3-5-sonnet-20241022-v2:0
temperature: 0
max_tokens: 2048
prompt_sha: 7f3c9a1e # sha256 of the prompt template file, checked at boot
import hashlib, logging, yaml
cfg = yaml.safe_load(open("config/models.yaml"))["extraction"]
prompt_text = open("prompts/acord_extract.md").read()
actual_sha = hashlib.sha256(prompt_text.encode()).hexdigest()[:8]
if actual_sha != cfg["prompt_sha"]:
raise RuntimeError(
f"prompt changed ({actual_sha}) but models.yaml still says {cfg['prompt_sha']}; "
"run the canary and update the pin"
)
logging.info("extraction call", extra={
"model_id": cfg["model_id"],
"prompt_sha": actual_sha,
"temperature": cfg["temperature"],
})
Two things fall out of that. The prompt is now versioned with the same seriousness as the model, because a prompt edit changes behaviour exactly as much as a model swap. And every extraction you write into the AMS can be traced back to a model id and a prompt hash, which is what you want when a CSR asks in September why the July drafts looked different.
Freeze a small case set
The pre-go-live harness used 200 closed cases. That is the right size for a decision to turn something on. It is the wrong size for a check that has to run on every pull request.
Cut a canary set of 40 to 60 cases out of it, chosen on purpose rather than at random:
- Every form version you actually see in the book. ACORD 125 revisions do not retire on schedule.
- The scanned and photographed documents, not just the clean digital PDFs.
- The cases the agent got wrong at go-live and the fix cleared. These are the regressions you will actually suffer.
- Two or three documents with adversarial text in them, so an injection defence regression shows up here rather than in production.
- Anything where a reviewer overrode the agent in the first 90 days of live running. Pull these from the approval log monthly.
Store the documents and the expected field values in the repo, redacted the same way your pipeline redacts them, with the insured names replaced by fixtures. The expected values are a reviewed artefact: a licensed person signed off on what the right answer is, and the file records who and when.
cases/
acord125-rev2016-clean-01/
document.pdf
expected.json # {"named_insured": "...", "fein": "12-3456789", ...}
provenance.json # {"source": "closed 2026-03", "verified_by": "...", "verified_at": "..."}
Score field by field, not document by document
A whole-document pass/fail rate hides the failure you care about. One field regressing from 0.98 to 0.71 while the document score barely moves is the normal shape of a model change.
def score(cases, extract_fn):
per_field = {}
for case in cases:
expected = case["expected"]
actual = extract_fn(case["document"])
for field, want in expected.items():
got = actual.get(field)
bucket = per_field.setdefault(field, {"hit": 0, "miss": 0, "abstain": 0})
if got is None:
bucket["abstain"] += 1 # said nothing: safe, still a cost
elif normalize(field, got) == normalize(field, want):
bucket["hit"] += 1
else:
bucket["miss"] += 1 # said something wrong: the expensive case
return per_field
Keep the three buckets separate. An abstain sends the case to a person and costs four minutes. A confident wrong answer on the named insured can put the wrong entity on a certificate. Those are not the same event and averaging them into one accuracy number destroys the distinction you need.
normalize is where most of the real work lives, and it belongs to the domain, not the model: FEINs compared with punctuation stripped, dates parsed to ISO before comparison, entity suffixes (Inc, Inc., Incorporated) folded, addresses compared on street number plus ZIP rather than string equality. Write it once, review it with the account managers, and do not let the model be blamed for a comparison bug.
Gate the change
Run the canary set against the candidate model, compare it to the recorded baseline, and refuse to move the pin on a miss-rate increase:
FIELD_TOLERANCE = 0.02 # allowed drop in hit rate per field
MISS_TOLERANCE = 0.0 # no increase in confident-wrong, ever
def gate(baseline, candidate):
failures = []
for field, base in baseline.items():
cand = candidate[field]
if rate(cand, "hit") < rate(base, "hit") - FIELD_TOLERANCE:
failures.append(f"{field}: hit rate {rate(base,'hit'):.3f} -> {rate(cand,'hit'):.3f}")
if rate(cand, "miss") > rate(base, "miss") + MISS_TOLERANCE:
failures.append(f"{field}: miss rate {rate(base,'miss'):.3f} -> {rate(cand,'miss'):.3f}")
return failures
Any failure stops the merge and prints the field names. Tolerating a small hit-rate drop is a judgment call you can defend; tolerating more confident-wrong answers is not, because that is the class of error that reaches a document with the agency's name on it.
Run the gate on three triggers: any pull request that touches a prompt or the extraction code, any change to the pinned model id, and a scheduled weekly run against the same pin. The weekly run is the one that catches a provider changing behaviour underneath a version string you did not touch. Note the cost: 50 cases times a handful of calls, once a week, is cents, and it is the cheapest monitoring in the system.
Then shadow it on live traffic
The canary set proves the candidate is not worse on documents you already understand. It says nothing about the documents arriving next week. Before you cut over, run the candidate in shadow: real live inputs, both models called, only the incumbent's output shown to the reviewer.
Log both outputs plus the reviewer's decision. After 200 live items you have three numbers that matter: how often the models agreed, how often the candidate matched what the reviewer approved, and how often the incumbent did. If the candidate's agreement with approved output is at or above the incumbent's, move the pin. If the two models disagree on more than a few percent of items, read those items before deciding anything; the disagreements are usually a document class nobody had catalogued.
Shadow mode doubles inference cost for the duration and writes nothing to the AMS. Two weeks is usually enough. Keep the incumbent's config in the repo, unchanged, so rollback is a one-line revert rather than a reconstruction.
What this does not cover
Be clear about the limits when you write this up for the agency:
- The canary set measures extraction, not judgment. Whether an additional insured request is actually supported by the policy is a separate check with its own tests.
- A frozen case set decays. Any book shifts class of business and carriers over a year; refresh the set quarterly from recent approvals or you are defending accuracy against a book that no longer exists.
- Passing the gate is not permission to remove the approval step. Nothing here changes the rule that the agent drafts and a licensed person approves the send.
- Provider-side changes to safety filtering can surface as abstains rather than errors. Watch the abstain bucket for the same reason you watch the miss bucket: a quiet doubling of items routed to humans is a staffing event.
Where this fits
Model pins, a canary set, and a shadow window are part of what we mean by agent operations: the same engineer who built the agent keeps it honest as carriers, forms, and models change underneath it. It is also what lets us answer the question a principal should ask on the first call, which is not "how accurate is it" but "how will you know when it stops being accurate".
If you are running an extraction agent against Applied Epic, HawkSoft, or EZLynx and there is no gate between a model change and your CSRs, contact us and tell us the workflow. A senior engineer replies within one business day.