Every agent we ship runs silently against the agency's live book before it touches a client. It reads real requests, matches real accounts, drafts real output, and sends nothing. Then we score it. The silent run is the only honest accuracy number available, because vendor accuracy figures were measured on someone else's documents, someone else's naming conventions, and someone else's AMS hygiene.
The part nobody writes down is how to score it. This tutorial is the harness we use: how to build the labeled set out of work your agency already closed, which numbers to compute, and where to set the threshold that decides whether an agent goes live at all. The code is plain Python and stdlib plus pandas; the method matters more than the tooling.
Build the labeled set from closed work
Do not write test cases. Take history. A commercial agency has thousands of finished items in the AMS and the mailbox, each one with an outcome a human already committed to.
For a COI Agent, pull 200 certificate requests from the last 90 days that were fulfilled and closed. For each one, the label is what the CSR actually did: the account, the policy, the ACORD form issued, the holder, and whether an endorsement had to be attached. For a Service-Request Agent, pull 300 inbound messages and label the classification and the account the CSR filed the activity to. For Quote-Intake, pull 100 submissions and label the fields that were keyed.
Three rules keep the set honest:
- Sample across the whole window, not the easy weeks. Include the renewal-heavy month. Include the two days the mailbox backed up.
- Freeze it. Copy the documents and the labels into a versioned location. If you tune against a set that keeps changing, the score means nothing.
- Do not let the agent's author write the labels. The CSR lead does that, or you validate a sample of the labels with the CSR lead.
Labeling 200 certificate requests takes an experienced CSR roughly four to six hours. That is the price of a defensible number, and it is cheaper than the first bad certificate.
Store each case as one record:
{
"case_id": "coi-2026-0417",
"source_uri": "s3://agency-eval/coi/coi-2026-0417.eml",
"truth": {
"account_id": "ACCT-88213",
"policy_id": "GL-4471902",
"form": "ACORD 25",
"holder_name": "Brightline Construction LLC",
"needs_additional_insured": true,
"waiver_of_subrogation": false
}
}
Score the fields the way the work fails
A single "accuracy" percentage hides everything you need to know. Score per field, and normalize before comparing, or you will spend a week chasing differences that no human would call an error.
import re, json
import pandas as pd
SUFFIXES = r"\b(inc|llc|l\.l\.c|ltd|co|corp|company|the)\b"
def norm_name(value):
if value is None:
return ""
v = value.lower()
v = re.sub(r"[.,&']", " ", v)
v = re.sub(SUFFIXES, " ", v)
return re.sub(r"\s+", " ", v).strip()
def norm_id(value):
return re.sub(r"[^a-z0-9]", "", (value or "").lower())
NORMALIZERS = {
"holder_name": norm_name,
"account_id": norm_id,
"policy_id": norm_id,
}
def compare(truth, pred):
row = {}
for field, expected in truth.items():
f = NORMALIZERS.get(field, lambda x: x)
got = pred.get(field)
row[field] = {
"expected": expected,
"got": got,
"match": f(expected) == f(got) if isinstance(expected, str) else expected == got,
"abstained": got is None,
}
return row
Now the three numbers that decide whether the agent is usable.
Field accuracy on non-abstained answers. Of the cases where the agent produced a value, how often was it right? This is the number that predicts what a CSR sees in the approval queue.
Abstention rate. How often did the agent decline to answer and route the case to a person? A high abstention rate is not a failure. An agent that answers 70% of certificate requests correctly and hands you the other 30% untouched is a good agent. An agent that answers 100% and is wrong on 12% is a liability.
Silent-error rate. Of the cases the agent answered confidently, how often was it wrong? This is the only number an E&O carrier cares about, and it is the one that has to be near zero on the fields that change coverage.
def score(results, field):
rows = [r[field] for r in results]
answered = [r for r in rows if not r["abstained"]]
correct = [r for r in answered if r["match"]]
return {
"field": field,
"n": len(rows),
"abstention_rate": 1 - len(answered) / len(rows),
"accuracy_when_answered": len(correct) / len(answered) if answered else None,
"silent_errors": len(answered) - len(correct),
}
fields = ["account_id", "policy_id", "form", "holder_name",
"needs_additional_insured", "waiver_of_subrogation"]
report = pd.DataFrame([score(results, f) for f in fields])
print(report.to_string(index=False))
Weight the errors, because they are not equal
A misspelled holder name costs a CSR ten seconds. A missing additional-insured endorsement on an ACORD 25 costs a claim. Classify every error before you average anything:
- Cosmetic: formatting, abbreviation, trailing entity suffix. Fix in normalization, not in the model.
- Rework: right account, wrong policy; right classification, wrong queue. Costs a human a minute.
- Material: coverage-affecting. Wrong account, wrong insured, an endorsement asserted that is not on the policy, a form issued that the request did not justify.
Set the go-live gate on material errors, not on the blended average. Ours: zero material errors on the frozen set, and any material error found later stops the rollout until the cause is understood. That is achievable precisely because abstention is allowed. The agent's job is to be right or to be quiet.
For classification work, a confusion matrix tells you more than a score. It usually shows the same thing: two classes that overlap in real life, and a routing rule that should merge them.
matrix = pd.crosstab(
pd.Series([r["truth"]["intent"] for r in cases], name="actual"),
pd.Series([r["pred"].get("intent") for r in cases], name="predicted"),
dropna=False,
)
print(matrix)
Compare against the human baseline, not against 100%
Score the same set the way the agency already performs. Pull the CSR's original filing on a sample of 50 cases and have the CSR lead review it blind, alongside the agent's output, without knowing which is which.
This step changes conversations. Human filing accuracy on service-request classification is not 100%; nobody's is. If the agent matches the human on rework errors and beats the human on turnaround, and produces no material errors, that is a result you can defend. If you hold the agent to a standard the agency has never met, you never go live, and the staff hours stay where they were.
Re-run it on a schedule
The score is not a one-time artifact. Carrier portals change, an AMS upgrade renames a field, a new producer starts writing accounts a different way. Keep the frozen set in version control next to the agent, run it in CI on every change, and re-sample 50 fresh cases each quarter and add them. Drift shows up first as a rising abstention rate, which is the failure mode you want: the agent gets quieter before it gets wrong.
Log every production decision in the same shape as the eval record: source document, matched account and policy, fields checked, output drafted, reviewer, decision, timestamp. Then the approval queue itself becomes next quarter's labeled set, because a rejected draft is a labeled error with a reason attached.
What this does not measure
The harness scores extraction, matching, classification, and drafting. It does not score judgment on cases outside the frozen set, it does not tell you whether the drafted email reads like your agency, and it does not predict what happens the first week the staff decide to click Approve without reading. Watch approval times during the supervised go-live. Approvals that get faster than a person can read the source document are a process problem, not an accuracy problem.
Surveys of agency technology plans keep showing the same split: most agencies intend to expand AI use, and a small minority actually have something running. The gap is not model quality. It is that nobody can say what the thing gets wrong. A silent run with a scored, frozen set is how you close it.
We build this harness into every engagement, and the numbers belong to the agency, along with the code. If you want a senior engineer to run a scored silent run against your book, contact us.