Bindmatic

Learn from the edits your CSRs make in the approval queue

A COI Agent goes live. Week one, the CSR approves 61 of 94 drafts untouched, edits 28, and rejects 5. Week six, the numbers are the same. Nothing improved, because nobody wrote down what she changed.

That is the most common failure we see in a supervised agent after go-live. The approval gate is doing its job: a licensed person reads every draft and nothing goes out unseen. But the gate is also the only place in the system where a trained human corrects the model on real work, and in most builds that correction is thrown away the moment the send button is hit.

This tutorial is how we capture it. Postgres, a diff, a short reason list, and a Friday review. No fine-tuning, no auto-updating prompts.

Capture the before and after

The agent writes a draft. The CSR opens it, changes the certificate holder address, adds a line about the waiver of subrogation, and approves. You need both versions and the identity of the person who changed it.

Store the draft at the moment it enters the queue, immutable, and store the approved payload as a separate row rather than an update in place. If the approved version overwrites the draft, there is no signal left to learn from.

create table agent_draft (
  id uuid primary key,
  task_id uuid not null,
  agent text not null,              -- coi | renewal | intake | service
  ams_account_id text,
  policy_id text,
  drafted_payload jsonb not null,   -- exactly what the agent produced
  model text not null,
  prompt_version text not null,
  created_at timestamptz not null default now()
);

create table agent_approval (
  id uuid primary key,
  draft_id uuid not null references agent_draft(id),
  outcome text not null,            -- approved | edited | rejected
  approved_payload jsonb,           -- null when rejected
  reason_code text,
  reviewer text not null,
  review_seconds int,
  created_at timestamptz not null default now()
);

prompt_version matters more than it looks. Six weeks from now you will want to know whether the address errors stopped after the change you made on the 14th, and that question is unanswerable if every row points at "current".

review_seconds is the other number worth having. Time-in-queue is what the principal actually bought. An agent whose drafts are approved untouched but take four minutes to read has not saved anyone anything.

Diff the fields, not the text

A character diff of two emails tells you nothing you can act on. Diff the structured payload field by field, because the fields map to the part of the pipeline that produced them.

def field_diffs(drafted: dict, approved: dict, prefix: str = "") -> list[dict]:
    out = []
    for key in sorted(set(drafted) | set(approved)):
        before, after = drafted.get(key), approved.get(key)
        path = f"{prefix}{key}"
        if isinstance(before, dict) or isinstance(after, dict):
            out += field_diffs(before or {}, after or {}, prefix=f"{path}.")
        elif normalize(before) != normalize(after):
            out.append({"field": path, "before": before, "after": after})
    return out


def normalize(v):
    if isinstance(v, str):
        return " ".join(v.split()).strip().casefold()
    return v

Normalize before comparing or your top edit will be whitespace. Then roll the diffs up weekly:

select d.agent, f.field, count(*) as edits
from agent_approval a
join agent_draft d on d.id = a.draft_id
cross join lateral jsonb_array_elements(a.field_diffs) as f(field)
where a.outcome = 'edited'
  and a.created_at > now() - interval '7 days'
group by 1, 2
order by edits desc
limit 20;

The first time we ran this at an agency on Applied Epic, the top row was holder.address with 34 edits in a week. Not a model problem. The holder addresses in Epic were stale because the CSRs had been correcting them in the outbound certificate for years and never writing them back. The fix was a write-back step, not a prompt.

That is the usual shape of the finding. Most edits are data, not reasoning.

Make the reason one click

A free-text "why did you change this" box returns empty strings. Give the reviewer six buttons, agent-specific, and make one of them required on any edit or rejection:

  • Stale AMS data — the agent used what was in the system; the system was wrong.
  • Missed document — the answer was in an attachment the agent did not read.
  • Wrong policy or account match — right task, wrong file.
  • Coverage judgment — the endorsement check or the risk call was wrong.
  • Tone or house style — content fine, wording not ours.
  • Should have abstained — the agent guessed where it should have flagged the gap and stopped.

Six codes, not twenty. Each one routes to a different owner: stale data to operations, missed document to the extraction pipeline, wrong match to the matching rules, coverage judgment to the engineer and the account manager together, tone to the template, abstention to the confidence thresholds.

"Should have abstained" is the one to watch. A rising count there is the leading indicator of a silent error reaching a holder, and it is worth paging on. Every other code can wait for Friday.

Turn edits into cases, not into training

Resist the obvious move. Do not pipe approved payloads into a fine-tune, and do not let a weekly job rewrite the prompt. Both make the agent's behavior change without a reviewable diff, which is exactly the property you need when an E&O carrier asks why a certificate said what it said.

What we do instead, in order of preference:

  1. Fix the data. Stale AMS fields, missing endorsement records, duplicate accounts. Highest yield, and it improves the agency's book whether or not the agent survives.
  2. Fix the pipeline. A missed document usually means a split or OCR failure upstream, not a reasoning failure.
  3. Add a rule. Deterministic checks for deterministic things. If this holder always requires a 30-day notice of cancellation, that is a row in a table, not a sentence in a prompt.
  4. Add a few-shot example. Last resort for tone and format, versioned with the prompt, capped at a number you can read in one sitting.

And regardless of which fix you pick: every edited draft becomes a regression case. Freeze the inputs, label the approved payload as the expected output, and add it to the same harness you used to score the silent run. After 90 days of live running, your real eval set is not the 200 historical cases you started with. It is the 300 edits your CSRs made since.

The Friday review

Thirty minutes, every week, for the first quarter. The engineer who built the agent, the CSR lead, and the top-20 field table on the screen.

Three questions:

  1. What is the top edited field, and is it data, pipeline, rule, or prompt?
  2. Did last week's fix move its count? If the count did not move, the fix was wrong; back it out.
  3. Is should have abstained going up or down?

Write the answers in the ticket, tag the prompt version, and ship one change. One. Two changes in a week and you cannot attribute the difference, and you will spend the following Friday arguing about which one worked.

What this will not do

It will not make the agent correct on coverage judgment by itself. Volume of edits does not teach a model to read a lease; a rule written by someone who has read a lease does.

It will not shorten the approval queue to zero. Our target is untouched-approval rate climbing and review time per item falling, with a licensed person still on every send. An agency that reads the edit log for a quarter usually gets from roughly 60% untouched to the high 80s on certificates. The remaining items are the ones that should have a human on them.

And it will not work if the reason codes are optional. If the button can be skipped it will be skipped at 4:45 on a Thursday, which is when the interesting edits happen.

If you are running an agent in an agency today and you cannot answer "what did your CSRs change last week," that is the first thing to build. It takes about a day.