Every agent in an agency pipeline ends the same way: something has to become a record. A certificate request becomes a holder, a requirement list, and a policy match. A loss run becomes five years of claims rows. A service request becomes an activity with a code, an account, and a proposed action. The model's prose is not the deliverable. The row is.
So the interesting failure is not a hallucinated sentence. It is a field that arrives as "12/1/26" when the write-back expects 2026-12-01, a limit that arrives as "$1M/$2M" when the AMS wants two integers, or a confident "effective_date": "2026-01-01" on a document where that date was never printed. Each one either throws at write time or, worse, lands quietly.
This tutorial covers the shape of the fix: declare the schema, make the provider enforce it, validate it again yourself, let the agent say "not present", and fail the item into a review queue instead of into the AMS.
Start from the write, not from the prompt
Write the schema that the AMS write-back needs, then work backwards to the model. If an Epic activity needs a code from your own list, that list is an enum, not a free string. If HawkSoft wants a log note under 4,000 characters, that is a maxLength. If a claim row with no paid amount is meaningless, paid is required.
A loss-run extraction schema for a submission looks roughly like this:
CLAIM_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["carrier", "policy_period", "claims", "source_pages"],
"properties": {
"carrier": {"type": "string", "minLength": 2},
"policy_period": {
"type": "object",
"additionalProperties": False,
"required": ["effective", "expiration"],
"properties": {
"effective": {"type": ["string", "null"], "format": "date"},
"expiration": {"type": ["string", "null"], "format": "date"},
},
},
"claims": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["claim_number", "date_of_loss", "status",
"paid", "reserved", "coverage", "source_page"],
"properties": {
"claim_number": {"type": "string"},
"date_of_loss": {"type": ["string", "null"], "format": "date"},
"status": {"enum": ["open", "closed", "reopened", "unknown"]},
"paid": {"type": ["number", "null"], "minimum": 0},
"reserved": {"type": ["number", "null"], "minimum": 0},
"coverage": {"enum": ["GL", "AUTO", "WC", "PROP", "UMB", "OTHER"]},
"source_page": {"type": "integer", "minimum": 1},
},
},
},
"source_pages": {"type": "array", "items": {"type": "integer", "minimum": 1}},
},
}
Three things in there are doing real work.
additionalProperties: false at every level. Without it a model will invent "insured_name_2" and you will discover it in production when a downstream mapper indexes a key that is usually there.
Enums instead of strings for anything you will branch on. status and coverage are the fields a scoring step reads. If they are free text you will get "Closed - subrogation", "CLSD", and "closed w/ payment" across three carriers in one book.
source_page on every row. The reviewer gets a page number next to each claim, so checking twelve rows takes a minute instead of re-reading the PDF. Decide that at schema time; bolting provenance on later means re-running extraction.
Make the provider enforce it
Every major provider now supports constrained decoding against a JSON Schema rather than "please reply in JSON". Use it. The grammar-constrained path makes a structurally invalid response very close to impossible, which removes the whole class of bugs where a stray sentence precedes the opening brace.
resp = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": page_text},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "loss_run",
"schema": CLAIM_SCHEMA,
"strict": True,
},
},
temperature=0,
)
Two constraints you will meet in the first hour. Strict modes generally require every property to be listed in required, so an optional field is expressed as a nullable type (["string", "null"]), not as an absent key. And several keywords are advisory even in strict mode: format: date, minimum, maxLength, and minLength constrain nothing at decode time. The model will happily emit "date_of_loss": "last March" inside a perfectly well-formed object.
Which is why the server-side schema is the floor, not the ceiling.
Validate again on your side
Run the response through your own validator before anything touches the AMS. Same schema, plus the checks the decoder never made.
from jsonschema import Draft202012Validator, FormatChecker
validator = Draft202012Validator(CLAIM_SCHEMA, format_checker=FormatChecker())
def validate(payload: dict, page_count: int) -> list[str]:
errors = [f"{'.'.join(map(str, e.path))}: {e.message}"
for e in validator.iter_errors(payload)]
for i, claim in enumerate(payload.get("claims", [])):
if claim["source_page"] > page_count:
errors.append(f"claims.{i}.source_page: page {claim['source_page']} "
f"does not exist in a {page_count}-page document")
if claim["paid"] is not None and claim["paid"] > 1_000_000:
errors.append(f"claims.{i}.paid: {claim['paid']} exceeds review threshold")
dol = claim["date_of_loss"]
if dol and not (period_start <= date.fromisoformat(dol) <= period_end):
errors.append(f"claims.{i}.date_of_loss: {dol} outside policy period")
return errors
FormatChecker is what turns format: date from documentation into a rule; by default jsonschema ignores formats entirely. The business rules underneath it are the ones no schema language expresses: a page that exists, a date inside the period, an amount above the threshold where a human should look regardless of confidence.
Keep these cheap and deterministic. This is not a second model grading the first one. It is arithmetic, and it runs in milliseconds on every item.
Let the agent say it does not know
The most expensive habit a constrained model has is filling a required field because it is required. A loss run that does not print a reserve gets a 0. A certificate request with no expiration gets today plus a year.
Two defences, and you want both.
Make unknown representable. Nullable types and an explicit "unknown" enum member give the model a legal way out. Then say so in the system prompt, plainly: if a value is not printed on the document, return null; do not infer it from other fields, from the file name, or from the request email.
And ask for the span. For fields that drive a decision, require a short verbatim quote from the source alongside the value, then check the quote actually appears in the extracted text:
if claim["paid_quote"] not in normalized_page_text(claim["source_page"]):
errors.append(f"claims.{i}.paid: quote not found on page {claim['source_page']}")
A model that cannot point at the characters it read usually did not read them. This single check moves more junk out of the AMS than any amount of prompt tuning, and it costs one substring comparison.
Repair once, then stop
When validation fails, hand the errors back for one retry. One.
def extract_with_repair(page_text: str, page_count: int):
payload = call_model(page_text)
errors = validate(payload, page_count)
if not errors:
return payload, "clean"
payload = call_model(
page_text,
repair_note="Your previous answer failed validation:\n"
+ "\n".join(errors)
+ "\nReturn a corrected object. Use null where the "
"document does not state a value.",
)
errors = validate(payload, page_count)
if not errors:
return payload, "repaired"
return None, "failed:" + errors[0]
A second repair almost never converges; it burns tokens and latency while the model restates the same guess. Let it fail. A failed item goes to the review queue with the errors attached, which is a worse day for one submission and a much better month for the book.
Log the outcome bucket per item: clean, repaired, failed. The repair rate is a real health metric. When a carrier changes its loss-run layout, repairs climb before failures do, and you get a week of warning.
Pin the schema with golden cases
Schemas change. A new AMS field, a new activity code, a coverage type you did not plan for. Keep a small fixture set, twenty to forty documents from the agency's own book, with the expected object checked in next to each one, and run it in CI on every schema or prompt change.
@pytest.mark.parametrize("case", load_cases("fixtures/loss_runs"))
def test_extraction_matches_golden(case):
payload, outcome = extract_with_repair(case.text, case.page_count)
assert outcome != "failed"
assert payload["claims"] == case.expected["claims"]
Version the schema and store the version on every extracted record. When a reviewer asks why a January record has no coverage value, "that field arrived in v4" is an answer; a shrug is not.
What this does not fix
Structured output is a plumbing control, not an accuracy control. A schema will not notice that the model read the wrong column of a claims table, matched the wrong one of two policies with the same effective date, or extracted the subtotal row. Those are caught by a scored silent run against real documents, by the quote check above, and by the licensed person who approves the output.
That is the division of labour we build to: the schema guarantees the record is well formed and that unknowns stay unknown, and a human makes the judgment call. Nothing leaves the agency unapproved.
We build this layer into Quote-Intake Agent and COI Agent engagements, in the agency's own AWS account, with the source code yours. If you are wiring extraction into Applied Epic, HawkSoft, or EZLynx and want a senior engineer on it, contact us.