A certificate request arrives with a contract excerpt attached. The holder wants to be named as additional insured on a primary and non-contributory basis, with a waiver of subrogation in their favor, on general liability and auto. Producing the ACORD 25 takes four minutes. Answering whether the policy actually does that takes twenty, because it means opening the policy PDF, finding the endorsement schedule, and reading the form.
Under volume, the second job is the one that gets skipped. The certificate goes out describing coverage the policy may not grant, and the agency owns the difference.
This tutorial covers the check, not the certificate. You already have the request parsed and matched to an account. What you need next is an answer to one question per requirement: is this on the policy, and where. The agent produces that answer with a citation. A licensed person reads it and decides. The agent does not issue, and it does not send.
What the check actually is
Strip the language and every holder requirement reduces to a small set of claims about the policy:
- An additional insured endorsement applies to this holder, by schedule or by blanket wording tied to a written contract.
- The additional insured status extends to ongoing operations, completed operations, or both.
- Coverage is primary and non-contributory.
- A waiver of subrogation applies in favor of the holder.
- Limits meet a stated floor, including any required umbrella or excess layer.
- Cancellation notice terms match what the contract demands.
Limits and dates come from structured AMS fields and need no retrieval. The rest live in endorsement forms attached to the policy, which in most agencies means a PDF in the document store and nothing in the database. That is the gap the retrieval step fills.
Step 1: build the endorsement index per policy
Do not build one index across the book. Build a small index per policy term, rebuilt when a document is added. Cross-account retrieval is how you end up citing another insured's endorsement.
Pull the documents attached to the policy through the AMS document API, filter to policy forms and endorsement packets, and split each PDF on form boundaries rather than on a fixed token count. Standard ISO and carrier forms carry a form number and edition date in the footer, usually in the shape CG 20 10 04 13 or CA 04 44 10 13. A regex over the extracted text finds those reliably, and each hit is the start of a new chunk.
import re
FORM_RE = re.compile(
r"\b([A-Z]{2}\s?\d{2}\s?\d{2})\s?((?:0[1-9]|1[0-2])\s?\d{2})\b"
)
def split_on_forms(pages):
"""pages: list of (page_number, text). Returns chunks with form metadata."""
chunks, current = [], None
for page_no, text in pages:
match = FORM_RE.search(text)
if match or current is None:
if current:
chunks.append(current)
current = {
"form_number": match.group(1).replace(" ", " ") if match else None,
"edition": match.group(2) if match else None,
"start_page": page_no,
"end_page": page_no,
"text": text,
}
else:
current["text"] += "\n" + text
current["end_page"] = page_no
if current:
chunks.append(current)
return chunks
Keep start_page and end_page. They are the citation. An answer a CSR cannot verify in ten seconds is an answer they will re-derive by hand, which puts you back where you started.
Embed each chunk, store vectors alongside the form number and page range, and keep the raw text. The store can be pgvector, OpenSearch, or whatever the single-tenant account already runs. At the scale of one policy term, ten to sixty chunks, the choice does not matter.
Step 2: retrieve on form number first, text second
Semantic search alone is weak here. Endorsement forms share vocabulary; CG 20 10 and CG 20 37 read almost identically, and they mean different things: ongoing operations versus completed operations. Getting that pair wrong is exactly the error the check exists to prevent.
So run two retrievals and merge.
The first is a lookup table. Each requirement type maps to the form numbers that usually satisfy it. If the chunk metadata contains one of them, that chunk goes into context regardless of embedding distance.
FORM_HINTS = {
"ai_ongoing": ["CG 20 10", "CG 20 33", "CG 20 38"],
"ai_completed": ["CG 20 37"],
"waiver_gl": ["CG 24 04"],
"waiver_wc": ["WC 00 03 13"],
"primary_nc": ["CG 20 01"],
"ai_auto": ["CA 20 48", "CA 04 49"],
}
Treat the table as a hint, never as the answer. Carriers issue proprietary equivalents with their own numbering, and manuscript endorsements carry no ISO number at all. A form number present is evidence to read the form; a form number absent proves nothing.
The second retrieval is vector search over chunk text using the requirement phrased as the contract phrased it. Take the top five. Union the two sets, cap the context, and pass it on with page numbers attached.
Step 3: ask for a verdict with three outcomes, not two
The model gets the retrieved chunks and one requirement at a time. One requirement per call. Batching them invites the model to reuse the reasoning from the first on the fourth.
The response schema has three verdicts, and the third one is the point of the exercise:
{
"type": "object",
"required": ["verdict", "form_number", "pages", "quote", "reasoning"],
"properties": {
"verdict": {
"enum": ["satisfied", "not_satisfied", "cannot_determine"]
},
"form_number": { "type": ["string", "null"] },
"pages": { "type": "array", "items": { "type": "integer" } },
"quote": {
"type": ["string", "null"],
"description": "Verbatim sentence from the retrieved text. No paraphrase."
},
"reasoning": { "type": "string", "maxLength": 600 }
}
}
Use the provider's structured output mode so the shape is enforced rather than requested. Then enforce the rest yourself in code:
- If
verdictissatisfiedandquoteis null, downgrade tocannot_determine. - If
quoteis not a substring of any retrieved chunk after whitespace normalisation, downgrade. A quote the agent invented is the failure mode that matters most, and a substring check catches it for free. - If
pagesfalls outside the page range of the cited chunk, downgrade.
Blanket wording deserves its own rule. A blanket additional insured endorsement grants status to any party the named insured has agreed in a written contract to add, executed before the loss. Whether that covers this holder depends on a contract the agent has not read and cannot verify. So when the only support is blanket wording, the verdict is cannot_determine with a note naming the form. That is not the agent hedging. It is the correct answer, and it is the answer a licensed person should be looking at.
Step 4: write the result where the CSR already works
The output is a short table on the certificate task, one row per requirement, each with a verdict, a form number, and a page link into the document in the AMS. Not a chat window, not a separate portal.
Holder: Northgate Construction LLC
Request: ACORD 25, GL + Auto + Umbrella
Additional insured, ongoing ops (GL) satisfied CG 20 10 04 13 p.14
Additional insured, completed ops (GL) not_satisfied - -
Primary and non-contributory (GL) satisfied CG 20 01 04 13 p.17
Waiver of subrogation (GL) cannot_determine CG 24 04 05 09 p.19 blanket, needs contract
Additional insured (Auto) satisfied CA 20 48 10 13 p.31
GL limit >= $2M aggregate satisfied AMS policy field
The CSR now works two lines instead of six. The completed-operations gap goes to the producer as a coverage conversation before the certificate is issued, which is the outcome the manual process rarely reaches in time.
Every run writes an activity to the account with the requirement list, the verdicts, the cited forms, and the model version. When a certificate is questioned two years later, the record shows what was checked and who approved it.
What we will not claim for this
The agent reads what is in the document store. If the endorsement packet was never attached to the policy in the AMS, and in most books some are not, the honest verdict is cannot_determine, and the fix is a document-coverage report before go-live rather than a smarter prompt.
It does not interpret the underlying contract. It does not decide whether the agency should issue the certificate. It does not touch the ACORD 25 fields, and it does not send.
Score it before you trust it. Pull 200 closed certificate requests where the policy documents are on file, have an account manager mark each requirement, and compare. Two numbers matter: how often a satisfied verdict was wrong, which is the number your E&O carrier cares about, and how often the answer was cannot_determine when the document was right there, which is the number that decides whether the agent saves anyone time. We hold the first under one percent before a supervised go-live, and we accept a high second number in exchange.
If you are working through this on Applied Epic, HawkSoft, or EZLynx and want to compare notes on document coverage in your own book, tell us which system you are on and roughly how many certificate requests you handle a week.