A commercial submission packet is not a clean document set. A workers' comp supplemental carries owner SSNs. A driver schedule carries licence numbers and dates of birth. A premium finance form carries a bank routing and account number. The ACORD 125 carries the FEIN. All of it arrives as email attachments and lands in the document store, and then an agent reads it.
The question an operations director will ask you, usually second or third in the meeting, is what leaves the agency's environment and where it goes. "Single-tenant AWS" is a real answer for storage and compute, but it is not an answer for the model call. This tutorial builds the layer that is: a redaction step between the document store and the model, deterministic where it can be, reversible where the draft needs the real value back.
You need Python, an AWS account with permission to call Amazon Comprehend and whatever model you use, and a document extraction step already in place. If you are extracting fields with Textract Queries, redaction sits after extraction and before any prompt.
Decide what actually needs to reach the model
Start here, before any code. Most agent work does not need the sensitive fields at all.
The COI Agent needs the named insured, the policy numbers, the limits, the endorsement wording, and the holder's requirements. It never needs an SSN. The Quote-Intake Agent needs entity name, address, class of business, payroll, revenue, prior claims, and the gaps in the packet. It needs the FEIN to exist and be well-formed so it can be keyed into the AMS; it does not need the model to reason about the digits. The Service-Request Agent needs enough identifying text to match an account.
So split fields into three buckets:
- Needed for reasoning. Insured name, addresses, dates of coverage, limits, class codes, claim descriptions. These go through as-is. Redacting the insured's name breaks account matching and produces a worse gap list.
- Needed in the output, not in the reasoning. FEIN, licence numbers, account numbers, dates of birth. Replace with a placeholder on the way in; put the real value back on the way out.
- Not needed at all. SSNs, full bank details, anything the agency keeps only because a carrier form asked for it. Replace with a placeholder and never re-hydrate it into a draft that goes to a third party.
Write that classification down per workflow and get the agency's operations lead to sign it. It is the artifact that answers the question in the meeting.
Deterministic patterns first
Run regex before you run anything statistical. SSNs, FEINs, and ABA routing numbers have fixed shapes, and a pattern match with a checksum is more reliable than a model on exactly the fields you least want to miss.
import re
import hashlib
PATTERNS = {
"SSN": re.compile(r"\b(?!000|666|9\d\d)\d{3}-\d{2}-\d{4}\b"),
"FEIN": re.compile(r"\b\d{2}-\d{7}\b"),
"ROUTING": re.compile(r"\b\d{9}\b"),
"DL": re.compile(r"\b[A-Z]{1,2}\d{6,12}\b"),
}
def aba_checksum_ok(number: str) -> bool:
w = (3, 7, 1, 3, 7, 1, 3, 7, 1)
return sum(int(d) * f for d, f in zip(number, w)) % 10 == 0
def token_for(kind: str, value: str, doc_id: str) -> str:
digest = hashlib.sha256(f"{doc_id}:{value}".encode()).hexdigest()[:8]
return f"[{kind}_{digest}]"
def redact_patterns(text: str, doc_id: str):
vault = {}
def substitute(kind):
def repl(match):
value = match.group(0)
if kind == "ROUTING" and not aba_checksum_ok(value):
return value
placeholder = token_for(kind, value, doc_id)
vault[placeholder] = {"kind": kind, "value": value}
return placeholder
return repl
for kind, pattern in PATTERNS.items():
text = pattern.sub(substitute(kind), text)
return text, vault
Two details matter more than they look.
The placeholder is stable per document and per value. The same FEIN appearing on page 1 and page 9 gets the same token, so the model can still tell that two forms describe the same entity. That is the difference between a usable gap list and one that flags a mismatch on every page.
The bare nine-digit routing pattern will hit things that are not routing numbers. The checksum filters most of it. Expect to tune DL: state formats vary, and a broad pattern will eat policy numbers. Test it against the agency's real documents before you trust it, which means running it inside their account, not on samples.
Amazon Comprehend for the shapeless rest
Names of individuals, dates of birth in prose, addresses inside claim narratives: no regex holds these. Amazon Comprehend's PII detection returns typed entities with offsets and confidence scores, which is what you want for a second pass. The PII detection documentation lists the entity types.
import boto3
comprehend = boto3.client("comprehend")
SENSITIVE_TYPES = {"SSN", "BANK_ACCOUNT_NUMBER", "BANK_ROUTING",
"DRIVER_ID", "PASSPORT_NUMBER", "DATE_TIME"}
MIN_SCORE = 0.80
def redact_entities(text: str, doc_id: str, vault: dict):
found = comprehend.detect_pii_entities(Text=text[:5000], LanguageCode="en")
spans = [
e for e in found["Entities"]
if e["Type"] in SENSITIVE_TYPES and e["Score"] >= MIN_SCORE
]
for entity in sorted(spans, key=lambda e: e["BeginOffset"], reverse=True):
value = text[entity["BeginOffset"]:entity["EndOffset"]]
placeholder = token_for(entity["Type"], value, doc_id)
vault[placeholder] = {"kind": entity["Type"], "value": value}
text = text[:entity["BeginOffset"]] + placeholder + text[entity["EndOffset"]:]
return text, vault
Replace from the end of the string backwards, or every offset after the first substitution is wrong. detect_pii_entities has a byte limit per call, so chunk long documents on page boundaries and keep the vault across chunks. And note DATE_TIME in that set: it is deliberately included because dates of birth show up as dates, and deliberately dangerous, because policy effective dates do too. Either scope it to the pages where a DOB is expected, such as a driver schedule, or leave it out and accept that DOBs in prose survive. We usually scope it.
Put the values back at draft time
The vault stays in the agency's account, next to the document, with the same retention rules. Re-hydration happens on the way out, after the model has produced structured output and before anything is written to the AMS or shown to a reviewer.
def rehydrate(draft: str, vault: dict, allowed_kinds: set) -> str:
for placeholder, entry in vault.items():
if placeholder not in draft:
continue
if entry["kind"] not in allowed_kinds:
raise ValueError(f"model asked to emit {entry['kind']} in a draft")
draft = draft.replace(placeholder, entry["value"])
return draft
The allowed_kinds check is the useful part. For an AMS submission record, {"FEIN"} may be allowed. For an outbound certificate or an email to a holder, the allowed set is usually empty. If the model has placed an SSN placeholder in a draft that is about to leave the agency, you want that to raise, get logged, and go to a person, not to be quietly filled in.
Any placeholder left in a draft is also a signal. A reviewer seeing [SSN_9f2c1a4b] in a queued item knows exactly what happened and can decide. That is better behaviour than silent stripping.
A second net, not the first one
If you are on Amazon Bedrock, Guardrails can apply sensitive-information filters to prompts and responses independently of your code, with either block or mask behaviour. The Guardrails documentation covers configuration.
Use it. Do not rely on it as the only control. Guardrails sit at the model call and know nothing about which fields your workflow needs, which document a value came from, or how to put a FEIN back into an AMS field. Two independent layers, one in your pipeline and one at the boundary, is the posture worth describing to an E&O carrier: the second catches what the first missed, and both are logged.
Test it like a connector, not like a prompt
Build a fixture set from the agency's own documents, redacted copies, held in their account, and assert on it in CI:
- Every known SSN in the corpus is tokenised. This is the test you cannot fail.
- Named insured, policy numbers, and limits survive untouched. Over-redaction quietly degrades the agent, and nobody files a ticket about a slightly worse gap list.
- The same value gets the same token within a document, and different tokens across documents.
rehydrateraises on a disallowed kind.- Placeholder round-trip: redact, re-hydrate with everything allowed, get the original text back byte for byte.
Run it on every change to the pattern set. Redaction is a connector, in the sense that it breaks when documents change, and carrier forms change.
Limits, stated plainly
- Recall is not 100%. Comprehend returns scores, not certainties, and a handwritten SSN on a scanned fax may never make it out of OCR in a matchable shape. Redaction reduces exposure; it does not eliminate it. Say so in writing rather than promising a clean boundary.
- Redaction does not de-identify a document. A named insured with an address and a class code is identifiable. This work protects specific field types, not identity.
- Regex on OCR text inherits OCR errors.
12-3456789read as12-345G789will not match. Extract structured fields where you can and redact those, rather than relying only on the raw text layer. - This is not legal or compliance advice. Which fields are in scope for an agency is a question for its counsel and its E&O carrier; your job is to make the classification enforceable in code.
- None of this changes who approves a send. The agent drafts, a licensed person approves.
Where this fits
We build this layer into every engagement that touches documents, because it is the first thing an operations lead asks about and the last thing anyone wants to retrofit. It runs in the agency's own single-tenant AWS account, the vault never leaves it, and the client owns the code.
It is part of how Quote-Intake Agent and COI Agent work get scoped. If you are standing up a document pipeline and want a senior engineer on the data boundary, contact us.