Applied Epic, HawkSoft, and EZLynx all have APIs. The carriers, mostly, do not. So the loss run for a renewal that binds in 90 days sits behind a portal login that one CSR has memorized, and the renewal status your agent needs is a table on page three of an underwriting site that was last redesigned in 2019.
This is the part of an agent that breaks. Not the model, not the AMS write-back: the connector to a website that someone else changes without telling you. This tutorial is how we build those connectors so a change costs an hour instead of a week, and so nobody discovers the break by finding an empty renewal file.
You need Node 20+, Playwright, somewhere to run headless Chromium on a schedule, and a secrets store. We use AWS Secrets Manager and Fargate tasks in the client's own account.
Before you write a line of code
Get permission in writing. The agency has a producer or service agreement with the carrier; automated retrieval of the agency's own book is usually fine, and sometimes explicitly supported, but the portal terms are the carrier's and the exposure is the agency's. We ask the agency to email their marketing rep, describe what will be fetched and how often, and keep the reply. Two carriers so far have answered by pointing us at a download endpoint or an SFTP drop we did not know existed. That is a better connector than any browser automation, and it takes one email to find out.
Also ask which credentials to use. A shared agency service account with read access is right. A named CSR's login is wrong: it ties the agent to one person's password rotation, and it makes the carrier's audit log say she pulled 400 loss runs at 3am.
Log in once, reuse the session
Most portals will let a session live for hours or days. Logging in on every run is slow, trips fraud heuristics, and multiplies your MFA problem by the number of scheduled runs. Do it once and persist the storage state.
import { chromium } from 'playwright';
import { GetSecretValueCommand, PutSecretValueCommand, SecretsManagerClient } from '@aws-sdk/client-secrets-manager';
const sm = new SecretsManagerClient({});
const STATE_SECRET = 'carrier/acme/session-state';
async function loadState() {
try {
const res = await sm.send(new GetSecretValueCommand({ SecretId: STATE_SECRET }));
return JSON.parse(res.SecretString);
} catch {
return undefined;
}
}
export async function withPortal(fn) {
const browser = await chromium.launch();
const context = await browser.newContext({
storageState: await loadState(),
userAgent: process.env.PORTAL_USER_AGENT,
viewport: { width: 1440, height: 900 }
});
const page = await context.newPage();
try {
await page.goto(process.env.PORTAL_HOME, { waitUntil: 'domcontentloaded' });
if (await isLoginScreen(page)) {
await signIn(page);
const state = await context.storageState();
await sm.send(new PutSecretValueCommand({ SecretId: STATE_SECRET, SecretString: JSON.stringify(state) }));
}
return await fn(page);
} finally {
await context.close();
await browser.close();
}
}
The session state is credentials. It goes in the secrets store, encrypted, with the same access policy as the password, and it is never written to the task's filesystem or a log line.
isLoginScreen should test for something the login page has and the landing page does not, a password field is the usual choice, rather than assuming the redirect URL stays the same.
MFA, honestly
There are three cases and only two of them are automatable.
- TOTP. The carrier hands the agency a QR code. Store the shared secret in the secrets store and generate the code at runtime with
otplib. This is the good case and it is worth asking for by name when the agency enrolls the service account. - Email code to a shared mailbox. Read it with Microsoft Graph, the same way we read a certificates mailbox in reading a shared mailbox with Microsoft Graph. Poll for a message from the carrier's sender received after the login attempt started, pull the six digits, and delete nothing, the mailbox is the audit trail.
- SMS to a personal phone. Not automatable, and you should not try. Ask the carrier to move the service account to TOTP or an email code. If they will not, the honest answer to the agency is that this carrier stays manual, and the agent files a task for a human instead of pretending it fetched anything.
We say that out loud in the first scoping call. A connector plan that quietly assumes every carrier will cooperate is how a project slips a month.
Selectors that survive a redesign
Portals are not built to be automated, so there are no test IDs. What there is, usually, is stable visible text: a link that says "Loss Runs", a column header that says "Policy Number". Text and roles change less often than the class names a framework generates.
// Fragile: dies the next time the vendor bumps their component library.
await page.click('.mat-tab-label-content:nth-child(3) > span.ng-star-inserted');
// Better: anchored to what a human reads on the screen.
await page.getByRole('link', { name: /loss runs?/i }).click();
await page.getByLabel(/policy number/i).fill(policyNumber);
await page.getByRole('button', { name: /^search$/i }).click();
await page.getByRole('table').waitFor({ state: 'visible', timeout: 30_000 });
Two more rules that pay for themselves. Never use waitForTimeout as a substitute for a real wait condition; a sleep that is long enough on a quiet Sunday is not long enough on renewal Monday. And put every selector for a given carrier in one module, so a redesign is one file to fix and one diff to review.
Downloading the document
Loss runs arrive as a PDF, sometimes generated on demand behind a spinner. Playwright's download event handles both.
const [download] = await Promise.all([
page.waitForEvent('download', { timeout: 120_000 }),
page.getByRole('button', { name: /download|export/i }).click()
]);
const tmp = await download.path();
const bytes = await fs.readFile(tmp);
const sha = createHash('sha256').update(bytes).digest('hex');
await s3.send(new PutObjectCommand({
Bucket: process.env.DOC_BUCKET,
Key: `carrier/acme/${accountId}/${policyNumber}/${sha}.pdf`,
Body: bytes,
ContentType: 'application/pdf',
Metadata: { carrier: 'acme', policyNumber, fetchedAt: new Date().toISOString() }
}));
Keying on the content hash gives you deduplication for free: re-fetching the same unchanged loss run costs one PUT and creates no second document for a producer to wonder about. It also means the extraction step downstream, the one described in extracting ACORD fields with Textract Queries, can be cached by hash and never runs twice on the same bytes.
Check what you downloaded before you file it. A 4KB PDF is usually an error page with a letterhead. Assert the byte length, assert the magic number is %PDF, and assert that the text layer or the OCR contains the policy number you asked for. If any of those fail, treat the run as a failure, not as a document.
The canary run
A connector that breaks silently is worse than no connector, because the agency stops checking. So every connector we ship has a second scheduled job that does one fetch of one known account, and asserts on the result.
Run it every morning at 06:00 local, before the first CSR is in. It logs in, pulls the loss run for a designated test policy, and checks three things: the login succeeded, the document arrived, and the document contains the expected policy number. Anything else pages the on-call engineer and posts to the agency's operations channel with a one-line status: ACME loss-run connector failed at 06:03: login screen after session restore, MFA code not received.
That message is the whole point. When the carrier redesigns their portal on a Saturday, the agency finds out Monday at 06:03 from us, not on Thursday from an underwriter waiting on a submission. Keeping those canaries green over time is agent operations, and it is most of what ongoing hours get spent on in year two.
Rate, retry, and being a good guest
One session at a time per carrier. Serialize the queue rather than running ten browsers, put 3 to 5 seconds between page actions, and cap the run at whatever volume the agency would plausibly do by hand in a day. Retry twice on timeouts with backoff, then stop. If a login fails twice, halt the connector entirely and alert, do not retry into a lockout: the recovery cost is a phone call to the carrier's help desk and a day without the connector.
Schedule the heavy pulls overnight and leave the business day for the fetches a person is actually waiting on.
What this does not do
- It does not submit anything to a carrier. Read-only retrieval is a different risk conversation from bind authority, and our quote intake stops at the AMS submission for exactly that reason.
- It does not rate. No connector we build enters data into a rating platform and accepts the number that comes out.
- It does not defeat controls. No CAPTCHA solving, no residential proxies, no logging in as a person who does not know it is happening. If the portal is telling you it does not want a robot, the answer is a conversation with the carrier, not a workaround.
- It does not decide anything from what it fetched. The loss run lands in the document store and the agent's summary lands in front of a licensed person, on the approval pattern in building a human approval gate with Step Functions.
What to expect
Plan on two to five days per carrier for the first connector, including the permission email and one round of surprises, and one to two days for each carrier after that once the harness exists. Budget maintenance: across a book of eight to ten carrier portals, assume one of them changes something that matters each quarter. That is not a defect, it is the cost of the carrier side of the workflow, and it is why we bill hourly and keep the engineer who built the connector on it.
If you want a senior engineer to look at which of your carriers can be automated and which should stay manual, contact us with your AMS and the two or three portals your team is in most often.