Industry surveys keep finding the same thing: most independent agencies say they plan to increase their use of AI, and a small fraction have actually deployed anything beyond a chat widget (Applied Systems, Liberty Mutual, and others published agency surveys to this effect in 2025 and 2026). The gap is not that agency owners are slow. It is that the products on offer are hard to evaluate, and the demos all look the same.
This is a list of the questions that separate a tool that will still be running in your agency in 18 months from one that will be a line item you forgot to cancel. It is written for a principal or operations lead at a commercial-lines agency on Applied Epic, HawkSoft, or EZLynx, evaluating either a vendor product or a custom build.
1. Where does the output land?
If the answer is "in our dashboard", ask why. Your team lives in the AMS. Your E&O trail is the AMS activity log. Your management reports read from the AMS. A tool whose output lives somewhere else creates a second place to look, a second place for things to fall through, and a second system to reconcile. Prefer tools that write activities, tasks, and notes into the AMS through its published API and keep their own interface minimal.
2. Which systems does it read?
A renewal, a certificate, and a submission each touch the AMS, at least one carrier system, the inbox, and a document store. A tool that reads only the AMS is a reporting tool with an AI label. Ask, for each workflow, which systems it reads and how: API, portal automation, email parsing, or "you upload the file", which means your staff are doing the integration by hand.
3. What does it send, and who approves?
Ask whether the tool can send anything (an email to a client, a submission to a carrier, a certificate to a holder) without a person approving that specific action. If it can, ask how you turn that off, and whether the approval is logged. In a commercial agency the E&O exposure is in the send. Any tool that removes the human from that step has moved your liability without telling you.
4. How was accuracy measured, and on whose book?
A demo on the vendor's sample data proves nothing. Ask for the process by which accuracy is established on your data before go-live. The right answer involves running the tool silently against your real recent history, comparing its output with what your team actually did, and tuning from the differences. If the vendor cannot describe that process, there is no accuracy number, only a hope.
5. What happens when a carrier portal changes?
Carrier portals change without notice, and any tool that reads them will break. Ask who fixes it, how fast, and whether that is included in the monthly fee. A tool that breaks silently is worse than no tool, because your team will have stopped doing the manual check.
6. What happens when your AMS vendor ships the same feature?
They will. Applied, HawkSoft, EZLynx, and Vertafore are all shipping assistants inside their products, and the single-step features (summarize this email, draft this note) will arrive natively. Ask the vendor which parts of their product they expect the AMS to absorb and what remains. A good answer names the cross-system workflows and the judgment steps. A bad answer is "our AI is better".
7. Who owns the code and the configuration?
For a custom build, insist on owning the source and the infrastructure definitions. For a product, ask what you can export if you leave: the rules you tuned, the mappings you built, the history. Lock-in is the vendor's business model, and it is not yours.
8. Where does the data go, and who trains on it?
Ask for a written answer to three things: where is client and policy data processed (region and provider), how long is it retained, and is any of it used to train any model. The answer to the third should be no, in writing, including for the AI model provider underneath the tool.
9. Single-tenant or shared?
Ask whether your agency's data and processing run in an environment dedicated to you or in a shared one. Shared is cheaper and is fine for many things. For a system that reads your whole inbox and your whole book, you should at least know, and your E&O carrier may want to.
10. What is the total cost?
Ask for the total: build or setup fee, monthly fee, per-user fees, per-policy or per-transaction fees, API program fees your AMS vendor charges for third-party access, and cloud costs. A tool with a low monthly fee and a per-transaction charge on certificates will cost more than it looks like in an agency that issues 200 a week. A vendor that states its fees plainly knows its own costs.
11. Who does the work?
For a custom build: are the people who scope it the people who build it, and where are they? Hand-offs to an offshore bench are where scope gets lost. For a product: who answers when it breaks, and what is their turnaround?
12. What is the smallest thing we can try?
A good vendor will name one workflow, a clear boundary around it, and a delivery date measured in weeks, and will tell you what it will not do. Be wary of an engagement that starts with a paid "AI strategy" phase. You do not need a strategy to find out whether a renewal list with risk reasons is useful. You need the list.
The real test
Every question above is a version of one question: is this tool built for the parts of the work your AMS will never do (crossing systems, checking against the policy, holding a human accountable for the send), or for the parts your AMS will do natively next year? Buy the first kind. Rent the second kind, if at all, month to month.
If you would like our answers to all twelve for our own work, they are on the approach and services pages, and you can hold us to them.