Every agent we build ends the same way: something has to land in the agency management system. A Renewal Radar packet becomes an Epic activity on the account. A COI approval becomes a log note in HawkSoft. A triaged service request becomes an EZLynx task assigned to the right CSR. If the output lands anywhere else, the account manager has a second place to look, and within a month nobody looks.
Write-back is the least glamorous part of the build and the one that generates the support tickets. Not because the API call is hard, but because the call gets made twice. A Lambda retries after a network timeout that actually succeeded. A queue redelivers. An engineer reruns yesterday's batch to fix one account. Now there are three identical "Renewal review started" activities on the same policy, the account manager stops trusting the agent, and you are explaining duplicates to the principal.
This tutorial is the write-back layer we put in front of every AMS: a stable idempotency key, a claim table, one adapter interface across the three platforms, and a no-op mode for the silent run.
Assume the API will not help you
Applied Epic, HawkSoft, and EZLynx expose different surfaces, on different contracts, with different auth. The endpoints change; treat the specifics as something you confirm with your integration contact, not something you hard-code across the codebase. What you can assume in all three cases:
- There is no
Idempotency-Keyheader. You cannot hand the platform a token and let it dedupe for you. - Creating an activity twice is legal. The AMS will happily store both.
- There is a rate limit, and you will find it during a backfill rather than in the documentation.
- There is no transaction spanning your database and theirs.
So idempotency is your job, on your side, before the call.
Step 1: derive a key that survives a rerun
The key has to be a pure function of the work, not of the run. A UUID generated at send time is worthless: rerun the batch, get a new UUID, get a duplicate. Hash the things that identify this piece of output for this account.
import hashlib, json
def write_key(*, ams: str, account_id: str, policy_id: str | None,
kind: str, source_id: str, day: str) -> str:
"""Stable across retries and reruns; different for genuinely new work."""
payload = {
"ams": ams, # "epic" | "hawksoft" | "ezlynx"
"account": account_id, # AMS account id, not the agent's guess at a name
"policy": policy_id or "",
"kind": kind, # "renewal_review" | "coi_issued" | "service_request"
"source": source_id, # message id, document id, or renewal cycle id
"day": day, # "2026-03-14" for recurring work; "" for one-shot
}
blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(blob.encode()).hexdigest()
Two judgement calls in that payload.
source_id is what makes reruns safe. For a COI, use the mailbox message id, not the certificate holder's name. For a renewal, use the policy id plus the renewal effective date, so next year's cycle is genuinely new work and this year's rerun is not.
day is a deliberate escape hatch for recurring output. A weekly service-request volume note should post once per week, not once ever. Bucket it. Leave it empty for anything that should exist exactly once for all time.
Do not put drafted text in the key. Reword the activity body and the key changes, and you are back to duplicates.
Step 2: claim before you call
The pattern is claim, call, confirm. Write the key with a conditional put; if the claim fails, someone else already owns this write.
import time, boto3
from botocore.exceptions import ClientError
ddb = boto3.client("dynamodb")
TABLE = "ams_writes"
class AlreadyWritten(Exception): pass
def claim(key: str, ttl_seconds: int = 60 * 60 * 24 * 400) -> None:
try:
ddb.put_item(
TableName=TABLE,
Item={
"writeKey": {"S": key},
"status": {"S": "IN_FLIGHT"},
"claimedAt": {"N": str(int(time.time()))},
"expiresAt": {"N": str(int(time.time()) + ttl_seconds)},
},
ConditionExpression="attribute_not_exists(writeKey) OR "
"(#s = :inflight AND claimedAt < :stale)",
ExpressionAttributeNames={"#s": "status"},
ExpressionAttributeValues={
":inflight": {"S": "IN_FLIGHT"},
":stale": {"N": str(int(time.time()) - 900)},
},
)
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
raise AlreadyWritten(key)
raise
def confirm(key: str, remote_id: str) -> None:
ddb.update_item(
TableName=TABLE,
Key={"writeKey": {"S": key}},
UpdateExpression="SET #s = :done, remoteId = :r, wroteAt = :t",
ExpressionAttributeNames={"#s": "status"},
ExpressionAttributeValues={
":done": {"S": "WRITTEN"},
":r": {"S": remote_id},
":t": {"N": str(int(time.time()))},
},
)
The claimedAt < :stale clause matters. A process that dies mid-call leaves the key IN_FLIGHT forever, and every later attempt is silently swallowed: the worst failure mode, because it looks like success. Fifteen minutes is long enough that no live call is still running and short enough that a stuck item recovers the same morning.
Set the TTL past a full renewal cycle. Thirteen months is a reasonable floor; a table of hashes is cheap, and an expired key means a duplicate on the anniversary.
Step 3: one adapter, three platforms
Keep the AMS-specific parts small and behind an interface. Everything above this line is identical across Epic, HawkSoft, and EZLynx; everything below it changes when a platform changes.
from dataclasses import dataclass
from typing import Protocol
@dataclass
class ActivityDraft:
account_id: str
policy_id: str | None
kind: str
subject: str # short, scannable in a list view
body: str # what the agent did, what it checked, what it needs
assign_to: str | None # AMS user id of the owner, never a shared queue
source_refs: list[str]
class AmsWriter(Protocol):
def create_activity(self, draft: ActivityDraft) -> str: ...
def write_once(writer: AmsWriter, draft: ActivityDraft, key: str) -> str | None:
try:
claim(key)
except AlreadyWritten:
return None
remote_id = writer.create_activity(draft) # retries live inside the adapter
confirm(key, remote_id)
return remote_id
Retries belong in the adapter, with exponential backoff and jitter, and only on 429 and 5xx. Never retry a 400: a malformed activity will be malformed on the fourth attempt too, and you have just spent your rate limit finding that out.
A detail that costs nothing now and saves an afternoon later: put the write key in the activity body itself, on its own line, as ref: 8f3c.... When someone reports a duplicate, you can search the AMS for the key and see immediately whether it is two writes of one key (your bug) or two keys for one event (a key-derivation bug). Those have different fixes.
Step 4: no-op mode is not optional
During the silent run the agent processes the live book and writes nothing. That is a hard requirement, and a config flag read at the call site is a bad way to enforce it, because there will be six call sites.
class DryRunWriter:
"""Same interface, no side effects. Used for the silent run."""
def __init__(self, sink): self.sink = sink
def create_activity(self, draft: ActivityDraft) -> str:
self.sink.record(draft)
return "dryrun:" + write_key(
ams="dryrun", account_id=draft.account_id,
policy_id=draft.policy_id, kind=draft.kind,
source_id=",".join(draft.source_refs), day="",
)[:16]
Inject the writer at the edge of the process. The agent code cannot tell the difference, which is the point: what you score during the silent run is exactly what would have been written. Wire the production credentials into one place, and make the dry-run writer the default so a missing config produces silence rather than 400 activities on a live book.
What this does not solve
- Duplicates the AMS already had. If a CSR hand-keyed the same activity ten minutes earlier, your key never saw it. Match on account, kind, and a time window before writing, and skip with a logged reason.
- Partial writes. Create an activity, attach a document, fail on the attachment: the activity exists. Make the activity the last step where you can, or record the remote id so the retry attaches rather than recreates.
- Ordering. Two agents writing to one account do not coordinate. If sequence matters, serialise per account, not globally.
- Deletes. Most write paths are append-only by policy, and that is the right default. Correcting a bad activity is a human action with a note, not an automated cleanup job.
None of this changes who decides. The write-back layer moves an approved draft into the system of record; a licensed person still approved it. What idempotency buys you is that approving once means it appears once, which is the only version of the story that survives an audit.
We build this layer inside the agency's own AWS account, against the agency's own AMS credentials, on every engagement: Renewal Radar, COI Agent, and Service-Request Agent all share it. If you have duplicate activities in Epic, HawkSoft, or EZLynx from an integration that is already running, tell us what you are seeing and a senior engineer will reply within one business day.