Running an agent in shadow on your past cases before cut-over

Every vendor demonstration is run on cases the vendor chose. Your process has its own quirks: a regional garage that always under-quotes, a dealer whose invoices never match the order, a tax code nobody has updated since 2019. The only way to know how an agent will handle your work is to give it your work, without letting it change anything.

What shadow means

In shadow, the agent receives exactly what your team received, makes its own decision and records it, and stops there. Nothing is written to a live system. Afterwards, its decisions are compared with what your people actually did. The comparison is the evidence for, or against, switching anything on.

How to run it

1. 01Choose a closed period of past cases, such as last quarter's motor claims or last month's purchase bills, large enough to include the awkward ones.
2. 02Mask personal data before the agent sees it, and keep the data in the region where it belongs.
3. 03Replay each case through the agent, recording its answer and its stated confidence.
4. 04Compare its answers with your team's decisions, case by case.
5. 05Review every disagreement with the process owner. Sometimes the agent is wrong; sometimes the record shows your team was.
6. 06With your risk owner, set the confidence line for each answer: above it the case can move, below it a person decides.
7. 07Run in shadow again on live cases, in parallel with your team, until the agreement holds.

What to measure

• Agreement, answer by answer. An agent can be excellent at deciding which queue a claim belongs in and poor at judging injury severity.
• Calibration. When the agent says it is 90% sure, is it right about 9 times in 10? A confidence score that does not mean what it says cannot set a line.
• Coverage. What share of cases would clear the line and move on their own, and what share would reach a person?
• Disagreements by type. Clusters show you a rule to write, a data problem to fix or a kind of case to keep with people.

No calibration, no automation

In elan1 Decide, shadow is not optional. A decision runs beside your current process, on your own data, and must match that process before any automation is switched on. The thresholds are signed by your risk owner, and every case below them fails toward a person, never toward an action. Our paper on the confidence line sets out the method in more depth.

The cut-over decision stays with you

Shadow produces evidence, not permission. The person who owns the process decides whether to cut over, for which answers and from which date, and that decision is recorded like any other. Start with one route, watch it for a few weeks, and widen it only when it holds. If it stops holding, move the line back. Nothing about switching on is permanent.