Research · Paper
The confidence line: where AI hands a claim to a person
A confidence score is only useful if it means what it says, and if someone has decided what a wrong answer costs. This paper explains how calibration and a signed cost matrix set the line at which an insurance claim moves on its own or reaches a handler.
- Author
- elan1 research team · Research
- Published
- Reading time
- 4 min read
Why a raw score is not enough
Most models produce a number between 0 and 1. Many teams treat it as a probability and pick a threshold, often 0.9, because it sounds safe. Two things can go wrong. The score may not be calibrated, so 0.9 does not mean right nine times in ten. And the threshold may have nothing to do with what a mistake actually costs.
Calibration first
A model is calibrated when, across all cases scored at 0.9, about 90% are right. elan1 Decide is designed to report calibrated confidence for each typed answer, whether it is a Boolean, a choice or a score. Calibration is checked on your own historical claims, in shadow beside your current process, before any automation is switched on. No calibration, no automation.
A signed cost matrix
Calibration says how often the model is wrong. The cost matrix says what each kind of wrong costs. For a first-notice-of-loss triage decision, the matrix might set out:
- Routing a simple claim to a handler unnecessarily: a few minutes of handler time.
- Fast-tracking a claim that needed a handler: rework, a delayed settlement and a possible complaint.
- Missing an injury flag: potentially serious harm and a regulatory conversation.
The values are a business judgement, and the person who owns the risk signs them. That signature is the point: the line is set by someone who can answer for it, not by a data scientist's default.
Where the line falls
With calibrated confidence and a signed matrix, the line is arithmetic. A claim moves on its own only when the expected cost of acting is lower than the cost of a person reviewing it. Everything else is routed.
Four routes, not two
A yes-or-no threshold wastes information. elan1 Decide is designed with four routes. Act, when confidence clears the line. Review, when a handler should confirm. Escalate, when a harder case needs deeper reasoning or a specialist. Hold, when inputs are missing or the system is unsure in a way the matrix did not anticipate. Fraud signals always go to a specialist; they never deny a claim on their own.
Watching the line move
Lines drift as claim mix, weather and repair costs change. The record of every decision, including the handler's answer where one was asked, lets the risk owner see whether the line still sits in the right place. Re-signing the matrix each quarter is a sensible default.
Regulators expect this discipline. IRDAI's policyholder protection regulations ask for timely and reasoned claim decisions, and the EU AI Act lists risk assessment and pricing in life and health insurance as high-risk. A line someone signed, with a record behind it, is easier to defend than a threshold nobody owns.
Key takeaways
- Confidence scores must be calibrated on your own data before any claim moves without a person.
- The risk owner signs a cost matrix; the line between act and review follows from it.
- Four routes (act, review, escalate, hold) use more of what the model knows than a yes-or-no threshold.
Further reading
Product capabilities named here carry their current status (available, in pilot or designed) on each product page and in Trust and security. Figures marked “Design target” are specifications, not measurements.
About the author
elan1 research team
Research
The research team writes elan1's papers and briefs. Each one states its method and its limits, labels design targets as targets, and cites third-party figures to their primary source. elan1 has no customer results to publish yet, and our research does not imply any.
