Conversational agentic systems, and their evaluation
EVA is the conversational agent of Groupe Mutuel, a Swiss health insurer. In the mobile app, it answers policyholders about what their plan covers, in four languages. And I built the harness that proves it answers right, every time.
EVA is multi-agent, multi-turn, in four languages, across dozens of products and plan levels. An answer that is right for one level can be wrong for another. The old check looked at one answer, one message, once. Models changed often, with no reliable way to catch regressions.
channel
Mobile app
iOS and Android. Where 115,000 policyholders ask their questions.
backend
API
FastAPI, async Python. Answers are streamed.
reasoning
Azure OpenAI
The GPT models every agent runs on.
knowledge
Azure AI Search
General terms, business rules, knowledge base.
memory
Azure Blob Storage
The history of every conversation.
EVA
● Guardrails and language detection first (my work). Then a router calls 0 to n of 7 specialised agents, in parallel, and a synthesis agent writes one answer in the policyholder's language.
business system
Policy API
Deductible, plan, premiums, invoices.
policyholder data
Database
The source of truth.
observability (my work)
Langfuse
Every LLM call, agent and tool traced, with cost and latency.
evaluation (my work)
The harness
Simulated policyholders, calibrated judges, pass^k. Gates feature rollouts.
data (my work)
Databricks
Nightly ETL of the traces; questions grouped by topic with BERTopic.
improvement
Expert review
Business experts review conversations and flag errors; dashboards in Power BI.
runtime
Azure Kubernetes Service
Everything above runs here, in Docker containers.
delivery
GitLab CI/CD
Lint, types, tests, images. Dev → int → staging → prod, plus a preview per branch.
health (my work)
Golden questions
A pod stays out of rotation until its latency is back under the threshold.
Each question goes through guardrails, then a router that calls zero to n specialised agents in parallel, then a synthesis agent that answers in the policyholder's language. A coverage question takes about ten LLM calls, and every call is traced with its cost and latency. We shipped 25 releases of this architecture in about 14 months.
situations
125 situations
One YAML file each: a goal, a contract, a persona, tags. Played with every relevant product, plus one that covers nothing.
personas
8 personas, 4 languages
Different ways of asking. Facts are only revealed when EVA asks for them.
Simulated policyholder ⇄ EVA
An LLM plays the policyholder and talks, turn after turn, with the real EVA. N situations × k runs, in parallel.
goal agent
Knows when to stop
After each turn it checks whether every part of the goal got an answer. A turn limit stops the rest.
output
conversations.jsonl
Messages, EVA's steps and the goal agent's decision. Kept, so they can be judged again without simulating again.
The LLM classifies, the code scores
Each use case has its evaluators. LLM judges give a typed verdict per claim; they are calibrated on human labels before their scores count.
control
Faithfulness check
If the simulated policyholder invented a fact, the conversation is excluded, not judged.
code
Checks and scoring
Which agents were called, whether the goal was reached; then a penalty table turns the judges' verdicts into a score.
humans
Review
A reviewer can overturn a verdict; the score is recomputed without calling the LLM again.
per use case
pass@k and pass^k
A run passes when its score clears the threshold with no critical error. Over k runs: at least once, or every time.
interface
Evaluation Lab
Runs, conversations and verdicts in one place. Arena compares two deployments blind.
production
Release gate
A new feature reaches policyholders only once its use cases hold.
- A simulated policyholder with a goal, a persona, and facts it only reveals when asked.
- Each situation is played with every relevant product, a level that doesn't cover it, and a product that covers nothing, so refusals are tested as much as answers.
- Judges are calibrated before their scores count. Every run is reproducible: code, documents and dataset are fingerprinted.
Chance of passing, for an agent that succeeds 70 % of the time
| k | pass@k, at least one success | pass^k, all succeed |
|---|---|---|
| k = 1 | 70 % | 70 % |
| k = 2 | 91 % | 49 % |
| k = 3 | 97 % | 34 % |
| k = 4 | 99 % | 24 % |
| k = 5 | 100 % | 17 % |
- pass@k
- At least one of k runs passes. It climbs fast: 100 % at k = 5. That is what a demo shows.
- pass^k
- All k runs pass. It falls just as fast: 17 % at k = 5. That is what a policyholder who asks again expects.
- policyholders have used EVA
- 115,000
- have access to it in the app
- 400,000+
- calls avoided in 6 months
- ~18,000
- estimated savings
- ≈ CHF360k
The harness itself: 219 tests, 125 situations × 30 products, 4 languages, 8 personas, built solo in about six weeks. It catches regressions and gates feature rollouts in production.
A real test situation from the harness. The simulated policyholder knows things EVA has to ask for, then judges check every claim of the answer.
- simulated policyholderI need to replace an old crown. Does my insurance pay for it?
- EVAIt depends on why it needs replacing. Was it caused by an accident, or by a cavity?
- simulated policyholderA cavity. My dentist estimates about CHF 2,500.
- EVABasic insurance doesn't cover dental care after a cavity. Your dental supplement pays part of it, up to its yearly limit. Do you want the details of your plan?
- 1. guardrails, language detected: English
- 2. router → coverage agent
- 3. coverage agent: basic insurance, dental supplement
- 4. synthesis → answer
- Asked for the cause before answeringexpected
- Basic insurance doesn't cover care after a cavitycorrect
- The dental supplement pays part of itcorrect
Goal met in 2 turns.
The same question, replayed 5 times
- Try 1: pass
- Try 2: pass
- Try 3: fail
- Try 4: pass
- Try 5: pass
- pass@5
- pass: one success is enough
- pass^5
- fail: all five must pass
The judges were wrong before EVA was: calibration showed they flagged the standard texts EVA adds to its answers as unsourced. We fixed the judges before trusting a single score.