Jude against PolyAI

Each PolyAI claim, the deterministic target set to exceed it, and Jude's measured result from the live harness. 1289 of 1289 synthetic conversations verified end to end across 5 runs.

Id	PolyAI claim	PolyAI figure	Target	Jude, measured		Proof
`P01`	At 90% per-turn accuracy only 12% of 20-turn conversations succeed (0.90^20 = 0.12)	12% conversation success	100% of conversations end with every promised outcome verified against the ledger, at 20, 40 and 60 turns, with failures injected	Single-run component evidence: 20-turn target: 24 observed, pass; 40-turn target: 47 observed, pass; 60-turn target: 67 observed, pass. One live fixed-template HTTP run at each length. Excludes ASR/TTS, Calls integration and Jude model inference. Earlier HTTP400 failures remain unresolved/intermittent; this is not a reliability or AEC9 claim.	simulation target met	20-turn run, 40-turn run, 60-turn run
`P02`	A customer service conversation runs to 20 turns	20 turns	Conversations measured at 20, 40 and 60 turns	Single-run component evidence: 20-turn target: 24 observed, pass; 40-turn target: 47 observed, pass; 60-turn target: 67 observed, pass. One live fixed-template HTTP run at each length. Excludes ASR/TTS, Calls integration and Jude model inference. Earlier HTTP400 failures remain unresolved/intermittent; this is not a reliability or AEC9 claim.	simulation target met	20-turn run, 40-turn run, 60-turn run
`P03`	Failure compounds from turn 6	turn 6	A corrupted turn 6 changes no state (state hash before = after) and the conversation still verifies	corrupted turn 6 in 0 conversations: state unchanged in all, 0 verified	not demonstrated	dlg-adv-corrupt-t6
`P04`	By turn 14 the model reasons from a broken thread	turn 14	Backend timeout from turn 14: zero facts stated while it is down, then recovery by retry or hand-off with a reference, 100%	timeout from turn 14 in 0 conversations: 0 facts stated while down, 0 recovered	not demonstrated	dlg-adv-timeout-t14
`P05`	Production means 200 simultaneous calls	200 simultaneous	At least 400 simultaneous conversations through the public address, every one verified	400 simultaneous: 400 of 400 verified	simulation target met	dlg-adv-concurrency
`P06`	Latency is 2 seconds per turn under load	2 s per turn	p95 turn latency under 500 ms at 200 and at 400 simultaneous conversations, measured by the caller through HTTPS	p95 108.6 ms at 200, 139.2 ms at 400	simulation target met	dlg-adv-concurrency
`P07`	Dead air reaches 40 seconds per conversation	40 s	Total wait per 20-turn conversation under 5 s at 200 simultaneous	longest total wait in a conversation at 200 simultaneous: 0.49 s	simulation target met	dlg-adv-concurrency
`P08`	Multilingual slip: an English reply to a Spanish caller	no figure	Zero replies in a language other than the caller's, including a mid-call switch and retrieval returning another language	0 language slips in 1289 conversations (6 languages, switches and cross-language retrieval)	simulation target met	dlg-adv-lang-switch, dlg-adv-retrieval-lang
`P09`	Hallucination when a backend API fails	no figure	Zero fabricated answers under injected timeouts and errors: the data is said to be not reachable, then retried or handed over	0 facts stated on failed-backend turns in 1289 conversations	simulation target met	dlg-adv-timeout-persistent
`P10`	Prompt layering causes contradictions and decay	no figure	Constraints are code, not prompt: a payment asked at turn 59 without verification is refused; policy at turn 60 equals policy at turn 1	turn-59 payment without verification refused: yes	simulation target met	dlg-adv-policy-depth
`P11`	Raven is fine-tuned on hundreds of thousands of production calls	SFT corpus	No trained generation in the reply path: every reply is a frozen template filled only from backend records	0 replies outside the frozen templates in 5232 turns	simulation target met	dlg-adv-long-60
`P12`	Three post-training stages (SFT, GRPO/DPO fusion, auto-reasoning)	3 stages	Every change to what Jude understands passes the full regression suite before it is live; harmful changes are rejected	regression gate: harmful change rejected, safe change accepted after the full suite	not demonstrated	dlg-adv-self-improve
`P13`	Auto-reasoning in fewer than 40 tokens	< 40 tokens	0 model tokens per turn; every decision is a named act with its sources in a signed receipt	0 model tokens per turn; 1289 of 1289 conversations with a valid signed receipt chain	simulation target met	dlg-adv-long-60
`P14`	Raven 3.5 reasons on about half of turns	about 50% of turns	100% of turns explained: act, facts and sources recorded and chained	100% of 5232 turns receipted with act and sources	simulation target met	dlg-adv-long-60
`P15`	Raven median latency under 300 ms	< 300 ms median	Median turn latency under 150 ms at 200 simultaneous conversations	p50 47.9 ms at 200 simultaneous	simulation target met	dlg-adv-concurrency
`P16`	Raven lists 24 languages	24 languages	30 languages served from frozen, hashed language packs	24 languages with complete frozen packs	not demonstrated	dlg-adv-lang-switch
`P17`	PolyAI recommends median under 500 ms and p95 under 1 s	p95 < 1 s	p95 under 500 ms at 200 and 400 simultaneous	p95 108.6 ms at 200, 139.2 ms at 400	simulation target met	dlg-adv-concurrency
`P18`	API price US$0.02 per connected minute	US$0.02 per minute	No per-minute fee; compute measured per turn	no per-minute fee; 2.54 CPU ms per turn on jude-1	simulation target met	dlg-adv-concurrency
`P19`	Token prices US$2.00 / 0.20 / 5.00 per million	per-token billing	0 model tokens in the reply path	0 model tokens per reply	simulation target met	dlg-adv-long-60
`P20`	API concurrency 2 calls, or 10 with a card	2 or 10	No concurrency cap; 400 simultaneous measured	no cap; 400 simultaneous measured	simulation target met	dlg-adv-concurrency
`P21`	ISO 27001 stated; SOC 2, PCI DSS, HIPAA, GDPR shown as logos	certifications	Every turn signed and hash-chained, verifiable by anyone at /api/dialog/receipts/<session>	every turn HMAC-signed and hash-chained; 1289 chains verified	simulation target met	dlg-adv-long-60
`P22`	Customer outcomes: PG&E, Carter's, SafeRide, UniCredit, Fogo de Chão, Audibel	case studies	First-customer use receipt: a completed journey run as X-X.TECH for the first customer, in the simulated regulated context	First-customer evidence and level 9 matrices are still required	not demonstrated	dlg-fin-billing
`P28`	Agent Studio trial for two months	60 days	Open now: chat and voice at https://dialog.judiverse.tech with no sign-up	open now, no sign-up	simulation target met	chat, voice

Harness runs

Run	Simultaneous	Conversations	Verified	Success %	Turns	p50 ms	p95 ms	p99 ms
spike-200-paced	200	200	200	100.0	770	47.9	108.6	147.1
spike-200	200	200	200	100.0	770	180.9	422.8	568.3
spike-400-paced	400	400	400	100.0	1556	49.7	139.2	195.2
spike-400	400	400	400	100.0	1560	320.3	981.5	1356.1
suite	89	89	89	100.0	576	216.9	1555.7	2437.4

Method

Fixed-template simulator with its own synthetic ledger. These measurements do not establish performance of Jude model-backed chat or Calls, speech recognition/synthesis, or superiority over PolyAI. Synthetic callers (app/caller.py) talk to Jude through https://dialog.judiverse.tech exactly as a client would, over one persistent HTTPS connection each, from jude-1. Each conversation is judged by its outcome, not its turns: every promised payment, booking, change, cancellation, ticket or answer is read back from the ledger and must match what Jude said; every reply must be made only of the frozen templates in the caller's language; every receipt chain must verify. Failures are injected at the transport (corrupted turn, backend timeout or error, retrieval in another language, a duplicate message, a barge-in). Latency is the caller-side time for one turn through Caddy and TLS; speech recognition and synthesis run in the caller's browser on the voice page and are not in these figures.

Generated 2026-10-23T13:17:55Z. Raw results: /api/dialog/benchmark.