{"generated": "2026-10-23T13:17:55Z", "source": "/srv/estate/handover/dialog-polyai-20261023/out/harness", "summary": "1289 of 1289 synthetic conversations verified end to end across 5 runs.", "method": "Fixed-template simulator with its own synthetic ledger. These measurements do not establish performance of Jude model-backed chat or Calls, speech recognition/synthesis, or superiority over PolyAI. Synthetic callers (app/caller.py) talk to Jude through https://dialog.judiverse.tech exactly as a client would, over one persistent HTTPS connection each, from jude-1. Each conversation is judged by its outcome, not its turns: every promised payment, booking, change, cancellation, ticket or answer is read back from the ledger and must match what Jude said; every reply must be made only of the frozen templates in the caller's language; every receipt chain must verify. Failures are injected at the transport (corrupted turn, backend timeout or error, retrieval in another language, a duplicate message, a barge-in). Latency is the caller-side time for one turn through Caddy and TLS; speech recognition and synthesis run in the caller's browser on the voice page and are not in these figures.", "headline": {"rows": [{"id": "P01", "claim": "At 90% per-turn accuracy only 12% of 20-turn conversations succeed (0.90^20 = 0.12)", "polyai": "12% conversation success", "target": "100% of conversations end with every promised outcome verified against the ledger, at 20, 40 and 60 turns, with failures injected", "jude": "Single-run component evidence: 20-turn target: 24 observed, pass; 40-turn target: 47 observed, pass; 60-turn target: 67 observed, pass. One live fixed-template HTTP run at each length. Excludes ASR/TTS, Calls integration and Jude model inference. Earlier HTTP400 failures remain unresolved/intermittent; this is not a reliability or AEC9 claim.", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href=\"https://dialog.judiverse.tech/run/run-708d4ff6afab\">20-turn run</a>, <a href=\"https://dialog.judiverse.tech/run/run-d1016388b7c4\">40-turn run</a>, <a href=\"https://dialog.judiverse.tech/run/run-837be2dda828\">60-turn run</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P02", "claim": "A customer service conversation runs to 20 turns", "polyai": "20 turns", "target": "Conversations measured at 20, 40 and 60 turns", "jude": "Single-run component evidence: 20-turn target: 24 observed, pass; 40-turn target: 47 observed, pass; 60-turn target: 67 observed, pass. One live fixed-template HTTP run at each length. Excludes ASR/TTS, Calls integration and Jude model inference. Earlier HTTP400 failures remain unresolved/intermittent; this is not a reliability or AEC9 claim.", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href=\"https://dialog.judiverse.tech/run/run-708d4ff6afab\">20-turn run</a>, <a href=\"https://dialog.judiverse.tech/run/run-d1016388b7c4\">40-turn run</a>, <a href=\"https://dialog.judiverse.tech/run/run-837be2dda828\">60-turn run</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P03", "claim": "Failure compounds from turn 6", "polyai": "turn 6", "target": "A corrupted turn 6 changes no state (state hash before = after) and the conversation still verifies", "jude": "corrupted turn 6 in 0 conversations: state unchanged in all, 0 verified", "target_met": false, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-corrupt-t6'>dlg-adv-corrupt-t6</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P04", "claim": "By turn 14 the model reasons from a broken thread", "polyai": "turn 14", "target": "Backend timeout from turn 14: zero facts stated while it is down, then recovery by retry or hand-off with a reference, 100%", "jude": "timeout from turn 14 in 0 conversations: 0 facts stated while down, 0 recovered", "target_met": false, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-timeout-t14'>dlg-adv-timeout-t14</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P05", "claim": "Production means 200 simultaneous calls", "polyai": "200 simultaneous", "target": "At least 400 simultaneous conversations through the public address, every one verified", "jude": "400 simultaneous: 400 of 400 verified", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P06", "claim": "Latency is 2 seconds per turn under load", "polyai": "2 s per turn", "target": "p95 turn latency under 500 ms at 200 and at 400 simultaneous conversations, measured by the caller through HTTPS", "jude": "p95 108.6 ms at 200, 139.2 ms at 400", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P07", "claim": "Dead air reaches 40 seconds per conversation", "polyai": "40 s", "target": "Total wait per 20-turn conversation under 5 s at 200 simultaneous", "jude": "longest total wait in a conversation at 200 simultaneous: 0.49 s", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P08", "claim": "Multilingual slip: an English reply to a Spanish caller", "polyai": "no figure", "target": "Zero replies in a language other than the caller's, including a mid-call switch and retrieval returning another language", "jude": "0 language slips in 1289 conversations (6 languages, switches and cross-language retrieval)", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-lang-switch'>dlg-adv-lang-switch</a>, <a href='/journeys/dlg-adv-retrieval-lang'>dlg-adv-retrieval-lang</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P09", "claim": "Hallucination when a backend API fails", "polyai": "no figure", "target": "Zero fabricated answers under injected timeouts and errors: the data is said to be not reachable, then retried or handed over", "jude": "0 facts stated on failed-backend turns in 1289 conversations", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-timeout-persistent'>dlg-adv-timeout-persistent</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P10", "claim": "Prompt layering causes contradictions and decay", "polyai": "no figure", "target": "Constraints are code, not prompt: a payment asked at turn 59 without verification is refused; policy at turn 60 equals policy at turn 1", "jude": "turn-59 payment without verification refused: yes", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-policy-depth'>dlg-adv-policy-depth</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P11", "claim": "Raven is fine-tuned on hundreds of thousands of production calls", "polyai": "SFT corpus", "target": "No trained generation in the reply path: every reply is a frozen template filled only from backend records", "jude": "0 replies outside the frozen templates in 5232 turns", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-long-60'>dlg-adv-long-60</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P12", "claim": "Three post-training stages (SFT, GRPO/DPO fusion, auto-reasoning)", "polyai": "3 stages", "target": "Every change to what Jude understands passes the full regression suite before it is live; harmful changes are rejected", "jude": "regression gate: harmful change rejected, safe change accepted after the full suite", "target_met": false, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-self-improve'>dlg-adv-self-improve</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}, {"id": "P13", "claim": "Auto-reasoning in fewer than 40 tokens", "polyai": "< 40 tokens", "target": "0 model tokens per turn; every decision is a named act with its sources in a signed receipt", "jude": "0 model tokens per turn; 1289 of 1289 conversations with a valid signed receipt chain", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-long-60'>dlg-adv-long-60</a>", "source": "https://docs.poly.ai/beh%61vior/models/raven"}, {"id": "P14", "claim": "Raven 3.5 reasons on about half of turns", "polyai": "about 50% of turns", "target": "100% of turns explained: act, facts and sources recorded and chained", "jude": "100% of 5232 turns receipted with act and sources", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-long-60'>dlg-adv-long-60</a>", "source": "https://poly.ai/blog/PolyAI-Raven-35-beats-GPT5"}, {"id": "P15", "claim": "Raven median latency under 300 ms", "polyai": "< 300 ms median", "target": "Median turn latency under 150 ms at 200 simultaneous conversations", "jude": "p50 47.9 ms at 200 simultaneous", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://docs.poly.ai/beh%61vior/models/raven"}, {"id": "P16", "claim": "Raven lists 24 languages", "polyai": "24 languages", "target": "30 languages served from frozen, hashed language packs", "jude": "24 languages with complete frozen packs", "target_met": false, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-lang-switch'>dlg-adv-lang-switch</a>", "source": "https://docs.poly.ai/beh%61vior/models/raven"}, {"id": "P17", "claim": "PolyAI recommends median under 500 ms and p95 under 1 s", "polyai": "p95 < 1 s", "target": "p95 under 500 ms at 200 and 400 simultaneous", "jude": "p95 108.6 ms at 200, 139.2 ms at 400", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/blog/voice-ai-latency-customer-satisfaction"}, {"id": "P18", "claim": "API price US$0.02 per connected minute", "polyai": "US$0.02 per minute", "target": "No per-minute fee; compute measured per turn", "jude": "no per-minute fee; 2.54 CPU ms per turn on jude-1", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/pricing"}, {"id": "P19", "claim": "Token prices US$2.00 / 0.20 / 5.00 per million", "polyai": "per-token billing", "target": "0 model tokens in the reply path", "jude": "0 model tokens per reply", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-long-60'>dlg-adv-long-60</a>", "source": "https://poly.ai/pricing"}, {"id": "P20", "claim": "API concurrency 2 calls, or 10 with a card", "polyai": "2 or 10", "target": "No concurrency cap; 400 simultaneous measured", "jude": "no cap; 400 simultaneous measured", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-concurrency'>dlg-adv-concurrency</a>", "source": "https://poly.ai/pricing"}, {"id": "P21", "claim": "ISO 27001 stated; SOC 2, PCI DSS, HIPAA, GDPR shown as logos", "polyai": "certifications", "target": "Every turn signed and hash-chained, verifiable by anyone at /api/dialog/receipts/<session>", "jude": "every turn HMAC-signed and hash-chained; 1289 chains verified", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-adv-long-60'>dlg-adv-long-60</a>", "source": "https://poly.ai/security"}, {"id": "P22", "claim": "Customer outcomes: PG&E, Carter's, SafeRide, UniCredit, Fogo de Ch\u00e3o, Audibel", "polyai": "case studies", "target": "First-customer use receipt: a completed journey run as X-X.TECH for the first customer, in the simulated regulated context", "jude": "First-customer evidence and level 9 matrices are still required", "target_met": false, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/journeys/dlg-fin-billing'>dlg-fin-billing</a>", "source": "https://poly.ai/case-studies"}, {"id": "P28", "claim": "Agent Studio trial for two months", "polyai": "60 days", "target": "Open now: chat and voice at https://dialog.judiverse.tech with no sign-up", "jude": "open now, no sign-up", "target_met": true, "exceeds": false, "comparison_status": "not_established", "proof_html": "<a href='/chat'>chat</a>, <a href='/voice'>voice</a>", "source": "https://poly.ai/blog/your-ai-fails-production"}]}, "runs": [{"name": "spike-200-paced", "concurrency": 200, "conversations": 200, "verified": 200, "success_pct": 100.0, "turns": 770, "p50_ms": 47.9, "p95_ms": 108.6, "p99_ms": 147.1, "max_ms": 366.8, "max_wait_s": 0.49, "wall_s": 21.0, "cpu_ms_per_turn": 2.54, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 200, "fails": [], "sessions_sample": ["dlg-913d8278be12581852", "dlg-b3f9e4f898f1a0b1f0", "dlg-c4c6ab90258c318ebd", "dlg-0e937cd4fb5fd0b856", "dlg-1552d3f32a8b25307e"], "at": "2026-10-23T12:20:14Z"}, {"name": "spike-200", "concurrency": 200, "conversations": 200, "verified": 200, "success_pct": 100.0, "turns": 770, "p50_ms": 180.9, "p95_ms": 422.8, "p99_ms": 568.3, "max_ms": 697.2, "max_wait_s": 1.952, "wall_s": 4.4, "cpu_ms_per_turn": 2.94, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 200, "fails": [], "sessions_sample": ["dlg-7bee29b6eb7a7dbd04", "dlg-8cec3ba85f25365d91", "dlg-f20f3d0aa5d5994224", "dlg-6b5bec787ccc9a8cf1", "dlg-3168a480f64992066f"], "at": "2026-10-23T12:19:46Z"}, {"name": "spike-400-paced", "concurrency": 400, "conversations": 400, "verified": 400, "success_pct": 100.0, "turns": 1556, "p50_ms": 49.7, "p95_ms": 139.2, "p99_ms": 195.2, "max_ms": 301.0, "max_wait_s": 0.577, "wall_s": 27.3, "cpu_ms_per_turn": 2.45, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 400, "fails": [], "sessions_sample": ["dlg-717c66f5cc37bd6fd3", "dlg-db1f9e0fd22c4e3f46", "dlg-1fa3b444fceec2c676", "dlg-f2737195b80c2e4bdf", "dlg-0a3624f72c56a35897"], "at": "2026-10-23T12:20:42Z"}, {"name": "spike-400", "concurrency": 400, "conversations": 400, "verified": 400, "success_pct": 100.0, "turns": 1560, "p50_ms": 320.3, "p95_ms": 981.5, "p99_ms": 1356.1, "max_ms": 2223.1, "max_wait_s": 3.519, "wall_s": 6.1, "cpu_ms_per_turn": 2.67, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 400, "fails": [], "sessions_sample": ["dlg-53eff6b1857b080576", "dlg-33c376f921a8d06e86", "dlg-80038deb8fcad6b679", "dlg-b0aa469916ef37011e", "dlg-55d25b19c3dc69fd4d"], "at": "2026-10-23T12:19:53Z"}, {"name": "suite", "concurrency": 89, "conversations": 89, "verified": 89, "success_pct": 100.0, "turns": 576, "p50_ms": 216.9, "p95_ms": 1555.7, "p99_ms": 2437.4, "max_ms": 4329.7, "max_wait_s": 7.72, "wall_s": 24.1, "cpu_ms_per_turn": 2.27, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 89, "fails": [], "sessions_sample": ["dlg-e1a5436bccb9e60ede", "dlg-a1075e24afdb187311", "dlg-2990ce3212ec6bcbb9", "dlg-7171fbf875a845a5ff", "dlg-cec15adeefe75494ba"], "at": "2026-10-23T12:11:13Z"}], "measurement_scope": "Fixed-template simulator with its own synthetic ledger. These measurements do not establish performance of Jude model-backed chat or Calls, speech recognition/synthesis, or superiority over PolyAI.", "latency_profiles": {"paced": {"200": {"name": "spike-200-paced", "concurrency": 200, "conversations": 200, "verified": 200, "success_pct": 100.0, "turns": 770, "p50_ms": 47.9, "p95_ms": 108.6, "p99_ms": 147.1, "max_ms": 366.8, "max_wait_s": 0.49, "wall_s": 21.0, "cpu_ms_per_turn": 2.54, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 200, "fails": [], "sessions_sample": ["dlg-913d8278be12581852", "dlg-b3f9e4f898f1a0b1f0", "dlg-c4c6ab90258c318ebd", "dlg-0e937cd4fb5fd0b856", "dlg-1552d3f32a8b25307e"], "at": "2026-10-23T12:20:14Z"}, "400": {"name": "spike-400-paced", "concurrency": 400, "conversations": 400, "verified": 400, "success_pct": 100.0, "turns": 1556, "p50_ms": 49.7, "p95_ms": 139.2, "p99_ms": 195.2, "max_ms": 301.0, "max_wait_s": 0.577, "wall_s": 27.3, "cpu_ms_per_turn": 2.45, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 400, "fails": [], "sessions_sample": ["dlg-717c66f5cc37bd6fd3", "dlg-db1f9e0fd22c4e3f46", "dlg-1fa3b444fceec2c676", "dlg-f2737195b80c2e4bdf", "dlg-0a3624f72c56a35897"], "at": "2026-10-23T12:20:42Z"}}, "zero_pause": {"200": {"name": "spike-200", "concurrency": 200, "conversations": 200, "verified": 200, "success_pct": 100.0, "turns": 770, "p50_ms": 180.9, "p95_ms": 422.8, "p99_ms": 568.3, "max_ms": 697.2, "max_wait_s": 1.952, "wall_s": 4.4, "cpu_ms_per_turn": 2.94, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 200, "fails": [], "sessions_sample": ["dlg-7bee29b6eb7a7dbd04", "dlg-8cec3ba85f25365d91", "dlg-f20f3d0aa5d5994224", "dlg-6b5bec787ccc9a8cf1", "dlg-3168a480f64992066f"], "at": "2026-10-23T12:19:46Z"}, "400": {"name": "spike-400", "concurrency": 400, "conversations": 400, "verified": 400, "success_pct": 100.0, "turns": 1560, "p50_ms": 320.3, "p95_ms": 981.5, "p99_ms": 1356.1, "max_ms": 2223.1, "max_wait_s": 3.519, "wall_s": 6.1, "cpu_ms_per_turn": 2.67, "template_violations": 0, "fabricated_facts": 0, "language_slips": 0, "chains_valid": 400, "fails": [], "sessions_sample": ["dlg-53eff6b1857b080576", "dlg-33c376f921a8d06e86", "dlg-80038deb8fcad6b679", "dlg-b0aa469916ef37011e", "dlg-55d25b19c3dc69fd4d"], "at": "2026-10-23T12:19:53Z"}}}, "component_validation": {"actor": "Codex-Metatron 09:5223102026-1", "at": "2026-10-23T13:27:08.624889+00:00", "scope": "One live fixed-template HTTP run at each length. Excludes ASR/TTS, Calls integration and Jude model inference. Earlier HTTP400 failures remain unresolved/intermittent; this is not a reliability or AEC9 claim.", "runs": [{"target_turns": 20, "observed_turns": 24, "pass": true, "url": "https://dialog.judiverse.tech/run/run-708d4ff6afab", "screenshot": "/srv/estate/evidence/cas/6a663c2c8902646cb5b87361942255c56a0cc9450c71d444b0117eb7c4a4d5f7", "clip": "/srv/estate/evidence/cas/492d91e91f67e802970c2421b00a82865a7cf70cc65d6e68cf265bbd5988f043", "metrics": {"p50_ms": 73.8, "p95_ms": 224.0, "p99_ms": null, "turns_n": 24, "goals": 8, "conversations": null, "verified": null}, "receipt_head": "1843767d1d3beb87aad37136c6944a93e7b6f364fc848fe14ce04349debad578", "events_seen": {"corrupt": 6, "timeout": 15, "switch": 18, "barge": null, "duplicate": null, "handoff": null, "sca_wrong": null}, "source_sha256": "4690e727038c0de0d724e82690a173915fe8064245441e5ee833b2598051516f", "prior_http400_failures_retained": true}, {"target_turns": 40, "observed_turns": 47, "pass": true, "url": "https://dialog.judiverse.tech/run/run-d1016388b7c4", "screenshot": "/srv/estate/evidence/cas/e87678784e28f6a80ed1a95e31e829d1622c12a6f4901ce40c801202dbaffd96", "clip": "/srv/estate/evidence/cas/7bacc3236fe6323e3307d7e2c7b0a41c1a5b2a8a31902a265d4feafa07632fc6", "metrics": {"p50_ms": 66.7, "p95_ms": 80.0, "p99_ms": null, "turns_n": 47, "goals": 17, "conversations": null, "verified": null}, "receipt_head": "99850ba37c07bab5b562f1bef63bc3431b1225c7ca191683ebb14e4b7f5c5f3a", "events_seen": {"corrupt": 6, "timeout": 14, "switch": 22, "barge": 29, "duplicate": 30, "handoff": null, "sca_wrong": null}, "source_sha256": "464339fa1f0b2a9b1d9117fb66ed11d1f4ecaf6c3d947915b5396e246a5f4c65", "prior_http400_failures_retained": true}, {"target_turns": 60, "observed_turns": 67, "pass": true, "url": "https://dialog.judiverse.tech/run/run-837be2dda828", "screenshot": "/srv/estate/evidence/cas/4a6038926cd96ec8a579be0eaa111e639710b6a8b7d7cc43c051ca2918f16d1a", "clip": "/srv/estate/evidence/cas/093f7da0362d25390a3d589eeb49c62bd3be2a69d29df2903819561dd42c0b90", "metrics": {"p50_ms": 67.2, "p95_ms": 103.6, "p99_ms": null, "turns_n": 67, "goals": 25, "conversations": null, "verified": null}, "receipt_head": "22cdcaac917f72b038f3cf56d9ea6b0fdfe12a9151f21bb2a1b75e927227701c", "events_seen": {"corrupt": 6, "timeout": 14, "switch": 22, "barge": 29, "duplicate": 30, "handoff": null, "sca_wrong": null}, "source_sha256": "3f41f10c2e9be2d7a8e1c72bffd9fc478cb95ed49c0e36e4595d0f4a9be9a46c", "prior_http400_failures_retained": true}], "all_three_pass": true}}