IEEE Big Data Cup 2026 - χ-Bench Challenge
Build an autonomous agent for prior authorization, utilization management, and care management in χ-World. This challenge evaluates whether AI agents can complete complex healthcare-administration workflows—not merely answer a question, but gather evidence, apply policy, use operational tools, communicate with simulated people, and drive a case to the correct outcome.
χ-Bench evaluates long-horizon agents inside χ-World, a high-fidelity simulator of U.S. healthcare operations. Agents work across healthcare applications exposed through MCP and REST tools while grounding their decisions in the 1,279-document Managed-Care Operations Handbook.
Tracks and Awards Agents are evaluated across three domains: Provider Prior Authorization, Payer Utilization Management, and Care Management. The public development set contains 75 tasks—25 per domain—and final standings use organizer-run evaluation on approximately 75 unreleased private tasks. Teams rank by private pass@1, then pass^3 (reliability), inference cost, and tool calls.
Competition Format The competition consists of a public development phase and a private evaluation phase: * Public Development Set: 75 tasks used for provisional leaderboard ranking based on pass@1. * Private Evaluation Set: Unreleased clinician-validated tasks where every complete, prize-eligible final entry is run for three independent attempts (pass^3). * Scoring: Each task receives a strict binary reward (1 or 0). A reward of 1 is given only when every required non-N/A verifier check passes. Final rankings prioritize higher private pass@1 and pass^3 scores, followed by lower mean agent inference cost and fewer mean agent tool calls.
Participation Requirements Participants must join the Kaggle competition, accept the Rules, and request access to the gated Managed-Care Operations Handbook before running χ-Bench. Any model, agent harness, prompting strategy, hardware, cloud provider, or local environment may be used, subject to the Rules and applicable licenses. Teams may contain up to five people.
Timeline The Kaggle competition opens on August 15th, 2026. The entry deadline is November 7th, 2026, with the final Kaggle submission deadline on November 15th, 2026. Final results will be announced on November 23rd, 2026, with finalist materials due December 7th, 2026. The event concludes at IEEE Big Data 2026 in Phoenix, Arizona, USA, from December 14–17, 2026.