Sit the paper. A Russian teenager walks into an exam room with a paper in front of them: a fixed number of tasks, one subject, no notes. They have never seen these particular items. Nobody has published the answers. That is this competition, with a model in the chair.
This is the same open bank of Russian national exam tasks (ЕГЭ and ОГЭ) — and the same fact that makes it worth measuring on: ФИПИ, the state institute that writes the exams, has never published the answers. The key here was recovered from the board's own checker and appears in no dataset anywhere.
The first competition hands you a labelled training set, so a TF-IDF baseline is a reasonable place to start. This one does not. There is no train.csv. No labelled examples of any kind, in any subject. That is the whole point. A model that scores here does so because it already knows the material — not because it fitted the shape of this particular test set. Zero-shot is not a constraint we added for flavour; it is what an exam is.
Tasks are grouped into papers: a paper column tells you which sitting a task belongs to, and position where it falls inside it. Every paper is one subject at one exam level, so a paper is a coherent thing to score, to analyse and to be embarrassed by.
Each task's answer space is small and fully enumerable — 3 to 6 options. Every option was submitted to the official ФИПИ checker until it accepted one. Because the space is exhausted rather than sampled, an answer was found for every task, not only for the ones some solver got right. This matters more than it sounds: a key built by verifying a model's guesses would quietly consist of exactly the tasks that model can already solve, and every benchmark built on it would flatter its own author's model. This one cannot.
Tasks needing a figure or an audio file were dropped — they are unanswerable from text. Duplicates were removed on normalised text. The task ids here are our own, and no task in this competition appears in the classification round.
The ФИПИ checker is still online. Do not go ask it. This is a community competition with no prizes — the leaderboard is the only thing it produces, and that is the thing cheating destroys.
Metric: Categorization Accuracy — the share of tasks answered correctly across the whole test set. In exam terms this is your первичный балл expressed as a fraction: every task is worth one mark, exactly as in the real paper. The entire test set is scored on the public leaderboard. There is no private split, no held-back rows and no final shake-up: what you see is your score.
Submission format: One row per task in test.csv, plus a header:
Id,Category
X00001,3
X00002,1
X00003,4
Id is the task id from test.csv. Category is the option number you pick — an integer from 1 to that row's n_options. Options are numbered as the option_1 … option_6 columns; a task with four options leaves option_5 and option_6 empty, and 5 or 6 is not valid for it.