The Poisoned Chalice of LLM Evaluation

Language models for code are evaluated on benchmarks that may already appear in their training data. When a model has memorised a test file, its score on that file measures recall, not capability. For models that publish their training corpus, this can be addressed by deduplicating the benchmark against the corpus. For models that do not, there is currently no reliable way to tell.

This competition asks you to build that capability. Given a code file and access to a model, decide whether the model saw that file during training. The task is file-level membership inference. This differs from most of the membership inference literature in two ways: the unit of inference is a whole source file rather than a short text fragment, and the domain is code rather than natural language. Both are underexplored, and attacks that work well on natural-language fragments do not obviously transfer.

Competition Format

The competition consists of two stages:

• Stage 1: Participants develop their attack against StarCoder2. A labelled set of member and non-member files is released for development, and predictions are scored on a held-out split from the same distribution. The leaderboard updates in real time. Stage 1 is for building and tuning; it does not determine the final ranking.
• Stage 2: Submitted code is executed unmodified against a different set of files and a target model that is not disclosed during Stage 1. The final ranking comes entirely from Stage 2. Teams wishing to participate in Stage 2 must submit a short report on their approach.

This design ensures that an attack tuned to the quirks of one model is useless for auditing models whose training data is undisclosed. Stage 2 measures whether your approach captures a general signal of membership. Scoring rewards both how well you separate members from non-members overall and whether you find members that previous attacks missed. Finding new vulnerable examples is worth more than re-finding the easy ones.

Evaluation

Submissions are scored on two components:

1. AUC-ROC: The standard Area Under the Receiver Operating Characteristic Curve.
2. Novelty Score: To encourage methodological diversity, this component rewards attacks that identify members not detected by reference attacks from previous editions of the competition. Members that were rarely identified by previous attacks contribute more to the novelty score, while members consistently identified contribute less. The final competition score is the arithmetic mean of the novelty score and the AUC-ROC score.

Submission Requirements

Each submission must include a report in a short paper format outlining the proposed approach, implementation rationale, and results on the open test set. Additionally, teams must submit a replication package (a Kaggle notebook or a repository with instructions) that produces the team's Stage 1 score.

Participants have the option to include their reports in the ICSE 2027 Competition Track proceedings. Submissions included in the proceedings undergo a lightweight peer review process with three reviewers. Reports must conform to ICSE formatting guidelines. Based on the assessment, participants may be offered an opportunity to revise their report or replication package. Accepted submissions must make their replication package public after acceptance. Competitors are welcome to participate without formally registering for the broader conference.