Loading...
Loading...
Contributors:
Dataset URI:
10.5281/zenodo.23865699CrossVal-100 is a benchmark of 100 research ideas (50 generated by an LLM pipeline, 50 drawn from published papers in comparable research areas), with all judge ratings and the evaluation harness used in "CrossVal-100: auditing integrity invariants in LLM-judged research-ideation comparisons" (submitted to Royal Society Open Science). This archive is an export of the frozen release tag crossval-100-v17 (commit 5677cc09ce22361c12ea4e300a23e4e87f69ec58). It contains the dataset (v3, with v2 retained for audit), the evaluation harness, all 300 panel ratings, the 1,200 judge-robustness ratings with per-call provenance sidecars, the serialization manifest, pre-registration and verdict documents, and the analysis and checking scripts. The archive also contains the surrounding IdeaGraph application code, which is not needed to reproduce the paper. Reproduce Tables 3 and 4 of the paper (Python 3 with numpy and scipy):python reproduce_tables.py --ratings data/crossval/eval_out/ratings.json --key data/crossval/eval_out/blinded_key.json Check that every cell of the released analysis report recomputes from the ratings:python check_released_outputs.py Licences: data under CC BY 4.0 (DATA_LICENSE.md); code under the MIT License (LICENSE).
Keyword(s):
benchmark integrity; evaluation methodology; LLM-as-a-judge; non-inferiority testing; reproducibility.
MeSH Term(s):
Grants and Funding:
No Grants or Funding Data Available
Associated Publication(s):
No Associated Publication Data Available
Supplementary Data:
No Supplementary Data Available
Related Dataset(s):
No Related Dataset Data Available
The National Library of Medicine (NLM) provides information about biomedical datasets. Inclusion on an NLM Web page does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: NLM Web Policies (nih.gov)