Loading...
Loading...
Contributors:
Dataset URI:
10.5281/zenodo.23248969This record provides de-identified derived data and reconstructed statistical analysis code accompanying the manuscript "Diagnosing multimodal model failures on K12 mathematics multiple choice questions with images". The main dataset contains 1,400 model-item outcome records from 200 image-based mathematics multiple-choice questions evaluated using seven publicly deployed AI assistant services across Zhongkao, AMC, SAT, and HK-DSE. It records extracted final-answer correctness and four diagnostic failure-category indicators: Visual Perception Failure (VPF), Spatial Reasoning Deficiency (SRD), Subject Knowledge Deficiency (SKD), and Logical/Symbolic Failure (LSF). The archive also contains 140 supplementary selective structured-prompt records, retained annotation-agreement contingency counts, response-status summaries, analysis scripts, statistical results, and automated tests. The Python code reproduces Cochran's Q, 21 exact pairwise McNemar comparisons with Holm adjustment, four item-clustered logistic GEE analyses with Holm correction across 24 non-intercept service contrasts, and descriptive error-label analyses. The scripts were reconstructed from retained research records; they do not re-run the model evaluations. Original examination images, complete question texts, and full model-response transcripts are not included. The selective structured-prompt results are descriptive and should not be interpreted as a fully retested causal effect.
Keyword(s):
MeSH Term(s):
Grants and Funding:
No Grants or Funding Data Available
Associated Publication(s):
No Associated Publication Data Available
Supplementary Data:
No Supplementary Data Available
Related Dataset(s):
No Related Dataset Data Available
The National Library of Medicine (NLM) provides information about biomedical datasets. Inclusion on an NLM Web page does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: NLM Web Policies (nih.gov)