Loading...
Loading...
Dataset URI:
10.3389/frai.2024.1474019.s001IntroductionAssessments of interactional competence have traditionally been limited in large-scale language assessments. The listening portion suffers from construct underrepresentation, whereas the speaking portion suffers from limited task formats such as in-person interviews or role plays. Human-delivered tasks are challenging to administer at large scales, while automated assessments are typically very narrow in their assessment of the construct because they have carried over the limitations of traditional paper-based tasks to digital formats. However, computer-based assessments do allow for more interactive, automatically administered tasks, but come with increased complexity in task creation. Large language models present new opportunities for enhanced automated item generation (AIG) processes that can create complex content types and tasks at scale that support richer assessments.MethodsThis paper describes the use of such methods to generate content at scale for an interactive listening measure of interactional competence for the Duolingo English Test (DET), a large-scale, high-stakes test of English proficiency. The Interactive Listening task assesses test takers’ ability to participate in a full conversation, resulting in a more authentic assessment of interactive listening ability than prior automated assessments by positing comprehension and interaction as purposes of listening.Results and discussionThe results of a pilot of 713 tasks with hundreds of responses per task, along with the results of human review, demonstrate the feasibility of a human-in-the-loop, generative AI-driven approach for automatic creation of complex educational assessments at scale.
Keyword(s):
automatic item generation; Duolingo English test; generative AI; interactional competence; interactive listening.
MeSH Term(s):
Grants and Funding:
No Grants or Funding Data Available
Associated Publication(s):
Supplementary Data:
No Supplementary Data Available
Related Dataset(s):
No Related Dataset Data Available
The National Library of Medicine (NLM) provides information about biomedical datasets. Inclusion on an NLM Web page does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: NLM Web Policies (nih.gov)