LLM-Based Multimodal Representations of Student Collaboration in Mathematics

Abstract

Assessing the quality of students’ collaboration at scale remains a key challenge in computer-supported collaborative learning, especially when integrating multimodal data sources. From a Quantitative Ethnography (QE) perspective, this challenge centers on how to systematically represent and measure the structure of collaborative meaning-making across heterogeneous data streams. This study examines how large language models (LLMs) interpret different combinations of behavioral logs and dialogue transcripts to evaluate collaboration quality in mathematics problem solving. Grounded in a QE-informed analytic framework, we developed a theoretically grounded rubric encompassing social, cognitive, and regulatory dimensions. We then compared six LLMs from the Gemini and GPT families on their ability to evaluate the quality of students’ collaboration using interaction log, dialogue, and human-coded label-based data sources from 22 college students (11 dyads) collaboratively solving 49 algebra problems in a digital learning platform. Results show that adding dialogue to log data improved agreement with expert ratings, while human-annotated dialogue labels offered no additional benefit. Dimension-level findings revealed stronger alignment on explicit communicative behaviors than on implicit indicators. Our findings highlight both the potential and limitations of LLMs for collaboration assessment from multimodal data sources.

Materials

• Full BibTeX (publisher)citation-needed.springer.com

Cite

@inproceedings{zhang2026llmbased,
  author = {Zhang, Shan and Li, Hongming and Lee, Seiyon and Liu, Xiner and Lee, Ji-Eun and Schroeder, Noah L. and Botelho, Anthony F.},
  title = {LLM-Based Multimodal Representations of Student Collaboration in Mathematics},
  booktitle = {Advances in Quantitative Ethnography (ICQE 2026)},
  year = {2026},
  volume = {3227},
  pages = {96--110},
  publisher = {Springer Nature Switzerland},
  doi = {10.1007/978-3-032-44026-6_7},
  url = {https://link.springer.com/chapter/10.1007/978-3-032-44026-6_7}
}