Current position: Home >> ICI NEWS
ICI International Classroom Analysis Webinar║Report by Dr. Jiang Yang from US ETS: Application of process data in large-scale educational evaluation.


On August 26, Dr. Jiang Yang from American ETS guested on the third webinar of the International Classroom Analysis Series and made a keynote report, namely Application of Process Data in Large-scale Educational Evaluation.

The forum was chaired by Associate Professor Shi Yuchen, deputy director of the International Classroom Analysis Laboratory. Professor Cui Yunluo, Professor Yang Xiangdong, and Associate Professor Yang Xiaozhe from ICI joined discussion and communication, which attracted nearly 500 audiences.

Report content

Dr. Jiang Yang works as a researcher and data scientist at the American Educational Testing Service. This report mainly introduces the application of big data in the field of education, including how to apply methods such as educational data mining and learning analysis to analyzing process data, so as to explore the process of student learning and test taking. The report further took the National Evaluation of Educational Progress (NAEP) as an example, showing the use of big data to analyze students' cognitive and meta-cognitive processes, problem-solving methods and strategies, and predict students' academic achievement. The report also explores how these research findings will influence the design of online learning platforms and evaluation systems.

Dr. Jiang took the National Evaluation of Educational Progress (NAEP) as an example to introduce the application of process education data in large-scale education evaluation. As the only long-term and nationally representative large-scale education evaluation system in the United States, NAEP is mainly used to evaluate the knowledge and ability of American primary and secondary school students in mathematics, reading, science and other subject areas.

After an introduction to the National Assessment of Educational Progress (NAEP), Dr. Jiang introduced three topics she chaired.

The first topic studies how to understand, through process data, students' cognitive processes and problem-solving strategies when they answer questions . The research used the mathematics assessment data of NAEP in 2017, which was also the first online NAEP test of mathematics, and obtained a large number of samples and data. The test process includes: firstly complete the tutorial about how to use the test system; secondly, complete two sets of questions A and B, each with a time limit of 30 minutes; finally, students complete a set of questionnaires about personal information and learning experience.

NAEP's digital assessment platform displays a wide variety of tools, including sticky notes and calculators, for students' convenience.

Dr. Jiang introduced the basic methods and principles of process data analysis by taking a drag-and-drop question on the eighth grader’s number operation ability as an example. In this question, each number can be divided into starting sources and ending targets. Students need to drag the starting point to the ending point one by one, and different sequences represent different problem-solving strategies. In the exported specific data, you can clearly see each step performed and the time taken by the students. By using the procedural data to "restore" the whole process of the students' problem-solving, you can finally know the students' answering strategies and process.

Afterwards, Dr. Jiang introduced her research questions which include exploring students' cognitive processes and problem-solving strategies when solving problems, and whether these cognitive processes and strategies are related to students' scores. The study found that large-scale process data can help us understand what and how many are students' problem-solving strategies and further analyze the relationship between students with different scores and the problem-solving strategies adopted.

Dr. Jiang pointed out that the sequence length of answering behaviors reflects whether students modify their answers or not. The study found that students who answered correctly were generally less likely to revise their answers, while those who answered incorrectly hesitated more, with behavior reflected in repeated revisions. And for students who revised their answers, the study found that they tended to change answers from wrong to correct. The response time to answer the question reveals whether students are guessing and the process of thinking. Students with higher scores spend more time planning and checking answers on the question, but less time for performing actions of drag-and-drop. These process information is not available in traditional paper-and-pencil tests.

Dr. Jiang went on to introduce the second topic, which attempted to answer how students use calculators and the correlation between calculator usage and students' performance on the subject.

In traditional paper-and-pencil tests, students are also allowed to use calculators, but researchers have no way of knowing whether and how students use calculators. The process data recorded by online tools can solve this problem. Research has shown that calculators can better help students get correct answers, and students with correct answers are more likely to use calculators to solve problems and use calculators more efficiently.

Dr. Jiang also introduced what sequential pattern mining is and the discoveries it can bring through specific cases. By analyzing the sequence of student keystrokes on the calculator and combining it with the student's performance, more rich, deep, and interesting discoveries can be made.

The third topic is an analysis expanding from a single item to the entire assessment data, and it attempts to explore how students use calculators in large-scale assessments, whether the intensity and quality of use vary by question types, and whether calculator usage can predict students' mathematical ability.

The data uses two sets of questions from the 2019 NAEP eighth-grade data test. Dr. Jiang explained how to classify question types through specific cases. Questions are divided into three types based on whether a calculator is required.

Then, the research team and subject experts have developed a reference sequence standard. By comparing this sequence standard with the actual operation sequence of the students, it can infer the students' usage strategies and efficiency of using calculators. The study further proposes four comprehensive evaluation indicators, including the mean of similarity, standard deviation of similarity, mean of efficiency and standard deviation of efficiency.

The study found that the results of two sets of questions in the 2019 NAEP 8th-grade math test were consistent, and that the use of calculators by eighth-grade students was consistent with the way experts estimated. Calculator usage could predict students' math competencies, which means student calculator usage varies by level of academic achievement. In addition, by building a regression model, the study further predicted students' academic performance.

Dr. Jiang also mentioned that, in addition to the analysis of mathematics, other disciplines can also understand the learning process and learning strategies of students in reading and writing through process data. Therefore, process data should not be regarded as an incidental product. Instead, it should be valued and systematically planned and analyzed in the design process to expose the "black box" of students' answering behavior and cognition. It helps teachers and students to provide learning feedback and continuously improve teaching and learning. Interestingly, process data can even be used to measure concepts that have traditionally been difficult to observe, such as meta-cognition, ability to use tools, and ability to collaborate.

In the interaction session, Professor Yang Xiangdong said that Dr. Jiang's report was thought-provoking, and the two sides further discussed how to use and analyze process data. Associate Professor Yang Xiaozhe had an in-depth dialogue with Dr. Jiang on the cooperation and improvement strategies of the research team in the large-scale assessment. Experts and scholars such as Professor Cui Yunluo and Associate Professor Shi Yuchen also discussed with Dr. Jiang, deeply inspiring those participated.