TIMSS consistently works to adhere to best practices in large-scale assessment and measurement methodology. Similar to technical reports of previous TIMSS assessment cycles, TIMSS 2023 Technical Report (Methods and Procedures) includes chapters documenting the methodology and processes used in all aspects of the study carried out over a four-year cycle, including instrument development, sampling, data collection, scaling, data analysis, and reporting, and the rigorous quality assurance steps involved in each aspect.

Chapters in this report are divided into three overarching sections, corresponding to the three major stages of the study over its four-year cycle:

I. Assessment Development: documents the development of mathematics and science assessments and context questionnaires, and the sampling design methodology

II. Data Collection Procedures: describes the survey operations procedures for preparing national instruments, administering the achievement tests, collecting questionnaire data, scoring achievement items, and preparing the international databases

III. Analysis and Reporting: documents the measurement methodology and implementation for evaluating the quality of the data, creating achievement and context measurement scales, conducting data analysis, and reporting the results of TIMSS 2023

The chapters in this report are structured to describe a) the methods that remain generally consistent across TIMSS assessments and b) the processes that are new or specific to the TIMSS 2023 assessment cycle. Because TIMSS is designed to measure trends in mathematics and science achievement, the general approach to conducting the study cannot change dramatically from cycle to cycle. Therefore, much of the chapter descriptions have remained the same or similar to those written for previous TIMSS technical reports. However, TIMSS continuously evolves with each cycle by capitalizing on advances in technology and measurement methodology or quality enhancements to improve the assessments. Each cycle has some unique characteristics, and improvements are made to ensure best practices are followed at every step. These characteristics are highlighted throughout the report.

Unique Characteristics of TIMSS 2023

Building on the rigorous methods and procedures of previous TIMSS assessments, TIMSS 2023 was remarkable for several reasons. The following innovations are highlighted across the chapters of this report:

● TIMSS 2023 is the first TIMSS cycle in which the fourth- and eighth-grade assessments were developed to be fully digital, completing the transition to a computer-based platform that began in TIMSS 2019. Paper-based assessments consisting of mathematics and science items developed for previous TIMSS cycles were offered for participating countries not yet prepared to conduct digital assessments. Three countries at each grade opted to participate in a paper-based national comparison study in addition to the full digital administration.

● TIMSS 2023 implemented a new group-adaptive assessment design to better measure the range of abilities found in each of the TIMSS participating countries.

● Further capitalizing on the operational efficiencies offered by digital assessment, TIMSS 2023 continued the development of automated scoring procedures that were first introduced in TIMSS 2019. A concerted effort was made to increase the degree of automated scoring to improve scoring accuracy and comparability across countries, including research into using machine learning to evaluate graphical and short written responses.

Acknowledgments

TIMSS 2023 was a collaborative effort involving numerous organizations and individuals around the world, including the TIMSS & PIRLS International Study Center, IEA Amsterdam, IEA Hamburg, RTI International, RM Results, and the National Research Coordinators and their teams in the participating countries. Contributing to this technical report, staff from the TIMSS & PIRLS International Study Center wrote chapters on instrument development, data collection procedures, scoring, data analysis, scaling, and reporting. Staff from IEA Amsterdam contributed to chapters on national instrument preparation and the international quality assurance program. IEA Hamburg data project managers wrote a chapter about creating the TIMSS 2023 International Database. RTI International and the IEA Sampling unit authored chapters on sample design and sampling implementation. Many report chapters contain descriptions that were carried forward from previous TIMSS cycles and are a credit to authors of earlier technical reports, particularly as it relates to instrument development, sample design, and variance estimation. Producing this report included a peer-review process of each chapter, involving the technical report editors, contributing authors, and staff at the TIMSS & PIRLS International Study Center, IEA Amsterdam, and IEA Hamburg. Additionally, each chapter underwent review by one or more members of IEA’s Technical Executive Group (TEG).