International large-scale education assessments, known as ILSAs, are widely used to compare the performance of countries' educational systems; however, a media-driven focus on country rankings can be misleading and serves as a poor basis for informing educational policy, argues Henry I. Braun of Boston College's Lynch School of Education and co-author Judith D. Singer of Harvard's Graduate School of Education in the April 6 issue of Science.
They note that because of overlapping margins of error, there may not be any meaningful statistical differences among the ranked average performances of many countries; furthermore, countries’ performances and the associated rankings may not be determined by education-system features so much as they reflect broader cultural attitudes and behaviors that strongly impact ILSA scores.
In a companion event, the National Academy of Education held a symposium on April 6 in Washington, D.C., to announce the release of a NAE report titled "International Education Assessments: Cautions, Conundrums, and Common Sense," edited by Professors Singer and Braun, and the NAE's Naomi Chudowsky. The NAE advances high-quality research to improve education policy and practice.
"Our hope is that the report and the Science article will bring renewed attention to the policy uses of ILSAs," said Braun, the Boisi Professor of Education and Public Policy at BC's Lynch School, and the director of its Center for the Study of Testing, Evaluation, and Education Policy. "We all bear responsibility in seeing that ILSA data and analyses are used responsibly to inform education policy around the globe."
Braun, and Singer, the James Bryant Conant Professor of Education and senior vice provost for Faculty Development & Diversity at Harvard University, assert that despite some differences in data quality and the challenges of achieving cross-cultural equivalence, nuanced analyses of ILSA results, in combination with in-depth within-country analyses, can be harnessed to identify problematic aspects of educational systems and suggest possible solutions. One such case is examining the strength of the relationship between socioeconomic status and student performance. For example, the relationship between test scores and socioeconomic status is noticeably weaker in Canada than in the United States, although Canada’s performance is much stronger. Such findings can prompt countries -- such as the U.S. -- with less equitable systems to experiment with strategies their neighbors have used to improve educational equity.
The authors concluded the Science article by outlining five suggestions for improved use of ILSA data, including linking ILSA data to other data sources and using findings from ILSA analyses to stimulate randomized field trials that can properly test the effects of specific interventions.
—Phil Gloudemans | University Communications | April 2018