International Trends

NUMBER 56, SUMMER 2009

Measuring Learning in Higher Education in a Globalization Era

Jeannie Kim, Josipa Roksa, and Richard Arum
Jeannie Kim is Program Coordinator of the Social Science Research Council. E-mail: jkim@ssrc.org. Josipa Roksa is assistant professor of sociology, University of Virginia. E-mail: jroksa@virginia.edu. Richard Arum is professor of sociology and education, New York University. E-mail: richard.arum@nyu.edu.

Higher education is coming under increasing scrutiny, spurred by growing enrollments and rising college costs. In particular, stakeholders are increasingly asking whether students are learning and whether institutions are providing a quality of service that justifies their cost. Although little agreement to date has been reached on how to assess learning or even on the utility of imposing a single standardized measure of learning across higher education institutions, the need for developing some standard of assessment is apparent.

These concerns have recently led to the launch of a pilot project by the Organization for Economic Cooperation and Development (OECD). The OECD has moved forward with an ambitious feasibility study called the Assessment of Higher Education Learning Outcomes (AHELO). The AHELO study seeks to test “the science of assessment” as well as the practicality of implementation, to consider various methods of measurement and their validity in an international context (www.oecd.org/edu/ahelo). To contribute to the conversation about assessment of learning in higher education, we describe two of the most common approaches currently used in the United States.

Contemporary US Models
The National Survey of Student Engagement (NSSE) is a widely used assessment of student learning and personal development in tertiary education (http://www.nsse.iub.edu). NSSE is built on the premise that what students do in higher education is crucial for their learning and personal development. Consequently, it focuses on measuring students’ engagement in college, including participation in activities inside and outside of the classroom, experiences in courses, and interactions with faculty. With respect to learning, students are asked to rate (on a 4-point scale from “very much” to “very little”) how much the experiences at their institution have contributed to their development of different skills such as “thinking critically and analytically” and “writing clearly and effectively.” Schools use this survey to get an indication of how student experiences can be improved upon to optimize learning.

One of the key critiques of NSSE has been that the assessment of learning is based on students’ self reports. Can students accurately identify or define learning or critical thinking skills? Moreover, are students able to objectively report their gains in learning, and in particular how likely are self-reports to inflate the gains in acquired skills? On the other hand, because NSSE is based on students’ responses to multiple-choice questions, it is relatively simple and cheap to administer, leading to widespread use and large sample sizes. Student self-reports can be a valuable and relevant, although not necessarily a definitive, source of data on students’ educational experiences at a particular institution. Data from NSSE have been explored by George Kuh and others to provide useful insights on institutional practices so as to improve student learning and personal development.

In contrast to NSSE, the Collegiate Learning Assessment (CLA) attempts to measure learning directly and does so through open-ended prompts (www.cae.org/cla). CLA has three components: make an argument (in which students need to support or reject a position on some issue), critique an argument (in which students are asked to evaluate the validity of an argument made by someone else), and a performance task (in which students are asked to use different materials such as memos, articles, news clips, etc., to respond to an open-ended question regarding a hypothetical but realistic situation). Through these approaches, the CLA aims to measure broad skills such as critical thinking, analytical reasoning, problem solving, and writing communication.

The CLA is seen as an exemplary model of what is called a value-added assessment—an assessment strategy that focuses on institutions rather than students and aims to provide a summative evaluation of the school’s contribution to student learning. The CLA attempts to accomplish this in two ways: 1) by measuring how well an institution’s students perform relative to similarly situated students at other institutions (i.e., those with similar admission test scores); and 2) by assessing the improvement of students’ skills over time at a given institution (usually by comparing the level of skills and knowledge of students when they enter higher education and right before they graduate).

The CLA is currently based on a voluntary sample of schools and students. As a consequence, student participation is not consistent across institutions, which raises questions about students’ motivation and effort in taking and performing well on this instrument. Critics of the CLA have also worried that schools using this tool will put too much emphasis on training their students to outperform other schools on the assessment while neglecting important skills that might not be measured by the CLA. Proponents of the CLA have noted that incorporating the types of questions used on the CLA in the classroom should strengthen skills that universities claim are important to their missions, such as critical thinking, analytical reasoning, and writing. Advocates of the CLA do not suggest that this instrument should be imposed on institutions or that it can measure the entire university learning experience. Rather, it is an assessment that should be used with other indicators like the NSSE.

For the past two and a half years, Richard Arum and Josipa Roksa, with the support of the Council for Aid to Education, have been conducting a study using the CLA as well as supplementary data collected from student surveys, college transcripts, and secondary sources of institutional data to generate a Determinants of College Learning longitudinal dataset. The study, so far, has yielded a set of intriguing findings on individual and institutional factors associated with learning in higher education. In order to learn more about the findings, an initial report can be accessed at: http://programs.ssrc.org/ki/pathwaystocollege/CLA_Report.pdf.

Future Considerations
Current challenges surrounding the measurement of learning in higher education are not novel. Throughout the history of education, educators and other stakeholders have often labored to develop a set of common outcomes that can be measured and evaluated. Given the challenges of this endeavor, commensuration—or the process of finding a common metric to measure characteristics that normally have different units—would undoubtedly be a necessity. Doing so offers a standardized way to compare values that might initially seem incomparable.

On the issue of commensurability, however, sociologists Wendy Espeland and Mitchell Stevens have highlighted the influence of such efforts on changing behaviors, molding expectations, and altering the very values of things. In education, regardless of what experts might know to be true about the inherent limitations of assessment indicators, quantification influences the behavior of students, parents, schools, administrations, and governments. This is evident in school rankings and high-stakes testing (i.e., testing situations that have important consequences for students, such as admission to colleges, or for schools, such as funding). Although the dangers of misuse are there, Espeland and Stevens remind us that it is a necessary part of life. Hopefully, the recent studies that have ventured to find a valid and reliable measure of student learning will be used to inform the search for proof that our institutions of higher learning are fulfilling their role of shaping a promising future.

[Online] Available: http://www.bc.edu/bc_org/avp/soe/cihe/newsletter/Number56/p3_Kim_et_al.htm