International Higher Education, Spring 2004

Success and Perils in Evaluating Brazilian Undergraduate Programs

Claudio de Moura Castro
Claudio de Moura Castro is a Brazilian economist who has worked for the International Labour Organisation, the World Bank and the Inter-American Development Bank. He is currently the president of the Consultative Council of Faculdade Pitágoras, a new Brazilian multicampus college. E-mail Claudioc@pitagoras.com.br.

It would seem natural that educational institutions be evaluated, since they have such a major impact on students’ careers. Yet, the taboos are hard to break--in Brazil and elsewhere. In a previous government, a most courageous initiative was taken by the minister of education in 1995. For each major undergraduate program a test was created to evaluate how much students learned from the official curriculum. A grade would be assigned to each program, based on the mean performance of its students in their last semester of school.

The minister played all his cards in order to manage the acrimonious opposition of students, faculty, university presidents, and even his own staff. A barrage of legal attempts to block the testing threatened, up to the last hour, the implementation of the examination. The decisive element came from a poll conducted by the prestigious newspaper Folha de São Paulo. Educated public opinion turned out to be massively in favor of the initiative, adding to the minister’s political backing.

The System
For each major program, the system calls for eight leading professors to define the parameters of the specific examination. Professional exam makers then prepare the questions following these guidelines. A private foundation was selected to do the work of final preparation, administering (with rigorous proctoring), grading, and producing the numbers for dissemination. Brazil has ample experience in large-scale (multiple-choice) testing for university entry. Therefore, the task of testing 400,000 students on the same day was not such a formidable hurdle.

In the absence of standards for how much a graduating student should know, it would have been pointless to establish a passing grade or some form of absolute grading. In addition, regional disparities would create a thorny political problem. Instead, the ministry simply ranked the scores and graded them on a normal curve. The top 12 percent get an A. Those between 12 percent and 18 percent get a B. The center of the distribution--40 percent of the respondents--get a C. Below C come D and E, with symmetrical cutting points.

Taking the annual test, nicknamed Provão (“big test”), is mandatory, and students receive their individual test scores. However, while negotiating the law creating the test, the minister was not permitted to include the scores in the students’ academic records. Therefore, some students protest by leaving the examination blank. They are not immediately harmed, because individual scores are not published. But blank tests do negatively affect the grades of programs. As a result, many administrators, while approving Provão, regret that some students are damaging the reputation of their schools. While this complication is certainly an annoyance, it only affects a small number of programs.

The Impact of Provão
Provão has become a major theme in higher education. Some students hate it, and occasional boycotts still occur. Left-wing educators and politicians relish in bashing it, since opposition to any form of testing remains a powerful issue in some groups. But the majority, not always very vocal, tend to approve the initiative. For instance, a recent document signed by the representatives of all associations of private higher education institutions offered positive comments on Provão, while being quite negative on other aspects of public policy.

Whether one likes it or not, Provão has become the ultimate gauge of the quality of a course of studies. Students understand the test’s impact well; the same goes for school administrators. Like the owners of restaurants losing stars in the Michelin Guide, managers of programs that are demoted in standing are mortified--and for good reason, since programs with improved grades see an increase in the number of candidates and those that go down one grade, see drastic reductions. Programs with the grade E become in effect higher education pariahs.

Particularly in the case of private education (which includes around 70 percent of enrollments), a lowered grade impacts a serious institution like a tornado. For lack of inspiration, some institutions react by painting their buildings. Dozens of anecdotes exist documenting drastic institutional reforms, and whatever managers can think of that might boost their standing in the next year. More concretely, the system has seen an abrupt increase in the formal qualifications of teachers since the first Provão.

The test does have a number of problems--the question of value added being one of them. Provão measures both the raw material, the students, and the process imparted by the college. A study conducted by the author of this article and others indicated that 80 percent of the variance in scores is due to differences the students already presented when they first entered higher education. The results are purely relative within each program type and thus do not allow comparisons from one major program to another, a fact that is puzzling to many observers. Grades are not comparable from one year to the next, because there is still much trial and error in preparing the tests. Graduates from programs that hardly have a specific labor market connection are judged by very strict and narrow examinations. The quality of the examination also varies from one program to another. Yet, these limitations are not sufficiently serious to invalidate the overall results.

Serious research evaluating the impact of Provão on higher education has never been conducted. However, most dispassionate observers consider the test a major advance.

The perils and the disgruntled
When the new government was elected in 2002, a group of professors and teachers union leaders took charge of the Ministry of Education who were seriously disgruntled by the education policies of the previous minister. The number one enemy was Provão. The presidency of INEP--the Statistics and Evaluation Office in the ministry that is in charge of Provão--was given to a union leader whose career was dedicated to destroying the teacher evaluation system of the prestigious University of São Paulo. However, this official was unable to eliminate Provão right away because it had been created by means of a federal law. Instead, he appointed a commission composed mostly of hard-core enemies of Provão. The commission came up with a report proposing the creation of an incredibly complex system of institutional evaluations and self-evaluations. While Provão did not disappear, it did become engulfed by the baroque complexity of the system. In addition, the report openly challenged the previous policy of ranking institutions, although the arguments seem technically weak in the opinion of the author of this article.

Provão was slated to almost disappear and be replaced by procedures requiring several committees and armies of experts to visit the programs. Whereas Provão involved the evaluation of results, the new policy is a return to the evaluation of the process, known to be highly vulnerable to politics, corruption, and influence. While there is nothing wrong in principle with institutional evaluation, the problem is that when the stakes are high, preventing fraud becomes a very complex and expensive process.

The middle-of-the-road public reacted negatively to the report. But more importantly, the new minister, Cristovam Buarque, was not happy with the direction taken by the report. He openly declared to the press that he was in favor of ranking institutions and was focused on the need to have additional ways of evaluating higher education.

After a number of internal discussions, a new proposal was produced: keeping Provão but basing its results on samples, rather than on all students. The test would be administered every third year, instead of yearly. The new system would keep all the heavy institutional evaluation apparatus but allowed the Provão results to be presented separately. It also prescribed that 30 percent of the questions be less narrowly focused on the specific programs--a definitely welcome change.

Provão defenders--this author included--were not happy with the new guidelines even though they are not as disastrous as those produced by the initial committee. The new system introduces elements making fraud and manipulation much easier, while Provão was practically immune to any such problems.

For better or worse, much has been left unstated and undecided in the new guidelines. The possibility remains that Provão will survive intact and, hopefully, to be of use. But it may be watered down to the point where it loses its most effective features. Unfortunately, the minister has not taken a clear stand one way or the other.