The Pisa methodology: do its education claims stack up?

The OECD's assessment of student performance across the world is certain to provoke debate. But its methodology has come under just as much scrutiny as the results themselves

Share186

inShare5 Email

Pupils recite Di Zi Gui, which translates as 'standards for being a good student and child', in the playground of Jiale Centre primary school in Hainan province, China. Photograph: China Photos/Getty Images The Programme for International Student Assessment (Pisa) is no ordinary test. The 510,000 students who took part came from 65 economies and the results are intended to reflect which educational systems perform the best and improve the fastest. Though it was conceived by the OECD in 1997, the first Pisa was not carried out until 2000. Since then, there have been three more assessments, each one prompting a slightly different debate about educational performance and the OECD's methodology.

This, the fourth Pisa to be released, is likely to do the same so policymakers, academics, teachers and parents may be interested in how the numbers were assembled.
What does it measure?

Pisa claims to provide the most comprehensive picture available of education in 80% of the world's economy (represented by 34 OECD member countries and 31 partner countries). It: assesses the extent to which 15-year-old students have acquired key knowledge and skills that are essential for full participation in modern societies But the study claims to measure more than students' ability to recite what their teachers have imparted it also attempts to capture how well they can apply what they have learned in school. This year's release comprises the results of more than half a million students between the ages of 15 years, three months, and 16 years, two months, who sat the assessment in 2012. The thematic areas were:

Mathematics Reading Science Problem solving Financial literacy The three first subjects have consistently featured in Pisa, which allows for easier comparison over time, though this year's report has a particular focus on mathematics. The latter two are minor areas of assessment.
How do they measure it?

Basic demographic information was compiled by asking students to answer a background survey that included questions on their homes, their schools and who they were. Headteachers were also given a questionnaire, through which they could provide context for their school system and the atmosphere in the classroom. In some countries or economies (for instance, Macau was included separately because its education system differs from that of

mainland China), questionnaires were also distributed to parents to garner information about their perceptions of their children's schools and career prospects. Finally, this was supplemented by one of two questionnaires completed by schools, which countries could choose between one focused on information and communication technologies, the other on the general background of a child's education to date. It is against this background information that the core of Pisa's assessment takes place. Assessments on the main three subjects lasted two hours and were paper-based. In several places an additional 40 minutes was given for a computer-based assessment of mathematics, reading and problem solving.

Teenage girls in a UK classroom. UK Photograph: Jeff Morgan 07/Alamy Students were faced with a mixture of multiple-choice questions as well as more open-ended ones that required a free-form response. Different students answered different question bundles, each of which were based on a passage of text describing a real-life situation. Students were ranked either on where they figured in their own country's percentiles or according to the OECD's own international assessment levels (1 to 6 with 6 being the strongest performers). The averages, which will affect each country's overall ranking, are based on a simple mean of each country's estimates, regardless of its size or how many children participated.

What criticisms have been made against it?

From the outset, Pisa has been met with scepticism, criticism and even outrage, most of which has stemmed from the claim that the study's findings are arbitrary. One such voice was Dr Svein Sjberg of the University of Oslo, who claimed that a small change in question choice or weightings could result in a big change in a country's overall rankings. Though the methodology for collecting the results might be clear, the way they are interpreted and analysed to become final results is less so. This lack of statistical transparency has also been a focal point of criticism levelled at Pisa and indeed the OECD. The last time they published a technical report to make their methods more open, they werent able to make it any more concise than 419 pages. The sheer breadth of Pisa has also raised questions about the comparability of results within it. Even those who accept it is fair to assess countries and economies alongside one another question some of the cultural factors that divide them. Joachim Wuttke, a teacher of computer science in Germany, is one such sceptic. Wuttke looked at which students failed to complete the test and which preferred to increase their speed even at the risk of increasing their errors. He found significant differences in student behaviour that affected student performance:

Dutch students try to answer almost every item. Towards the end of the test they become hasty and increasingly resort to guessing. Austrian and German students skip many items, and they do so from the first block on, which leaves them enough time to finish the test without much accelerating their pace. Greek students, by contrast, seem to be taken by surprise by the time pressure near the end. In the first block, their correct-response rate is better than in Portugal and not far away from the US and Italy. In the last block, however, non-reached items and missing responses add up to 35%, bringing Greece down to one of the last ranks. At a more fundamental level, some have pointed to the difficulties of testing students in such a wide array of languages. If such a thing as a perfect translation exists (few would argue it does) it might still not be enough to control for students' differing interpretations of the same set of instructions.

There is also a temptation to grab at averages from Pisa to understand the performance of a country. Doing so would obscure the vast differences that can occur regionally within a country such an oversight can be particularly dangerous, say critics, for understanding the true academic opportunities available in a country. Meanwhile, countries such as Finland, a consistent top-performer in the Pisa league tables, has been accused of failing miserably on other international academic tests such as TIMSS. Why? Gabriel H Sahlgren, the author of a book on the dubiousness of the Finnish miracle claims it is because its centralised curriculum has ignored certain concepts that are not tested in Pisa. What unites the critical voices of Sjberg and others like him is a refusal to accept the basic claim of the assessment that it is able to accurately capture the full range of students' abilities and compare them across the world.
What rebuttals have been made to support it?

Not all are convinced that Pisa officials should don the dunce cap. Conceding that several of the criticisms are valid, John Jerrim, a lecturer at the Institute of Education, points to the strong correlation between student performance in Pisa and key stage 3 maths exams and between Pisa and other tests such as TIMSS. The jury is still out and no doubt they will have new evidence with this years release in reaching an assessment of the validity of Pisa. While they deliberate, the academic community seems split between those concerned that national education policy is being dictated by an OECD statistical release with little public input and those who argue instead that politicians will always be guided by what is financially possible and politically popular, Pisa or no Pisa.