The exams are over and the results are in. And like a gaggle of anxious students, education ministers around the world have been checking their nation’s performance in the latest Programme for International Student Assessment (Pisa).
Set up in 2000 by the Organisation of Economic Cooperation and Development (OECD), Pisa is a triennial test of the abilities of more than half a million 15 year olds across 65 countries. And its findings always provoke controversy.
To no one’s great surprise, the best performing nations came from east Asia, with children in Shanghai, China, topping the league table.
More surprising, perhaps, was the middling performance of theUnited States, Britain and most European nations.
For the UAE, in the bottom quartile of the league table, the message might seem to be “must try harder”, although “keep up the good work” may be more accurate.
As Humaid Al Qattami, the Minister of Education, pointed out, the UAE was one of only six countries to show a big improvement in scores since the last survey.
So why are all children not as smart as those in Shanghai, whose scores are 20 per cent higher than the Pisa average? Cue a host of anecdotes and conjectures.
Most teachers would envy the status and autonomy enjoyed by teachers in Shanghai. Whether they could take the relentless pressure to perform and the continuous retraining is, however, another matter.
The same goes for the students, most of whom undergo yet more teaching from private tutors.
And not all parents would welcome the regular calls from school to ensure their offspring were working at home – or, apparently, the advice on how to be better parents.
Still, the education mandarins of Shanghai are confident all of this is worthwhile, believing education is the surest route to economic success.
In that, at least, there seems to be widespread agreement – along with a belief that the Pisa results must be taken seriously. Yet both these convictions may well be misplaced.
You might think that the rankings of each country are based on the results of every student in each country performing the same test.
Not so. To save time and money, each student typically answers fewer than half the reading questions, and the national ranking is extracted from a mathematical model of “plausible” scores the students might have got, had they answered all the questions.
This may sound dubious, but any child in Shanghai will tell you that mathematical models can do a good job of “imputing” missing data.
But the model used for the Pisa survey has been put to the test – and not only has the model been found wanting, the impact of its failings on league table rankings is not pretty.
This year, Prof Svend Kreiner, of the University of Copenhagen, showed that the method used by Pisa to estimate reading scores introduced so much uncertainty that actual league table rankings could vary by 20 places or more.
Even if the model was perfect, it is not clear the rankings would mean much. As Prof Kreiner told the Times Educational Supplement: “It is meaningless to try to compare reading in Chinese with reading in Danish.” He could have added Arabic.
Even if the rankings were completely reliable, it would be dangerous to read too much into them.
The claim that every nation can thrive by adopting practices that have led some Asian nations to achieve high Pisa rankings is surely mistaken.
After all, the belief that there is more to childhood than cramming for exams runs deep in many cultures.
It is often backed by stories about how Newton and Einstein were lousy students, and how the Britain – mid-table in the Pisa rankings – has bagged 96 science Nobels, compared with China’s measly tally of four.
Such arguments barely stand up to scrutiny, of course. Britain has been a world power for centuries, while China is still emerging from agrarianism. Who knows how many would-be Newtons spent their lives tending Maoist-run paddy fields.
But there is a serious point lurking behind such anecdotes. The controversy over the Pisa tables has focused largely on overall scores, which supposedly give an indication of average national performance.
But this is only half the story. The performance of students follows the famous bell curve, for which the variance – roughly speaking, the spread of the curve – is no less important.
The bigger the variance, the squatter the bell-curve, and the more extensive its “tails”.
And it is entirely possible for one bell curve to have a lower average value than another, but a bigger variance – thus giving it thicker tails.
Translated into academic performance, that means one nation could have a relative low average score and yet still have a relatively large proportion of people of the highest ability.
This helps to explain the apparent anomaly of nations such as Britain and the US having lower average scores than east Asian countries, and yet utterly trouncing them on measures of outstanding creativity such as Nobels.
That is not to say the average performance of a nation is irrelevant to economic success. The UAE is surely right to aim for a population with decent average levels of literacy and numeracy.
But the drive to achieve this could come at the expense of too little variance – generated by an overly prescriptive educational system that aims for success at Pisa-style tests, while deterring independent thought.
The French philosopher Voltaire warned “le mieux est l’ennemi du bien” – perfection is the enemy of the good. The maths of the bell curve warn that, in education at least, the average can be the enemy of the exceptional.
Robert Matthews is visiting reader in science at Aston University, Birmingham, England