worried about PISA?
Lately the results from the Programme of International Student Assessment (PISA) tests have been increasingly used to justify policy and political rhetoric in the UK and elsewhere. Many of us: teachers, professors, researchers, employers, harbour serious doubts about the foundation that PISA is built on. There is much to learn from PISA, of course, but many doubt its suitability as a platform for any international comparative league tables. Nor is it any kind of valid basis for criticisms of the weaknesses of educational systems - not that education is perfect of course. Because it is too often used and abused in those ways, we regard it as dangerous and indeed potentially damaging.
This web page - which seems to be attracting a lot of visits - brings together, and will grow, a number of useful papers and references to help people who, like us, are astonished at the claims made for PISA's usefulness in policy making. Test results vary in the rankings of course. The 2007 TIMSS report suggested that UK students were "top" in Europe, for example. We are not though suggesting that policy makers pick and choose their tests, any more than they should pick and choose findings capriciously from within those tests.
This all matters because you only get one chance at being a child and if a politician or policy maker messes up that chance with a capricious cherry picking of policy from what is already a deeply flawed comparative vehicle then we all need to shout out loud and clear that this is wrong. As many others have already observed, with scholarship to back their reflections, any policy making based on the PISA data (whether about school structures, standards or the curriculum) simply cannot be justified. Of course even where PISA with all its faults throws up a "league leader" the smart ones can see that its approach is less than useful in this new millennium.
This web page hopefully helps everyone to be smart about the dangers of PISA. is just a list, lightly annotated. Others have done the hard work, we are simply pointing a lens at their efforts and reflections. But it is a damning collection that should concern any parent, teacher, politician or student hearing the words PISA and policy together in the same speech.
Ironically, one of the 'certainties' that are claimed to have emerged from PISA, although certainly ignored completely here in the UK in 2011, is the suggestion that the 'best' achievers are characterised by having collaboration rather than competition between their schools. Perhaps the very many critics of PISA should learn to collaborate more to magnify their voices of dissent.
This is not a counsel of despair. Kenneth Chen is the inspirational Under Secretary for Education in Pisa top performer Hong Kong. Speaking at the recent regional leadership forum there in March 2011 he commented that: "learning to learn is the cornerstone of our education reforms in Hong Kong", and "Hong Kong does not come out on top of science teaching globally by continuing to teach science in the old ways", and "I would encourage you all to be moving away from a focus on Content Knowledge". He also reminded us that Singapore's mantra is currently "teach less, so that we might learn more". Wisdom indeed, and very quotable.
Of our own criticisms (all of the below and more!) we would perhaps highlight the mismatch between the "met before" curriculum of PISA and the world of uncertainty and surprise that its learners are going out into. New technology allows people to move closer to the margins of what is possible and at those margins things fail spectacularly as we have seen often this century: volcanic ash clouds, deep ocean oil well leaks, a banking collapse. To be able to address these unexpected and surprising problems children need ingenuity, imagination and agility. PISA claims that they test "... students on their ability to adapt the knowledge they acquire at school to real-life situations as opposed to how they master a specific curriculum". The evidence is that it doesn't, or doesn't with enough validity to be useful for policy making.
This chapter is a great starting point. Stefan T. Hopmann/Gertrude Brinek's introduction to their (2007) book "PISA According to PISA – Does PISA Keep What It Promises?". Their introduction is a scholarly overview of everything from the cultural skew, through to PISA's criticised research conduct and significant flaws in the response rates and sampling (with its resulting bias and uncertainties).
Gerald W. Bracey - writing in Dissent Magazine in August 2008. "The Leaning (Toppling?) Tower of PISA: Facts and Doubts about International Comparisons in Education".
This article takes a bit of time to get going, but has some useful criticisms buried within it. For example
"PISA questions ramble discursively and sometimes contain irrelevant information and factually incorrect material. PISA's long questions, administered to 15-year-olds, mean that its assessment of science and math is hopelessly confounded with reading. The correlation between PISA's math and reading assessment, for instance, is .77, indicating a great deal of overlap."
Gerald offers some amusing examples about language to back that up (search article for "scholar's wife")...
Yong Zhao is currently Presidential Chair and Associate Dean for Global Education, College of Education at the University of Oregon, where he also serves as the director of the Center for Advanced Technology in Education (CATE). He is a fellow of the International Academy for Education. He has an honourable track record of criticism of PISA - see for example this blog post:
or this one: http://zhaolearning.com/2009/09/03/nature-article-interpreting-international-test-scores-is-the-u-s-really-behind/
or if you prefer video try No Child Left Behind and Global Competitiveness? Part of a film series produced by the Mobile Learning Institute's called "Technology and 21st Century Learning"
This misuse of PISA data is not necessarily PISA's own fault. OECD do offer caveats, whcih are then too often ignored. For example in the documentation which accompanies the 2009 PISA data the OECD state that no trends can be drawn between the latest data and those of a decade ago. They say: "Trend comparisons, which are a feature of the PISA 2009 reporting are not reported here because for the United Kingdom it is only possible to compare 2006 and 2009 data".
See a reflection on this from Full Fact . org "Promoting accuracy in public debate", here:
Pass rates in SATs in the UK have doubled in the last 15 years, but this is not matched by PISA data (for all the above reasons, and...). Here is the Local Schools Network's Francis Gilbert exploring "PISA versus SATs — which assess reading the best?"
We'd do well to take our slice of the Pisa with a large pinch of salt - Peter Wilby towards the end of 2010 writing in the Times Educational Supplement's Connect section.
"The truth is that Pisa cannot offer a guide to the relative success of different school systems. It tests 15-year-olds on the "cumulative impact of learning ... experiences both in school and at home"
Donald Clarke doesn't pull any punches with his "Leaning tower of PISA – 7 serious skews" - but it covers some familiar criticisms with bravura.
You may need cheering up after all this. We both enjoyed John Sener's Washington Post article Standardized tests prove I'm better than Michael Jordan in April 2011.
still more to come, this is enough for now...
this page last updated Thursday, April 14, 2011 9:18 AM