PISA Under Examination: Gerry Mac Ruairc's Keynote Speech

Gerry Mac Ruairc gave a keynote speech at the symposium entitled, "Ticking the Boxes: A Critical Examination of the Process of PISA Testing from a Student Perspective."

Here are the notes from the speech:

Testing:

• technologies of power surveillance
• impacts of testing: high-stakes and low-stakes

Unintended consequences of policy developments:

• what are they?
• need to watch them
• no one sets out to damage, but unintended consequences do that
• alliances: neo-liberal, neo-conservative, neo-Middle class
• accountability
• measurement

Policy of testing in Ireland

• began in 2008 with 7/8 year olds and 11/12 year olds
• results not centrally collated - within schools
• reviewed during Whole School Inspection
• must report scores to parents
• vertical system of testing
• 15 year olds and 18 year olds subject to more standardized testing
• TIMSS, PIRLS, PISA

Ireland

• small schools
• 8 schools have less than 50 students, 291 schools had greater than 500 students, and 238 students had 300-499 students.
• voluntary secondary schools in Ireland: privately owned, charge fees but publicly funded, Catholic or the Church of Ireland
• community and comprehensive schools
• vocational schools with a working class profile

General trends:

• commitment to knowledge economy
• commitment to education

Ireland in PISA:

• above average in math and science
• 5th or 6th place in reading

PISA 2006 Ireland:

• 165 schools randomly selected
• mix from different schools, e.g. secondary, vocational
• 4585 students
• 5 different grades (Irish 1-5)
• high level of agreement to participate
• Reading Literacy Level 1 or below: 12.2% (OECD average: 20.1%)
• Science Literacy Level 1 or below: 15.3%

Socio-economic groups and testing:

• language and linguistic discontinuity
• choice of language use intended or unintended?
• social class achievement in PISA: working class - lower achievement, middle class - higher achievement
• curriculum vs. culture
• multiple-choice answers - linguistic choices of the working class

Research on PISA 2009 - "PISA girls"

• disadvantaged, inner-city girls
• "opportunistic" sampling
• didn't know what the test was looking for
• Principal talked about the timing and the intensity of the process
• school PISA coordinator talked of "students wilting," "too intense" process, "what monster is this feeding," "rat in a lab" scenario
• ticking the boxes just to "get rid of it"
• girls' view - upset by answering questionnaires about lifestyle
• impact on engagement - interested at first, but then just ticking boxes
• issues of confidentiality - names of booklets
• didn't try as hard because of not getting results back
• the need for calculators - five grade levels in the room. Students were too embarrassed to ask for calculators
• personal nature of items on questionnaires: too "nosy," felt swayed by the socio-economic nature of questions
• negative sense of self: not willing to write about their parents' professions, some lied, bias of jobs listed for parents - not "normal" jobs in a deprived area

Conclusion:

• need to engage with student perspectives on the testing process
• need to dig deeper an enrich our understanding of socio-economic level
• need to consider the impact of socio-economic status

PISA Under Examination: Katharina Maag Merki's Keynote Speech

Katharina Maag Merki gave a keynote speech at the symposium entitled, "Central Exit Exams as an Instrument to Improve School Effectiveness? Results of an Empirical Study in Germany."

The notes from the speech are as follows:

• Exit exams - comparison of student achievement under centralized and decentralized exit exams.
• Students' motivation and learning strategies have no negative effect in the individual learning of students.

There is no consistent picture on the effects of implementation of exams on school processes.

• Germany vs. the USA - differences in testing systems.
• Bremen, for example, has decentralized local exams but centralized state exams.
• There is an overlap in achievement in Germany between the Hauptschule, Realschule, and Gymnasium
• There is a lack of standards in assessment

Problems:

• students with the same achievement levels are recommended to different tracks

Proposed:

• central "A-level" exams
• e.g. Bremen in 2008 had centralized exams for basic and advanced studies

There was no real difference in achievement between centralized and decentralized exams at the basic level.

Advanced math increased in achievement.

Advanced English showed improvement in the first year, but a decrease afterwards.

Conclusion: There is no effect of centralized exit exams on achievement.

PISA Under Examination: David Scott's Keynote Speech

David Scott gave a keynote speech entitled, "PISA, International Comparisons, Epistemic Paradoxes."

Here are the notes from the speech:

The capacity and ability to know each other's minds, in terms of PISA.

Two forms of knowledge:

1. knowledge sets, skills, dispositions, capacities
2. reformed knowledge sets, skills, dispositions: performative capacities, e.g. exams

Performative knowledge: Wash-back consequences across:

• teaching and learning
• the individual
• nations

Washback consequences: different in different cases and situations.

Assessment: the traditional view:

• true score
• errors in testing

False psychometric beliefs:

• direct process of examination
• no background to knowledge
• no internal transformation: when a person is tested, no change occurs when they are tested. The original knowledge set transforms to a performative knowledge set
• no washback effect: no external transformation. The curriculum is reformed because of standardized tests.
• unidirectional linear process: no bidirectionality. There is forward and backward flow. The original assessment is unreliable.
• conflating lower and higher level skills: knowledge of facts vs. synthesis of basic facts - an example of the construct being tested
• conflating capacity with performance: "translate" for purposes of test
• culture-free testing vs. national values
• no problem with transfer

PISA:

• develops a certain type of knowledge
• curriculum-free testing doesn't really exist
• comparison of different curricula, pedagogies, testing approaches
• different samples, different values to knowledge, different views of evidence, different national idioms
• the task of PISA is to iron out these differences
• "imperfect caricatures" of all knowledge bases under consideration. (Fair testing is very hard to do)

Indicators measured in PISA:

• quiet place to study: what does this mean?
• number of books at home: socio-economic status correlation. Number does not equal influence.

Indicators are hard to develop. Testers search for indicators which have:

• Perceived Task Value: washback effect
• Motivational effect - difference among children and nations
• Ironed out ambiguity (this, however, reduces complexity and depth)

A test is a performance. Response to a test is in relation to the types of answer they think are required.

PISA has become a league table ordering a nation's achievement. A league table cannot provide us with useful information. There is a need to design instruments that take into account other variables, a table for improvement, not achievement.

Foucault: "Examination" - Discipline and Punishment

• 10 pages about education
• a test: hierarchy, normalizing judgement
• allows society to construct individuals in certain ways
• transforms the economy of visibility into the exercise of power
• introduces the individual into the field of documentation
• making each individual a case

Conclusion:

• hierarchical organization

PISA Under Examination: David Berliner's Keynote Speech

David Berliner gave a keynote speech entitled, "Contexts of High-Stakes Testing: The 'Quality' of the PISA Results and the Challenges of PISA to Definitions of School Knowledge"

Here are the notes from the speech:

• PISA - not high-stakes testing in the USA, unlike Germany
• State tests (50 of them) have sanctions
• NAEP (National Assessment of Educational Progress), TIMSS (Trends in Mathematics and Science Survey), AP (Advanced Placement), SAT (Scholastic Aptitude Test), ACT (American College Testing) are indicators of quality
• No Child Left Behind
• Same context of interpretation: uncritical acceptance of scores as validation of teachers and school quality
• Scores - a prophesy: transmitting scores into predictions
• Media focus on USA's poor performance in scores
• Press attention - US students "below average"
• PISA 2006 focus on science: Glee in the USA, as newspaper writers like bad news, and it promoted vouchers and charter schools
• High PISA scores come from countries with strong public schools and national standards. *Finland does not*
• High-stakes testing popular in the USA - competition
• Last half of the 20th century was American-dominated in terms of education
• No longer the case as illustrated by PISA
• Perpetuates the myth of failed education
• David Brooks of the New York Times: PISA scores are shaping the nation
• Tom Friedman of the New York Times: USA in PISA is like Slovakia or Portugal, not like Canada or The Netherlands
• USA in PISA at the international average

Media does not report:

• high scorers in PISA's Level 6 above the OECD average
• 240,000 Finnish students are at Level 6
• 7.4 million US students are at Level 6
• this does not indicate decline in education for the USA
• trouble in finding jobs: in the USA, 25% of the top scorers were at Level 6 in scientific literacy. In 2001 there were 758,000 bachelor's degrees in science, although 1/3 of these students did not stay in science. In 2001 there were 160,000 master's degrees in science, but 2/3 of these students do not have science jobs.
• PISA tests not what is learned in schools, but what is learned over life
• USA - wrong performance analysis. PISA purposely unrelated to curriculum and not related to school
• PISA - GDP per capita and number of youth (percentage of school age population) can predict PISA scores, for example in Finland
• USA - breakdown of race: White students (40% of the school-age population) scored similarly to the Japanese in PISA. Minority students score lower.
• in the USA, SAT scores directly correlate to family income, e.g. wealthy students perform well in PISA, the SAT, etc.; poor, minority students do not perform well; Anglo countries have much child poverty

US education policy: by 2015, children must be "proficient," but these things are not taken into consideration:

• the possibility of cheating, giving answers, etc.
• high-stakes testing corrupts schools
• Campbell's Law: whenever a social indicator takes on too much value, the actors/creators will become corrupt
• the narrowing of the US curricula: No Child Left Behind led to an increase in reading and math instruction. The more time devoted to learning must lead to more achievement, but with the exception of No Child Left Behind. Too much drilling? Something is going wrong.
• No Child Left Behind was created to reduce the gap in learning, but the gap has not closed

England replicated the policy and received the same results:

• narrowing of curriculum
• only rewarding academic achievement
• narrow view of the "educated child"
• taking time from other subjects to focus on mathematics and reading. In the USA, time was taken from recess, social studies, art, music, and lunch.

USA has an "apartheid curriculum," for example, areas of low poverty can focus on art and music, while areas of high poverty concentrate on test preparation, mathematics, and reading.

This is VUCA (Volatile, Uncertain, Complex, Ambiguous)

• we are heading to this world
• we need breadth in the curriculum
• a VUCA world would seem to demand breadth. We need breadth to survive. We won't evolve properly. We need variation of talents and skills to survive.

19th century goals of education:

• love of learning
• thinking for oneself

Howard Gardner:

• schools have a responsibility to teach morals

USA data has led to:

• stopping of napping
• the seven minute lunch

PISA Under Examination: Clara Morgan's Keynote Speech

Clara Morgan gave a speech entitled "The OECD's PISA: Unravelling a Knowledge Network"

The notes are as follows:

This gives a public policy and political science perspective on the matter.

Introduction

• intelligence testing
• international testing
• PISA in the Canadian context
• PISA in the knowledge network
• analysis of education in the international context
• transnational governance - a web of transnational networks - operating at different scales
• "scientization" takes out politics
• political interests
• circulation of ideas
• PISA - active implementation on all levels, e.g. ideas of how schools should be run, boomerang pattern of networks

Rise of Education at the OECD

• 1990s - central to OECD work
• renewed interest in human capital theory - the necessary skills to compete in society
• 2002 - Directorate of Education
• educational governance - neo-liberal policies, e.g. competition, parental choice, testing (high-stakes)
• American influence - focus on the indicators of quality of education, e.g. A Nation at Risk, wanted a comparison with similar economies
• link with adult literacy and human capital, e.g. Statistics Canada and ETS adopted by PISA
• creation of PISA
• OECD replaced UNESCO and IEA as leader of education
• 1995 - Lahti, Finland - "strategy of student achievement analysis"

Constructing the PISA Knowledge Network

• PISA an exclusive network, e.g. OECD - exclusive economies, educational experts and researchers, governing board and policy makers
• 1990s trend towards large scale assessments
• cultural bias
• OECD needed to get educational expertise: tendering process with Borgogne, ACER, and IEA. ACER won, turned into International Consortium
• PISA creators - advocating a different outcome of education, but not prescribing what to teach

Governing Education Through PISA

• PISA infrastructure - administration, analysis, and dissemination of results
• Media coverage - "naming and shaming"
• "naming and shaming" - rankings, league tables
• results inform OECD reports
• domestic policy influence

Canadian Context

• Federal government - little control over compulsory education
• Federal interest - learning, development skills for a knowledge-based economy, transition from school to work supports PISA, not IEA tests
• Provincial interest - performance of compulsory education systems, curricular outcomes

Student Achievement Testing in Canada

• Provinces/Territories
• National - SAIP, PCAP
• International - PISA, TIMSS, PIRLS
• international benchmarking
• linkages across assessments - promise of improving provincial assessments
• international tests cannot capture the complexities of learning

Conclusion

• PISA - reinforces a scientific and technical approach to politics of education
• grown to inform educational policy
• has stopped public debate use international testing as a basis for education policy
• democratic vision of school can come from the ordinary

PISA Under Examination: Tom Popkewitz's Keynote Speech

Tom Popkewitz gave a keynote speech entitled, "PISA - Numbers, Standardizing Conduct, and the Alchemy of School Subjects"

The notes from his speech are as follows:

A content/context-free test is an illusion.

The Practical Knowledge of PISA helps us "see" historical assemblages and cultural theses about modes of life. It is like a recipe. Ingredients lead to a final product. PISA does not see the "ingredients." It is seen as a whole, but knowledge is not practical. PISA has become a cultural phenomenon, it is talked about and acted upon.

The Reason of PISA - Transnational Governance, National Salvation, and the Political Numbers as a Telling truth. Here, numbers are an important part of social science. "Facts" are uniform, objective, and rigorous. Democracy has a correlation with measurement. There is a paradox of numbers: they appear as fact through the making of that fact.

Numbers and Assessment in Making Equivalences: Global Positioning Systems. PISA monitors student improvement and student performance. A GPS always knows its position. Constant measurement and performance helps locate oneself in the world (Simons) and gives navigational tools (Lindblad). This does not act directly on people.

PISA: The Alchemy and School Subjects. "The alchemy" is an analogy to medieval practices to think about pedagogical translations. PISA has little relation to interaction and communication of the academic fields. The "practical knowledge" assembles a cultural thesis about the "lifelong learner," a mode of life who the child is and should be.

PISA as Cultural Thesis of the Lifelong Learner: Who Made the Recipe of Practical Knowledge? A lifelong learner is flexible, active, and collaborative. An uncertain world needs problem solving. This works in a global world with no finishing line. Life continues responsibly. The only choice is to choose.

Science: has textbooks to emphasize greater participation and problem solving to learn the symbolic systems of science.

Mathematics: has modeling and predicting of real-world phenomena

Disciplinary knowledge: stable, logical, and analytic structures separated from the social and cultural fields in its production.

Scientific Literacy: no consensus across countries. It is NOT a universal language (in textbooks)

PISA, School Subjects, and Historical Context: The Planning of Science in the Redemption of Society. Science uses the future to reorganize the present. 19th century science went from the mastery of the physical world to the mastery of the social world. Science as planning: it was to predict the future.

PISA and the "Reason" of Practical Knowledge: Exclusion and Abjections in the Impulse to Include. Alchemy: double gestures -- hope for the future and fears of dangers. Education for all: "all" assumes a unity and a consensus. "All" erases differences and the inscription of psychological and social differences.

Conclusions.

I. PISA is not merely measures of practical knowledge of the future. What is practical knowledge? There are different contexts.

II. The politics of school.

• Alchemy: knowing occurs as "practical."
• Installation of "shepherds" to guide what is "practical."
• PISA: hope for the future, the dangers, and dangerous of populatoins.

III. The limits of the present.

IV. Historical situations: thinking about the present with implications and consequences.

PISA Under Examination: Ulf P. Lundgren's Keynote Speech

Ulf P. Lundgren gave a keynote speech entitled "PISA as a Political Instrument: the history behind formulating the PISA Programme"

Ulf P. Lundgren is a Professor of Policy and Philosophy of Education at the University of Uppsala and the Director-General of the National Agency of Education in Sweden.

The notes from his speech are as follows:

The history behind PISA has two interpretations:

1. a short history
2. reforms in policy followed Zeitgeist, the changes in society

PISA really started in the 1980s.

Educational measurement began in the 19th century. In 1862, England began assessment and inspection related to outcomes. The measurement related to intelligence testing, along with educational and psychological testing.

Education is important for society and individuals. For individuals, education is a link to a new life. Education also has a link to the labor market. The modern evaluation is a social context where everything could be questioned.

Assessment techniques developed. The international Examination Inquiry of the 1930s formed an international network of assessment, but few with comparative ambitions.

In 1957, Sputnik occurred, and President John F. Kennedy announced that a US man must be first on the moon. The "space race" tied in education to space outcomes.

The IEA was founded officially in 1967 and began comparative study. International comparisons became features on political agendae and international comparativeness was born. Human capital theory was established.

The 1970s oil crisis increased international competition. International assessments became more important, and international assessments were seen as tools for educational improvements.

The Cambridge manifesto in 1972, which encouraged broader methods of educational research (including qualitative research) paralleled the 1970s trend which attacked school systems and statistics in education.

In the 1980s, the Welfare Society came under attack. New policies were implemented. Globalization gave new solutions, such as decentralization and marketization of schools.

In the US and UK there was a decentralization of policy, but a centralization of inspection.

In 1989 the Berlin Wall fell, and in 1992, the USSR ended, which marked the end of the Cold War. Competition was not about domination. A new context for competition began: economic. ICT began to emerge. Teachers began to be seen as professionals.

International assessments have changed its context over time.

In 1968, the OECD's Centre for Education Research and Innovation (CERI) was started. In the 1980s, the OECD's INES, the education indicators program began. Statistics began to impact upon education policy.

INES built up connections and networks in different countries. In 1992, Education at a Glance was born. However, there were problems in finding outcome data. At the time there were only the IEA studies.

Tom Alexander of CERI said the OECD needed a new program, as there were validity problems with the IEA studies. They wanted a test that was "content free." ETS (Educational Testing Service) created a broad literacy concept test. Statistics Canada handled the design and data collection.

From this came PISA, first administered in 2000, and with those results released in 2001.

PISA Under Examination: Opening of the Symposium

The opening of the symposium was by a panel with Miguel Pereyra, Robert Cowan, Guadalupe Gonzalez Tano, Pilar Teresa Diaz, and German Gonzalez.

Guadalupe Gonzales Tano spoke of:

• a clear imbalance in education
• cautions about these surveys
• how surveys such as PISA show places to improve within an education system
• the benchmarks they provide

Pilar Teresa Diaz spoke of:

• PISA as an international phenomenon
• how PISA highlights the main problems of society and education
• how PISA diagnoses global problems
• the sensitivity needed with these surveys
• the controversy of PISA, as PISA leaves nobody indifferent
• the fundamental problems in comparative education
• how PISA promotes improvement in education
• how PISA needs to encourage in-depth analysis
• the positive transformation of education
• how PISA points out the problems we face

German Gonzalez spoke of:

• PISA as a popular phenomenon
• his role as director of the museum of education
• Plato's views on education: the greatest beauty of body and soul
• how today's world is interested in other aspects of education, as in the economic side
• how PISA needs people with know-how

Miguel Pereyra spoke of:

• the founding of CESE in London (1961)
• the symposium, which focuses on PISA, the most complete international assessment thus far
• the media storm following the release of PISA scores
• the OECD creating the best form of international comparison
• how perhaps PISA has become the OECD's "unwanted child"
• the International Education Statistics, InES, which reoriented the efficiency of international indicators in 1986
• PISA's birth in 1986
• how PISA is often misinterpreted
• the difficulty in actual comparison
• how PISA is more than a ranking, it is a tool for the governance of education, and someday will be an international tool for world governance
• the symposium would highlight the problems of PISA and the puzzle that PISA represents
• the problematic scope of PISA -- it illustrates the systems that aren't working
• the media's interpretation of PISA as an X-Ray or snapshot of schools and education systems
• PISA's definition of "literacy," not concerned with didactics
• the "horserace" that PISA has generated
• how PISA measures not only performance, but also school functions
• how PISA's measurement of education is problematic: it comes with rich data and a reservoir of empirical data
• how PISA goes far beyond a test of economic or educational achievement
• how PISA permeates everything

PISA Under Examination: Changing Knowledge, Changing Tests, and Changing Schools (Part Two)

The CESE Symposium held on the island of La Palma also hosted a poster exhibition. These are the posters presented at the Symposium:

• A Portrait of European Top Performers in Science by Fabio Alivernini, Laura Palmerio, Valerie Tortora, National Institute for the Evaluation fo the Education System, Frascati, Italy
• Thinking About PISA and the Participation of the Parents by Claudio Almonacid Aguila, Ana Jimenez Saldana, Daniel Rios Munoz, University of Santiago de Chile
• An Investigation of Reasons for Finland's Success in PISA by Jennifer H. Chung, Liverpool Hope University
• From the Static to the Dynamic: PISA Items as an Educational Resource by Jose R. Galo Sanchez, Descrtes Project, Institute of Education Technologies, Ministry of Education, Spain; University of Cordoba
• PISA, Finland and the Autonomous Community of Madrid by Maria Jose Garcia Ruiz, UNED
• Digital Skills Appear in PISA Report: Are Students and Teachers Prepared? by Ana Garcia-Valcarcel, Francisco Javier Tejedor, Anunciacion Quintero, University of Salamanca
• Difference of Gender in the Educative Practices and Expectations of Students by Carmen Nieves Perez Sanchez, Moises Betancort, University of La Laguna
• PISA on Front Page by Ariadne Runte-Geidel, Diego Sevilla, University of Grenada
• The Educational and Sociolabour Guidance Program: An Instrument to Facilitate the Sociolabour Transition by Lidia E. Santana Vega, Jose A Santana Lorenzo, University of La Laguna
• Similarities and Differences Between Spanish and Finnish Teachers: Organization of School Time and Space, Selection and Salary by Begona Zamora Fortuny, University of La Laguna
• The New Challenges Posted by PISA for a Graduate Programme in Upper Secondary Education Teaching at the National Autonomous University of Mexico by Juan F. Zorilla, UNAM, Mexico

PISA Under Examination: Changing Knowledge, Changing Tests, and Changing Schools

The Comparative Education Society in Europe (CESE) held in international symposium in La Palma, Canary Islands, Spain from the 23rd to the 26th of November 2009. This was organized by Miguel A. Pereyra, President of CESE and Professor of Comparative Education at the University of Granada.

The Symposium featured these Keynote Speakers:

• David C. Berliner of Arizona State University
• Antonio Bolivar of the University of Grenada
• Julio Carabana of the Complutense University of Madrid
• Robert Cowan of the Institute of Education, University of London
• Ulf P. Lundgren of the University of Uppsala
• Katharina Maag Merki of the University of Zurich
• Gerry Mac Ruairc of University College Dublin
• Clara Morgan of Carleton University
• Donatella Palomba of the University of Rome "Tor Vergata"
• Anselmo Roberto Paolone of the University of Rome "Tor Vergata"
• Thomas S. Popkewitz of the University of Wisconsin - Madison
• Javier Salinas of the Complutense University of Madrid
• David Scott of the Institute of Education, University of London
• Daniel Santin of the Complutense University of Madrid
• Hannu Simola of the University of Helsinki
• Petra Stanat of the Freie Universitat Berlin
• Daniel Troehler of the University of Luxembourg