PISA Under Examination: David Scott's Keynote Speech

David Scott gave a keynote speech entitled, "PISA, International Comparisons, Epistemic Paradoxes."

Here are the notes from the speech:

The capacity and ability to know each other's minds, in terms of PISA.

Two forms of knowledge:

1. knowledge sets, skills, dispositions, capacities
2. reformed knowledge sets, skills, dispositions: performative capacities, e.g. exams

Performative knowledge: Wash-back consequences across:

• teaching and learning
• the individual
• nations

Washback consequences: different in different cases and situations.

Assessment: the traditional view:

• true score
• errors in testing

False psychometric beliefs:

• direct process of examination
• no background to knowledge
• no internal transformation: when a person is tested, no change occurs when they are tested. The original knowledge set transforms to a performative knowledge set
• no washback effect: no external transformation. The curriculum is reformed because of standardized tests.
• unidirectional linear process: no bidirectionality. There is forward and backward flow. The original assessment is unreliable.
• conflating lower and higher level skills: knowledge of facts vs. synthesis of basic facts - an example of the construct being tested
• conflating capacity with performance: "translate" for purposes of test
• culture-free testing vs. national values
• no problem with transfer

PISA:

• develops a certain type of knowledge
• curriculum-free testing doesn't really exist
• comparison of different curricula, pedagogies, testing approaches
• different samples, different values to knowledge, different views of evidence, different national idioms
• the task of PISA is to iron out these differences
• "imperfect caricatures" of all knowledge bases under consideration. (Fair testing is very hard to do)

Indicators measured in PISA:

• quiet place to study: what does this mean?
• number of books at home: socio-economic status correlation. Number does not equal influence.

Indicators are hard to develop. Testers search for indicators which have:

• Perceived Task Value: washback effect
• Motivational effect - difference among children and nations
• Ironed out ambiguity (this, however, reduces complexity and depth)

A test is a performance. Response to a test is in relation to the types of answer they think are required.

PISA has become a league table ordering a nation's achievement. A league table cannot provide us with useful information. There is a need to design instruments that take into account other variables, a table for improvement, not achievement.

Foucault: "Examination" - Discipline and Punishment

• 10 pages about education
• a test: hierarchy, normalizing judgement
• allows society to construct individuals in certain ways
• transforms the economy of visibility into the exercise of power
• introduces the individual into the field of documentation
• making each individual a case

Conclusion:

• hierarchical organization