This video was sent to me by Alan Davies, who was unable to make it in person, to use as introduction to the symposium on “Assessing the academic literacy of additional language students” at the AILA 2008 conference in Essen, Germany. Andrew Cohen graciously agreed to act as discussant in Alan’s place.

The papers were subsequently published in a special issue of the Southern African Linguistics and Applied Language Studies, Volume 27, Issue 3,  2009.

In the transcript given below, the later articles, based on the papers to which Alan Davies refers to in his address, are given in square brackets.

AILA 2008

Essen Symposium.  Assessing the academic literacy of additional language students

Introduction by Alan Davies

“Assessing academic literacy is not the same as describing it, although, of course, some initial description may be helpful in deciding what to assess. Assessment is about setting a standard. Hence all the emphasis on frameworks, levels, scales, bands and standards themselves. Hence too the modern concern for ethics since setting a standard assumes a judgement and that judgement is open to an ethical critique. The well-known distinction between the high jump (in athletics) and the long jump can help us compare assessing and describing. In the long jump, the interest is in the distance the athlete can jump: it is as though we are asking the question: how far can athletes jump, a descriptive question. In long jump competitions we usually place the contestants in order of distance: that, then, becomes a norm-referenced assessment. In the high jump, benefiting no doubt from previous description, the interest is in whether the athlete can reach a given a standard, height. This is an assessment question, often termed a criterion referenced assessment.

“In academic assessment we are less interested in what candidates can do, the undoubted range and variety of their proficiency gifts. Such range, such variety, are very evident in educated native speakers. We assume that in spite of their great differences from one another educated native speakers will possess adequate academic literacy in their first language. As I suggest later, this is an assumption which may not be justified, since educated native speakers, like educated non-native speakers, have to acquire the advanced literacy skills needed for academic survival and eventual success. But what is that adequate? What is the threshold in the language medium needed for academic viability? Notice that I do not say academic success since that is influenced by many factors such as subject knowledge, character, health and perhaps luck. But basic academic language literacy is essential to make that success possible and so the question we face is just what is that basic language proficiency: how do we encapsulate it in our tests?”

[Weideman, Albert. 2009.  Constitutive and regulative conditions for the assessment of academic literacy. Southern African Linguistics and Applied Language Studies, 27(3): 235-251.]

“Weideman expands this argument by pointing out that ‘a language test has two terminal functions: its leading function is to be found in its technical design and its founding function in the analytical or theoretical mode’. What Weideman’s theoretical mode determines is just what is the basic language proficiency to which I have referred. That is validity while the technical design of the test, the micro decisions on what the test is to include, the tasks and items that constitute it, that is validation.

“Weideman’s concern is to problematise validity as the foundation for all language test considerations. He pleads for a wider view, one which takes account of the foundations of the field of applied linguistics and of its philosophical base. This is a timely intervention for two reasons. It points to the need to view applied linguistics as a unified field, a view which in recent years has been opposed by those who prefer to see the field as fragmented. And it reminds us too that language testing is more than a technical exercise: its philosophical and ethical justification requires us to query the wider and wider uses of language tests for purposes for which they were not designed.”

[Van der Slik, Frans & Albert Weideman. 2009. Revisiting test stability: further evidence relating to the measurement of difference in performance on a test of academic literacy. Southern African Linguistics and Applied Language Studies, 27(3):253-263.]

“But language test constructors must take care of the technical qualities of their tests and it is those qualities that Frans van der Slik makes clear in his paper which celebrates the stability over time and across populations of two South African literacy tests, the TALL and the TAG. The importance of stability to test constructors is not just that they can rely on their results but that they can also claim to be measuring something that matters, or perhaps I should say possibly matters. After all, what TALL and TAG are measuring, what remains stable over time and across the three universities of interest may be no more than shoe size. But we know, or think we know, from other evidence that it is language proficiency, in this case academic literacy not shoe size. Which is another way of saying that validation requires multiple sources of evidence.”

[Butler, Gustav. 2009. The design of a postgraduate test of academic literacy: accommodating student and supervisor perceptions. Southern African Linguistics and Applied Language Studies, 27(3):291-300.]

“Butler’s investigation of face validity in relation to the design of a test of academic literacy for postgraduates offers a methodology for ensuring that such a test has face validity while at the same time not compromising other ‘crucial considerations in test design’. What Butler reminds us of is that tests do not exist on their own, they are not intended for use in the laboratory. It is salutary, therefore, to be given the kind of evidence Butler provides of stakeholder concerns and views, particularly when there is dissonance between – in this case – supervisor perceptions and those of students. This realisation compels the test constructor to consider whether there is any way to reconcile these perceptions by changes in the operationalising of the construct.”

[Cliff, Alan & Hanslo, Monique. 2009. The design and use of ‘alternate’ assessments of academic literacy as selection mechanisms in higher education. Southern African Linguistics and Applied Language Studies, 27(3): 265-276.]

“One important stakeholder is the receiving institution, often a university such as the University of Cape Town, the home of the Placement Test in English for Educational Purposes (PTEEP), the focus of the paper by Cliff. Their careful analysis demonstrates the disadvantage Black students from resource-poor secondary schools face. Their analysis leads Cliff to argue that allowance needs to be made for the disadvantaged group. The authors draw attention to ‘the importance of disaggregating data such that there is compensation for the Black student group’. But the last thing that Cliff  wants is to abandon use of school leaving achievement information: to do that could be to exchange one disadvantage for another. No, what they recommend is ‘using measures (such as PTEEP) in addition to school achievement for students from diverse educational backgrounds’. As with the validity conclusion earlier, it seems that a range of information sources is necessary.

“Cliff’s solution is to combine measures of achievement (school leaving exams) and potential, a kind of aptitude (PTEEP). But while disaggregating makes good sense, is it not incumbent on test constructors to decide beforehand what approach to disaggregating to pursue? Not to do so removes decision making from the central stakeholder, the test constructor. But one area Cliff does not explore is the extent to which race and poor schooling can be separated: what happens to white children from poor schools and black children from good schools?

“In the discussion, which I regret not being part of, I hope it may be possible to consider how far academic literacy is a particular problem for second language learners and how far it is about the acquisition of a skill, the skill which native speakers as well as non-native speakers have to acquire in order to do being academically literate.”