Methods of Assessment
22. What are the problems of assessing communication in a L2?
It is neither possible nor desirable to separate speaking skills from listening skills in many tests of oral production. They are interdependent skills.
This increases the difficulty of analysing precisely what is being tested at any one time. Without a tape recording for subsequent checking, spoken language is transient. The examiner has to make subjective judgements rapidly and often under considerable pressure.
Limited time: it is often impossible to administer this kind of oral test to large numbers of students.
The nature of language: language is interaction-based. Even activities such as letter-writing are affected by interactional factors in real situations.
Unpredictability: processing unpredictable data in real time is a vital aspect of language use. Communication is also affected by the context of the situation, including the physical environment, the roles and status of the participants, and the degree of formality.
Purpose: every utterance is made for a purpose. Communicative performance involves successive approximation towards a successful outcome.
Authenticity: a candidate's communicative ability includes the ability to come to terms with the unknown, including unsimplified authentic language. Artificially simplified or "watered-down" test items may fail to measure genuine communicative ability.
Behaviour-based assessment: communicative ability is concerned with behavioural outcomes and with the primacy of content over form in language use. A test of communicative ability therefore needs to be criterion-referenced against the operational performance of a set of authentic language tasks.
Performance tests: these may involve a global task together with the enabling skills needed to complete it. However, a candidate may be able to demonstrate all the enabling skills separately but still be unable to mobilise them effectively in an actual situation. Driving a car provides an obvious analogy: knowing how to perform each individual operation does not necessarily mean being able to drive successfully. This illustrates the need for performance tests.
Summary of Problems and Book Recommendations
- Which performance operations should be tested? What level of proficiency is anticipated? Which enabling skills are involved? Should these skills be tested individually or together? How should the content and format of the test be specified?
- How can we avoid hopelessly subjective measures of production while also setting receptive tasks appropriate to different levels of language proficiency?
Note: B. J. Carroll's 1977 specification included:
- Size and complexity of the text
- Range of enabling skills, structures and functions handled
- Speed at which language can be processed
- Flexibility in dealing with changes of topic
- Accuracy of structures
- Appropriacy of functions
- Independence from reference sources
- Repetition and hesitation in processing text
Recommended Reading
- Testing Communicative Performance: An Interim Study by B. J. Carroll. This is a theoretical book suitable for applied linguistics study or curriculum planning, but perhaps less directly useful for lesson planning.
-
Communicative Syllabus Design by John Munby (Cambridge, 1978).
The book provides a detailed taxonomy of language skills, including examples such as:
- discriminating and articulating sounds in isolated word forms and connected speech;
- discriminating and articulating stress patterns within words and connected speech;
- producing intonation patterns;
- recognising the script of a language and discriminating graphemes;
- deducing the meaning of unfamiliar items through stems and roots;
- understanding conceptual meaning and communicative value.
This is again a theoretical book, commendable for its thoroughness, but the question remains: which institution or teacher will have the time to implement such a comprehensive system? It is nevertheless well worth consulting.
- Developing Reading Skills by Françoise Grellet (Cambridge, 1981). This is a practical book which summarises reading skills succinctly and links the theory to practical examples of exercise types. It is particularly useful for teachers interested in lesson planning and materials selection or production.
- Comprehension and Comprehension Tests by Lunzer, E. et al. (1979).
- Research in Comprehension in Reading, pp. 499–545, by Davis, F. B. (1968).
- Reading Research Quarterly, Vol. 3: "Identification of Subskills in Reading Comprehension by Maximum Likelihood Analysis" by Spearritt, D. (1972).
- Reading Research Quarterly, Vol. 8: "A Model Program for Teaching Advanced Reading to Students of EFL" by Eskey, D. E.
- Language Learning, Vol. 23, No. 2, pp. 169–184: "Reading as Problem Solving: An Investigation of Strategies" by Olshavsky, J. E.
- Industrial English by T. Jupp and S. Hodlin (1975). This presents both theory and practice in functional language teaching. It was one of the earliest attempts to offer a communicative syllabus to people already working in an English-speaking environment who needed basic survival English for integration and for carrying out their jobs more effectively. It is particularly interesting for its connections between theory and practice.
- ESU Framework by B. J. Carroll and R. West (Longman, 1989). This contains operational objectives for language skills and sub-skills and attempts to chart the standards set by different British examination boards within a single framework. It includes scales and specifications for testing all four language skills and related sub-skills.
- Communication in the Classroom: Applications and Methods for a Communicative Approach, edited by Keith Johnson and Keith Morrow.
Problem Areas and Possible Solutions
- The relationship between the test and the curriculum, including a purposive test framework such as those associated with Van Ek and Munby.
- Test content and procedures.
- Levels of performance.
- Methods of data analysis.