Skip navigation

Category Archives: Uncategorized

By Peter Preston, Poland

Teachers do calculate the average score from tests, but then nothing serious is done with it. Even when the average score is close to the pass mark little statistical comment is made about the glaring problem that this represents. For example, if the average and the pass mark are the same and the population is normally distributed around the average, this means that 50% of the students fail. Can it be considered acceptable for 50% of the candidates to fail an end-of-the-year examination or even worse an end-of-the-course examination?

In fact at our college the last third-year UoE exam failed 80% of the students. Now you would think that a statistically-minded person would immediately start asking questions about validity of the exam. Construct validity – did the items set test the points intended to be tested? Course validity – did the items tested figure in the course syllabus? Is there a proper tie-up between the course syllabus and the test specifications (if the latter exist at all)? Did the distribution of correct responses discriminate between the weak and strong candidates? Were the items either too easy [not in this case] or too difficult? Is there any objective reference to competence standards built into the teaching programme? To ask just a few relevant questions.

I would love to hear that other institutions do use statistical analysis of exam data and look at the variance between different exam sittings using the same exam or different ones, but I wonder if small institutes can ever bring together the required expertese to carry out such work either before the exam goes live or afterwards. It would be great to conduct a poll on this matter to try to assess the use of statistics in the analysis of exam data at as many institutes as possible.

Peter Preston's students in Poland

My own experience inclines me to believe that exams are in fact not so much an educational evaluation of the work being done as a policy instrument to give face validity to the programme. As such one does not need to worry about the quality of the exam since one can adjust the results before publication. Or in the case of my institute the exam can be repeated by order from above until the teachers get the message.

I do not like the cynical manipulation of exam data, so having good quality statistical information and quality control of all documents involved in the course would be the start to a reevaluation of the course and teaching methods. By accurate assessment at the beginning of a course it should be possible to predict the level students could get to after a given number of teaching hours, taking into account the realities of life. By keeping proper statistical records over a few years one would accumulate powerful information. This is what insurance companies do to calculate their premiums.

Advertisements

By Amy Shipley, Academy of Art University, San Francisco, USA

We use a rubric that’s based on our course learning outcomes for all our writing and speaking assignments. I give them the rubric when I assign the task. I also put students in groups and give them two things: the rubric itself and a blank rubric. I have them paraphrase the requirements in each grading category so they fully understand what I’m looking for.

For speaking tasks, I videotape all formal presentations, so I have a record of what they’ve done. But it’s also for the students to evaluate (and grade) themselves. During the presentation, I also assign students in the audience to specific speakers, and have them evaluate and give feedback to them at the end of the presentation. When students grade themselves (which I check after I’ve graded them), I get feedback on my grading. It helps me to know if they understand the criteria and whether or not I explained it well.

By Erlyn Baack, now retired, formerly at ITESM, Campus Queretaro, Mexico http://eslbee.com

Both the IELTS and the TOEFL are proficiency tests that measure overall proficiency. They are both global in nature. I do not think they should be seen as achievement tests to be used at the end of a semester of study. Instead, they may be used to inform the achievement rubrics that should be developed within successive levels within an English program. Likewise, these proficiency exams should not be used as placement exams either because there are better placement exams available. There is not a single question on the TOEFL, for example, that discriminates the difference between English ONE, TWO, and THREE levels for instance. So for placement, even Michigan’s very old English Placement Test (if it is still available) would be better than the TOEFL for placement.

That said, the IELTS and the TOEFL should inform the achievement (and the rubrics in each of the four skills, ideally) that teachers and/or course administrators want to achieve at each level within an English program. Teachers and/or course administrators have to decide the curriculum at each level: For example, in developing the curriculum for English ONE, teachers and/or course administrators must ask and answer the following questions: At the end of the semester, (1) What do we want the students to know (or achieve, or be, or be able to do)?, (2) How are we going to teach it?, and (3) How are we going to test it?

Teachers and/or administrators are then responsible for designing a curriculum and an ACHIEVEMENT exam, _with rubric_, that measures the level of student achievement throughout the semester. By definition, all students should have the ability to STUDY or PRACTICE the curriculum within the semester that would lead to higher achievement scores meaning there would be a high correlation between (1) the number of hours a student studies and (2) his/her final semester score. Those achievement scores, then, would affect the TOEFL and the IELTS only indirectly.

I think it is helpful to distinguish between various exams and what they measure.

(1) Placement exams contain questions at all levels to place students within an English program. Michigan’s EPT is an example.

(2) Proficiency exams measure overall proficiency. The IELTS and TOEFL are examples, and they are used by universities, generally, to determine whether proficiency is sufficient for university studies.

(3) Achievement exams measure the level of student achievement within a semester of study. A major monthly exam, a mid-semester exam, or a final exam are examples of those. Did the student “achieve” what was supposed to have been taught and learned within a given week or month or semester?

>By Simon Gill – Olomouc, Czech Republic

At the IATEFL Conference in Edinburgh in 1999, the opening plenary was given by Mike Wallace, who told the story of the origin of the Berlitz method. I may have got the details wrong but it went something like this. Charles Berlitz, who was trilingual, was on the verge of opening his school when he fell seriously ill and had to employ a stand-in teacher, who spoke only English. On his return from his illness he was amazed to discover that the substitute had achieved great success using an English-only method that was very different from the grammar-translation approach Berlitz himself had intended and, being a sharp businessman, took it over and took all the credit too. So essentially the Berlitz Method is, like so many other successful inventions, an accident.

>Martin McMorrow – Massey University, New Zealand

There have been some interest in the historical subjects of Direct Method and BASIC English over the last week or so. A good source for information about both of them is Howatt’s History of ELT. Howatt points out among other things that BASIC stands for “British American Scientific International Commercial” and it’s intended to be a kind of language in its own right, simply using English words and grammar as its raw ingredients in the same way that Zamenhof had earlier boiled down lumps of Latin, chunks of Czech etc into his Esperanto gruel. The way in which Ogden selected the vocabulary items for BASIC brings to mind the approach of Roget’s Thesaurus – though Jeremy Bentham’s philosophy was said to be a major influence. As Howatt describes it, the 850 BASIC words were made up of 150 qualities, 600 things and 100 operations.

In order to stretch this limited lexical stock, Ogden used three main strategies: circumlocution (eg if you can say ‘put a question’, you can get rid of ‘ask’); conversion (lots of the 600 ‘things’ could be used verbally) and combination (phrasal/ prepositional verbs such as get in etc increased the verbal resources exponentially!). Of course, in doing all this stretching, Ogden hugely increased the number of lexical items in BASIC while, strictly speaking, restricting himself to the meagre 850 words. As Howatt neatly puts it “one language learning problem, the number of new words, has been exchanged for another, the multiplicity of meanings that each new word is required to carry” (1984, pp 253-4)

Howatt also points out the weakness of Ogden’s project in practical, sociopolitical terms. BASIC was neither fish nor fowl – neither English, nor not English. There were no native speakers of BASIC English and how would you persuade teachers who could speak ‘proper’ English to ‘break the rules’ in order to speak BASIC (eg by missing off the -s endings, which were ungrammatical in BASIC). (By the way, I think this is also a contemporary issue in relation to EIL/ELF). Howatt uses Malinowski’s notions of ‘context of situation’ and ‘speech community’ (which were developing at the same time as BASIC) as a framework for this critique. According to this perspective, BASIC English was “a lost code looking for a speech community” (1984, p. 255). As such, BASIC English in itself was something of a dead end. In the long term it was Malinowski – through Firth and later Halliday – who had the deeper and more enduring influence on language study and teaching.

That said, one aspect of BASIC English – its strongly visual emphasis – links it to the much more successful Direct Method. One key crossover was the book “English through Pictures” by IA Richards and Christine Gibson – surely one of the most influential and enduring EFL publications – even though, it did seal the breach between Richards – an early champion of BASIC English- and Ogden – its purist originator. In BASIC English, preference was given to qualities and things that could be represented visually – and much the same can be said about the DIRECT METHOD. Whenever we mime in class or point to visual aids, aren’t we walking with our Direct Method dinosaurs? Howatt has some great quotes from Sauveur’s seminal Direct Method work – his 1874 “Introduction”. For instance, “Here is the finger. Look. Here is the forefinger, here is the middle finger, here is the ring-finger, here is the little finger, and here is the thumb. Do you see the finger, madame? Yes you see the finger and I see the finger. Do you see the finger, monsieur? – Yes, I see the finger. – Do you see the forefinger, madame? – Yes, I see the forefinger ….” (Sauveur, 1874, p. 10 as cited in Howatt, 1984, p. 200). I think I’ve done that lesson myself – haven’t we all? In fact, I’m sure I’ve told hundreds of trainee teachers that the first place to look for visual aids is in the mirror. Does this make me a bit of a TEFLosaurus myself – not strictly “Direct” but, at the very least, Direclectic?

About the Direct Method itself, Howatt charts the different stages of its evolution and outstanding success. Howatt explains that the term ‘direct method’ seems not to have been coined by anyone in particular, but to have “emerged (rather like our contemporary ‘Communicative Approach’) as a useful generic label to refer to all methods of language teaching which adopted the monolingual principle as a cornerstone of their beliefs” (1984, pp. 207-8). One interesting fact that I’d forgotten was that, apparently, Berlitz himself never referred to the Direct Method – preferring the term Berlitz method. I suppose the generic nature of the term ‘Direct Method’ ruined it for marketing – brands need ownership and boundaries. If “a language is a dialect with an army and a navy” (Weinreich, 1945, p. 13), then perhaps a language teaching Method is an approach with a lawyer and a PR rep!

It must have been the huge success of Berlitz (nearly 200 schools worldwide by 1914), that propelled the term ‘method’ into its lengthy commercial life in ELT. It’s been tacked onto any number of language school names to suggest the uniqueness of the experience on sale within. Judging by the Callan Method, the Schenker Method etc etc, you might assume the world was awash with innovative, unique ways of learning English. Mind you, a similar interpretation of ‘Whoppers’ ‘Big Macs’ etc available on any High Street, might lead you to conclude that we benefit from a rich and varied diet! If only!

Be that as it may, I suspect that the commercial allure of the term ‘Method’ has long since worn off. Marketing loves language to death. Having consumed ‘method’, marketing seems to have switched its affections to ‘solution’ (if the English magazine Private Eye is to be believed – it runs a weekly column on the subject!). Soon, if not already, we can expect to see: MySpeech: a multilevel, multimodal organic language Solution! You read it here first!

Howatt, A.P.R. (1984) A history of English language teaching. Oxford: Oxford University Press (NB an updated second edition is now available) Weinreich, M. (1945). YIVO and the problems of our time. Yivo-bleter 25
(1)
(I found this reference at: http://www.edu-cyberpg.com/Linguistics/armynavy.html – which also gives the original Yiddish version: “A sprakh iz a diyalekt mit an armey un a flot”

>Welcome! TEFL EDU is a blog for English teachers and administrators in ESL and EFL workplaces, including pre-K-12, 2- and 4-year institutions of higher learning, and adult education. Each of these areas has teachers with a wide range of experience and expertise, making for a broad and diverse mindset.