iBT Speaking and Writing - how do they stack up to the real thing? (Part 1)

As an English language educator based for a long time in South Korea, I have to confess to being rather heartened when rumors began to circulate in the market about the arrival of iBT - a new "next generation" Internet-based version of the TOEFL.

Having gone through the experience of preparing Korean teenagers for TOEIC, then CBT (Computer-Based TOEFL) - as the market swung around somewhat chaotically through such test fads - I was looking forward to the idea of more integrated skills and equal emphasis (in terms of points) for the four basic skills. Tests are everything in this context (as they are in many others), so in many ways what we as teachers get to apply in class and how we are judged depends greatly on the 'tests of the day.' To be quite frank, I am a very firm believer in communicative and task-based language learning, and preparing students for TOEIC and CBT almost made me want to give the profession away at times. Students stopped speaking - because they didn't need to. The tests, and as a result the whole general market, were telling them that it was okay not to speak or communicate. Just study, memorize, and analyze in your L1 until you have enough patterns and rules wedged in your head to ace the test on the day. I did achieve considerable success in the field, with several students obtaining perfect TOEIC scores and many with high CBT scores (including one perfect 300), but I was becoming a trainer - not an educator. I always walked away from these classes feeling that whatever success had been achieved had been somewhat shallow and was coming at a price that students and the market/country in general would pay for later.

So suffice to say I was pretty enthusiastic about the arrival of the iBT. I started checking out the speaking side of things a good three years ago, and I wasn't all that shocked to see that the speaking portion of the iBT was pretty much a re-jigging of the existing TSE (Test of Spoken English) already offered by ETS as a stand-alone component. The new writing task on the iBT - integrated writing - also appears to be an extension into writing from the pre-existing question 4 of the TSE. So the new portions of the test hardly came out of the blue, and a well-connected colleague of mine assured me that ETS had actually been trying to get the iBT going for some 10 years or more before they finally announced it officially.

So given the considerable time for development and research and pre-testing, a huge question now is: how well do these speaking and writing tasks stack up to what might be expected of foreign students studying at English-speaking universities? The more complicated way of asking this in testing terms is: how valid is the test? Do the speaking and writing portions of the iBT test what they claim to test - the academic and real-world proficiency in speaking and writing required to be able to handle study at tertiary level in an English-speaking context?

I'll chew on this particular pie piece by piece, shall I? For the moment, I am leaving alone issues like test reliability and timing/format. They are also interesting questions to pursue, but I'll do so in separate postings. For now let's just look at each task and think a little about (1) what it asks students to do, and (2) how well this relates to using speaking and writing skills in the 'real' college/university world.

Starting with the speaking tasks...

Speaking question 1 asks the test taker to talk about something based on personal experience. It is termed "independent open choice" because, based on a question, the student can choose what to talk about. It could be a person, place, thing or event that is familiar to the speaker. Okay - I have no major problems with this one. It is very general and I'm not sure exactly how it relates to university-oriented talking, but the generality of it is what makes it attractive to me. Official ETS guides for the test also encourage the idea of describing and giving reasons for the choice the speaker makes. Not bad in my mind - these are general discourse skills that will serve a student well not just in a tertiary setting but in all sorts of other situational settings as well. I also think the task is fairly appropriate as a test-opener. It is generally accessible and sends a nice message: you need to make your own choices, elaborate what you mean, and say why. Okay, I'm not falling out of my chair in admiration for the task, but I'll take it.

Speaking question 2 is another independent task, but now involves a 'closed choice'. That is, here are two options. Choose one and explain why you prefer it. I like this one as well. Very general, and therefore has a wide range of situational and communicative applications. It could have relevance to talking with friends about plans, but it could also have application in tutorials, for example, where students are encouraged to have opinions and be able to back them up from an argumentative point of view. Keeping the question general and personal gives it good scope, with the speaking and language selection skills involved having a nice wide range of potentially possible situational applications. I'll take this one, too. I might even applaud softly at the choice.

Speaking question 3. Mmmmm. Read a campus-related notice. Listen to two students talking about the notice. One of the students feels strongly about the notice (in a positive or negative way). Say how the student feels and explain his/her reasons for feeling this way. Mmmmm. Okay, I'll be honest. Question 3 started off with the right idea and then went off somewhere into la-la land. Sure - looking at campus notices is realistic enough. Yes, it's reasonable to expect to hear students talking about it and having positive or negative reactions. So why now does the test taker have to 'report'? Is it so that they can tell a friend about a notice they saw and then pass on the crucial additional information regarding two complete strangers talking about it, with one of them having a particular reaction of some kind? I'm sorry, but I can honestly say I never needed to do this in all my days on campus, and I cannot for the life of me see why a foreign student would need to talk about it either. Yeah sure - we could realistically expect to pass on information we read from a notice, or heard other students talking about, especially if it is important. But why the idea of passing on the unknown student's opinion and reasons? I think this could have relevance as a listening task, in that listening to other students talking about notices could help a foreign student better grasp the message's meaning and gravity. But, aside from the fact that as far as I am aware there is a very well catered-to section in the iBT for pure listening skills, I'm left with the feeling that a positive chance to speak productively here has been lost. Why not have the test taker paraphrase the information in the notice to a friend, or weigh in with his/her own opinion in relation to the notice? Okay, the context may be too specific and the speaker doesn't know enough to reliably form his/her own opinion. However, a very valid speaking application here could be for the speaker to ask additional questions about the notice, or to make a prediction about how other students might be affected or feel about the announcement. To me the task as it stands is basically 'integration' of reading and listening for the sake of it. While the setting and situation are potentially viable, what the student needs to do in response is a bit out there. In all, speaking question 3 needs some adjustment before it can expect my vote of approval (and as you can no doubt see, ETS never did nor ever will ask people like me to approve their test items!).

Speaking question 4 involves reading some academic material, then listening to a lecturer talk as a follow up to the reading. The idea is to link the two inputs and identify how the lecture relates to the reading (illustrating or challenging it in some way), and to then explain this relationship with supporting details and connections. Nice job on this one - it's very relevant to a college or university setting. It does, in fact, resonate with many of my own experiences as a university student, where what you read in texts is not always the full or even entirely accurate story. Now, while I sympathize with the poor Asia-based learners, many of whom have spent their entire lifetimes up to this point in a transmission-based education model where everything is poured into their heads and memorized for unchallenged regurgitation - usually directly out of a textbook, things are generally (or at least should be) different in western tertiary settings. If they want to learn in that environment, they need to be capable of handling readings and lectures that either (a) follow up with a real world illustration or example of some kind, or (b) directly challenge it on the basis of alternative evidence or findings. AND, students will need to be able to see the nature of the connections between the two AND report on this - hopefully with a view to discussing it in more detail. I'm pretty strongly for this one. In my mind speaking question 4 starts to press the right buttons.

Speaking question 5. Mmmmmm. You've heard me 'mmmmm' in scepticism before and I'll 'mmmmm' again here (but maybe not quite as loudly this time). The idea for this one is to listen to two students talking about a campus/study-related problem one of the them has. In response to details about the problem, the other person makes a couple of suggestions about actions that could be taken to address the issue. The test taker now needs to relate the problem, the two solutions offered, choose one of the two solutions, and say why he/she prefers it. As per speaking question 3, I'm not against the situational context presented here, or for that matter the language involved (and this one has the more attractive requirement of a personal opinion to be expressed), but I do scratch my head at the prospect of having the speaker basically report the information and apply their own opinion in a very unfocused (even abstract) way. Why does the speaker have to talk about the other students as complete strangers and deliver their own opinion to what is basically Internet ether? Is this another example of where ETS expects foreign students to stand around campus listening to native speakers talk about something so that they can then run off to tell someone else about it? A very simple change to the requirements of the task would appease me, that being that the speaker needs to suggest a third alternative directly to the person with the problem, OR - the speaker weighs into the conversation and tells the person with the problem which of the solutions suggested sounds better and why. These sound to me to be more realistic and useful sorts of spoken language application given the situation as it is presented (though I confess we may need to plug the idea the student is talking with fictional on-campus friends). It would be even better if the conversation wound up with one of the students 'turning' to the test taker and saying "well, what do you think?".

Question 6 in the speaking section involves listening to a longish lecture, then summarizing and paraphrasing the key points from it in response to a targeted question. I like this one as well - very valid when I recall tutorials following up from lecture hall deliveries, or sitting around in the library either discussing a lecture with classmates or telling the friend who didn't go to the lecture (because he or she was too hung-over at the time... and yes, I confess, sometimes that was me listening to other friends recounting the missed lecture).  All in all, I'm very happy to see this task included in the iBT.

So, doing the math, you can see the iBT speaking section gets roughly 4 votes out of 6 from me. It is slightly more complicated than that, however. I have problems with the fact that nowhere in the speaking section are test takers encouraged to frame formal questions/inquiries, make requests, or offer direct suggestions or interpretations, all of which I think are absolutely essential language skills in spoken discourse - especially in an academic setting, and especially for 2nd language learners who ought to be encouraged to ask for or offer things like additonal explanations and clarifications (amongst a host of other things). There is too much abstract reporting that doesn't seem to have a realistic and/or useful purpose, and not enough basic situational or even academic interaction. The iBT is testing speaking skills, just not enough of the ones that could or should really count.

As for writing, well I've run out of steam here. I think I'll tackle that one in a separate post sometime, and settle for calling this Part 1 of 2...

Comments

[Eric Roth]

Excellent critique!

Teaching in Los Angeles at an elite private university (USC), I have frequently encountered students who "passed" the TOEFL and struggle with essential academic tasks. What did ETS spend all that money on?

[Alex Case]

I totally agree with Eric (even though he still owes me a review). How do ETS manage to put all that time and effort into things like updating the TOEIC and in no way come out of it with a better test??

Loved this bit:

"Question 3 started off with the right idea and then went off somewhere into la-la land"

If you ever get the chance to see Grant Trew tear TOEIC apart, I highly recommend it

[Cumanda]

I think the Toefl is only a business. What goverment institution supervise them? Has ever one of the american Deans have reviewed the materials or tried taking the test?
People are so scared of looking "ignorant" that they don't have the courage to confront them, to ask them questions, to read the materials, to ask explanations why too many students "fail", what were the parameters and where such a big amount of money goes to. $170 each time!
That is a shame.
Americans are intelligent but they are helping to make rich to a few who don't own an explanation to anyone. Fooling to everybody with stupid tests that if they were to take them for sure will fail.