Evaluating CALL courseware: My thoughts on Hubbard's (1988) framework

Just as I was getting to grips with the some of the ideas presented in Levy's (1997) article regarding the CALL courseware development process that we read about in Week 5, I came across the word 'evaluation' in the title of this week's reading article and the first thing that came to mind was "Oh no, not now, please." (lol) You see, up until a little while ago, I still hadn't decided which piece of technology or courseware would form the focus of my EN6482 assignment let alone think about how I would develop and implement it. So, to think about 'evaluation' seemed like, at first, a step ahead too far for me. Later I realized, however, that I had misinterpreted 'evaluation' as meaning the evaluation of students' learning as result of CALL courseware implementation instead of the courseware itself. (Obviously I should have read the title of Hubbard's article more carefully!).

Let's be honest, when it comes to evaluating CALL courseware, no evaluation scheme can possibly be more comprehensive and more flexible than the evaluation framework that was put forward by a linguistics expert from Stanford University named Philip Hubbard (1988) more than two decades ago. It is comprehensive in that it contains sections that cover every possible angle as far as the evaluation of computer-assisted language learning and teaching is concerned, including 'operational description', 'learner fit' and 'teacher fit', all of which themselves have a number of distinct components which need to be looked at in any courseware evaluation procedure. The evaluation framework is also flexible because, as Hubbard explains, it provides the tool through which the courseware evaluator can create his or her own questions or build some other evaluation scheme according to the evaluator's needs.

Some courseware evaluators may be put off by the apparent complexity of Hubbard's evaluation framework (with all those arrows and boxes drawn in) and may also harbour worries about the the length of time it takes to evaluate a single courseware package. "Fear not," I would say to them, for the author provides the reassurance that it is only those that need to compose reviews of a package that doesn't appear to be suitable who need to go through the full evaluation procedure. Ordinary teachers like you and me just need to use the framework as a guiding tool to quickly weed out courseware packages that do not fit the bill. As Hubbard's mentions, even if we just address the question of whether the courseware fits our students' needs and interests, it will go a long way toward making an informed decision. Flexible it is indeed.

Second Life

I was one of the unfortunate few who could not get Second Life to run properly on the computer last Monday, so I am not in the best of positions to comment on its suitability for use in the language classroom. However, having witnessed it running smoothly on Rubi's PC, my first impression was that this kind of virtual world is nothing short of amazing in terms of its scale and the vast array of things that your avatar can explore and do in it. If I were to use Second Life with my upper primary students, whose English proficiencies are well below the territory average, I would introduce its different elements sloooowly over a period of time, perhaps in my literacy workshops during the post-examination period.

For starters, I would assist my in creating a single avatar representative of the whole class because it would be easier in terms of classroom management than allowing every student to have their own. Of course, my students would very much prefer to have their own avatar given the choice but I am little concerned about the steep learning curve they would have to go through individually and the fact that some (especially the boys) might choose to meander off task. Creating an avatar can be an English learning opportunity itself - you could get students to verbalize the reasons for wanting to choose a particular last name or starting look (using adjectives to describe appearance or quality), for example. Once the avatar is co-created, the students, with expert guidance from the teacher, could begin exploring one of islands. As we discovered last Monday you can easily lose track of what you're supposed to be doing in this virtual world so there needs to be a concrete task for the students to complete and it needs to be explicitly stated in both spoken and written form.

There has to be a million and one ways to exploit the Second Life virtual world for language learning but my brain is obviously nowhere near operating on all four cylinders on this Chinese New Year's day (by the way, Kung Hei Fat Choi to you all!) I guess the first thing that comes to mind is to have the students explore together an island that you have already taken the time to explore yourself and then get them to provide a running commentary of what they are seeing or write about what they have seen. Or set them the task of finding certain things in certain places e.g. find three things that you would like to see in real life in the Shakespeare's Globe Theatre and describe them. (I think somebody else might have already mentioned this.) It goes without saying that Second Life provides a natural and engaging stimulus for students to speak and write. I think I'm sold.

Theory-driven CALL and the development process by Levy (1997) - a reflective summary

In his article, Levy (1997) draws from a number of theoretical frameworks underlying CALL projects as well as the findings of an international CALL survey to explore the nature of theory-driven CALL.

According to the writer, there are two main and distinct groups of CALL practitioners. Those who believe that completed research is necessary for successful CALL development belong to the group known as formalists and they tend to make theory the starting point of any project. Those who believe that research need not be completed right before the start of the development process but can be integrated into that process belong to the group known as proceduralists and they tend to make discoveries as they go along.

One problem with the formalist view is that, since there exists a plethora of CALL theories influenced by a diverse range of disciplines, seldom do we find that research based on a particular theory is complete. In fact, theories can and do undergo constant revision during the course of CALL development, often because they haven't yet fully encompassed the CALL context at hand. Thus, one has to question the need for any CALL development project to be driven by theory.

On the other hand, proceduralists, whose CALL development projects are mainly technology-driven, need to be careful not to become totally fixated on the capabilities of a particular technology and neglect research altogether, because the latter can play a vital role in their projects by providing new theoretical insights into their own work.

In a nutshell, the best CALL development approach for CALL practitioners to adopt is one that proceeds in a circular, rather than linear fashion but places equal significance on theory (of instructional design, teaching or learning) and application (of a particular technology). CALL practitioners should strive towards the desirable goal of establishing a successful fit between the two.

Despite the fact that both the current capabilities of technology and language pedagogy have evolved considerably since the publication of Levy's article thirteen years ago, I believe his notion of 'fit' is relevant more so now than ever. Although I am not an avid CALL practitioner by any means, judging from my previous experiences of using IT in language teaching (e.g. creating word puzzles from online puzzle generators) I would consider myself to be a proceduralist with very little consideration for theories of language pedagogy. The essay written by Levy (1997) serves as a timely wake-up CALL (excuse the pun), because it has already increased my awareness of the need to consider both theory and application in equal measures when I am developing CALL materials for my students.

I end this post by posing a question for the writer. If the international CALL survey were to be conducted again today (almost two decades after it was conducted the first time) how do you think the results would turn out? I just thought this might be interesting.

Language Learner Behaviour in a Virtual Environment - A Reflective Summary

Roed (2003) investigated the behaviour of university students of Danish while they were engaged in a communicative language task within a synchronous virtual learning environment (i.e. chat room). She found that the task resulted in changes in behaviour; students who were normally very quiet in class, for example, behaved differently in the chat room, playing a considerably more active role. She observes that this kind of environment helps to reduce language anxiety and attributes this to the fact that computer-mediated communication (CMC) creates a rather anonymous environment where the computer serves as a shield from being onstage. The benefit of CMC, she notes, is its ability to function as scaffolding: 'gradually giving anxious students more confidence to embark on conversation in the target language'.

The author also stresses the fact that when operating in virtual environments, not only does learners' behaviour changes, but also the perception of learners' behaviour. Thus, it is possible for normally reserved students, for example, to appear to behave in a brash manner in the eyes of other chat room users. Online language learning certainly has its fair share of disadvantages as well as advantages.

Roed's findings do seem to support the view that online chatting facilitates language learners to communicate as a result of lowered self awareness while engaging in public discourse. However, in my own teaching context (Chinese ESL students, many of whom are shy, passive learners with low English proficiency - in a small primary school in the New Territories) a significant number of students would have to face the double handicap of having only limited vocabularies and limited keyboard skills. I would imagine that such students would go off task easily. Therefore, if I were to set up an online communicative task for my students to complete, in a similar manner to that described in Roed's (2003) article, it would have to be well-planned with clear and achievable goals laid out. Appropriate scaffolding according to my students' linguistic and technical needs would also be necessary. Yet, despite taking these measures, I would still need to address the issue of catering for learner diversity. How, as a teacher, would I be able to keep students of a wide range of abilities happy and ensure that they participate in equal amounts without constantly intervening and interfering? I suppose that for the latter I could always tell my students at the beginning of the task that every one of them has to at least post a certain number of posts (either questions, answers or comments) to their peers. But in terms of keeping everybody happy, no matter what the lag time is between hitting the enter key and the appearance of text on the screen, there will always be those students who become easily frustrated by a sluggish system or the failure of others to respond promptly to their posts. On the other hand, there will be other students who become overwhelmed by the pace of it all.

Nevertheless, I do think that, given their greater linguistic and technical experience, my older students (say at P6 level) would gain more from the experience of being immersed in virtual environments than the younger ones. I would certainly not rule out the possibility of trying out online chatting in English with my older students now that I am aware of some of the drawbacks of the approach. [Update: for a different slant on the issue of using online communication tasks with primary students, click here.]

Dotsub - alternative to Yappr

Aha, I've just found out from Nik Peachey's blog that Dotsub is similar to Yappr but you don't need to pay a subscription fee in order to input subtitles beneath video clips i.e. students can transcribe video clips for free! One idea of Nik's is really fab - you can ask students to create their own video clips (with digital cameras or webcams), upload them to Dotsub and then get other classmates to transcribe them.
I would add that you could then get students to embed these transcribed videos to their own blogs and then evaluate each other's contributions through the commenting feature.
Nik's idea could easily translate into a group project which has enormous potential for developing students' all-round English skills. Just think of the possibilities! The video clips could be based on a topic or based around a target language structure that is currently being covered in the syllabus, thus reinforcing what they learn (if indeed, they do learn!) from their (may I dare say, mundane) textbooks. I think my P6 class could handle this project as long as I put them into mixed-ability groups (interestingly enough, those that I consider to be tech-geeks also happen to be above-average in English - I know who they are because I did a survey with my class when they were in P5 last year). However, one problem that would most likely occur in this scenario would be the unequal contribution by individual members of a particular group. Or, even if the students did take equal turns to transcribe a video clip, how would you deal with the technologically or linguistically inept? One would have to figure this out in advance but for starters teachers running this project would no doubt need tonnes and tonnes of patience!

Yappr - Transcribed Videos for ESL

Just found this link via Nik Peachey's quickshout blog which may be of interest to some of you (if you haven't visited the link already, that is). Useful albeit small selection of free video clips of different genres to suit students of all ages and abilities. Every clip has an accompanying script which appears in a window alongside, to enable students to listen AND read while watching the clip. Have a look at http://en.yappr.com/welcome/Video.action?videoGuid=10CEB2F8-8FB4-4225-8F06-54713BB83B7C to see exactly how it works. A lot more clips are available upon paying a subscription fee. I can't wait to try Yappr out myself!

Internet for English Teaching (Warschauer, Shetzer & Meloni)

This article introduces a set of pedagogy-related principles which Warscauer et. al. believe can help educators to effectively incorporate the Internet into the language classroom. The authors classify these principles under 'Learning Goals', 'Teaching Guidelines' and 'Planning Tips'. In terms of Learning Goals, they say that teachers should help students

• master new technologies in active and creative ways
• develop autonomous learning strategies
• learn collaborative communication and teamwork skills
• learn in a variety of cultural contexts
• develop critical thinking skills

The Teaching Guidelines are intended to facilitate the achievement of the above Learning Goals and the authors classify them under five headings, namely

• Dual Immersion (where students are immersed in learning the skills of language AND technology simultaneously)
• Integration (where computers are fully integrated into the curriculum)
• Project-Based Learning
• Student-Centred Learning
• Learning With a Purpose

The list of planning tips is extensive but not exhaustive. Perhaps the most useful and relevant tip for me is the one about creating a friendly home page to provide low-ability ESL students a secure platform from which to begin their Web-browsing activities. Other useful tips include having students work in groups in a one-computer classroom and focussing on e-mail when students are conducting Internet projects in low-tech environments.

Warschauer et. al. end their article with an informative selection of sample web projects. The Primary School Webfolios seems to be a great idea and the tasks would appear to be perfectly achievable for my average P6 student. However, I would like to know how the teacher (Rachel Arenstein) actually handled her low-ability students who had a variety of learning needs such as those who had difficulties with spelling, little knowledge of reading/writing skills and strategies, etc. The E-mail Cultural Exchange Project is something that I have always wanted to set up and try out with my class, but I have been thwarted by my futile attempts at trying to find willing and suitable partners. In the past I have set up pen-based exchange projects in my school only to find that the correspondence dried up soon after set-up. The crux of the problem was that there had often been a mismatch between my students' levels of English and those of their pen-friends. Quite simply, you're not going to reply to somebody's letter if they appear to put less effort than you or if what they write intimidates you because of its scale and undecipherable meaning. Nonetheless, I can see from the example that the teacher has a crucial facilitative role to play which can make or break such a challenging project. Certainly provides some food for thought.

Somewhat unfortunately for me, the majority of the sample web projects provided by the authors are based on secondary or tertiary educational contexts which, in my opinion, seems to suggest that (at the time of writing at least) Internet-based teaching and learning in the English classroom could only be fully exploited if the students had at least several years of studying English. In other words, there is a threshold level of English beyond which students can truly benefit from Dual Immersion. Anybody would like to challenge me on this?