Evaluating CALL courseware: My thoughts on Hubbard's (1988) framework

Just as I was getting to grips with the some of the ideas presented in Levy's (1997) article regarding the CALL courseware development process that we read about in Week 5, I came across the word 'evaluation' in the title of this week's reading article and the first thing that came to mind was "Oh no, not now, please." (lol) You see, up until a little while ago, I still hadn't decided which piece of technology or courseware would form the focus of my EN6482 assignment let alone think about how I would develop and implement it. So, to think about 'evaluation' seemed like, at first, a step ahead too far for me. Later I realized, however, that I had misinterpreted 'evaluation' as meaning the evaluation of students' learning as result of CALL courseware implementation instead of the courseware itself. (Obviously I should have read the title of Hubbard's article more carefully!).

Let's be honest, when it comes to evaluating CALL courseware, no evaluation scheme can possibly be more comprehensive and more flexible than the evaluation framework that was put forward by a linguistics expert from Stanford University named Philip Hubbard (1988) more than two decades ago. It is comprehensive in that it contains sections that cover every possible angle as far as the evaluation of computer-assisted language learning and teaching is concerned, including 'operational description', 'learner fit' and 'teacher fit', all of which themselves have a number of distinct components which need to be looked at in any courseware evaluation procedure. The evaluation framework is also flexible because, as Hubbard explains, it provides the tool through which the courseware evaluator can create his or her own questions or build some other evaluation scheme according to the evaluator's needs.

Some courseware evaluators may be put off by the apparent complexity of Hubbard's evaluation framework (with all those arrows and boxes drawn in) and may also harbour worries about the the length of time it takes to evaluate a single courseware package. "Fear not," I would say to them, for the author provides the reassurance that it is only those that need to compose reviews of a package that doesn't appear to be suitable who need to go through the full evaluation procedure. Ordinary teachers like you and me just need to use the framework as a guiding tool to quickly weed out courseware packages that do not fit the bill. As Hubbard's mentions, even if we just address the question of whether the courseware fits our students' needs and interests, it will go a long way toward making an informed decision. Flexible it is indeed.

Second Life

I was one of the unfortunate few who could not get Second Life to run properly on the computer last Monday, so I am not in the best of positions to comment on its suitability for use in the language classroom. However, having witnessed it running smoothly on Rubi's PC, my first impression was that this kind of virtual world is nothing short of amazing in terms of its scale and the vast array of things that your avatar can explore and do in it. If I were to use Second Life with my upper primary students, whose English proficiencies are well below the territory average, I would introduce its different elements sloooowly over a period of time, perhaps in my literacy workshops during the post-examination period.

For starters, I would assist my in creating a single avatar representative of the whole class because it would be easier in terms of classroom management than allowing every student to have their own. Of course, my students would very much prefer to have their own avatar given the choice but I am little concerned about the steep learning curve they would have to go through individually and the fact that some (especially the boys) might choose to meander off task. Creating an avatar can be an English learning opportunity itself - you could get students to verbalize the reasons for wanting to choose a particular last name or starting look (using adjectives to describe appearance or quality), for example. Once the avatar is co-created, the students, with expert guidance from the teacher, could begin exploring one of islands. As we discovered last Monday you can easily lose track of what you're supposed to be doing in this virtual world so there needs to be a concrete task for the students to complete and it needs to be explicitly stated in both spoken and written form.

There has to be a million and one ways to exploit the Second Life virtual world for language learning but my brain is obviously nowhere near operating on all four cylinders on this Chinese New Year's day (by the way, Kung Hei Fat Choi to you all!) I guess the first thing that comes to mind is to have the students explore together an island that you have already taken the time to explore yourself and then get them to provide a running commentary of what they are seeing or write about what they have seen. Or set them the task of finding certain things in certain places e.g. find three things that you would like to see in real life in the Shakespeare's Globe Theatre and describe them. (I think somebody else might have already mentioned this.) It goes without saying that Second Life provides a natural and engaging stimulus for students to speak and write. I think I'm sold.

Theory-driven CALL and the development process by Levy (1997) - a reflective summary

In his article, Levy (1997) draws from a number of theoretical frameworks underlying CALL projects as well as the findings of an international CALL survey to explore the nature of theory-driven CALL.

According to the writer, there are two main and distinct groups of CALL practitioners. Those who believe that completed research is necessary for successful CALL development belong to the group known as formalists and they tend to make theory the starting point of any project. Those who believe that research need not be completed right before the start of the development process but can be integrated into that process belong to the group known as proceduralists and they tend to make discoveries as they go along.

One problem with the formalist view is that, since there exists a plethora of CALL theories influenced by a diverse range of disciplines, seldom do we find that research based on a particular theory is complete. In fact, theories can and do undergo constant revision during the course of CALL development, often because they haven't yet fully encompassed the CALL context at hand. Thus, one has to question the need for any CALL development project to be driven by theory.

On the other hand, proceduralists, whose CALL development projects are mainly technology-driven, need to be careful not to become totally fixated on the capabilities of a particular technology and neglect research altogether, because the latter can play a vital role in their projects by providing new theoretical insights into their own work.

In a nutshell, the best CALL development approach for CALL practitioners to adopt is one that proceeds in a circular, rather than linear fashion but places equal significance on theory (of instructional design, teaching or learning) and application (of a particular technology). CALL practitioners should strive towards the desirable goal of establishing a successful fit between the two.

Despite the fact that both the current capabilities of technology and language pedagogy have evolved considerably since the publication of Levy's article thirteen years ago, I believe his notion of 'fit' is relevant more so now than ever. Although I am not an avid CALL practitioner by any means, judging from my previous experiences of using IT in language teaching (e.g. creating word puzzles from online puzzle generators) I would consider myself to be a proceduralist with very little consideration for theories of language pedagogy. The essay written by Levy (1997) serves as a timely wake-up CALL (excuse the pun), because it has already increased my awareness of the need to consider both theory and application in equal measures when I am developing CALL materials for my students.

I end this post by posing a question for the writer. If the international CALL survey were to be conducted again today (almost two decades after it was conducted the first time) how do you think the results would turn out? I just thought this might be interesting.