Summary of week two activity

This week several people have contributed to the discussion about the ADDIE model and quality. Views about the usefulness of the ADDIE model of education design have been mixed, although consensus appears to be that it can provide some guidance for design and development. However, theory is believed to be only a starting point, and the ADDIE model is really just a "project management tool" (Alages and Tom), and other factors need to be considered, for example, time, quality and budget" (Rosanne). An excellent point was made that it is important not to get too restricted by one model such as the ADDIE as flexibility is needed when thinking about design of online systems (Mark).

Also, some people have used the model in their design work (Rosanne, Alages), or something similar to it (Sandra H, Mark and Sandra W), and that it is best not to become too linear oriented when designing online learning experiences as it is a dynamic process (Alages). Awareness about the challenges in undertaking evaluation are apparent in the discussions, and it is an area which is often left because of time constraints. Also, the need for evaluating during the design and development etc. phases is regarded as important and mentioned by most people.

I have referred people to the PADDIE model of design which we use at Otago Polytechnic, and as you will see it is still a work in progress - I am too busy doing the actual projects to finish the theoretical framework.  A case in point. The information about the ADDIE model for instructional design provided by Alages has more information about how to use the ADDIE model - for each phase details are provided about the type of evaluation you might undertake.

Only two people are familiar with the eLearning Guidelines, and everyone thought they would be useful although having time to explore them could be problematic. Some people mentioned they used other benchmarks but did not provide information. At Otago Polytechnic we use the Good practice checklist for benchmarking quality in Moodle courses. It is based on others' work as listed on the document. To access this file, you need to login as a Guest. A five stage model for measuring organisational quality in eLearning which some of you may be interested in is the eLearning Maturity Model for NZ.

• Maturity Model Getting Started 
• Marshall, S. (2006). E–learning Maturity Model Process Assessment Workbook. There are five main areas for assessing quality and benchmarking eLearning:

1. Learning Processes that directly impact on pedagogical aspects of e-learning; 2. Development Processes surrounding the creation and maintenance of e-learning resources; 3. Support Processes surrounding the oversight and management of e-learning; 4. Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle; 5. Organisation Processes associated with institutional planning and management.

 Additional benchmarks for measuring quality can be found on the Quality Principles site.  Are any of you these resources familiar to you?

Summary Week 2: Quality and evaluation

Image: Butterfly My First by photogirl7
What a busy week you have all had. Everyone pretty much has their blog all set up and ready to go and several people have posted responses to the questions for Week 2: Quality and evaluation. Hence my summary has lots of substance and questions for you to ponder. I am very pleased with the progress everyone has made especially those of you for whom blogging is a new experience. It has been great to talk to several of you online or on the telephone.
The questions for your response in Week 2 were:
- Why is evaluation important to you and how do you define it;
- What sort of evaluations mentioned on the presentation are familiar to you already and why?
- Why is quality important in eLearning?
Answers to all these questions will vary depending on your personal views and perspectives and the context in which you work. For example, Adrienne's blog post is very relevant to her work, and she has provided us with a practical list of items which she considers when designing eLearning. There are some very useful things for you to look at when planning your evaluation projects. She also makes a very important point about the link between quality and student retention - a subject close to us all I reckon.

DebraM in her post has listed several of the evaluation items she was familiar with from the presentation and makes a good point about evaluation, "it can be used to 'pat yourself' on the back, as well as 'allowing room for improvement'. She has presented a very comprehensive discussion about quality - it is all in the "eye of the beholder" well the user's opinion anyway. And check out the first eLearning Guideline to be selected.

Sam in her post has offered a different slant on quality again and brings information about two different organisations and their views of quality to our attention - Concord Consortium versus Articulate. I love her discussion about the idea of rapid eLearning offerings and the demise of quality if we just "throw-up online" without taking time and care to provide a thoughtful product. Lots of ideas for debate in this post. See what you think.

Herve's post refers to the presentation as did DebraM, and he says the concepts about evaluation were all familiar. One thing I found useful from Herve's post was the perspective that evaluation in eLearning may be even more important than in a traditional classroom because the interactions are different. He and Joy are on a similar wavelength - do you agree with Herve and Joy?

Joy alludes to the way we evaluate constantly when teaching and others have mentioned this as well. She has also provided us with some definitions from the web as well as her own take on evaluation. Joy is a fan of the ADDIE model of design, and gives us a good overview of why it is a useful model to follow to ensure quality in eLearning. Joy, like Herve, also tells us how the different ways we interact online compared to face-to-face classrooms, can make it hard for lecturers to gauge how students are understanding or enjoying their learning. Therefore, evaluation is even more important in eLearning.

Iain like many of you, is focused on evaluating to ensure his students are getting the best learning experience possible. Yes we do want them to keep coming back for more don't we? I really like the description of the ways in which he uses learning outcomes as a focus at the end of each lesson. It encourages students to self-evaluate their understanding and also helps Iain measure how well the class is doing. Iain's company does a very comprehensive evaluation of student satisfaction and success and it sounds like evaluation is integrated into all areas of the student learning. Well done! This sounds like a great model. I will be following your trail with interest.

Rachel in her week 2 post, as well as some insightful statements about quality being necessary for a range of areas, e.g., needs assurance, cost effectiveness, student retention and positive outcomes for students, Rachel provides us with a thought provoking question - “Why is quality important in eLearning?’ – and ask is quality not important in all learning?, should there be a distinction." Good on you Rachel asking this extremely important question.

This has spurred me on to provide a lengthy reply. Quality as you mention Rachel has always been important and in all areas of education, not just eLearning. However, the reason quality became such a sticking point in eLearning harps back to the early days of online teaching. In the early days, people new to online teaching often took the pages and pages of lecture material they used in class and just "slapped" it online. Design principles were poorly adhered to (graphics, chunking of text, formats, navigation, instructions etc) and approaches were often inconsistent re tools, instructions, formats and layouts. Then the "bells and whistles" brigade came along and produced fancy multimedia which no-one could open (software issues) or you could only use with a Nasa-designed computer. :)

Now of course we are much better informed, but the dilemma in the online classroom which several people have mentioned, is the lack of personal interaction in an online classroom - lack of body language or facial expression, unable to observe students and answer queries immediately. And it is more difficult to teach unprepared online - how many of you experienced teachers have walked into a f2f classroom not as prepared as you would like, but comfortable in your knowledge and expertise on the subject and had a great session? I believe it is harder to do this online because people can get so lost and intimidated. They seem to like schedules, instructions, structure and clarity. Do you agree?

This means we have to be much more slick with the way we organise online courses. This is where evaluation in the design and development stages becomes so important - so we get the design of the online learning as good as possible before we impose it on students. And then we need to find out if the approach was effective. I guess it is not quality which is different but they way in which we ensure we are meeting particular standards or guidelines which help us produce quality learning.

Krishan's post also gave me plenty to write about. Krishan makes some very valid points. I agee that having materials with well written instructions is a very important aspect of quality. It is so easy to get lost online. I agree with him about the predominance of summative evaluation in education. In which case, the evaluation is far too late to keep at-risk students in a course. There is nothing more disruptive or soul destroying than a class of disgruntled students. Do you agree?

Krishan mentions his familiarity with evaluative observations - they certainly are an excellent technique for evaluating skills, however I still shake when thinking of being watched while I undertook nursing tasks.

In evaluation of multimedia including web-based courses, observation is a good way to evaluate the clarity of how materials are presented and laid out. By observing how students navigate through, and also how they use the materials, the bugs can quickly be located and fixed. What may seem to us to be an obvious pathway or icon to click is not always so to others. Do you agree?

Minhaaj in his week 2 post, mentions some very insightful ideas, and I certainly agree with his statement about the reason we need to evaluate: "without the evaluation process, learning will be restricted to a traditional non-interactive, one-way process which is not only monotonous but kills the enthusiasm and interest levels of participants." He mentions an evaluation rubric used in week 3 of a course development called Enhancing lessons, so participants could measure the quality of the materials they were creating. It is fair to say than some sort of structured approach to measuring quality is important if educational materials are to have credibility. I do wonder though whether a rigid evaluation framework stifle would creativity and innovation. Is there a balance and a compromise? I believe there is but it needs to be carefully considered - do you agree?

So in week 2 we certainly had a wide range of opinions about quality and evaluation. It was helpful that you shared your previous experience with evaluation and also the material you found when exploring the topic.

TIP: It is good to see references at the end of some posts - it is also good to have hyperlinks in body of the post especially when quoting or mentioning items you have obtained from your explorations and reading. Some of you are doing this very well.