Summary: Weeks seven and eight - negotiate and write an evaluation plan,

Image: Up In Flames by Ted Percival

Evaluation plans are swirling like lonely cinders as Margaret Mahy said in her story about the witch and the cherry tree.

Catherine's most recent post has brought something very important to the forefront about customising the eLearning guidelines to suit your project. Remember you can write your own elearning guidelines to suit the purpose of your evaluation. I have given Catherine some feedback on her draft ideas for eLearning guidelines. I have added an example of a guideline written from "scratch". Once you settle on your eLearning guidelines you will need to write some sub-questions for each - 1 to 4 is a good number. This way you will be able to get the answers you need - remember to get the right answers you have to ask the right questions. Please look at my feedback to Catherine for some examples of what I mean by this. Don't be put off by the length of my feedback. You also need to be very clear about the purpose of your evaluation - if you write down what you are trying to find out, this will help to clarify whether the guidelines you have chosen are appropriate.

Adrienne's plan for an effectiveness evaluation is shaping up nicely. The guidelines she has chosen are excellent and very relevant and just need some sub-questions. For example for TD4 - What makes an effective online discussion? Example sub-questions: 1. Which types of online discussion promote learning? This will enable her to look at the email discussion forum, discussion on the course blog and participant blogs. 2. what are the features of an engaging interaction?

Decisions and outcomes
Remember everyone your decisions must be directly related to the outcomes of the evaluation, that is the answers to your questions. For example, in Adrienne's case, if the use of participant blogs inhibits regular interaction between course participants is because they prefer to have face-to-face or email discussions, then Adrienne will have found out that blogs may not be a good way to promote discussion and email is, therefore as one of the outcomes may be to find the best means of communication, the recommendation may be todiscontinue blogs for communication and use email.

Minhaaj has made a good choice with regard to summative evaluation and the Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm for your evaluation project. It is refreshing to see his own beliefs (paradigm) regarding statistics coming to the fore. He now has a plan for evaluating the Flat Classroom project based on a qualitative approach where he will interview participants who have been involved in a collaborative learning environment.

Kay has made a change to an elearning guideline and asks some pertinent questions about academic process. Her approach is quite different as she is going to be testing criteria to measure quality, as well as getting students to look at usability.

Hervé on an earlier post in two parts has provided an excellent background to his project, proposed methods and articles. He has posted his draft plan as well for a formative evaluation - great guidelines - and is seeking feedback.

Joy has posted her guidelines and information about her ideas for an effectiveness evaluation. Once again I am suggesting some sub-questions. For TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?) Sub-question: Are learning tasks relevant and do they motivate learning? To help more with this it would be good to know more about the course and the aspects she wishes to evaluate. I wonder, are there particular areas of concern - such as too much content, too little communication between students and lecturer, is the assessment working well? etc.

Debra M has some great sub-questions accompanying her guidelines. She has been encouraged to keep the project small with a handful of students and an expert reviewer. Her next step is to develop some guiding questions for the reviewer. Her evaluation will be looking at usability.

Heather has been working hard over the weekend (along with several of you) and now has her draft plan posted. Some really good thinking emerging around the needs assessment she intends to carry out with students - investigating the introduction of a web-based "game-like" quiz into a written communication course - along with other strategies.

I have mentioned a possible "fish hook" which may concern others as well. Is getting information easier when you have a bond with participants in the evaluation or are they just telling you what you wish to hear? Anonymous surveys can alleviate this to some extent and encourage honesty.

Krishan is also looking at conducting an effectiveness evaluation and may have some interesting findings considering the course - Operate a Personal Computer - which no longer relies on a textbook as all the resources are now online.

Michelle also has a draft plan posted for a needs assessment of an online software training course to find out if an eLearning approach is going to be feasible for the learners. She has posted an excellent example of eLearning guidelines with sub-questions. It looks like Michelle has her work cut out preparing a sample course for staff to evaluate using usability criteria.

Pradeep has a draft plan for a needs assessment to introducing eLearning into the mechanical engineering course. A great variety of approaches to gather data including student focus groups and survey and expert review. It is good to see his notes under the guidelines. He has also outlined on another post some of the issues for introducing eLearning and the student-centred focus in his organisation. There is also an interesting overview of an article about an engineering evaluation project (EASEIT-ENG Project) on another post.

Rachel has some excellent ideas for her evaluation; a needs analysis associated with teaching hotel management. As part of this, he is looking at strategic documents for the organisation and interviewing "experts" so she can investigate the best direction for introducing flexibility. At this stage, time constraints will make it difficult to gather data from students. of course this can be done outside this evaluation project.

So we have a variety of evaluation project approaches: effectiveness evaluations, needs assessment - with some focussing mainly on students' perceptions and others looking at strategic documents and expert opinion. We also have a few usability evaluations for new online courses and also formative evaluations. Well done everyone! I am impressed with your activity and hard work. Now you can get on with the fun part - talking to people and gathering the data. It is always a bit like xmas finding out what others think of your ideas and approaches.

Some others are not mentioned here but have been working on their ideas and I am sure they will appear very soon.

Weeks 7 & 8: Negotiate and write an evaluation plan

Image: Pink Ball by DeclanTM

Welcome back. I hope you managed to have some relaxation time over the holiday break.

Over the next two weeks you will need to firm up your evaluation planning and prepare a formal plan - information about a template is in the Things to Do list below (taken off the wiki). By now you should have a pretty good idea of what you intend to evaluate. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment. And do contact me with your ideas and draft plan before submitting this part of the assessment as we need to negotiate your plan first. Read on for the fun stuff....

Guest speaker: On Tuesday 21 April 19.00-20.00 NZST - Helga Wientjes from the Department of Conservation presented to the class. Here is the Elluminate recording for your viewing pleasure. - she did a wonderful evaluation project for her workplace when she was taking the evaluation of eLearning for Best Practice course in 2008. It was a formative evaluation to measure effectiveness of a web conferencing training module.

Things to do:

• Decide on two eLearning guidelines against which you will evaluate.
• Establish the paradigm and model for your evaluation.
• Look at this example of an evaluation plan - Usability evaluation
• Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
  • You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
  • Remember this can be modified to suit your situation.
• Explore the Evaluation Cookbook to decide on some sampling methods for your project.
• Prepare some draft ideas and post them on your blog.
• Contact your lecturer to negotiate your plan.
• Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Summary Weeks five and six: Evaluation methods

Image: Eggs-ibit 2004 by annessasgirl

Well it looks like - needs analysis/assessment/Front End analysis - might be a popular choice of evaluation method for several of you. This has come up in some conversations on Elluminate and some phone conversations I have had. This is a logical way to go, when you are thinking about introducing eLearning into your teaching as opposed to actually already being in the throes of using eLearning.

Some people are also thinking about Effectiveness evaluations as well, but most are still trying to clamber over the paradigm hump. I hope you can all leave this behind after Easter and focus on the type/method of evaluation and the ways in which you will collect your data. - to fit with the eLearning guidelines you have chosen. Watch this space for a link to an evaluation plan template.

You will all be pleased to hear that the MIT library is getting the course textbook back and it will be on desk copy. I also have some chapters I can send out if you need something let me know on email after you have looked at the Table of Contents. Use the library distance services in the first instance.

Debra M has found an article which helped her sift through the paradigms: Phillips, R. (2005). We can’t evaluate e-learning if we don’t know what we mean by evaluating e-learning! Interact, 30, p 3 - 6. Learning Technology Support Service, University of Bristol.
It is easy to read and helps everything fall into place a lot easier.

Sam found some wisdom about evaluation following an altercation with a fierce mother hen and a shopping trolley. "Listening to the ideas of others and sensitivity to others’ parameters for success." - very relevant in evaluation.

Joy has plans for a mixed methods approach which looks suspiciously like an Effectiveness Evaluation as she intends to measure whether learning objectives are being met. Here is a link to an example of an Effectiveness evaluation plan by Hilary from the 2008 class.

Michelle has also chosen a mixed methods approach. More to come soon.

Rachel has written a very thoughtful and comprehensive post about her beginning ideas about conducting a needs analysis. She has invited feedback and it would be great to see you giving her some help. The question is - should she sample students who know little about eLearning or would she be best to investigate support systems in place for students who will have to access eLearning? This would fit with one of her chosen guidelines: ST1 - Do you have a way to identify needs and respond to them?