Summary: Weeks seven and eight - negotiate and write an evaluation plan,

Image: Up In Flames by Ted Percival

Evaluation plans are swirling like lonely cinders as Margaret Mahy said in her story about the witch and the cherry tree.

Catherine's most recent post has brought something very important to the forefront about customising the eLearning guidelines to suit your project. Remember you can write your own elearning guidelines to suit the purpose of your evaluation. I have given Catherine some feedback on her draft ideas for eLearning guidelines. I have added an example of a guideline written from "scratch". Once you settle on your eLearning guidelines you will need to write some sub-questions for each - 1 to 4 is a good number. This way you will be able to get the answers you need - remember to get the right answers you have to ask the right questions. Please look at my feedback to Catherine for some examples of what I mean by this. Don't be put off by the length of my feedback. You also need to be very clear about the purpose of your evaluation - if you write down what you are trying to find out, this will help to clarify whether the guidelines you have chosen are appropriate.

Adrienne's plan for an effectiveness evaluation is shaping up nicely. The guidelines she has chosen are excellent and very relevant and just need some sub-questions. For example for TD4 - What makes an effective online discussion? Example sub-questions: 1. Which types of online discussion promote learning? This will enable her to look at the email discussion forum, discussion on the course blog and participant blogs. 2. what are the features of an engaging interaction?

Decisions and outcomes
Remember everyone your decisions must be directly related to the outcomes of the evaluation, that is the answers to your questions. For example, in Adrienne's case, if the use of participant blogs inhibits regular interaction between course participants is because they prefer to have face-to-face or email discussions, then Adrienne will have found out that blogs may not be a good way to promote discussion and email is, therefore as one of the outcomes may be to find the best means of communication, the recommendation may be todiscontinue blogs for communication and use email.

Minhaaj has made a good choice with regard to summative evaluation and the Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm for your evaluation project. It is refreshing to see his own beliefs (paradigm) regarding statistics coming to the fore. He now has a plan for evaluating the Flat Classroom project based on a qualitative approach where he will interview participants who have been involved in a collaborative learning environment.

Kay has made a change to an elearning guideline and asks some pertinent questions about academic process. Her approach is quite different as she is going to be testing criteria to measure quality, as well as getting students to look at usability.

Hervé on an earlier post in two parts has provided an excellent background to his project, proposed methods and articles. He has posted his draft plan as well for a formative evaluation - great guidelines - and is seeking feedback.

Joy has posted her guidelines and information about her ideas for an effectiveness evaluation. Once again I am suggesting some sub-questions. For TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?) Sub-question: Are learning tasks relevant and do they motivate learning? To help more with this it would be good to know more about the course and the aspects she wishes to evaluate. I wonder, are there particular areas of concern - such as too much content, too little communication between students and lecturer, is the assessment working well? etc.

Debra M has some great sub-questions accompanying her guidelines. She has been encouraged to keep the project small with a handful of students and an expert reviewer. Her next step is to develop some guiding questions for the reviewer. Her evaluation will be looking at usability.

Heather has been working hard over the weekend (along with several of you) and now has her draft plan posted. Some really good thinking emerging around the needs assessment she intends to carry out with students - investigating the introduction of a web-based "game-like" quiz into a written communication course - along with other strategies.

I have mentioned a possible "fish hook" which may concern others as well. Is getting information easier when you have a bond with participants in the evaluation or are they just telling you what you wish to hear? Anonymous surveys can alleviate this to some extent and encourage honesty.

Krishan is also looking at conducting an effectiveness evaluation and may have some interesting findings considering the course - Operate a Personal Computer - which no longer relies on a textbook as all the resources are now online.

Michelle also has a draft plan posted for a needs assessment of an online software training course to find out if an eLearning approach is going to be feasible for the learners. She has posted an excellent example of eLearning guidelines with sub-questions. It looks like Michelle has her work cut out preparing a sample course for staff to evaluate using usability criteria.

Pradeep has a draft plan for a needs assessment to introducing eLearning into the mechanical engineering course. A great variety of approaches to gather data including student focus groups and survey and expert review. It is good to see his notes under the guidelines. He has also outlined on another post some of the issues for introducing eLearning and the student-centred focus in his organisation. There is also an interesting overview of an article about an engineering evaluation project (EASEIT-ENG Project) on another post.

Rachel has some excellent ideas for her evaluation; a needs analysis associated with teaching hotel management. As part of this, he is looking at strategic documents for the organisation and interviewing "experts" so she can investigate the best direction for introducing flexibility. At this stage, time constraints will make it difficult to gather data from students. of course this can be done outside this evaluation project.

So we have a variety of evaluation project approaches: effectiveness evaluations, needs assessment - with some focussing mainly on students' perceptions and others looking at strategic documents and expert opinion. We also have a few usability evaluations for new online courses and also formative evaluations. Well done everyone! I am impressed with your activity and hard work. Now you can get on with the fun part - talking to people and gathering the data. It is always a bit like xmas finding out what others think of your ideas and approaches.

Some others are not mentioned here but have been working on their ideas and I am sure they will appear very soon.

Weeks 7 & 8: Negotiate and write an evaluation plan

Image: Pink Ball by DeclanTM

Welcome back. I hope you managed to have some relaxation time over the holiday break.

Over the next two weeks you will need to firm up your evaluation planning and prepare a formal plan - information about a template is in the Things to Do list below (taken off the wiki). By now you should have a pretty good idea of what you intend to evaluate. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment. And do contact me with your ideas and draft plan before submitting this part of the assessment as we need to negotiate your plan first. Read on for the fun stuff....

Guest speaker: On Tuesday 21 April 19.00-20.00 NZST - Helga Wientjes from the Department of Conservation presented to the class. Here is the Elluminate recording for your viewing pleasure. - she did a wonderful evaluation project for her workplace when she was taking the evaluation of eLearning for Best Practice course in 2008. It was a formative evaluation to measure effectiveness of a web conferencing training module.

Things to do:

• Decide on two eLearning guidelines against which you will evaluate.
• Establish the paradigm and model for your evaluation.
• Look at this example of an evaluation plan - Usability evaluation
• Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
  • You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
  • Remember this can be modified to suit your situation.
• Explore the Evaluation Cookbook to decide on some sampling methods for your project.
• Prepare some draft ideas and post them on your blog.
• Contact your lecturer to negotiate your plan.
• Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Summary Weeks 7 & 8: Negotiate and write an evaluation plan,

The highlight this week was a presentation by John Milne, project manager of the eLearning Guidelines project at Massey University. You can view the recorded session on Elluminate here.

John took us through a very interactive session about the eLearning guidelines and tried out some of the features on Elluminate e.g. polling. I really liked the way he got us all to place our locations on the map at the start. People voted to keep the format of the guidelines as questions rather than a checklist - as working through the questions helps people to think more about what they are doing in an organisation. They can then develop their own guidelines once they have thought through the fish hooks.

John's session was great for clarifying how to use the guidelines - the way we have used them in this course as overarching questions is different and you could also use some of the guideline questions to develop a questionnaire or checklist.

As mentioned in my last summary, several people have evaluation plans underway. For example:

Rika has an evaluation plan shaping up for a needs analysis and wants to find out if an "existing marine programme should be developed into an electronically delivered programme". My suggestion was to undertake a needs analysis to determine how the existing marine programme could be re-developed using digital materials and electronic communication methods which will "best meet the needs of key stakeholders and the primary target audience".

The reason being, if the answer to the first part was no - then the eLearning needs analysis would not go ahead fully. Also sometimes people are not aware of eLearning options until an exploration is undertaken. In Rika's latest post she has asked for more feedback on her updated plan. I hope you can take the time to do this.

Gordon in this week's post is making some changes to his plan, following my feedback, to gather evaluation data more in line with a maintenance evaluation - this is where you investigate whether an existing resource or course is meeting the current needs of an organisation. He also mentions in his post that "a high scoring example from a previous course" would be more useful than the example currently provided - good suggestion. Only thing is the criteria have changed slightly.

And each year I find the standard seems to increase, and so far we have seen some excellent plans. I tend to push for excellence when I see potential as well, something I have seen in all plans submitted so far as part of the negotiation process. Thank you Gordon for your astute suggestions and attention to detail in your planning.

Gordon also makes the point that the assessment criteria give the best guide to what is required. Agreed. Perhaps I also need to develop a template specifically for this course which is more in line with the criteria. That is, insist on inclusion of a list of objectives and outcomes.

Jennifer has been doing some catching up in her latest post, and needs some help to decide - whether her formative evaluation is going to be about the usability of the new learning tool - expert review and student feedback, and peer review, or should she conduct a summative evaluation and look at the students reactions (level 1) and measure what they have learned (level 2) - kirkpatrick's model? Could Jennifer get away with adapting Kirkpatrick's model, level one for a formative evaluation?

She has presented some indepth critique about a needs analysis she conducted previously, and also about an evaluation conducted on some online information literacy modules.

Yvonne mentions in her post this week that she has her evaluation plan pretty much sorted, but is worried the potential participants might be disappearing. If this does happen to any of you, be prepared to move to plan B so it is a good idea to have several options for data collection in your plan.

Helga says in her latest post that she is about to conduct her evaluation and is going to provide us with a presentation in the near future. She is providing a high standard of role modeling in how to conduct an evaluation project - I am impressed. Helga mentions her dilemma about revealing who has developed the resource - my feeling is that although you want honest feedback you may also need to be transparent about the development. I found a similar situation occurred when observing users of the online information literacy modules.

What you are after is constructive feedback anyway, and people seem able to be able to provide this regardless.

There is also some great discussion going on around Hilary's plan as several people have added to the comments on her post. I am impressed with the way you are all helping each other.

Anupam in his latest post, mentions he has selected four guidelines to evaluate elearning from an organisational perspective - they are excellent; this is a different approach to others' plans where courses or resources are being investigated. These guidelines will be relevant in his situation as he is not involved directly in any eLearning. For example,MO17 - how does the institution monitor the impact and effectiveness of eLearning? I have made several suggestions about people in the organisation who might assist and participate in his evaluation.

Oneteachersview08 assures us in his last post that he is still with us and it was good to have him join John Milne's presentation this week. Like several of you, he has found work and other PD gets in the way.

For those people who have gone a bit quiet in the last few weeks there is still time to negotiate your plans and I hope you can have something in draft soon - even some very brief ideas for others to comment on will be helpful for you. And we would also like to know you are still with us.

If you take a look at Sue's latest post you will see how some "thinking out loud" about your ideas can help you clarify things. I have left Sue a very long comment, but she stimulated lots of suggestions from me. All her ideas have value and I was pleased I could help her untangle some of the areas she mentioned. It looks like Sue is heading towards a summative effectiveness evaluation. The guidelines will help you sort out the real purpose of your evaluation as they provide the over-arching questions. I am looking forward to her plan unfolding. Some really good thinking going on there.

I found this site, sent by a colleague quite useful and you may also find some of the tips helpful.
Tips on how to write better blog posts

To not have them this long is one tip, but what is a gal to do when you are doing such great work? :)