Summary : Weeks ten and eleven: Conducting your evaluation

Some very interesting posts this week with presentations of evaluation plans appearing. There are presentations from Hilary, Gordon and Yvonne. Gordon has posted some of the feedback he has been getting on his plan and his discussion around the suggestions makes for interesting reading.

Helga has emailed a Flash presentation to everyone so if you did not get it you had better let her know. Helga had some difficulty finding a website where she could upload a Flash file, and the one I suggested did not work. This demonstrates how it is not always easy to display one's work in an open environment, even though we would like to do this.

Hilary had some glitches with Myplick when uploading the presentation, and it was very time consuming doing the recording; finding the right tool for the job is always a tricky one. Hilary's finished plan is now available as well.

Rika has posted her second plan and good on her getting it adjusted so quickly after being let down by red tape. It is good to hear that Sue is still with us after being out of action for a while. Watch out for her updated plan which will be coming soon.

I have to say I am impressed by the variety of the evaluations being conducted by everyone, and the range of approaches. The support you are giving each other is fantastic and there are some wonderful ideas coming through. It is also heartening to see your perseverance in learning new tools to create and upload your presentations. For most of you the next couple of weeks will be busy with data collection. It would be good to see an update on how it is going on your blogs, and do not hesitate to ask for help. There are plenty of heads here to help you solve the problem.

Recording audio
I find Audacity is always a good bet and is available as a free download. It enables you to edit and export the file as an mp3 format - essential for ensuring everyone can access and listen to the recording one has spent so much time on.

Discussion about measuring grades in an evaluation - a synopsis
There has also been some interesting discussion in the comments area on Yvonne's blog around whether measuring grades is a good indication of the effectiveness of a course. In my opinion better grades unfortunately do not always indicate a deep level of learning - it depends on how the assessments are designed. So I am dubious about using grades as a key factor in measuring effectiveness.

In the early days of elearning researchers focussed on grade comparisons between f2f and online learning. They used scientific experimental models of research design with control groups. Little evidence was shown that one mode was better than the other. Nowadays it is often more important to find out if students are using the tools, how they are using them and if they are learning through using them. Enjoyment may be a big factor in learning as Yvonne is about to establish in her evaluation.

Weeks ten and eleven 19 May to 1 June: Conducting your evaluation

Over the next two weeks, if you already have a plan in place, you will be conducting your evaluation with your target group.

There are a few things to do:

- Recruit participants - this may have been done previously.
- Collect data using the sampling tools you have stipulated in your plan.

It would be great to see some updates of how it is going in the field as you carry out your data collection. If you have not reached this point yet, keep working on your plan and post your ideas to your blog so others in the class can give you some feedback. Remember you need to negotiate your ideas for your plan with me.

Summary Week 9: Prepare and present an evaluation plan

Thank you everyone who has responded to the polls. It looks like 3/7 would like to have a question and answer session on Elluminate in the next two weeks and 5/7 are not sure.

Suggestion: I will run another poll with some dates and times for an Ellminate session.

The presentations have been found to be helpful by 3/7 and 4/7 found them sort of helpful. Solution: I will now run a poll to find out what sort of synchronous sessions you would like.

Well this week has been steady with a few plans appearing in their final iterations, and Gordon is the first person to post a presentation for us to look at - excellent work Gordon. It is good to know that this course has been influential in getting him to conduct some evaluation into the use of a computer-based product. Now I have watched the presentation I am much clearer about how he is going to use the "blender application". It sounds like a useful tool to measure eLearning. I also liked his example of why he is going to do some cross-tabulation. That is the perceptions of senior staff versus less senior staff.

Gordon should get some excellent data to manipulate and the mixed methods approach will make up for the small sample. As a pilot evaluation it is excellent and can easily be tweaked if he finds it is necessary to evaluate the product further.

Gordon mentions some free software for audio recording - My Mp3 Recorder - and also Myplick for loading presentations and audio synching; this is the same facility used for the evaluation presentation you viewed in the second week of the course. Another free utility for audio is Audacity. I would be interested to know which one people find easier to use.

Helga is very happy as she has started gathering data for her evaluation, and is preparing a presentation for us to look at it. A public version of her final plan is yet to come. She also invites some discussion on her latest blog post about the use of surveys at the end of a course - How to motivate participants to fill out the survey (or happy sheet). I have moved her initial discussion post from the main WikiEducator discussion page to the course discussion page. There is already some great discussion happening on the topic on Helga's blog in the comments section.

Rika has got a few gremlins in the electronics and in her original plan due to organisational "hiccups", and she is working towards getting her presentation to us. How about some encouragement for her on her blog. In the meantime her plan is posted on her blog and ready for you to give her some feedback.

Yvonne's approach is original - she is asking about enjoyment and creativity in relation to the use of a Moodle learning Management System in a blended course. On her latest blog post, there is a link to her final plan and a very good one it is too. Refreshing to see this, as so often teachers are too focused on getting "down to business" and only want to find out how something was used - not how much enjoyment or creativity the students were exposed to as a result. I hope she manages to get the sample she would like. Good to see a mix of likert-type scales in use and open-ended questions.

This reminds me - if anyone is planning to use an online survey - Survey Monkey is free and easy to use.

Sue has her draft evaluation plan ready for people to view on her blog and is ready for you to give her feedback. As part of the evaluation, Sue is going to investigate the effectiveness of existing online formative assessment methods through a range of data collection methods involving students and tutors. Looking very good Sue. :)

So far we have six people marching onwards and getting close to collecting evaluation data with one already doing this. Others have some great ideas for their plans and their plans are coming. Others are still with us and trying to clear work commitments so they can get on with the course work. I will be in touch with those of you who have been very quiet.

Do not hesitate to get in touch if there is anything I can do to help you progress with your evaluation plans and projects. I am absolutely thrilled to see such a range of evaluation types and your ideas and plans are superb. It is encouraging to see how each of your plans, your ideas for evaluations are all very original and interesting and based on thorough research and thinking.

Week nine: Prepare and present an evaluation plan

This week you will need to spend time preparing a digital presentation of your plan. This is so you can gather feedback on your ideas. Have a think about how you might do this. Perhaps a poll on your blog or an elluminate session? I can help you organise this. Make sure you have checked out the schedule.

Prepare your plan and present it according to the Assessment and Marking Schedule. (Remember you need to have have negotiated your plan with me first and posted it on your blog.) Provide feedback to at least two plans. Once you have incorporated the feedback you received, you are ready to submit your written evaluation plan (Part two A: Negotiate and write an evaluation plan) in the Digital Dropbox in the course Learning Management System or on your weblog or via email - but do email me even if you use the DDB. If you prefer you can put your plan in Google docs and post a link to it on your blog - remember publish it and give me access.

I will be running a poll this week to find out if anyone would like a question and answer session on Elluminate and also to find out if people have found the presentations to date helpful. Please get in touch if you are stuck on anything.

Summary Weeks 7 & 8: Negotiate and write an evaluation plan,

The highlight this week was a presentation by John Milne, project manager of the eLearning Guidelines project at Massey University. You can view the recorded session on Elluminate here.

John took us through a very interactive session about the eLearning guidelines and tried out some of the features on Elluminate e.g. polling. I really liked the way he got us all to place our locations on the map at the start. People voted to keep the format of the guidelines as questions rather than a checklist - as working through the questions helps people to think more about what they are doing in an organisation. They can then develop their own guidelines once they have thought through the fish hooks.

John's session was great for clarifying how to use the guidelines - the way we have used them in this course as overarching questions is different and you could also use some of the guideline questions to develop a questionnaire or checklist.

As mentioned in my last summary, several people have evaluation plans underway. For example:

Rika has an evaluation plan shaping up for a needs analysis and wants to find out if an "existing marine programme should be developed into an electronically delivered programme". My suggestion was to undertake a needs analysis to determine how the existing marine programme could be re-developed using digital materials and electronic communication methods which will "best meet the needs of key stakeholders and the primary target audience".

The reason being, if the answer to the first part was no - then the eLearning needs analysis would not go ahead fully. Also sometimes people are not aware of eLearning options until an exploration is undertaken. In Rika's latest post she has asked for more feedback on her updated plan. I hope you can take the time to do this.

Gordon in this week's post is making some changes to his plan, following my feedback, to gather evaluation data more in line with a maintenance evaluation - this is where you investigate whether an existing resource or course is meeting the current needs of an organisation. He also mentions in his post that "a high scoring example from a previous course" would be more useful than the example currently provided - good suggestion. Only thing is the criteria have changed slightly.

And each year I find the standard seems to increase, and so far we have seen some excellent plans. I tend to push for excellence when I see potential as well, something I have seen in all plans submitted so far as part of the negotiation process. Thank you Gordon for your astute suggestions and attention to detail in your planning.

Gordon also makes the point that the assessment criteria give the best guide to what is required. Agreed. Perhaps I also need to develop a template specifically for this course which is more in line with the criteria. That is, insist on inclusion of a list of objectives and outcomes.

Jennifer has been doing some catching up in her latest post, and needs some help to decide - whether her formative evaluation is going to be about the usability of the new learning tool - expert review and student feedback, and peer review, or should she conduct a summative evaluation and look at the students reactions (level 1) and measure what they have learned (level 2) - kirkpatrick's model? Could Jennifer get away with adapting Kirkpatrick's model, level one for a formative evaluation?

She has presented some indepth critique about a needs analysis she conducted previously, and also about an evaluation conducted on some online information literacy modules.

Yvonne mentions in her post this week that she has her evaluation plan pretty much sorted, but is worried the potential participants might be disappearing. If this does happen to any of you, be prepared to move to plan B so it is a good idea to have several options for data collection in your plan.

Helga says in her latest post that she is about to conduct her evaluation and is going to provide us with a presentation in the near future. She is providing a high standard of role modeling in how to conduct an evaluation project - I am impressed. Helga mentions her dilemma about revealing who has developed the resource - my feeling is that although you want honest feedback you may also need to be transparent about the development. I found a similar situation occurred when observing users of the online information literacy modules.

What you are after is constructive feedback anyway, and people seem able to be able to provide this regardless.

There is also some great discussion going on around Hilary's plan as several people have added to the comments on her post. I am impressed with the way you are all helping each other.

Anupam in his latest post, mentions he has selected four guidelines to evaluate elearning from an organisational perspective - they are excellent; this is a different approach to others' plans where courses or resources are being investigated. These guidelines will be relevant in his situation as he is not involved directly in any eLearning. For example,MO17 - how does the institution monitor the impact and effectiveness of eLearning? I have made several suggestions about people in the organisation who might assist and participate in his evaluation.

Oneteachersview08 assures us in his last post that he is still with us and it was good to have him join John Milne's presentation this week. Like several of you, he has found work and other PD gets in the way.

For those people who have gone a bit quiet in the last few weeks there is still time to negotiate your plans and I hope you can have something in draft soon - even some very brief ideas for others to comment on will be helpful for you. And we would also like to know you are still with us.

If you take a look at Sue's latest post you will see how some "thinking out loud" about your ideas can help you clarify things. I have left Sue a very long comment, but she stimulated lots of suggestions from me. All her ideas have value and I was pleased I could help her untangle some of the areas she mentioned. It looks like Sue is heading towards a summative effectiveness evaluation. The guidelines will help you sort out the real purpose of your evaluation as they provide the over-arching questions. I am looking forward to her plan unfolding. Some really good thinking going on there.

I found this site, sent by a colleague quite useful and you may also find some of the tips helpful.
Tips on how to write better blog posts

To not have them this long is one tip, but what is a gal to do when you are doing such great work? :)

Week eight: Negotiate and write an evaluation plan

Welcome back from the holiday break. This week we continue on with Negotiate and write an evaluation plan - see the post I made before the holidays.

A few people have been negotiating their plans with me and I have given them feedback. You can take a peek at the evaluation plans and some of my feedback for Yvonne, Helga, Hilary, Rika and Gordon. It will be good to see more plans this week and next.

On Thursday 8 May at 730-8.30 pm there will be a presentation by John Milne from Massey University (Wellington) about the eLearning Guidelines project he is leading. He will be able to give us some examples of evaluation projects around the country which are measuring against a range of eLearning Guidelines. It will be good to talk to you all again.

Click on the meeting link.