Weeks 5 & 6: Evaluation methods

Over the next two weeks you will be exploring the different functions/types of evaluation i.e. methods you could use in your project. The method you choose will depend on the purpose of the evaluation. You have already heard the words formative and summative, needs analysis etc.

Well what does it all mean? Lots to look at on WikiEducator. The focus this time is to explore some projects in the literature and the types of evaluation which others have done. It is a good idea to choose projects in a discipline similar to your own.

You are now getting close to preparing an evaluation plan in preparation for carrying out your evaluation.

Poll about study hours each week
There is a poll running at the moment, as some of you have noticed, about the time you are spending each week. It would be great to find out how many hours you are all spending on average.

Presentation
Remember there is a presentation on Thursday night, 3 April 7.30-8.30 pm. Bonnie and Suzie from Northland Polytechnic will be speaking about the evaluation project they did for this course - DVD resources in nursing. There will be ample time for questions and discussion. Click on the meeting link to access the web conference on Elluminate.

It is best if you have a headset with microphone, and/or standing microphone so you can speak, but if this is not possible, then you can still listen and use the chat facility.

Make sure you test the link well in advance - it is open now, and if any problems contact me.

Summary Week Four: Evaluation paradigms and models

Some steady activity has been happening over the Easter weeks. I am impressed that most of you are demonstrating your progress very well in your blogs and communicating with each other. Keep up the great work. If you prefer to move at your own pace that is okay as well. The course schedule is a guide for keeping pace with the group.

Rika has provided an excellent critique of the different paradigms with some examples of situations where they might be used - and has helped us to decipher the long words! I am very excited to see that she has actually used the Critical Theory Neomarxist post modern praxis paradigm for evaluating "education resources from a Treaty / Bicultural perspective".

Jennifer in her week 4 post questions the usefulness of the Constructivist-Heremeneutic-Interpretivist-Qualitative Paradigm; check out Rika's blog post - for an example of when it can be used: "Evaluating from this perspective I would immerse myself in the world of the subject and try and understand how they had learnt and how the course had affected their construction and interpretation of reality".

Jennifer also suggests how the Eclectic-Mixed Methods-Pragmatic Paradigm matches the 360◦ approach used in management/human resource surveys. Do you agree? There is also some very useful critiquing of different models in both Jennifer's post and Helga's post.

Helga has mentioned that Patton’s Qualitative Evaluation Model might be useful for her needs due to the use of a range of qualitative methods. I wonder which paradigms this model fits under? Any ideas everyone?

Hilary expresses the need for evaluation in relation to paradigms very well: "I can particularly relate to the ‘pragmatic’ aspect which reflects the fact that nothing is ever perfect in education but can be improved through design and recognition of possible flaws."

If you are looking for some easy way to make sense of all this paradigm stuff, Gordon had done some great paraphrasing in a table which he has linked on his blog. He has picked through the dense theoretical information and come out "alive" at the other end, armed with resources to make sense of it.

If people wish to read further about needs analysis and Stake's Responsive Evaluation Model there is a very informative article (Sleezer, C. M., Kelsey, K. D., & Wood, T. (2005). Assessing safety training needs at AVN 300: A case study) in the Resources area on the wiki.

Most people appear to favour the Eclectic mixed methods pragmatic paradigm, and can see the benefits of the rich data you can obtain from an evaluation under this paradigm. As Rika points out - it could be complicated to design, but well worth the effort.

Remember that through the use of mixed methods under the Eclectic-Mixed Methods-Pragmatic Paradigm , there is ample opportunity to incorporate both quantitative and qualitative approaches. In the real world it is also important to have versatile options and to investigate from several angles. Which is your paradigm and model of choice for your project?

It looks like some of you have got so tied up with the paradigms that you forgot about the models - it would be great to see some posts on possible models you might use in your projects.

You might also like to check out Oneteachersview08 - a creative take on the use of eLearning guidelines and competencies for secondary schools and a post about an elearning action plan. For example, competencies for "the ICT area and the Technology Curriculum and about being actively involved in communities. The guidelines he has chosen reflect this.

Wrap up
Paradigms and models did not mix well with chocolate eggs and rabbits for some people. I hope you manage to stay with the group and keep up with the material, but no worries if you wish to go at your own pace. Please try and post something short, at least weekly about your progress, in any case, so we know you are still with us.

If I have missed anyone out, please let me know here and I will add something about your post.

Summary Week Three: eLearning Guidelines

Some great posts in Week three about two issues relating to eLearning and some eLearning Guidelines which relate to these. It looks like everyone has managed to find a diverse selection of guidelines and had no qualms about getting carried away. It is hard to choose just two or three. The following paragraphs outline some of the guidelines and issues you have all mentioned.

Yvonne has mentioned issues around blended learning and managed to find several guidelines which relate to this area in her post. She has also managed to find an educational situation perfect for evaluating. I wont give the game away entirely here, and you can read all about it on her blog. The eLearning guidelines Yvonne eventually settled on are interesting ones which can be used to really tease out what is important for making learning creative and effective where technology is involved. Here is one example - "ST7 Will the e-learning foster students’ curiosity and creativity?".

For Gordon in his post, there are issues around whether industry "are open to blended delivery options" for delivery of unit standards, the development of online communities, and also in relation to a suitable Learning Management system and staff training. For example, "TO1 Are staff development programmes for e-learning developed in consultation with teaching staff, in order to identify and meet staff needs?" An interesting discussion has started in the comments area.

Online discussion and timely feedback and retention rates are the areas selected by Hilary in her post and discussion about the associated issues and the relevant guidelines. For example, "TD10 Should students present work using online discussion tools?" The big issue here, which many of you will be able to relate to, is one about plagiarism and people taking stuff straight from the Internet.

And you may also be able to relate to this issue raised by Hilary, "..any structured course that contains several modules and/or is self paced, the highest drop-out will occur within the first few weeks," and the guideline associated with it. See for yourself and add to the discussion.

Lesley has chosen two guidelines which are very complimentary to each other, and could have very important implications for addressing the issues she has identified in her post. These relate to negotiated learning goals and completion of training modules, and getting honest and constructive feedback from trainees. Is it about motivation or about the quality of the materials? Again some great comments which are extending the discussion Lesley has commenced.

Helga has chosen several guidelines and in the first post about this topic she has discussed three main guidelines to be included in an evaluation as well as several subsidiary guidelines. For example, "SO7 Is it clear at the start of the course what support teaching staff will provide students?"

This is particularly important for an organisation which is offering online learning for the first time. The discussion of issues illustrates how it is probably best to pre-empt issues rather than always trying to put "bandaids on the gaping wounds" - have a look at Helga's post and see what you think.

In the part 2 post Helga mentions the issues around how the guidelines could be applied: "the guidelines can be applied at two levels. One is the whole computer training / induction framework, the other is the individual courses that are part of this framework." Helga is as yet undecided as to which approach will be taken, and has done a lot of preliminary thinking about the approach which will be most appropriate at this stage.

Barbara has three guidelines which compliment each other very well. TD1 - learner-centred and achievement of learning outcomes and TT6 - good practice and self assessment. TT13 also fits well as it is about meeting students' needs. I will be interested to read some more about the issues associated with these guidelines - perhaps some of you could chip in with your own experiences in this area.

Important - Describe the issues then select the guidelines
Some people have spent a lot of time outlining why the guidelines they have selected are important, however, they have not always described the issues which led them to select the guidelines or mentioned how the guidelines might help address the issues. It is important to look at the issues first, then choose the guidelines. The description about the issues will form the background in your evaluation plan so it is an important step in this process.

Article about an evaluation study
Also some of you have commented on the article of an evaluation study of podcasting methods introduced to dentistry students studying microbiology. It was conducted at the University of Michigan (U-M) School of Dentistry by Brittain, Glowacki, Van Ittersum and Johnson, 2006. This is a useful example to read as it has some description about the methodology used in the study.

Some advice
When honing down the number of guidelines (I recommend you end up with two or three) which will be used in your evaluation project, you all need to make a decision about the most important issues for your organisation at the present. Your evaluation project could help address these issues, so you need to choose the most appropriate guidelines against which you can evaluate and describe how they might address the issues. If you try and address too many issues and select too many guidelines, your project will become unmanageable, and time is limited as you are all well aware.

I now feel sort of caught up and hope most of you will get some time over Easter to do the same. Watch out for the Easter bunny.

Summary of Weeks One and Two: Introductions, evaluation, elearning and quality

Great progress everyone. Like some of you I am playing catch-up having spent the last couple of weekends saturating myself with music and performance. More about that on my blog over the next few days.

Most people have a blog up and running and several people have posted responses to the weekly activities. There has also been a small group of you reading and responding to others' posts.

This aspect is very important as it is our main means of communication in the class. If you feel overloaded by the thought of this, it is best to select two or so different blogs each week to read and respond to and rotate until you get into the rhythm. You will soon find that there will be a cluster of blogs you prefer reading because they are posting about areas of eLearning you are most interested in. I will do my bit by summarising (more regularly from now on) each week or two's activities, and pointing you to people's posts where they have made statements and initiated discussion of note on the topic.

The range of opinion about the importance of evaluation is wide ranging and here is a sample of what people have said:

• "..a comfort blanket" (Jennifer).
• "..need to be sure that the e-learning is an effective use of the available resources" (Gordon- elearning solutions).
• "..a means to measure effectiveness of learning. Also to check that goals are being achieved and if not, why?" (Hilary).
• "..how do we improve on what we are doing or even know that something needs to be improved" (Lesley).
• "..want to create WOW programmes and resources.." (Rika).
  

Defining evaluation was not so easy. Here are some I particularly like - you may have found others, if so please let us know here -

• "..a thoughtful consideration of events which can inform future events for the better"(Yvonne ). "
• "..define evaluation as a constant checking and fine tuning of all kinds of processes that happens most of the time unconscious and/or automatic and sometime in a planned way"(Helga).
• "an assessment of the efficiency, effectiveness and quality of a learning experience, as viewed from the perspective of the relevant stakeholder"(Gordon).

Quality and eLearning
There has been a remarkable amount written about why quality is important in eLearning.

Jennifer's statement on her blog post about the importance of quality in eLearning is a goer - "I think quality is important in all learning. Is eLearning any different? " - and opens up an opportunity for debate. I would like people to comment on her blog and have a bit of discussion about this question. I have made a start there already. Yvonne has also mentioned her opinion about this "..to enable all students to learn effectively the teacher must be aware of what works and what doesn't." She also mentions costs and the need for skilled staff.

"Launch and Track' type e-learning" is described by Gordon who refers to learning without an instructor always present. The necessity for quality resources and learning experiences which engage and provide for remote learners and is cost-effective is also discussed - "more dependent on the material being clear and effective as the ability to engage synchronously with others, to clarify points and engage in spontaneous discussion, is limited."

Also these statements by Rika are very poignant about the importance of quality, "elearning resources have to be able to stand up by themselves." And "what's the point in putting energy into creating stuff that isn't any good and doesnt work for people?"

Hilary mentions the difficulties associated with transferring a f2f course to an online one and how necessary it is to overcome people's barriers to eLearning.

The presentation about evaluation has ended up providing an example-in-action of quality which you can all relate to as you had to watch it and experience it. There is more about this in the next section.

Quality of the evaluation presentation
Some people have also provided some very good feedback about the evaluation presentation. It seems several of you enjoyed it and found it useful. The 20 minute length caused some problems around concentration for that long in one sitting - I can relate to that. Luckily it could be paused for note-taking and the format enabled people to stop and start and return to it. The inclusion of humour (Flight of the Conchords) captured the interest and has been mentioned by a few of you. It would probably benefit from being broken up into three shorter sections - I might run a poll about this. Thank you all. :)

As well as the discussions about the importance of evaluation and quality and eLearning, several people have provided great examples of what they are already doing in this area in their organisations. A detailed example was posted by oneteachersview who has provided us with an excellent overview of how evaluation is conducted in a secondary school classroom using an "evaluation technique called SII (Strengths, Improvements, Insight)" - worth a look. I am interested to see what sort of guidelines will be appropriate for the secondary school context.

As you can see, weeks one and two have been busy and everyone who has posted has "waxed lyrical" about the need for evaluation, and what quality means for eLearning. It is great to see everyone is already familiar with evaluation of some kind and doing something. Good work team!

Week four: Evaluation paradigms and models

Over the next two weeks we will be examining the different paradigms of evaluation so you will be able to decide on a theoretical umbrella under which your evaluation will sit. The design of your evaluation project will be based on a model which will determine the methods you will use to collect data.

Firstly , ask yourself whether you are interested in rich, descriptive information, or do you prefer to "crunch numbers"? Do you want to find out how people are rating certain features of your new eLearning system? Or do you want to find out how they are using something by hearing about it in their own words?

For example, under the analytic-empirical-quantitative paradigm, you would use an experimental model and quantitative methods e.g surveys (Likert-type scales) to collect numerical data. Whereas under a constructivist-hermeneutic-interpretivist-qualitative paradigm, you would select a constructivist (Guba and Lincoln or Pattons) model and collect descriptive data, using methods such as interviews and focus groups or content analysis of documents to gather data.

It is best not to get hung up on the big words. Read through the material and ask for clarification on the course blog if unclear about anything.

Week three: eLearning guidelines for quality

This week you will be expanding your ideas about quality and investigating the eLearning Guidelines. These were developed for the tertiary sector and may not be absolutely suitable for private training organisations (PTOs) or secondary schools. They should, however, give you some ideas about how you can go about doing some checks and balances about quality in your workplaces. If anyone finds anything more relevant to PTOs we are keen to find out and you could add the information to the Resources page on the wiki and alert the class on the Discussion there.

As you blog, please tag/label each post with eval08 as well as any other tags/labels you want to add. That way you will be able to do a search in Google and find all the posts relating to this course.

Four speakers in April
Here is some great news. Four alumni from previous iterations of this course, have agreed to present their evaluation projects as 10 min presentations with discussion. All sessions will be added to the course calendar with the web conference link.

All presentations will be after Easter on the following dates and times - and will be recorded providing permission is given:

• Bonnie and Suzy - 3 April, Thursday - 7:30-8:30pm - evaluating the effectiveness of a DVD in Nursing - meeting link open now for testing;
• Selena - 10 April, Thursday - 7:30-8:30pm - title to be announced;
• Sandra - 17 April, Thursday - 5:30-6:30pm - topic is on the effectiveness of online discussion in a midwifery nutrition course. Sandra is speaking from Vancouver, hence the earlier time.
  

The presentations are each week and we are deviating slightly from the fortnightly real time sessions, but this is so you can get the info early to help with your ideas.
Bronwyn

what happened in the web conference session 6 March

On Thursday evening five people joined a web conference on Elluminate. We had a lively discussion about quality and evaluation in general, and the topics for week2 and week 3. Yvonne had to whisper for the first part so she did not wake hubby - well it was 6:30 am in the UK. Most people got the hang of the push-to-talk feature but there were a couple of hiccups until people realised you need to let the microphone button go.

Those of you may wish to have a look and listen to the recording. You will need to download Elluminate to access this so be patient. You will be prompted - just make sure you click the pop up window if it appears at the top of your browser.

An interesting point came up when Anupam asked for some suggestions about ways he can find out if his students find the online materials helpful. We talked about how it is hard for a lecturer to obtain honest answers when the students are worried about offending the lecturer; they may worry about their grades being affected, especially if it is a small class. It is often difficult for feedback to remain anonymous. Also we often forget to ask for feedback and carry on regardless.

There are ways to gather feedback during a course or when a new product is introduced, and also at the end of a course; the time when most people seem to ask for input. Helga made the point that it is often easier to get honest feedback on how usable something is, during the development phase rather when the product is finished; perhaps people do not feel as awkward being honest.

Hilary mentioned feeling a bit isolated not having a discussion on something like Blackboard as blog discussion did not seem as interactive. The group had a bit of discussion about this, and Bronwyn asked that people try getting involved in blog discussion by looking at each other's blogs (setting up a RSS feeds such as http://bloglines.com can help with this) and making sure you tick the email alert box.

Yvonne mentioned that it did take a while to get used to communicating in this way, and lesley explained why group email had been a bit problematic in her experience - nonprofessional use was made of it in some instances. Bronwyn said that some people found communication via list serv email groups hard to manage as it tended to flood their inbox - but there were ways to manage it.

Bronwyn said that if the email alert function is ticked for comments you make on blogs people will find that discussion will occur more naturally; people will be alerted as others post comments to their blogs and can respond. Otherwise people will have another place to have to go to post apart from their blogs.

Of course if people wish to have a discussion away from the blogs, there is also the Wiki discussion area and anyone can start a new forum anytime. At the moment we have introductions still happening on there. So maybe it is time for a discussion around Quality - but make sure you post to your blogs first each week as this is Assignment One and a progressive activity.

Have a great weekend.
bronwyn

Week 2: Quality and evaluation

I am absolutely thrilled to have met several people already on teleconference, email and on blogs. Seven out of fifteen people have set up their blogs and introduced themselves and also notified the link on the wiki Discussion. Nine people have posted introductions on the wiki Discussion. This is a wonderful start. Don't forget to click the watch tab on the wiki so you get an alert when someone has posted. It is great if people can go back to the Discussion regularly and respond to people.

Also keep an eye on others' blogs so you can comment on what people are posting. Watching each others blogs is easier if you set up an account on Bloglines and add all the blog links there - then you only need to watch to see if the numbers change. Be sure and access some help on using a RSS reader such as Bloglines if you need it. I have decided only to have a link to participant blogs on the course blog, rather than a feed coming in there. Note: To read the writing on the image just click on it.

This week the topic is all about quality.
The presentation this week will give you an overview of the importance of evaluation and the relationship to quality. Hopefully it will stimulate some ideas for you to use in your evaluation project. There is also some light reading on the importance of evaluation, and some questions to respond to on your blog. If you look in the resources area there are several online journals which you might like to start exploring.

The first two weeks are light to allow people time to get set up and confident using the course wiki and blog, and to start posting on their own blogs. It is okay to move faster through the schedule if and when you have the time. You will find you will move through faster some weeks, and this is good, so when you need to take some time out, for whatever reason, you will not get left behind.