Monday, March 30, 2009

class meeting recording (paradigms, models and methods) and next meeting

At the last last class meeting, people indicated that Thursday evenings 7 - 8 pm were a good time for a get together. On Thursday 2 April between 7-8 pm NZST, six of us had a discussion about: the theory and practice of evaluation - paradigms, models and methods.
Here is the 2 April class meeting recording.

If you are confused about all the choices and the terms associated with evaluation this is your chance to clarify some things. There was a short presentation and discussion.
The next class meeting will be after the holiday break - 23 April 7-8 pm NZST. Meeting link

I am organising some dates with previous course participants to talk about their evaluation projects. This should help you with your ideas. Once I have confirmed the speakers, I will put up a list with dates and time. Hope to see you there, and as usual recordings will be available.

Tuesday, March 24, 2009

Weeks five and six: Evaluation methods

Image: Twitter Bird by tashmahal

For this topic you will need to spend two weeks - this will take you up to the holiday break over Easter - 6 to 19 April 2009.
It is very important you explore the options for conducting evaluations carefully, and choose a method appropriate for your evaluation project, because in weeks seven and eight you will need to start writing your evaluation plan. Some of you may wish to use some of the holiday break to start organising this. Some people have already been talking about doing formative evaluations or needs analysis and there are other terms such as usability, effectiveness evaluation, impact and maintenance evaluation with which you need to become familiar. Talk to Bronwyn if you are not sure what sort of evaluation to do.

Things to do:

1. Read the short article evaluation methods in the eLearning Guidebook.
2. View the presentation by Professor Tom Reeves - Six Facets of Instructional Product Evaluation, (Google Books, 2008)
  • It is based on the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications - available in the Manukau Institute of Technology library. Remember you can make a request for a chapter - might I suggest you choose one which is relevant to the type of evaluation you wish to conduct. The Table of contents was sent out on email.
3. Find an article or two in the literature which fits with the type of evaluation you are interested in for your project. You will have access to the online databases at MIT.
4. Post responses to the following on your weblog:
  • Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
  • Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

Wednesday, March 18, 2009

week 3 Summary: eLearning guidelines

Image: Soccer Practice by mill56

Isn't it great to see all the soccer players out on the playing fields at this time of year. This summary will be an overview of the main activity around the eLearning guidelines. People who did not feature in the previous summary will now get a mention. Again I am in danger of getting carried away as there are so many very interesting blog posts on this topic. It is also good to see that several of you are communicating with each other on your blogs and giving feedback. Impressive! :) The recording for the last class meeting is also available. We had a good discussion about quality and also evaluation projects, in particular, needs analysis.

As you can see from Deborah R-W's blog she has been attending an evaluation seminar for academic quality managers. It sounds fascinating. Deb you got me curious and now I can't stop! :) I did a Google search for Jane Davidson and her book: EVALUATION METHODOLOGY BASICS The Nuts and Bolts of Sound Evaluation - came up. I really like the idea of values-based evaluation which is Jane's approach. Watch for more on this in posts later on...

Elaine has been doing some serious catching up before disappearing for a short while. In her week 2 post she mentions some different methods she has used to collect data in the past when evaluating. There is further consolidation of the theme around quality, and the impact it has on student retention. This has been mentioned by several of you and clearly is a concern for all of us in the online teaching field. The week 3 post has two very relevant guidelines for the courses which Elaine is looking at re-organising in tourism, hence the need for some evaluation. I really like the ideas she has for redesigning the courses to include some authentic project work. See what you think and do give her some well deserved feedback.

Craig in has been doing some serious exploring and writing about eLearning Guidelines which are suitable for his project. In the My Guidelines post Craig has shown us how he worked through several options to find two guidelines which were most suitable. The guidelines will enable him to "monitor the quality of study material", and evaluate a "communities of practice approach to course development". This is a great example of the use of critical analysis (thinking and reasoning and evaluative judgement to solve a problem) is demonstrated in a blog post.

Craig has also posted a slideshare presentation about the Quality Matters rubric used to evaluate distance education. There are forty specific elements you can look at for some ideas. Be careful you read the copyright statement on the website before using the material as is, as unlike the eLearning Guidelines, the material is not freely available for use. Though they cannot control the ideas you might obtain from it. :) You might like to compare the Quality Matters rubric to the Wright and MacEwan criteria which Adrienne told us about in week one.

Pradeep has also been busy posting and in his first post on quality for week 2 sums it all up with the words: "..evaluation is the tool to checking quality." He also makes the point that assessments help to evaluate whether curriculum design is adequate. We do get excited when students pass at a high level and despondent when there is a high fail rate, and this certainly stimulates teachers to look at how their material is presented and to try different strategies to improve outcomes doesn't it?

In his week 3 post on guidelines, Pradeep has selected two examples and I have suggested he add the word, How, to one of them so that he has more scope for his evaluation project. SD3 - [How] do students gain knowledge relevant to employment and/or current thinking in their field? You might like to bear this in mind for the guidelines you choose.



Debra M
has written an excellent overview of her plans for evaluation and has critiqued some guidelines and her own "big picture" questions for her proposed evaluation. Several people have been leaving comments on her blog post and giving her feedback and encouragement. See what you think about Debra's ideas for making the pilot run of the new course interactive and for evaluating its success.

Kay has also been busy posting - week 3 and has chose two guidelines from the teaching area to address a lack of internal quality control procedures. Certainly a checklist for design and delivery is a helpful resource to have in eLearning. I wonder what others use? Adrienne has mentioned a resource she uses. Kay also posted her ideas around quality in her week 2 post and shows us how to connect the dots with others' posts. She mentions "specific indicators for measuring quality such as: assessment of student learning, feedback from students, peers and external reviewers and institutional accreditation procedures.

In contrast, Adrienne in her post has chosen guidelines from the student area with respect to choice, interaction and technologies. It is good to see you are back on board with a working computer.

A few people have found the cube of eLearning Guidelines a tad annoying to use. You can read their comments on Hervé [if] students are not clear about the expectations, processes, resources and outcomes of the course they are taking. From facilitator to students, it can be anyone's fault but in anycase it deprives the course of its spirit. Krishan, similar to a few of you, has found a focus on student learning around motivation and technology issues to be important with his choice of the following guidelines -TD2:Do students get clearly defined learning objectives that assist them in focussing on their learning activities? - and TD5: Have a representative sample of students tested the e-learning materials, and any necessary modifications been made?
OTARA design model
mentioned by Michelle in the comments area.
Weeks five and six: Evaluation methods, and there is two weeks holiday break from 6 to 17 April.

Week 4: Evaluation paradigms and models

Image: Bedouin Jerry Can Band by Michael Flynn and the Taranaki Daily News

You can blame the lateness of this post on my pilgrimage to Womad. I see there has been a lot of action on your blogs while I was away, so tomorrow I will post a summary of week three activity. This week is probably the driest one with regard to evaluation theory. You do need to have an understanding about the main inquiry paradigms and models which currently influence evaluation. If you can see it as a set of terms regarding current beliefs about how evaluation is done, things will seem easier, though there are some very long terms to get your larynx around. There is no need to memorise the terms though as I will certainly not be testing you. Hopefully you will get this posting as an email.

Things to do:

Week 4: Evaluation paradigms and models

Image: Bedouin Jerry Can Band by Michael Flynn and the Taranaki Daily News

You can blame the lateness of this post on my pilgrimage to Womad. I see there has been a lot of action on your blogs while I was away, so tomorrow I will post a summary of week three activity. This week is probably the driest one with regard to evaluation theory. You do need to have an understanding about the main inquiry paradigms and models which currently influence evaluation. If you can see it as a set of terms regarding current beliefs about how evaluation is done, things will seem easier, though there are some very long terms to get your larynx around. There is no need to memorise the terms though as I will certainly not be testing you. Hopefully you will get this posting as an email.

Things to do:

Monday, March 9, 2009

Week 3: eLearning guidelines for quality

Image: Lines & lights by tanakawho

This may be week 3, but if you are not up with the schedule do not worry. Most people have just completed the week one tasks of introducing themselves and setting up a blog. The best thing for everyone to do now is to follow the course weekly schedule. Please read important notice about my leave dates at the end of this post. Welcome to Pradeep who has joined us and is located at Polytechnic of Bahrain.

This week we are scheduled to look at some eLearning Guidelines which have been developed for New Zealand. These were used by 21 tertiary organisations as part of a Ministry of Education funded and sector wide project on implementing e-learning guidelines. This national project provided a mechanism for developing, introducing, trialling, evaluating, and sharing examples of e-Learning Guidelines. Check out the list of things to do for week 3. The eLearning Guidelines were developed specifically to address some of the issues around quality in online teaching and learning. Some of the factors impacting on quality in eLearning were discussed in the week 2 summary on the course blog.

Make sure you have read the information about the assessments and make contact with me if you have any questions. Remember my recommendation for keeping up with the course schedule is to spend approximately 5 hrs per week on it. Lots of people have found that marking dedicated time slots on the calendar works a treat and helps you fit your study into your busy weeks.

Optional fortnightly class meeting: 19 March 2009 - 7 to 8 pm. I hope you can attend the web conference. This will be a discussion about quality and eLearning Guidelines and will be recorded. If this time does not suit, please email times and days to the email group so we can find a more suitable time and perhaps add another day and time to the mix.

Process for Questions and comments
General questions and discussion can go to the email forum: evaluation-best-practiceATgooglegroupsDOTcom and personal email messages can be sent to bronwynDOThegartyATgmailDOTcom (letters are used on here to deter robots in case you are wondering). You can also generate discussions by using the comments area at the end of people's blog posts. It is alway great to get a comment when you have spent time and effort posting on your blog.

My Skype name is at the end of the group email, and if you would like me to Skype you please send me your user name and some suggestions for times and days to make contact. Generally, each week I will post a summary on the course blog with links you may wish to follow and this will help you pull some of the information together for the week. I also make an announcement at the beginning of each week regarding the main direction for the week.

I look forward to hearing about your ideas for evaluation and working on your projects with you. I am on leave from 12 to 17 March so will not be responding to emails over that time. Please contact Oriel Kelly, programme manager of the Grad Cert at MIT if anything comes up which just cannot wait.

Summary Week 2: Quality and evaluation

Image: Butterfly My First by photogirl7
What a busy week you have all had. Everyone pretty much has their blog all set up and ready to go and several people have posted responses to the questions for Week 2: Quality and evaluation. Hence my summary has lots of substance and questions for you to ponder. I am very pleased with the progress everyone has made especially those of you for whom blogging is a new experience. It has been great to talk to several of you online or on the telephone.
The questions for your response in Week 2 were:
- Why is evaluation important to you and how do you define it;
- What sort of evaluations mentioned on the presentation are familiar to you already and why?
- Why is quality important in eLearning?
Answers to all these questions will vary depending on your personal views and perspectives and the context in which you work. For example, Adrienne's blog post is very relevant to her work, and she has provided us with a practical list of items which she considers when designing eLearning. There are some very useful things for you to look at when planning your evaluation projects. She also makes a very important point about the link between quality and student retention - a subject close to us all I reckon.

DebraM in her post has listed several of the evaluation items she was familiar with from the presentation and makes a good point about evaluation, "it can be used to 'pat yourself' on the back, as well as 'allowing room for improvement'. She has presented a very comprehensive discussion about quality - it is all in the "eye of the beholder" well the user's opinion anyway. And check out the first eLearning Guideline to be selected.

Sam in her post has offered a different slant on quality again and brings information about two different organisations and their views of quality to our attention - Concord Consortium versus Articulate. I love her discussion about the idea of rapid eLearning offerings and the demise of quality if we just "throw-up online" without taking time and care to provide a thoughtful product. Lots of ideas for debate in this post. See what you think.

Herve's post refers to the presentation as did DebraM, and he says the concepts about evaluation were all familiar. One thing I found useful from Herve's post was the perspective that evaluation in eLearning may be even more important than in a traditional classroom because the interactions are different. He and Joy are on a similar wavelength - do you agree with Herve and Joy?

Joy alludes to the way we evaluate constantly when teaching and others have mentioned this as well. She has also provided us with some definitions from the web as well as her own take on evaluation. Joy is a fan of the ADDIE model of design, and gives us a good overview of why it is a useful model to follow to ensure quality in eLearning. Joy, like Herve, also tells us how the different ways we interact online compared to face-to-face classrooms, can make it hard for lecturers to gauge how students are understanding or enjoying their learning. Therefore, evaluation is even more important in eLearning.

Iain like many of you, is focused on evaluating to ensure his students are getting the best learning experience possible. Yes we do want them to keep coming back for more don't we? I really like the description of the ways in which he uses learning outcomes as a focus at the end of each lesson. It encourages students to self-evaluate their understanding and also helps Iain measure how well the class is doing. Iain's company does a very comprehensive evaluation of student satisfaction and success and it sounds like evaluation is integrated into all areas of the student learning. Well done! This sounds like a great model. I will be following your trail with interest.

Rachel in her week 2 post, as well as some insightful statements about quality being necessary for a range of areas, e.g., needs assurance, cost effectiveness, student retention and positive outcomes for students, Rachel provides us with a thought provoking question - “Why is quality important in eLearning?’ – and ask is quality not important in all learning?, should there be a distinction." Good on you Rachel asking this extremely important question.

This has spurred me on to provide a lengthy reply. Quality as you mention Rachel has always been important and in all areas of education, not just eLearning. However, the reason quality became such a sticking point in eLearning harps back to the early days of online teaching. In the early days, people new to online teaching often took the pages and pages of lecture material they used in class and just "slapped" it online. Design principles were poorly adhered to (graphics, chunking of text, formats, navigation, instructions etc) and approaches were often inconsistent re tools, instructions, formats and layouts. Then the "bells and whistles" brigade came along and produced fancy multimedia which no-one could open (software issues) or you could only use with a Nasa-designed computer. :)

Now of course we are much better informed, but the dilemma in the online classroom which several people have mentioned, is the lack of personal interaction in an online classroom - lack of body language or facial expression, unable to observe students and answer queries immediately. And it is more difficult to teach unprepared online - how many of you experienced teachers have walked into a f2f classroom not as prepared as you would like, but comfortable in your knowledge and expertise on the subject and had a great session? I believe it is harder to do this online because people can get so lost and intimidated. They seem to like schedules, instructions, structure and clarity. Do you agree?

This means we have to be much more slick with the way we organise online courses. This is where evaluation in the design and development stages becomes so important - so we get the design of the online learning as good as possible before we impose it on students. And then we need to find out if the approach was effective. I guess it is not quality which is different but they way in which we ensure we are meeting particular standards or guidelines which help us produce quality learning.

Krishan's post also gave me plenty to write about. Krishan makes some very valid points. I agee that having materials with well written instructions is a very important aspect of quality. It is so easy to get lost online. I agree with him about the predominance of summative evaluation in education. In which case, the evaluation is far too late to keep at-risk students in a course. There is nothing more disruptive or soul destroying than a class of disgruntled students. Do you agree?

Krishan mentions his familiarity with evaluative observations - they certainly are an excellent technique for evaluating skills, however I still shake when thinking of being watched while I undertook nursing tasks.

In evaluation of multimedia including web-based courses, observation is a good way to evaluate the clarity of how materials are presented and laid out. By observing how students navigate through, and also how they use the materials, the bugs can quickly be located and fixed. What may seem to us to be an obvious pathway or icon to click is not always so to others. Do you agree?

Minhaaj in his week 2 post, mentions some very insightful ideas, and I certainly agree with his statement about the reason we need to evaluate: "without the evaluation process, learning will be restricted to a traditional non-interactive, one-way process which is not only monotonous but kills the enthusiasm and interest levels of participants." He mentions an evaluation rubric used in week 3 of a course development called Enhancing lessons, so participants could measure the quality of the materials they were creating. It is fair to say than some sort of structured approach to measuring quality is important if educational materials are to have credibility. I do wonder though whether a rigid evaluation framework stifle would creativity and innovation. Is there a balance and a compromise? I believe there is but it needs to be carefully considered - do you agree?

So in week 2 we certainly had a wide range of opinions about quality and evaluation. It was helpful that you shared your previous experience with evaluation and also the material you found when exploring the topic.

TIP:
It is good to see references at the end of some posts - it is also good to have hyperlinks in body of the post especially when quoting or mentioning items you have obtained from your explorations and reading. Some of you are doing this very well.

Thursday, March 5, 2009

Recordings of this week's meetings

Image: Satellite Dishes by watchsmart

This week, we had a couple of good sessions on web conference using Elluminate and a good turnout to both sessions. Unfortunately it was not straightforward for some people who could not access Elluminate from their organisations. Hopefully the IT issues will be resolved by the next time we have a session in a fortnight's time. There were others who had other things come up, or could not get Elluminate to stabilise, but don't worry, the sessions are optional and recorded (not the same though is it), and I look forward to you all being able to connect at some time or other.

The Tuesday session was also pretty unfortunate for me as I kept getting booted off Elluminate and thankfully the group was able to keep conversations going without me. My Internet provider is looking into the problem. Apparently it has to be really bad in the analysis Telecom does on the line for them to actually do a lot. Looks like I might be investing in a satellite dish or six! :(

The recordings of this week's sessions are now available and are also on the wiki.
Tuesday 3 March at 3- 4pm - the recording of the meeting.
Wednesday 4 March at 7- 8pm - the recording of the meeting

It was good to clarify the assessment requirements and discuss people's backgrounds in elearning and their interests in evaluation. Some good ideas came through last night about quality in elearning and evaluation as well.

Last night's group asked for another session in a fortnight at the same time - 18 March 7-8pm. So hope you can make it. I will not be able to run the Tuesday session that week as I will be away, but suggest Tuesday 24 March 2- 3pm. The discussion topic will be announced on the blog on the week with the meeting link.

Also in about a month's time I will start the first in a series of presentations from previous participants in this course who have kindly agreed to talk about their evaluation projects using Elluminate. More to come about this...

Tuesday, March 3, 2009

Week 2: Quality and evaluation

Image: Rowan by kevinzim
This week, we are taking a look at what constitutes quality in eLearning and discussing this topic. As part of this, you will be investigating the importance of quality, and establishing a definition of evaluation for eLearning which fits your professional context. There is a presentation to watch and some reading to do. The presentation will introduce some terms associated with evaluation, e.g. formative and summative, and hopefully give you another perspective about the subject.
  • There are three questions for you to ponder and post to your blog. A short email about your main points would be a bonus.
  1. Why is evaluation important to you and how do you define it?
  2. What sort of evaluations mentioned on the presentation are familiar to you already and why?
  3. Why is quality important in eLearning?
Do check out Adrienne's post which will start the week off for you on the subject. Note the resource she has shared with us - this could be very useful when you are thinking about which aspects of eLearning you will evaluate. For example, will it be the course design or the support given to students or something else?
Do you agree with her reasons for ensuring quality in her courses and do you have the same issues?

Summary week One

Hawthorn berries (cratageous monogyna) - Image by Dave-F almost retired

I just love this time of year when all the berries are appearing. We have some lovely juicy blackberries near us which my Briard and I love to eat. of course she prefers that I feed her the berries but the prickles don't seem to bother her too much.
There has been a good steady start to the course. Most people have set up their blogs and sent the links to the email group. Still missing a few so it would be great to get you all linked to the course blog. A few couple still to get connected on the email group.

Some of you may be waiting for the "real time" sessions to clarify some things about your blogs. I am looking forward to talking to everyone. If you can't make the sessions on Tuesday 3-4 pm and Wednesday 7-8pm, there will be a recording available. I will also be in touch by phone from time to time just to check in that everything is going ok.

Chocolate fish to the people who have set up blogs for the first time and others of you have re-activated your blogs for this course. A great effort!

We have a good mix of people from around the country - apologies if I have missed anyone. Two people from the South Island - Rachel (Dunedin) and Liz (Wanaka). Eight people from Auckland, four of whom are from Manukau Institute of Technology (MIT) - Michelle, Krishan, Joy, Herve - with Craig, Sam and Debra RW from private training organisations (PTO) in Auckland. Four people from UCOL in Palmerston North - Kay, Debra, Elaine, Heather (Bulls). One person from a PTO in Wanganui - Iain. And Adrienne from Kamo in Whangarei. We also have Minhaaj joining us from Pakistan.

I have learned something new today re Kamo - "The name is a Māori word meaning "eyelash",[1] but has also been said to mean "to bubble up", referring to hot springs in the area." (Wikipedia.) Perhaps Adrienne will tell us more about her hometown on her blog. It will be great to get so many different perspectives from the mix of people from different backgrounds in the course. All links to people's blogs that I have so far are on the course blog and the snippet feed will update as you post - I hope.