Evaluation of eLearning for Effective Practice

Weeks 16 and 17: Evaluation reporting

2011-11-28T10:14:00.001+13:00

http://jigsawpuzzlesgames.net/

In the last two weeks of the course, you are putting the puzzle together and preparing the evaluation report (Assessment Three). If waiting for feedback on Assessment Two, the Results, you can still work on the draft report, and also ensure your blog is up to date (Assessment Four). The final date for submitting these assessments is week 17 (11 December) - I can extend this to 14 December if you want these marked before Xmas. If this is not possible, please make contact and negotiate a date in the new year. If Assessments One and Two are outstanding, you need to make contact asap to negotiate submission dates.

Please fill out the poll about the weekly hours you have spent on this course.

Activity Thirteen - Evaluation reporting – preparation of a draft report

Access the Evaluation report template and add to your chosen document application - Word processing application, project wiki, or Google docs.
Prepare the final sections of the report – conclusion and recommendations, executive summary and reference list - collaborate in your groups.
Invite feedback - I suggest you email the lecturer and others in the class to alert them.
Adjust sections of the Evaluation plan so they are suitable for the report - rewrite in third person, past tense, and include feedback from the marker. The sections are: Introduction, Background and rationale, Aim or Purpose and Evaluation questions, Decisions, Methodology.
Adjust Results (Assessment Two) and add to the report.
Make sure the Appendices are in order.

Activity Fourteen -Finalise the evaluation report (Assessment Three), and your reflections about the evaluation process on your blog (Assessment Four).

Submit the final evaluation report in pdf format. You can email this to the lecturer, and link to the report on your blog.
Summarise your reflections about the evaluation process for Assessment Four on your blog.
Notify the lecturer and the rest of the class when this is ready.
Sit back and relax - you did it.

Weeks 14 and 15: Evaluation reporting – Discussion

2011-11-15T10:19:00.000+13:00

Round Tables at the Roundhouse by mikecogh

The online meeting will be scheduled in week 15 at 19:00 - 20:00 on Thursday 24 November 2011 - Evaluation reporting - (not week 14 as previously indicated on the course schedule). I realise that most of you may not be at this stage in your evaluation projects as yet. Even so you can start thinking about this stage and hopefully join the online meeting.

Activity Twelve - Prepare a draft discussion about your evaluation results, report this to the class and invite feedback.

Collaborate to prepare a draft report with discussion of your findings – add to your project wiki.
For your discussion - think about how your findings have answered the evaluation questions.
Explain why the results obtained from the data you collected and analysed have or have not answered the evaluation questions.
Refer to the evaluation literature you have annotated to compare your findings to others' research.
What recommendations can you make to assist the decisions you listed in the evaluation plan?
Report your draft discussion to the class and invite feedback.

For example, if you were evaluating the design of an online course, you may comment on the key features of the design that students and staff found useful for learning, and mention similarities or differences from the research literature - it might be common that students dislike using discussion forums, or that navigation and the clarity of instructions are key areas which 'trip' students up when learning online.

Week 13: Evaluation literature

2011-11-09T14:17:00.000+13:00

Quill & Books by aurostar739

This week you are probably preparing the second assessment and writing up your findings. If you haven't got that far don't give up as I realise some groups have had a few delays. We are now up to reporting and information as well as activities to guide you in this phase is on the wiki. We begin with an activity which will help you to critically analyse your findings, and prepare the discussion section of your report. The discussion forum is set up on emit for you to discuss the information you are reading in the articles.

Activity 11
Discuss examples of evaluation literature that can be used in your report.

Search for and select suitable articles or reports to discuss on the eMIT forum. (Hint: it is advisable to find evaluation studies that use a methodology similar to your project.)
Annotate the articles - include enough information to be useful later on if you refer to the research in the Discussion section of your report.

The following prompts may assist:

Which evaluation theories underpin the research?
What methodologies were used - design, sampling methods, sample sizes?
What were the main findings and recommendations?
How similar or different to your results are the research findings in the articles?

Weeks 11 and 12: Conducting evaluation

2011-10-25T12:54:00.000+13:00

Numbers By lrargerich

Did you manage to get a break? Over the next two weeks, you will hopefully be collecting data and exploring methods of analysing and interpreting your data. I will be in the Wimba classroom on Thursday 27 October from 7 - 8pm to discuss the analysis and interpretation of data and the presentation of Results and to go over the process. I hope you can make it. In the meantime, I will be marking. You have prepared some wonderful plans.

Activity ten:
Analyse data and write up the findings – prepare tables, graphs and explanations – add to your project wiki.
Submit final results (wk12 - negotiable). A pdf is the best way to submit your results as this preserves the formatting.

Once you start adding your results to a new project wiki page - we will follow the same process as the plan - where I provide formative feedback before you submit your final results. You can add the final pdf on the wiki and/or email it to me. Information about preparing results should be up on the wiki by the end of the day.

Week Nine and Ten: Finalising plan and reflection on the group evaluation process

2011-09-26T14:59:00.000+13:00

Spring Wellington by Newsbie Pix

Countdown this week - please get your draft plans ready for me to give feedback. I will be on leave from 1 to 17 October, and will mark the final plans on my return. However, to make sure you are ready to contact participants in readiness for collecting data after the course break - 10 to 21 October - I need to provide feedback on your evaluation design this week. I hope you can meet the deadline. I have included two activities - this week's and next.

Online meeting this week
The online meeting in Wimba will be on Thursday 19:00 to 20:00 for last minute queries about the planning process and collecting data.

Activity Eight- Finalise and submit the final plan.

Incorporate feedback into the plan
Organise collection of data – send out invitations to participate.

Activity Nine- Reflect on the group evaluation processes you have undertaken to develop your plan.

Check the criteria for assessment four in the course handbook.
Update your blog according to the criteria.
In particular, on your blog - report and reflect on the personal contributions you have made to the group process during the development of the evaluation plan - which sections were you responsible for, what did you contribute etc.
Indicate what you learned about working in a group.

Week Eight: Planning evaluation –draft plan

2011-09-19T09:39:00.000+12:00

Big sister watches the young uns

Yes these are my lambs - I couldn't resist. It was touch and go when the younger ones were born because it was freezing in Dunedin. Luckily they were born in the shed and didn't move out of there for several days.

It is excellent to see your plans evolving, and I understand there is lots of activity happening as you meet in groups to put your plans together. This is probably the hardest part of the process - getting the evaluation design just right. Be sure and organise a Skype or Wimba session with me if you need to discuss your plans.

Your blogs
A quick reminder that you need to contribute regularly to your blogs for Assessment Four - Reflect in an on-going manner about your contribution to the evaluation process, and post on your individual blog. Please read the criteria in the Course Handbook - they are posted at the end of this message. This is the main way you can indicate the extent of your contribution to the group process, and this assessment must be passed to pass the course. Tom is posting regularly and I am sure he would appreciate some comments - check out his wonderful photo taken as he flew over the motorway. :)

What should you be doing this week?
This week you need to continue developing your evaluation plan and request feedback. By the end of this week you should expect to have an easily, readable draft plan with all the sections completed on the project wiki, ready for others to give feedback. Make sure you read the Analysis of data section on the wiki to sort out how you will analyse the data that you will collect so this can be included in the plan. This is also where the instructions for Activity Seven are located.

Activity Seven - Group work

Prepare other sections of the plan to complete a draft plan – limitations, logistics and time line, budget, references and appendices.
Finish writing the Introduction.
Add to your project wiki and request feedback on the project forum.

Criteria for Assessment Four - the blog

Demonstrates an understanding of evaluation (theoretical and practical) in an individual and professional context.

Produces evidence of learning and reflections about evaluation - including eLearning guidelines (or other benchmarks) and quality in a relevant context.

Reports and reflects, from an individual perspective, on the phases (plan, data collection, analysis and interpretation, and Results), progress and processes involved in the group evaluation project.

Reports and reflects on personal contributions to the group process, and what was learned about working in a group.

Demonstrates evidence of individual reflective learning about the process of evaluation.

Critiques the process used in one other group project, and describes the feedback provided at each stage.

Merit: Critical analysis of the evaluation process demonstrates evidence of reflective practice (effective reflection and professional learning[1]).

[1] Hegarty, B. (2011) A Framework to Guide Professional Learning and Reflective Practice. Unpublished Doctoral thesis. University of Wollongong, NSW.

Weeks Six and Seven: Planning evaluation methodology

2011-09-05T09:22:00.001+12:00

Daffodils in Bournville by ell brown

Over the next two weeks, 5 to 18 September you will continue planning your evaluation project, and will explore and prepare the evaluation design (methodology) - sampling methods, instruments, triangulation, and methods you will use for the analysis of data. Activity six and seven are relevant for these two weeks. Remember to look on Wikieducator (you can also get there via Course activities in eMit) in the Planning section for more detailed information about the activities. You will notice I have changed things around a bit. I am currently working on the other two sections which will go up - Results and Reporting.

Online meeting
This week we are very privileged to have a wonderful guest speaker, Professor Tom Reeves who will speak about evaluation and the connection to good design. The online meeting this week is on Thursday 8 September 2011 12:00 - 13:00 using a guest Wimba link, not the usual classroom. I will post more details about the link on eMit, email and on Wikieducator. I hope you will be able to make it. Apologies for getting the day wrong earlier.

Activity six:

Collaborate in your groups to design the methodology for the evaluation – add to your project wiki.
Use the resources to further develop your evaluation plan.
Add the methodology section to your project wiki.

Activity seven:

Continue developing the plan and request feedback.
Prepare other sections of the plan to complete a draft plan – limitations, logistics and time line, budget, references and appendices.
Finish writing the Introduction.
Add to your project wiki and request feedback on the project forum.

Using the project wiki

Remember, the beauty of using a wiki to develop your plans is that it becomes very much a work in progress - enabling you to keep revisiting and reworking the plan as your ideas unfold. You can expect your ideas to change all the way through as you develop the plan - if they don't I will be worried. So don't expect to get each section perfect before moving on to the next section. When you prepare the methodology you will probably have to go back to the other sections and change some things. This is easily done on a wiki, and it is easier to see how each section is fitting together doing the development in this way.
The finished plan is due in the week 26 September so keep working towards achieving this date. You can do it! Please do get in touch if you are stuck or getting sticky. I am happy to Skype with you individually or in groups.

Week Five: Planning evaluation - purpose and evaluation questions

2011-08-28T20:58:00.000+12:00

New Zealand sheep by 111 emergency

Great news! Professor Tom Reeves has agreed to be a guest speaker and share some of his expertise about evaluation. I have scheduled him for Wednesday 7 September at 12:00 -13:00 NZST. I do hope lots of you can make it over the lunch hour. As usual we will meet in the Wimba classroom, and the session will be archived. Do come ready with questions about evaluation. I will let him know beforehand some of the topics people from the previous online meeting asked to have explained.

Where should you be by now?
By now you should have decided on an evaluation project, and have prepared the background and rationale, goals and decisions. You should also have found a research article from the literature which relates to your plan. This will be useful in informing your evaluation design. I deliberately left writing the purpose of the evaluation (Aim) until you have had time to think about the type of evaluation you will do. You need to choose an evaluation approach which fits the reasons for carrying out the evaluation and the audience involved. The audience is usually the participants as well as the managers and colleagues who have an interest in the outcomes of the evaluation. You can leave the Introduction for now, as this is best done once the rest of the plan is completed.

What is happening this week?
Continue with your evaluation plan. All the activities and resources related to Evaluation Planning are now located in the section called: Planning on WikiEducator (via Course Activities from eMIT). Other sections (Results, and Reporting) are not visible as yet as they are under development. (Evaluation processes has met its demise, and resources from there have moved.)

This week you need to decide clarify the purpose of the evaluation, and write the Aim and Evaluation questions. These are the 'big picture' questions. Some of you may have one evaluation question, and others may have two or three. Under each of the broad evaluation questions, you can list one or two sub-questions which are more specific.

Activity Five- Aim and Evaluation questions.
    * Prepare the aim and two or three evaluation questions, and sub-questions for your plan to fit the decisions and goals.
    * Add to the project wiki.
    * Use the project forum to ask for feedback from the class.

I hope you can provide feedback to each other. I will be checking the project wikis later on in the week, but do email me once you have a draft Aim and Evaluation questions ready.

Week 4: Evaluation planning

2011-08-19T12:00:00.001+12:00

Crocus by Sids1

The most popular day and time for online meetings appears to be Thursday evenings 19:00-20:00. The next meeting will be on 25 August at 19:00 hrs in the Wimba classroom. We will discuss evaluation planning and your project ideas. Remember to make a time to skype with the lecturer if you wish to discuss your evaluation projects.

Activity Four
You will begin developing your evaluation plan over the next few weeks, and a series of activities will be carried out, starting with Activity Four. For this activity you will need to describe the background and rationale, goals, and decisions - on your project wiki using the evaluation plan template (on eMIT in Course Activities).

Email the lecturer with the name of your group, and the members so that a Group space can be allocated with the project wiki.
Decide on the type of evaluation - formative or summative - based on the Reeves and Hedberg model, or another one you wish to use (please discuss this with the lecturer).
Explore the reasons for your evaluation project, goals and the decisions to be made - begin the plan.Add the evaluation plan template to your wiki space in your group.
Remember to post your ideas about your evaluation project on your blog.

Resources to help with this are available on WikiEducator in the section - Evaluation Processes - also via Course Activities in eMit. Have a great week.

Week Three: Methods of evaluation and projects

2011-08-14T19:38:00.000+12:00

Snow at home in August 2011 by bronwynannh

Well it is easy enough to see that I will be playing in the snow this week. Luckily all the neighbours' wee lambs are tucked up safely in their big shed. My sheep are playing the hungry card, poor things.

Progress reminder
By now you should have your introduction on your blog and have contributed to the Activity two - evaluation and quality discussion forum.

What's on this week?
This week we are looking at methods of evaluation - needs analysis, usability, effectiveness, impact and maintenance - described by Professor Tom Reeves and Professor John Hedberg in their 2003 book on evaluation - Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications - Chapter Three. Also, on Google books is a preview of the book. The information is summarised in the different Resources accompanying Activity Three.

Online meeting

Evaluation methods will be discussed in the online meeting this week: 16 August - 19:00–20:00 on Wimba. Make sure you test the link well in advance of the session. Please take note of the telephone call number and student pin code so you can enter via your telephone if all else fails. Access via eMit - Online meetings/Wimba Classroom. Please text me if you have difficulties (contact details can be found via the Lecturer link on eMit).

Activity Three
The information about this week's activity and the resources accompanying it can be found via Course Activities/Evaluation Methods.

Use the resources to find out about different types of evaluation.
Explore the literature about evaluation which is relevant to your context. A lot of material is compiled in the References section of the wiki, and you may also wish to find your own.
Post a summary of the key ideas to your blog.
Examine the range of potential evaluation projects on offer (a link to them is available on eMit in the Projects discussion forum), and either choose one of these or advertise your own.
Organise yourselves into groups for the evaluation projects - use the Projects discussion forum to join a project group and to negotiate your methods of working collaboratively.

I hope you find a project to suit you. If you have an idea for your own evaluation project, make sure you email me with the overview - using a similar structure to the one used for the list of potential projects. This information will be used to inform your evaluation plan. Have a great week, and I hope to meet lots of you on Tuesday evening.

Summary of week two activity

2011-08-14T19:07:00.001+12:00

This week several people have contributed to the discussion about the ADDIE model and quality. Views about the usefulness of the ADDIE model of education design have been mixed, although consensus appears to be that it can provide some guidance for design and development. However, theory is believed to be only a starting point, and the ADDIE model is really just a "project management tool" (Alages and Tom), and other factors need to be considered, for example, time, quality and budget" (Rosanne). An excellent point was made that it is important not to get too restricted by one model such as the ADDIE as flexibility is needed when thinking about design of online systems (Mark).

Also, some people have used the model in their design work (Rosanne, Alages), or something similar to it (Sandra H, Mark and Sandra W), and that it is best not to become too linear oriented when designing online learning experiences as it is a dynamic process (Alages). Awareness about the challenges in undertaking evaluation are apparent in the discussions, and it is an area which is often left because of time constraints. Also, the need for evaluating during the design and development etc. phases is regarded as important and mentioned by most people.

I have referred people to the PADDIE model of design which we use at Otago Polytechnic, and as you will see it is still a work in progress - I am too busy doing the actual projects to finish the theoretical framework. A case in point. The information about the ADDIE model for instructional design provided by Alages has more information about how to use the ADDIE model - for each phase details are provided about the type of evaluation you might undertake.

Only two people are familiar with the eLearning Guidelines, and everyone thought they would be useful although having time to explore them could be problematic. Some people mentioned they used other benchmarks but did not provide information. At Otago Polytechnic we use the Good practice checklist for benchmarking quality in Moodle courses. It is based on others' work as listed on the document. To access this file, you need to login as a Guest. A five stage model for measuring organisational quality in eLearning which some of you may be interested in is the eLearning Maturity Model for NZ.

Maturity Model Getting Started
Marshall, S. (2006). E–learning Maturity Model Process Assessment Workbook. There are five main areas for assessing quality and benchmarking eLearning:

1. Learning Processes that directly impact on pedagogical aspects of e-learning; 2. Development Processes surrounding the creation and maintenance of e-learning resources; 3. Support Processes surrounding the oversight and management of e-learning; 4. Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle; 5. Organisation Processes associated with institutional planning and management.

Additional benchmarks for measuring quality can be found on the Quality Principles site. Are any of you these resources familiar to you?

Week Two: An overview of evaluation and quality - Activity two

2011-08-08T10:21:00.000+12:00

Winter fog by bronwynannh

How is this for a photo from a basic cellphone? It takes a pretty good pic and certainly shows you what it has been like around my place lately. I thought this would be a good starting point to talk about quality and the reasons we need to evaluate. This week it is a good idea to start checking out the Exemplars of previous students. I expect to have a list of potential projects available later in the week. If you have ideas already, please email me as soon as possible to discuss them.

Activity two
Detail about this week's activity (Activity two) can be found via Course activities in emit. You will end up looking at the WikiEducator page. The timetable and also up and coming activities will be listed on the Home page as a quick reminder. To find the information about this week's work, click on the tab called: Why evaluate?

This is where you will find the following instructions for Activity two, plus a list of resources.

Activity two

Use the resources to develop your knowledge about the topic.
Participate in the discussion about the evaluation of eLearning - on emit.

1. What do you think about the ADDIE model of educational design?

How much do you know about it? Please tell us especially if you use it.
How adequate is it for guiding effective eLearning design?

2. What type of benchmarks are you currently using to measure quality?

Have you used the eLearning Guidelines? Do tell us more.
What other benchmarks do you know about or use for evaluating teaching and learning?

If you wish to contribute other resources for this topic, please share them on the discussion forum and also if you wish on your blog.

Summary of week one activity

2011-08-08T09:56:00.000+12:00

Confused by Kristian D

The week has started well and eleven people have an individual blog up and running for activity one with an introduction, and there are some wonderful photos to look at. I am now getting a sense of who you all are, and where you work, and also have more understanding about your previous experience with evaluation. Look and see how many people are working in the Hair and Beauty Schools at MIT - I can feel a project coming on in that area. We also have people from Nursing and Health areas, and lecturers in ESOL , electronics and mathematics, and business communications as well as a programme coordinator with a role in quality assurance and a distance learning manager. The blogs of seven others in the course are still to come, so it looks like we have a hearty number studying the course this year. It is fabulous to see that a number of you know each other already from other courses, and some of you are close to finishing the Graduate Certificate.

Most people are using the emit blog, and two people - Oonagh and SandraH have chosen to use external blogs. I will be interested in hearing about your preferences about the emit and external blog platforms once we have got underway. The links to peoples' external blogs are listed on the course blog - see to the right, and both Oonagh and Sandra have put a link in the individual blog area. Fred has also invited us to view his Personal Learning Environment.

Online meeting archive
We also had the first online meeting, but there was a hiccup and one person could not get the technology working on the day - so make sure you run through the wizard on the Wimba classroom well in advance. The archive of the recording is available - click on Online meetings in emit, then Wimba Classroom, and once Wimba downloads scroll down the page past Enter Room and all the phone numbers until you see: Online classroom meetings - 08/04/2011 13:38 - future archives will appear there.

Poll about a time for online meetings
Chocolate fish to the three people who responded to the Doodle poll. We also had a chat about times in the first online meeting. Please respond to the poll and indicate your choice of day and time for the regular fortnightly online meetings. It would be great to find a time which suits most of you. So far, Tuesday 5pm and Thursday 5 pm or 7pm have been mooted.

Are you confused?
Some of you may find the use of emit and external platforms for the Course blog and Course activities a tad confusing. The best plan is probably to enter the course via emit and use the course timetable to guide you with each week's activity. I will also post an overview on the Course blog each week and send a reminder on email. Once we get going you will hopefully get the hang of it. Make sure you ask questions if you are not sure what is expected - on the Questions and Answers discussion forum in emit so everyone can see, or on email to me, though brave people may wish to use the Comments area on here. ...Off now to prepare this week's blog post.

Week One: Welcome to the 2011 class

2011-08-01T10:25:00.000+12:00


Sunrise on the mountain by Kiwi Flickr

Welcome everyone to the first week of the course. I am located in Dunedin - hence the picture of snow - and I work at Otago Polytechnic in the Educational Development Centre. I was snowed in so couldnt go to work for two days because I live up near Mt Cargill. I was one of the original developers of the grad cert programme, and have been teaching this course since it started in 2006, mainly because I really really enjoy teaching the course, and working with each new group of students.

So why a course blog? I like using a blog for posting a weekly announcement, to remind you about the activities which need to be completed over the week. It also means I can have fun selecting pictures. Also, the blog is also a good way to summarise each week's activities. People who contribute during the week always get a mention on the blog.

By now, hopefully, you have found your way around emit, and downloaded the course handbook. Once we get the business of the assessments out of the way, you will be able to relax and get on with starting the course.

I hope most of you can attend the inaugural online meeting of the class in the emit Wimba classroom - Thursday 4 August - 16:00 - 17:00. I will be emailing out a link soon, asking for the most suitable times for future online meetings.

First steps
First things first - this week is pretty relaxed, and you can take your time familiarising yourself with what is required, looking around emit and the course blog and wiki (where the course activities are outlined), and meeting the others in the class. So far there are eleven people enrolled. Once you all get your introductions posted to your blogs, either in emit or externally (if you wish), we will get a sense of where you are all located, and find out something about you and your work. Hopefully, the instructions on the Announcements page of emit helped you find your way around. Next week is where we really get down to it and start exploring the meaning of evaluation in eLearning.

Valuable bits of paper to look at first
The course handbook outlines the four assessments, and informs you how to get a merit for the course. The other valuable piece of paper to look at as soon as possible is the course timetable (see Course activities in emit) and you may wish to pin this on your wall.

I am really looking forward to meeting you all, and hope to see you in the Wimba classroom on Thursday.

almost the end

2010-12-07T14:01:00.001+13:00

Kiwi xmas tree by W J (Bill) Harrison

Thank you all for your participation and for making this course so varied and interesting. The course officially ends this week, but I am taking reports up until 15 December for marking this year. If you have negotiated an extension, I will be in touch next year. If you want to catch up don't forget to take in the class sessions in the archive of web conference recordings.

I am going back through your blog posts now for marking Assessment One - so if you are feeling almost finished, don't forget to check out the marking criteria for this assessment in the Assessments section. Here is a Hint: A summary post with links back to previous posts is a good idea to make sure I don't miss items you have described under each of the marking criteria.

At this moment in time, one report is in and about to be marked, and I know there are a few others which are very close to being submitted. A couple of people are still at the data collection stage, and others are in the throes of data analysis. So you are all working on different points in the continuum from evaluation plan, collecting data to analysing data and reporting. I am happy to give feedback on draft reports.

Now I have a dilemma for next year. How to get everyone to the evaluation plan stage, and collecting data as suggested in the schedule. Planning the evaluation projects seems to be the biggest 'stumbling block'. I would like to find out from you - what would have made it quicker and easier for you to get the planning phase completed? Having a buddy seemed to work well, but aligning projects set in different places may have been a bit tricky for some of you. Yes?

In the final web conference, we discussed some aspects about the design of the course. One thing which came up was that people would have liked more idea about what they were in for, right at the beginning - my assumption was that the exemplars would help with this - but they didn't seem to be enough. I am interested in your ideas about this. How do you think students could be informed about the big picture evaluation processes involved in this course right from the start?

Week 15: Evaluation reporting - new resources

2010-11-18T11:26:00.001+13:00

There is always music amongst the flowers in the garden, but our hearts must be very quiet to hear it by krikit

It was wonderful to meet some of you at MIT last Tuesday. People are very different in person, and we had a wonderful time fighting crab claws and eating lunch as well as discussing evaluation projects. Some people are readying themselves to write up the findings in a report, and others are finishing off their plans, and are poised to collect data. No worries about getting extensions for the course to February next year - just let me know your timeline. I would like to get all plans finalised by the 17 December this year though.

See: Writing up the findings for new material on the course wiki showing some examples of Tables and Figures. See: Preparing an evaluation report. A template is now available for writing the evaluation report.

The last web conference for the year is on Wednesday 1 December - 18:30 to 19:30. This is your chance to tell the class in person about your findings and present a summary of your report if you wish. We will be finishing with virtual wine.

Week 14: evaluation reporting

2010-11-08T00:40:00.000+13:00

gecko by wollombi

The pic this week depicts my status this last few days - cruising over the weekend in Brisbane - land of heat and geckos. Three days were spent in Melbourne at the eportfolio conference which was pretty useful for finding out where everyone is presently situated in Australia regardng this form of eLearning. More on that later.

So where should you be heading in the course? This week is scheduled to be the second week of your evaluation reporting. I know several of you have started to collect data and some of you may have started analysing material. It is important to decide how you will present this data in your evaluation report. Tables are always good to show frequencies of responses to survey questions. If you think it is easier to see the trends, I suggest you also prepare some graphs. Bar graphs are great for showing frequencies, and if you have small numbers of responses keep your data as numbers. Larger amounts of data work well if converted to % frequencies.

Make sure you check out the exemplars to see how previous students in the course have presented their data. It will be good if you can upload your findings into Google documents, or something similar and share it with me so I can add feedback. When you are ready you can publish as a web page, and put the url on your blog so others in the class can see your results.

Once your tables and graphs are ready describe what is in them as the Results section, then prepare a Discussion of findings; Conclusions; and Recommendations. Check out how it is done in the exemplars and use their headings as your template - of course you may wish to use your own. Be guided by the assessmnt criteria for the evaluation report . Have a great week and see some of you Tuesday.

Week 10: Conducting evaluation

2010-10-08T18:24:00.000+13:00

by Dominic's pics

Welcome back. I see some of you managed a break and took in Sharefest. I was very disappointed I had to withdraw at the last minute - pressing work stuff. Some of you have completed your plans (marking now), and others have asked for extensions. I am on leave myself this week, so it is a self-directed week where you may be finishing your plans or organising to run surveys or interviews and starting to collect data.

I will be back on 18 October, ready to mark plans then. I strongly advise you not to start collecting data until, I have given feedback on your plans especially the survey and/or interview questions. You can organise your participants though. There are three weeks scheduled for collecting data so still plenty of time.

It is great to see that most of you are still posting updates to your blogs even though it was the "holidays". I promise I will get round them when I get back on 18 October, so it would be great if you could look after each other in the meantime. Have a great week.

Week Nine: Quality in eLearning

2010-09-20T15:06:00.002+12:00

sugar magnolia By emdot marya

This week you need to keep working on your plans and also have a look at Quality in eLearning so you can incorporate some of this into your final plan. The due date for your evaluation plan is 4 October 2010 - for publishing finished plan on the wiki project page with a link to it posted on your blogs. I am open to negotiation on this. Dates for the other assessments are now on the wiki in Assessments. Remember your plans also need to include the actual sampling tools you are going to use, e.g., survey, interview etc. I can then give feedback on your survey and interview questions, if you are going down that track. So your tasks this week are to:

Explore quality benchmarks for eLearning.
Define quality in your context.
Arrange participants for the evaluation.

Web conference

Michael Winter from CoreEd will talk to us about his research evaluation of the SLENZ Second Life project on 22 September - 18:30 – 19:30 on Elluminate - Click for the Online meetings. This will be recorded if you can't make it. You may wish to read: Winter, M. (2010). Second Life Education in New Zealand: Evaluation Research Final Report .

Quality in eLearning
There is lots of material about benchmarking guidelines and frameworks. Of course we have the eLearning Guidelines which you have been exploring, and as we know quality is a complex area and a lot broader than a few guidelines. A good place to start for the New Zealand context is the E-Learning Maturity Model
which incorporates a wide range of eLearning areas. The model includes checklists you can use, each type depending on which area of eLearning in your organisation you are measuring for quality. Some of you may already have been involved in this exercise.

In the first instance you need to decide on the type of outcome do you want from benchmarking and this is covered in the Maturity Model Getting Started. The E–learning Maturity Model Process Assessment Workbook describes five main areas for assessing quality and benchmarking eLearning: For example:

Learning Processes that directly impact on pedagogical aspects of e-learning;
Development Processes surrounding the creation and maintenance of e-learning resources;
Support Processes surrounding the oversight and management of e-learning;
Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle;
Organisation Processes associated with institutional planning and management.

I have organised material in the References section on the course website. See: 1.1.1 Benchmarking eLearning material. There is no topic called quality in the wiki as all the material you need to look at is in the References section. If you wish to define quality in your context on your blog this would be great, but you can also incorporate this aspect into your plans which is where I would rather see it.

What's next?
This week I will be looking at your aim/purpose, big picture questions and decisions to make sure these pieces of the puzzle fit well together. Remember there is a holiday break 27 September to 10 October. Aim to start collecting your data at the end of the holiday so get organising with participants once your plans are submitted. Have a great week.

Week Seven: more planning

2010-09-13T13:42:00.001+12:00

Sweet choices by by moriza

This week the post is going to be short and sweet as I wanted to get you started this week with a quick message of encouragement. This week is when we really get into the swing of the negotiation phase. Your plans are starting to develop personalities and substance - great work. I am looking forward to reading your them and providing feedback, and it looks like there has been lots of activity on your project pages. I will work on a summary of your activities over the next few days.

Survey for feedback and dates for assessments
I will send out a short survey on email so I can get some interim feedback about how the course is going for you. The official due dates for the assessments will be up by the end of the day - aim to have your plans in draft form on the wiki so myself and your class mates can give you feedback - I recommend you do this by 19 September so you are ready to gather data from 11 October after the break.

Recordings of the web conferences
All the recordings of the web conferences are now archived as (1.2 Web conference archive) in the References section on the wiki. I have also added a link from the Home Page under Web conferences.

I have sent out some material to assist with some of the theory and the processes surrounding the types of evaluations you are doing. If you need more than one chapter, you will have to request this from the MIT library due to the 10% copy rule which restricts me sending out any more. You can make a request from the distance services - you can order articles and textbooks as well. Do make use of this service - use it or lose it. See: http://library.manukau.ac.nz/distance/index.php

Most importantly of all, I hope you have a great week.

Week Six: Summary - Planning and more planning

2010-09-06T16:59:00.000+12:00


Scene from Christchurch earthquake by Benjamin Humphrey

Lets start the week with words of support for everyone affected by Saturday's big shake up. I hope people get back to the status quo as seamlessly as they can. The clean up will be incredibly challenging.

Several evaluation plans are starting to appear on the project wiki pages. If you are linking from your blog - make sure you put the link to the actual project page, to save people hunting around. here is the summary of activity for this week. Excellent to see the feedback you are giving each other.

Kevin has put up a lovely message to people re the earthquake, and a wonderful summary about some blog posts he has been reading. It looks like Kevin and Jon have made a good start on their project page.

Rachel has illustrated some of the hassles associated with collaboration and technology - nothing is simple it seems. It looks like Rachel and Alex have given Misha and co some good ideas for collaboration.

Lisa has put up an excellent cartoon which really illustrates some of the dilemmas we find ourselves in regarding eLearning. Look out for the 10 tips for eLearning.

Meena and Misha have been evaluating the Digital Information Literacy module. It is a very thorough evaluation, and they provide different perspectives. They make a very good point about the level of interactivity which is reiterated by Kevin. Alex has asked a very important question about how the module could be used as is. The reader is given a good overview of the module with some very helpful ideas for improvement. A great evaluation of the resource.

Misha has also posted a summary of her email discussion about issues of time for development - if you stand up and state the actual hours you work on developments - are you going to start a revolution?

Katie is on a roll with her sheet of paper - a great way to plan the "big picture" questions and the project. I am looking forward to seeing them appear on your project page.

Dana and Louise appear to have a few challenges with finding uninterrupted time to meet. This can certainly make it very challenging to get the ideas and creativity to flow. It will be good to see their plan unfold on the project page. Louise says it is tricky to "clarify the distinction between an aim and a purpose, and it can be "a grey area". I believe they have found the solution. " ...an aim is a more broader approach to evaluation in relation to our topic and that the purpose is more specific".
I look forward to the story unfolding further.

Keep planning team - you are doing great work.

Summary of week Five

2010-08-31T16:02:00.000+12:00

Daffodils by di_the_huntress

Spring is here and you have all got a spring in your step as well. You have been very busy indeed with the mock evaluations, and thinking about your evaluation projects. The thorough approach you took in the tasters and mock evaluations has made my heart skip a beat. It is very exciting anticipating what you are going to come up with when doing the real thing. There has been some shuffling of partners, and hopefully everyone is feeling more confident about the collaborative approach to the projects. The details are on the wiki in the section about Evaluation Planning.

Mareena uses a wonderful analogy for evaluation, which all of us will be able to relate to - clothes buying. See if you agree that when she is measuring the quality of the clothes "in terms of price" she is evaluating from a cost-effectiveness perspective. I believe this must influence the decision to buy, and will vary in different contexts and situations. For example, you may spend more than normal on one item if buying a garment for a wedding. How does the context influence different types of evaluation?

Veronique and Mareena have prepared a comprehensive description about the Searching for Information module which they evaluated. They have identified some items which could cause big frustration for students on dial-up connections. At the level of digital information literacy for which this module was designed - lower level users - such problems could be very off-putting. This demonstrates the necessity of getting the balance of "fancy stuff" and "usable stuff" right. They have done a good job on this, some constructive suggestions balanced with the "fish hooks" they encountered. I like the way each person has referred to the other's perceptions about the module and Veronique also referred to others' evaluations, e.g., Alex's suggestion about the length of modules like this.

Lisa and Alex have identified some important factors about the computer taster and have taken an educational perspective which includes the provision of interaction opportunities. It is interesting that they have picked up on the need for online communication to supplement the f2f lectures and online and computer-based activities. I agree that the evaluation can be used to find out what "would work" as well as what is working. They have made some good suggestions - does anyone know do they match with what the evaluator discovered in his report in the Exemplars section?

Alex and Jon and Misha are the first people to put up their choice of eLearning Guidelines. The explanation Alex has provided with each one indicates how new eLearning is to her organisation and some of the challenges they face. Jon has taken a different approach about his choice of guidelines and re-described them to give a bigger picture view of the issues from his perspective. Misha has a mix appropriate for students and staff. Remember you can re-word the guidelines to suit your context, and also make up your own using the framework.

Jon mentions feeling some "excitement" about learning creeping in. He makes some valid points about the danger of over-analysing. My experience is that it has been this fear which has either led to, either no analysis at all, or very poor estimations of what is possible or necessary. Getting the balance right is key to progressing, as he says, and likewise, I believe it is the end user who must be considered pivotal to the whole process. How well do you believe educational organisations do this or are they too concerned with economics?

Katie, Alex and Rachel teamed up to evaluate the Digital Information Literacy OIL module. They were very creative and organised their evaluation on wikispaces witht the different perspectives - corporate and primary school - side by side. See if you agree with them? I agree that offering the module as part of a blended approach would be much more powerful, and making the module more user friendly for dipping in and out would also work. Excellent evaluation!

Lisa and Alex have also evaluated the Business module. They have highlighted some of the issues which can arise with pre-packaged learning objects. First of all, it is very hard to design something which "suits all the people all the time" - in terms of navigation, content, layout, learning preference and style. Although there are style sheets for this type of thing, the minute the developer wants to get creative, they may become irrelevant. As you say, working through a self-paced module like this is boring if the level of interaction isn't just right for each individual - it is like reading a text-book from start to finish. They make good points about the length of the module and the amount of content.

The one thing which I find frustrating with these modules, is the hoohah required to customise them. That is why my preference has moved away from static learning objects such as this, customisable as they may in theory be, towards open developments such as those on a wiki. One of the modules started it's life on a wiki - Digital Information Literacy - but the project brief was to create Learning Objects. One of the developers wanted to push the wiki option but was outnumbered - I guess he was right all along. What are your thoughts?

Dana has posted an overview of the reasons for his choice of evaluation - it will be an effectiveness evaluation - and he will evaluate "the smiley face approach" This is a resource Dana has developed to teach the eukaryotic cell in the bachelor of Nursing to help them understand and always remember the basis of life. I really like the reflective element which is emerging in these posts, and the next one - Working together with Louise - is particularly reflective and describes the difficulties which can occur when teachers are trying to be students, and also the challenges which can arise when working collaboratively on projects.

Katie has formulated an excellent big evaluation question. It appears she will not only need to find out what sort of technologies/tools the teachers would like to use, but also what they will need to use to keep up with educational trends, and maybe later she can find out more about what is going to be relevant for students to use. I have suggested that a good starting point might be Derek Wenmouth's post - 100 Ways Google Can Make You a Better Educator. Some of you may know him - he critiques the latest strategies and technologies for the school sector. Katie has made a great start with the thinking for her part of the evaluation plan. Where will she head next?

Misha is feeling behind the 8 ball, but her post makes it seem like she has been doing this blogging lark for a while. The guidelines she has chosen are very appropriate for the situation described. Being a "fingernail ahead" must be excruciating - hanging off the cliff alone and wondering if anyone is going to come and save her. She mentions the old adage "time" - there is never enough and staff are always concerned with what is priority - so that generally their up-skilling gets left until last. Why I wonder is course development always seen as "the thing you do" on top of everything else - then the word quality gets rolled out as an after thought? Haarumph! It makes me grumpy too!

Her plan to do "a Needs analysis around the digital literacy levels of our students and more particularly staff" is an excellent choice. I like the 3 Fs! People may be interested in some research I have recently completed with Oriel and others on Digital Information Literacy capability- staff and students - Digital Information Literacy: Supported Development of Capability in Tertiary Environments.

We found that time to play was essential for supporting people to develop confidence in using a range of digital technologies in our information rich world.

I hope you can all see evaluation as a way to celebrate what is working well as well as seeing it from a problem-based focus. Remember whether planning a needs analysis or an effectiveness or maintenance evaluation you can look at something from several different angles and perspectives. However for this project, you need to keep your approach focused and not let it get too big and unwieldy. When choosing your sampling tools for collecting data make sure you enable triangulation and more reliable information to be obtained. So far theire is a mix of needs analysis, effectiveness and maintenance evaluation projects planned.

Week Five: Planning evaluation

2010-08-23T15:47:00.001+12:00

Crocuses by Gilles Gonthier CC by attribution

Can you believe we have now reached week five. Spring is almost here and the signs are everywhere. It is great to see there has been some discussion on the email forum about the two questions I put up:

Is it always necessary to undertake a Needs Analysis before beginning any eLearning development?
What are the circumstances which might dictate this approach and when is it not necessary to do one?

So what is on this week?
This week you are scheduled to start the topic: Evaluation planning. From now on you should be focussing on the activities in the Processes of evaluation area on the wiki. We are spending four weeks on planning, and at the end of this time, you should have a pretty robust plan in place for your evaluation project. The due date is negotiable in case you were wondering. However, I would advise you aim to have a draft plan ready on your wiki project page (under class projects) to negotiate with me by 12 September, and a final version for marking by 26 September. That way you can start collecting data in October.

Quick tip
Get your purpose and the type of evaluation sorted first, then formulate the 'big picture' questions. Once you have a draft plan, make sure you obtain feedback from your classmates and integrate it in your plan before notifying me it is ready for marking. Posting about your progress on your blog as your plan evolves is also a great way to get feedback and it is surprising how helpful this can be.

The activities for this topic are located under Evaluation planning. You will also need to revisit the eLearning Guidelines for New Zealand: http://elg.massey.ac.nz/ to help formulate your 'big picture' questions. View the assessment guidelines to Negotiate an evaluation plan and check out the Marking criteria.

Web conferences
There are two web conferences scheduled:

25 August - 18:30 – 19:30 - the guest speaker is still to be confirmed. If the speaker is not available, we will discuss evaluation planning and I will help with some tips for the process, and discuss your ideas.

8 September - 18:30 – 19:30 - Rachel Byars, previously a 2009 student, will tell us about her evaluation project - a needs analysis. She is a senior lecturer in the School of Applied Business at Otago Polytechnic. Her exemplars are available for you to look at.

Summary weeks one to three - Part three

2010-08-17T15:20:00.001+12:00

Misha uses a wonderful analogy about her challenge to find a definition for evaluation: " ...it was a bit like trying to stuff a wriggling octopus into a small box...." Don't you love it. I was interested to read, with a reference no less, about the newness of evaluation - it is not a process, in the formal sense, which has had a lot of time to evolve. What do you think?

octopus tattoo by kusine

How did our ancestors gather feedback and evaluate their activities? For example, is the lack of evaluation the reason why chamberpots were happily tipped out of windows and into the streets in the US and Europe for many years up until the mid 1900s? Or was evaluation the primary reason why the practice was changed? The initiators of change - Ardern and Lockett (1914) - by describing the activated sludge wastewater treatment process to their peers invited critique and review; and through this evaluative process enabled the technique to rapidly become the most widely used biological wastewater treatment process.Read more: Apologies I digress.....

Meena and Misha have also looked at the Career Development taster and come up with a number of solutions which are different again. They suggest both a needs analysis and an effectiveness evaluation and more. I really like the suggestion: "Know our learners - Perform a diagnostic survey and pre-technology survey to analyse learners' learning style and technological understanding to troubleshoot through learning process." The other types of evaluation which are suggested are: Usability- how easy it is to navigate around and not lose your path. Viability. Review-Done during design /Development phase.Usability test before the module goes live. Expert review. You are going to be busy - which kind is most pertinent do you think?

Some questions and suggestions to you
When you do the Evaluate This exercise take notice about how you decide what to look for, and what criteria you use and why? When you evaluate formally these will need to be explicit. The differences between your evaluations of the tasters illustrate that some things will stand out more than others for each of you. When it comes to evaluation, everyone brings a different perspective so that is why a careful and structured approach to evaluation helps to remove any subjectivity which may creep in.

Don't forget, even if you are working in pairs, what you post on your blog needs to include some material which reflects your uniqueapproach - Kevin and Jon demonstrate what I mean by this and to some extent Dana and Louise.

A tip about using images on your blogs
One thing to be careful with when using images is to make sure you reference them, and make sure you are only using images without copyright restrictions unless you have obtained permission to use them. For example, I only use images which are: Public Domain and Creative Commons by Attribution or have other some rights reserved licensing. A good place to find material is: http://www.flickr.com/creativecommons/

Weeks one to three summary - Part two

2010-08-17T15:16:00.004+12:00

art at Denver Art Museum by jkonrath

The summaries continue with information about two different perspectives about the Engineering Mechanics taster.

Rachel in her post about evaluation processes has provided some interesting examples of evaluation - some worked and some did not. She illustrates how evaluation when it is inappropriate is a waste of time, and where important aspects are missed because the right questions are not being asked.

Rachel and Katie have chosen the Engineering Mechanics taster. Look out for the link to the finished product. They have prepared this on Wikispaces. They have decided that a needs assessment (also called needs analysis) is an appropriate evaluation method. One of their suggestions was to use "Gilly Salmon's five-stage method as it would allow the [lecturer] to move the students through the e-learning meeting and [evaluate] their needs within the course".

Check out Katie's cartoons about evaluation - they are great. Katie has also defined evaluation using a back stitch analogy: "To move the stitch forward you must always go back first and revisit where you have just been...and when it is going well you check up less often!" How true is that! Check out how Katie's eLearning Guidelines align with quality on her post about quality.

Mareena and Veronique have decided that a formative evaluation in the form of a needs analysis is required for the Engineering Mechanics taster. I really like the way Veronique has hyperlinked to Mareena's post and to the case study: "I hooked up with Mareena for this exercise. We looked at the Engineering Mechanics case study." They are very clear about why the evaluation process they suggest is appropriate because it would "...help to inform teaching staff of the usefulness and effectiveness of eLearning as well as identifying what additional support students and staff might need, and ensure time and effort is not wasted on implementing and developing resources that may not be useful."

One of the solutions suggested by both pairs which have looked at the Engineering mechanics taster was to find out how the learners were using the Merlot resource. Rachel and Katie suggest using an experimental model with two groups, a control group without access to the resource, and another with access, and then to investigate if there was any " ...difference in their formative or summative assessment scores." Veronique and Mareena in contrast suggested: " ...trial the eLearning resource with students; this could be done by observing the students using the resource and providing a follow-up interview or questionnaire." Which approach appeals to you?

Weeks one to three summary of blog posts - Part one

2010-08-17T15:04:00.002+12:00

Wonderful Fish! by Katie

There have certainly been some golden achievements in the last three weeks, and I am really enjoying the way you have all been swimming around exploring the murky waters of evaluation - it is great to read your ideas and discoveries on your blogs. Everyone appears to have completed the tasters - great going - and hats off to Dana and Louise who have completed the Evaluate This exercise. I have tried to summarise some of the key points you have made and attempted to point you to others' posts to highlight some of them. Lets continue the debate on the email forum. The post is broken up into three parts otherwise it will be longer than I really like, but you have done so much...........

Jon in his post about the importance of evaluation has modeled good design and made some great points. His blog certainly catches the eye, and the posts are colourful, and interesting. Certainly I find short chunks of text interspersed with images and headings, help me read the blog more quickly. They also draw the eye to the important items and key messages you are putting to the reader. He has some experience with using evaluation to reach some of the holders of funding, and can see the need for it in teaching and learning.
Jon aligns his assessment of the tasty taster about Career Development with soup - a great metaphor for formative evaluation. I really like the suggestion of using the KISS rule first (Keep it simple). See if you can discover how his solutions are different to those provided by Dana and Louise.

Jon's partner in crime, Kevin struggled off his death bed to complete the taster. He sure looks ill. See if you can spot whether the delerium of man-flu has crept into his post. The Career Development taster was chosen as Kevin says because there was more potential for finding solutions. He suggests a comprehensive list of strategies such as staircasing and PD for the teachers.

Mareena describes one of the most important indicators of learning - the assessment process. The effectiveness of the teachers is also an important part of this, and she has illustrated how the terms evaluation and assessment are one and the same. In eLearning there has been a strong focus on evaluating the look and feel of materials, and the instructions for learning as well as the strategies. We have become aware that the success of students depends strongly on the environment and their experience and there are many factors which can impact on this.

Alex has come up with some great explanations about the differences between assessment and evaluation:
" ...you assess a learner and evaluate the course. So, assessment is identifying the capability of an individual (or group) to perform a task. Evaluation is identifying how well the instruction or resources supporting the learning and assessment of a task/concept/process worked for the individual/group." It is interesting how a company, in comparison to a tertiary institution, is more likely to undertake evaluations to gather evidence to investigate the return on investment (ROI) in eLearning. Now Alex has found a partner, we will be hearing more about the mock evaluations.....

Louise and Dana have both posted the work they have done on one of the tasters and the Evaluate This exercise. Dana and Louise have also chosen the evaluation taster about Career Development. See if you can spot the different perspectives. For example Dana says: " the most important issue is the lack of knowledge and skill in how to evaluate an on-line course, its design of materials and how to investigate the level of support required for the students". In contrast to Jon, they have discussed both formative and summative evaluation as possibilities, and have decided that the developer needs to know more about:

In the Evaluate This exercise, both Louise and Dana have used screen shots well to illustrate the points they are making, There is an explanation about why the Essay writing module is appropriate for the Bachelor of Nursing. They have chosen eLearning Guidelines for learner centred and teaching relationships. Dana has also posted two statements to illustrate why these were chosen, and Louise has provided some rationale for them. See if you can find his sayings. Dana defines evaluation as:

Evaluation requires deciding on the purpose of the evaluation and can involve the following questions and responding appropriately:

Week Four: Methods of evaluation

2010-08-16T22:35:00.016+12:00

Image: Diagram from a presentation by Professor Tom Reeves

Your explorations of the evaluation tasters and the Evaluate This exercise have hopefully helped you to understand that there are different reasons for doing evaluation. The evaluation type or method you choose for your project will depend on why you are doing the evaluation. During this topic over the next week, you will be investigating both formative and summative evaluation methods such as: needs analysis, usability, effectiveness, impact and maintenance evaluation. These methods are outlined in one of this week's activities - the presentation by Professor Tom Reeves - it depicts "Six Facets of Instructional Product Evaluation. Review. Effectiveness. Maintenance. Formative. Needs Assessment. Impact. Development Activities".

The presentation is based on the book by Reeves, Thomas, C. and John G. Hedberg (2003), Google Books, 2008 Interactive Learning Systems Evaluation (click preview to view chapters), Englewood Cliffs: Educational Technology Publications - available in the Manukau Institute of Technology library. You can request a copy of one chapter to be emailed as well.

Remember evaluation is generally done as a measure of quality.

Diamond (Pink) by Clagnut

On the email forum we will be discussing different types of evaluation, taking into account the reasons for doing the different evaluations. Sometimes people who do not fully understand the processes of evaluation, jump straight in and talk about doing surveys or interviews, which is a common mistake; these are data collection methods, and only a small part of a bigger evaluation process.

If the broader evaluation method, that is, whether it is formative or summative is discussed and planned thoroughly, very useful information can be obtained. If you are clear about the phase of eLearning to be evaluated, the method to be used will become clear. For example, if eLearning is in the early planning stages, a needs analysis will be appropriate. However, if an eLearning course or resource has already been developed and has been launched as a finished product, then an effectiveness evaluation is more relevant.

As you have probably already realised since viewing the presentation - Why is evaluation so important? - evaluation is an ongoing process from the ideas phase right through design, development and implementation. This concept is reiterated in this week's topic with a more detailed explanation about evaluation and the ADDIE model on an audio I have made. There are lots of activities this week, and they are listed in the Methods section of the course wiki. They will prepare you for the next phase - planning your evaluation project so buckle yourself in for the ride.

First of all though - would you eat these pizzas? Watch and decide for yourself.

Week 3: Continuing - What is evaluation and why evaluate?

2010-08-10T12:02:00.006+12:00

Image: Winter fog by bronwynannh

This pic is a view from my house in the hills - feeling cold? We are now in week three of the course, and continuing the same topic from last week. So if you are feeling a bit behind, you are not, as two weeks were scheduled for this topic. It is very good to see you are managing to find partners and participate in the activities. Here are some stats to keep you up to date: there are 15 participants in the course; fourteen of you are subscribed to the group email, and eleven people have provided the link to their blog. If you have not set up your blog already, make it priority this week - more instructions further on in the post.

We are continuing to look at evaluation and find out why it is necessary. In this post, I summarise what you need to have done to get started, and remind you what you should be doing this week. If you are still unsure after reading this post, the best thing to so is email me (as some of you have done), or email the group

(ee4ep@googlegroups.com) for clarification - you need to have subscribed already - Go to: http://groups.google.com/group/ee4ep - Make sure tell me who you are so we don't get any mystery guests.

Getting a partner

If you don't already have a partner - put out a request to the email group. If you still have no luck, email me - we do have an uneven number so we may need a group of three.

What you need to have done to get started

Firstly read the instructions in the Introduction page on the wiki. Priority is to:

Set up your blog and post the link to the Wiki discussion for the Introduction page or the Group email. I will add it to the Course blog.
Join the email Discussion Group.
Get a partner.
Check out the tasters with your partner.
Read about the assessments.
Say hello on the email group.

What to do this week
Look at the Course timetable - we are continuing the topics about evaluation - the nuts and bolts of evaluation - you can find the information and activities on the wiki at: Why evaluate?
As part of this you are doing a mini evaluation project - Evaluate this!

Attend the web conference if you can. 11 August - 18:30 – 19:30. Check out the instructions on the Home page. This will be a general discussion about evaluation in eLearning from your own experience.

Week 2: What is evaluation and why evaluate?

2010-08-02T12:04:00.001+12:00

Image: Snowy is about to dive by Daveynin

What is on this week?
We are starting to look at evaluation more closely to find out what exactly it is and why you should do it. So check out the material for the topic - Why evaluate? There are several activities, and the ones I want to highlight are the 20 minute presentation about the importance of evaluation and also the mock evaluation exercise I would like you to do - Evaluate This. There is a case study to look at and material to evaluate. So partners need to buddy up with another set of partners to do this exercise. There are also sample evaluation plans and reports from previous students to look at - Exemplars. The reflection exercise will guide you with some of the information you can post on your blog. This topic runs over two weeks so take a deep breath and jump in.

What happened last week?
The first week has flown by, and there are now15 of you in the course. Don't forget to find yourself a partner to work with in the course; some people have taken the plunge. You might like to break in your "evaluation teeth" with your partner by checking out the tasters, and discussing the type of evaluation which would fit. As you can see by the posts rolling in to this blog, six people have got their blogs sorted. It is a good idea to get your blog up and running as soon as you can because history has shown that the longer you leave it, the bigger the thorn it becomes in your side.

What did the punters say?
Kevin's sense of humour is shining through on his first post which has a great pic of him - but is it? Kevin also tells us, interspersed with great pics, what he has started looking at in the course. Katie has also given us a peep into her world, and already has a great idea for an evaluation project. Dana has posted a reflection about the Elluminate web conference, and his observations. You may like to listen to the recording of the session. There were a few technical hitches but we got there, and people introduced themselves, and I talked through the assessment requirements and people got to tell us about their area of work. Rachel gives us an overview of her past experiences with evaluation on her blog post. A great start folks!

How to manage all the blogs
Although the course participants' blogs are feeding into the course blog, you might also like to subscribe to each blog using Google Reader. This is a good way to manage the blogs because you can just quickly read the new posts. Otherwise just keep one eye on the course blog, and read each new post as it rolls in. It would be great to see you giving each other some feedback - make sure you have Comments enabled on your blog so your class mates can do this.

Welcome to the Evaluation course - Introductions

2010-07-25T18:57:00.000+12:00

Image: Introductions by Wonderlane

Hello everyone it is fabulous to see there are 12 people enrolled already. You might like to see who your compadres are by having a gander at the class list. I can't wait to see your introductions, and find out who you all are and where you work. These can be on your blog, on the wiki or on the email forum - instructions are in the Introduction section on the Course Home page (http://wikieducator.org/Evaluation_of_eLearning_for _Effective_Practice_Guidebook). There is a short introduction on there to tell you how I got into the field of eLearning.

First things first - please read the information on the Home Page carefully.

For example, check out the course timetable - this gives you the list of weekly topics and activities. Make the timetable your pin up on the wall for the next 17 weeks. It is wise to try out the link to the web conferencing as soon as you can. The first sessions will be so we can meet each other and start getting to know each other. There are two choices for web conferences this week: Monday 26 July 16:00 to 17:00, OR Wednesday 28 July 18:30 to 19:30. After that there will be fortnightly class sessions where myself or a guest speaker will do a short presentation, and then we will discuss the scheduled topic of evaluation.

Email Discussion Forum: once you Apply for membership to the class email group, I will use that as a means of getting information to you quickly. We can also use the email forum to discuss stuff about evaluation, and for questions and answers. This blog will be used for the weekly update and to summarise the week's activity, and a link to new posts will be sent out on the email group.

The other places to look at on the Home Page are the tabs called: Introduction | Assessments
All the information you need to get started is there. If you are unclear about anything, or get stuck when setting up your blog or joining the email forum or the wiki, email or call me as soon as possible. The phone information is in the Handbook you were sent, and my email is on the Home Page. So lets get down to it.

Evaluation of eLearning for Effective Practice course starting 26 July 2010

2010-06-21T14:18:00.005+12:00

Image: Kereru (New Zealand Wood Pigeon) by philip_wgtn_nz

Another round of the Evaluation of eLearning for Effective Practice course is about to start on 26 July 2010. You will discover how to plan and carry out an authentic evaluation project relevant to your work. There are many wings to the evaluation process. To obtain valid and reliable information an evaluation must be planned carefully, and involves a lot more than running a survey. You will be encouraged to look at the big picture and explore a variety of methods to find the answers you need. This is a very practical course and is based on up-to-date models. The course is facilitated online by Bronwyn Hegarty (Educational Developer) from Otago Polytechnic and administered by Manukau Institute of Technology.

Evaluation findings can be very useful if you are trying to make quality improvements in your practice and/or in your organisation or verify the strengths of your innovation. There is nothing like hard data and a well written report for influencing management decisions.

In the course you are supported to choose the appropriate evaluation project for your context. Some of the choices include : Needs Analysis - what sort of eLearning approach is suitable; Usability - testing of courses and products under development; Effectiveness evaluation - an estimation of how well a course or product is working. Every fortnight the class will participate in real-time web conferencing sessions about specific topics as listed in the schedule. Plus there is an opportunity for anytime online discussion. The course schedule is currently being re-organised on the course website.

To enrol, please contact the administrator at Manukau Institute of Technology (MIT)- details about the course and the programme are on the website.

Evaluation time for the evaluation course

2009-07-13T21:22:00.004+12:00

Well the course is now officially over for the semester. A few people have completed their reports, and several of you are not far off, either in the throes of writing up your data or preparing your reports. No worries for those of you who still have to collect data and write it up. There are plenty of excellent examples to look at which will inspire you to keep going.

It is now evaluation time for the evaluation course. Some of you have completed your final assignment and others are at various stages. No matter. If you expect to complete the course within the next two months, I would very much appreciate if you could fill out the End of Course Survey for Evaluation of eLearning for Best Practice 2009. Click Here to take survey Please fill out the survey as soon as possible by 17 July 2009.

If you are planning to carry on through semester two and have made arrangements with me, please ignore this request as another survey will be conducted at the end of the year.

Week 15 and summary of weeks so far

2009-06-15T11:57:00.006+12:00

Image: Snow Mountain by ritaoksa

It is great to see so many excellent plans and evaluation projects underway. We are in week 15 of the course which is scheduled along with weeks 14 & 16 for writing your reports. No worries if you are taking it a bit slower than the schedule. There are a few people with draft plans which just need submitting so I can mark them and it would be good to get these. If you have not already done so, please let me know when I can expect your plans for marking, and your draft results and reports if you are working a different timeline to the one you initially indicated on your plans. It gives me some idea of when the marking is coming in so I can plan the next few months. I am happy to grant extensions, but it is best not to lose your momentum if you can keep going steadily.

So far three of you, Rachel, Michelle and Debra have put up some draft results for feedback. Rachel has posted the findings from a student survey on her blog and this is a good example of how to lay out your findings. It is easy to read and has enough graphs to help the reader visualise what is going on. I have made some suggestions as comments: 1. comment one and 2. comment two on her blog and also in a Google doc, and no doubt Rachel will make them available to you all. I hope Google docs works this time. Sometimes Internet Explorer does block Google docs as it regards it as a pop-up, so check the opt of your browser for this if it wont open and click and you will be away. Another option is to use Scribd and a URL can also be generated on this web facility for posting to your blog.

Michelle has extricated some excellent information from her participants, and explained it on her blog post. The findings from an expert review and surveys are promising. Once the responses from the survey are collated into a Table of the number of responses for each rating on the Likert scale and % frequencies they will be good to go. There is an example of how to do this on the course wiki. (http://wikieducator.org/Evaluation_of_eLearning_for_Best_Practice#Weeks_twelve_and_thirteen:_Analysis_of_data_and_results)

Michelle has also discovered one of the dilemmas associated with using a five or four part Likert scale - uncertainty when people choose the neutral option. My initial feedback is on the blog and I will give detailed feedback once I get access as a collaborator.

Debra has posted a description about how the data collection process went and some of the good things as well as some of the challenges. For example, the ease of getting date exported from Moodle and the time factor for people participating in the evaluation. Debra said she found it difficult to get enough people to respond to the surveys, and Hervé also mentions this. I wonder if anyone else found this an issue. Although it is okay to have a small number of respondents in a project for this course, in a "real" organisational-wide evaluation the response rate could be critical. We have found that some motivational announcements from the CEO help stimulate people to respond. Happily Debra feels as if her sub-questions have been answered and she is promising some draft results in the near future.

Hervé mentions some of the tricky bits associated with Survey Monkey in his post, and is almost there with his findings. Kay has been carrying out her evaluation and has posted something about her data collection process. Krishan is close to posting his results and tells us about his progress on his blog. Pradeep has the final version of his plan on his blog.

Some people are seeking feedback on their plans. Joy has a plan for an effectiveness evaluation of a computing course and needs help with the questionnaire and interview questions. Elaine has come back into the course burning rubber and would love some feedback on her draft plan, thanks to helpful colleagues. Adrienne has a draft plan up and might be feeling a bit lonely so please give some feedback on her plan. And it is not too late to give feedback to Heather on her draft plan.

Sam also has a draft plan up on her blog waiting for feedback, and we need to catch her before she shoots off to her new job - read her news on her latest post. Minhaaj would love some feedback on his plan I am sure and is currently interviewing a group of students who engaged in a collaborative online project called the Flat Classroom Project. Catherine has submitted a final plan and is awaiting marking. And in case you were wondering where the others have gone - Craig, Iain, Liz and Debra RW are extending into semester two. Please let me know if you need to do this as well.

Weeks twelve and thirteen: Analysis of data and results

2009-05-25T20:53:00.002+12:00

Image: Numbers_graph.png by surachetch

Although the next two weeks are scheduled for you to analyse the data you have collected, most of you will not be at this stage. However, you can still have a look at some of the material and start thinking about how you are going to analyse and organise your results. Remember to give feedback to at least two others on their evaluation plans.

Things to do:

View the presentation called Preparing evaluation results - questionnaire responses.
To view the presentation in full screen click the icon at the bottom right of the slide.
You can also look at the spreadsheet Descriptive statistics % Frequencies which accompanies the presentation to see how some samples of data have been analysed.

Look at Helga Wientjes' draft evaluation report. Unfortunately some of the Figures are not appearing - unfortunately Helga is away so permission cannot be obtained to access the other Figures but there are some there to look at.
Investigate some other examples of results and reports in the Resources area to see what others have done.
Decide how you will present your data - tables, graphs (what type of graph).

Post your initial ideas to your blog and ask for help if you need enlightenment.

Write descriptively about your results - again check out some examples.
Submit a draft of your results to the lecturer via Digital Dropbox, your blog or via email.Remember to notify the lecturer on email when this is done.
Post a summary of your results to your blog.
Provide feedback to at least two other people on their blogs.

Weeks ten and eleven: Conducting your evaluation

2009-05-11T10:35:00.005+12:00

Image: Focus Group Discussion by Singapore2010

The next two weeks are scheduled for conducting your evaluations - no worries if you are a little behind with finalising your plan - you can still organise times to collect data from your participants. Some of you have negotiated extensions and will be at a different phase to the class, and this is fine. There are some pointers for your plans further on in this post. Remember to make contact if you need advice.
Presentation
There was a remarkable turn out to Gordon Robinson's presentation about Maintenance Evaluation and you can listen to the recording if you were unable to make the session. It was very informative and interesting and Gordon and Craig told us about the Blender resource they use to ascertain the suitability of their training courses for eLearning. More on this later.

Plans
It is great to see so many plans up and running for us to give feedback on. People have been sending out email notices and links so make sure you give feedback to at least two people's plans. I will be giving detailed feedback on your draft plans so make sure if you have your plan on Google documents that you provide access for me - add me as a collaborator - I wont be re-writing your plan but will be able to add comments. Thanks to those of you who have done this already.

Some general tips
A few things have come to light as I am reading your plans.

Check the assessment and marking criteria to make sure you have included everything - my feedback will help with this.
Remember to prepare the plan as if you are writing a proposal to present to your managers, and/or people who know nothing about your work.
Write in third person not first person
Introduction - this needs to give a brief overview of the evaluation and lead the reader into the structure of the proposal. See Michelle's introduction for her plan plus you need to outline the sections which will be in the plan which Michelle has not yet done, but which Rachel has done well in her plan.
Sections can be shuffled around, for example - combine purpose and questions, combine sample and instrumentation etc. Whatever suits your approach to writing a plan as long as the items are included.
Timeline and budget need to be as real as possible. For example if mentioning your time - estimate the hours and add a $ amount - this gives you some practice in thinking through how long this sort of project may take.

Theory and articles - where possible base your plan on a similar type of evaluation which you have read about.

- the task you were given in previous weeks was designed to prepare you for this - so if you have not read an article or two about a current evaluation project you may be disadvantaged. Not only regarding time but also in having something theoretical on which to base your project - we can learn a lot from others' approaches.
References - use APA referencing - include at least one article.
Searching for articles - there is lots of stuff in the Resources area on the wiki so this is a good starting point.
Internet searching - there are lots of open and free articles around.
Remember to save to delicious and tag with eval09 so they go directly to the course wiki - material just in - area for others to see. You can also search delicious.
MIT library searching - have a look on the library site. You can also request articles if not available electronically or are in other libraries in NZ.
Google Scholar searching - search and if you have to pay for articles look for them in the MIT library journal database - failing that ask me as I have access to a University database.

Week nine: Prepare and present an evaluation plan

2009-05-05T14:52:00.004+12:00

Image: Presentation area by Plutor
Now that you have your ideas for your evaluation projects coming together you can gather them up into a formal plan. I am impressed by the level of activity so far. Once you are happy with your draft plan you can present it to the class for feedback on your ideas. Some of you have already done this on your blogs, and others are waiting for my feedback before "going live". By the time I get round everyone, week 9 may well be over, however this is not critical as you can still begin organising the logistics for the evaluation for the next couple of weeks and some of you still have questionnaires etc to develop.
Things to do:

Thursday 7 May 7-8pm NZST - attend a guest speaker presentation with Gordon Robinson via the Elluminate Meeting link. Gordon will speak about the evaluation plan he used for the 2008 course when he carried out a maintenance evaluation. The title of his talk is: Evaluation plan for the Hazardous Substances Awareness course.
Choose a medium for your presentation. Straight on your blog is fine or you may wish to link from Google documents or create a separate slide show presentation.

Check the suggestions for the presentation in the Assessment and Marking Schedule.
Provide feedback to at least two other plans.
Incorporate feedback and submit the written evaluation plan (Part two A: Negotiate and write an evaluation plan) in the Digital Dropbox in the course Learning Management System (let me know on email if you do this) or on your weblog or via email to the lecturer.
If you prefer you can put your plan in Google docs and post a link to it on your blog - remember to publish it as a web page and give the lecturer access as a collaborator.

Summary: Weeks seven and eight - negotiate and write an evaluation plan,

2009-04-26T16:58:00.013+12:00

Image: Up In Flames by Ted Percival

Evaluation plans are swirling like lonely cinders as Margaret Mahy said in her story about the witch and the cherry tree.

Catherine's most recent post has brought something very important to the forefront about customising the eLearning guidelines to suit your project. Remember you can write your own elearning guidelines to suit the purpose of your evaluation. I have given Catherine some feedback on her draft ideas for eLearning guidelines. I have added an example of a guideline written from "scratch". Once you settle on your eLearning guidelines you will need to write some sub-questions for each - 1 to 4 is a good number. This way you will be able to get the answers you need - remember to get the right answers you have to ask the right questions. Please look at my feedback to Catherine for some examples of what I mean by this. Don't be put off by the length of my feedback. You also need to be very clear about the purpose of your evaluation - if you write down what you are trying to find out, this will help to clarify whether the guidelines you have chosen are appropriate.

Adrienne's plan for an effectiveness evaluation is shaping up nicely. The guidelines she has chosen are excellent and very relevant and just need some sub-questions. For example for TD4 - What makes an effective online discussion? Example sub-questions: 1. Which types of online discussion promote learning? This will enable her to look at the email discussion forum, discussion on the course blog and participant blogs. 2. what are the features of an engaging interaction?

Decisions and outcomes
Remember everyone your decisions must be directly related to the outcomes of the evaluation, that is the answers to your questions. For example, in Adrienne's case, if the use of participant blogs inhibits regular interaction between course participants is because they prefer to have face-to-face or email discussions, then Adrienne will have found out that blogs may not be a good way to promote discussion and email is, therefore as one of the outcomes may be to find the best means of communication, the recommendation may be todiscontinue blogs for communication and use email.

Minhaaj has made a good choice with regard to summative evaluation and the Constructivist-Hermeneutic-Interpretivist-Qualitative Paradigm for your evaluation project. It is refreshing to see his own beliefs (paradigm) regarding statistics coming to the fore. He now has a plan for evaluating the Flat Classroom project based on a qualitative approach where he will interview participants who have been involved in a collaborative learning environment.

Kay has made a change to an elearning guideline and asks some pertinent questions about academic process. Her approach is quite different as she is going to be testing criteria to measure quality, as well as getting students to look at usability.

Hervé on an earlier post in two parts has provided an excellent background to his project, proposed methods and articles. He has posted his draft plan as well for a formative evaluation - great guidelines - and is seeking feedback.

Joy has posted her guidelines and information about her ideas for an effectiveness evaluation. Once again I am suggesting some sub-questions. For TD2 (Do students get clearly defined learning objectives that assist them in focusing on their learning activities?) Sub-question: Are learning tasks relevant and do they motivate learning? To help more with this it would be good to know more about the course and the aspects she wishes to evaluate. I wonder, are there particular areas of concern - such as too much content, too little communication between students and lecturer, is the assessment working well? etc.

Debra M has some great sub-questions accompanying her guidelines. She has been encouraged to keep the project small with a handful of students and an expert reviewer. Her next step is to develop some guiding questions for the reviewer. Her evaluation will be looking at usability.

Heather has been working hard over the weekend (along with several of you) and now has her draft plan posted. Some really good thinking emerging around the needs assessment she intends to carry out with students - investigating the introduction of a web-based "game-like" quiz into a written communication course - along with other strategies.

I have mentioned a possible "fish hook" which may concern others as well. Is getting information easier when you have a bond with participants in the evaluation or are they just telling you what you wish to hear? Anonymous surveys can alleviate this to some extent and encourage honesty.

Krishan is also looking at conducting an effectiveness evaluation and may have some interesting findings considering the course - Operate a Personal Computer - which no longer relies on a textbook as all the resources are now online.

Michelle also has a draft plan posted for a needs assessment of an online software training course to find out if an eLearning approach is going to be feasible for the learners. She has posted an excellent example of eLearning guidelines with sub-questions. It looks like Michelle has her work cut out preparing a sample course for staff to evaluate using usability criteria.

Pradeep has a draft plan for a needs assessment to introducing eLearning into the mechanical engineering course. A great variety of approaches to gather data including student focus groups and survey and expert review. It is good to see his notes under the guidelines. He has also outlined on another post some of the issues for introducing eLearning and the student-centred focus in his organisation. There is also an interesting overview of an article about an engineering evaluation project (EASEIT-ENG Project) on another post.

Rachel has some excellent ideas for her evaluation; a needs analysis associated with teaching hotel management. As part of this, he is looking at strategic documents for the organisation and interviewing "experts" so she can investigate the best direction for introducing flexibility. At this stage, time constraints will make it difficult to gather data from students. of course this can be done outside this evaluation project.

So we have a variety of evaluation project approaches: effectiveness evaluations, needs assessment - with some focussing mainly on students' perceptions and others looking at strategic documents and expert opinion. We also have a few usability evaluations for new online courses and also formative evaluations. Well done everyone! I am impressed with your activity and hard work. Now you can get on with the fun part - talking to people and gathering the data. It is always a bit like xmas finding out what others think of your ideas and approaches.

Some others are not mentioned here but have been working on their ideas and I am sure they will appear very soon.

Weeks 7 & 8: Negotiate and write an evaluation plan

2009-04-20T21:01:00.004+12:00

Image: Pink Ball by DeclanTM

Welcome back. I hope you managed to have some relaxation time over the holiday break.

Over the next two weeks you will need to firm up your evaluation planning and prepare a formal plan - information about a template is in the Things to Do list below (taken off the wiki). By now you should have a pretty good idea of what you intend to evaluate. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment. And do contact me with your ideas and draft plan before submitting this part of the assessment as we need to negotiate your plan first. Read on for the fun stuff....

Guest speaker: On Tuesday 21 April 19.00-20.00 NZST - Helga Wientjes from the Department of Conservation presented to the class. Here is the Elluminate recording for your viewing pleasure. - she did a wonderful evaluation project for her workplace when she was taking the evaluation of eLearning for Best Practice course in 2008. It was a formative evaluation to measure effectiveness of a web conferencing training module.

Things to do:

Decide on two eLearning guidelines against which you will evaluate.
Establish the paradigm and model for your evaluation.
Look at this example of an evaluation plan - Usability evaluation
Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
- You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
- Remember this can be modified to suit your situation.
Explore the Evaluation Cookbook to decide on some sampling methods for your project.
Prepare some draft ideas and post them on your blog.
Contact your lecturer to negotiate your plan.
Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Summary Weeks five and six: Evaluation methods

2009-04-06T14:18:00.009+12:00

Image: Eggs-ibit 2004 by annessasgirl

Well it looks like - needs analysis/assessment/Front End analysis - might be a popular choice of evaluation method for several of you. This has come up in some conversations on Elluminate and some phone conversations I have had. This is a logical way to go, when you are thinking about introducing eLearning into your teaching as opposed to actually already being in the throes of using eLearning.

Some people are also thinking about Effectiveness evaluations as well, but most are still trying to clamber over the paradigm hump. I hope you can all leave this behind after Easter and focus on the type/method of evaluation and the ways in which you will collect your data. - to fit with the eLearning guidelines you have chosen. Watch this space for a link to an evaluation plan template.

You will all be pleased to hear that the MIT library is getting the course textbook back and it will be on desk copy. I also have some chapters I can send out if you need something let me know on email after you have looked at the Table of Contents. Use the library distance services in the first instance.

Debra M has found an article which helped her sift through the paradigms: Phillips, R. (2005). We can’t evaluate e-learning if we don’t know what we mean by evaluating e-learning! Interact, 30, p 3 - 6. Learning Technology Support Service, University of Bristol.
It is easy to read and helps everything fall into place a lot easier.

Sam found some wisdom about evaluation following an altercation with a fierce mother hen and a shopping trolley. "Listening to the ideas of others and sensitivity to others’ parameters for success." - very relevant in evaluation.

Joy has plans for a mixed methods approach which looks suspiciously like an Effectiveness Evaluation as she intends to measure whether learning objectives are being met. Here is a link to an example of an Effectiveness evaluation plan by Hilary from the 2008 class.

Michelle has also chosen a mixed methods approach. More to come soon.

Rachel has written a very thoughtful and comprehensive post about her beginning ideas about conducting a needs analysis. She has invited feedback and it would be great to see you giving her some help. The question is - should she sample students who know little about eLearning or would she be best to investigate support systems in place for students who will have to access eLearning? This would fit with one of her chosen guidelines: ST1 - Do you have a way to identify needs and respond to them?

class meeting recording (paradigms, models and methods) and next meeting

2009-03-30T23:09:00.003+13:00

At the last last class meeting, people indicated that Thursday evenings 7 - 8 pm were a good time for a get together. On Thursday 2 April between 7-8 pm NZST, six of us had a discussion about: the theory and practice of evaluation - paradigms, models and methods.
Here is the 2 April class meeting recording.

If you are confused about all the choices and the terms associated with evaluation this is your chance to clarify some things. There was a short presentation and discussion.
The next class meeting will be after the holiday break - 23 April 7-8 pm NZST. Meeting link

I am organising some dates with previous course participants to talk about their evaluation projects. This should help you with your ideas. Once I have confirmed the speakers, I will put up a list with dates and time. Hope to see you there, and as usual recordings will be available.

Weeks five and six: Evaluation methods

2009-03-24T16:52:00.003+13:00

Image: Twitter Bird by tashmahal

For this topic you will need to spend two weeks - this will take you up to the holiday break over Easter - 6 to 19 April 2009.
It is very important you explore the options for conducting evaluations carefully, and choose a method appropriate for your evaluation project, because in weeks seven and eight you will need to start writing your evaluation plan. Some of you may wish to use some of the holiday break to start organising this. Some people have already been talking about doing formative evaluations or needs analysis and there are other terms such as usability, effectiveness evaluation, impact and maintenance evaluation with which you need to become familiar. Talk to Bronwyn if you are not sure what sort of evaluation to do.

Things to do:
1. Read the short article evaluation methods in the eLearning Guidebook.
2. View the presentation by Professor Tom Reeves - Six Facets of Instructional Product Evaluation, (Google Books, 2008)

It is based on the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications - available in the Manukau Institute of Technology library. Remember you can make a request for a chapter - might I suggest you choose one which is relevant to the type of evaluation you wish to conduct. The Table of contents was sent out on email.

3. Find an article or two in the literature which fits with the type of evaluation you are interested in for your project. You will have access to the online databases at MIT.

Check the Resources page on WikiEducator for some examples.

4. Post responses to the following on your weblog:

Outline some information about the type of evaluation you believe will best suit your project with your reasoning.
Summarise the article(s) you have found (300 words) - full reference, evaluation design (paradigm, model, questions, methods), findings and how it aligns with the evaluation project you would like to conduct.

week 3 Summary: eLearning guidelines

2009-03-18T21:44:00.022+13:00

Image: Soccer Practice by mill56

Isn't it great to see all the soccer players out on the playing fields at this time of year. This summary will be an overview of the main activity around the eLearning guidelines. People who did not feature in the previous summary will now get a mention. Again I am in danger of getting carried away as there are so many very interesting blog posts on this topic. It is also good to see that several of you are communicating with each other on your blogs and giving feedback. Impressive! :) The recording for the last class meeting is also available. We had a good discussion about quality and also evaluation projects, in particular, needs analysis.

As you can see from Deborah R-W's blog she has been attending an evaluation seminar for academic quality managers. It sounds fascinating. Deb you got me curious and now I can't stop! :) I did a Google search for Jane Davidson and her book: EVALUATION METHODOLOGY BASICS The Nuts and Bolts of Sound Evaluation - came up. I really like the idea of values-based evaluation which is Jane's approach. Watch for more on this in posts later on...

Elaine has been doing some serious catching up before disappearing for a short while. In her week 2 post she mentions some different methods she has used to collect data in the past when evaluating. There is further consolidation of the theme around quality, and the impact it has on student retention. This has been mentioned by several of you and clearly is a concern for all of us in the online teaching field. The week 3 post has two very relevant guidelines for the courses which Elaine is looking at re-organising in tourism, hence the need for some evaluation. I really like the ideas she has for redesigning the courses to include some authentic project work. See what you think and do give her some well deserved feedback.

Craig in has been doing some serious exploring and writing about eLearning Guidelines which are suitable for his project. In the My Guidelines post Craig has shown us how he worked through several options to find two guidelines which were most suitable. The guidelines will enable him to "monitor the quality of study material", and evaluate a "communities of practice approach to course development". This is a great example of the use of critical analysis (thinking and reasoning and evaluative judgement to solve a problem) is demonstrated in a blog post.

Craig has also posted a slideshare presentation about the Quality Matters rubric used to evaluate distance education. There are forty specific elements you can look at for some ideas. Be careful you read the copyright statement on the website before using the material as is, as unlike the eLearning Guidelines, the material is not freely available for use. Though they cannot control the ideas you might obtain from it. :) You might like to compare the Quality Matters rubric to the Wright and MacEwan criteria which Adrienne told us about in week one.

Pradeep has also been busy posting and in his first post on quality for week 2 sums it all up with the words: "..evaluation is the tool to checking quality." He also makes the point that assessments help to evaluate whether curriculum design is adequate. We do get excited when students pass at a high level and despondent when there is a high fail rate, and this certainly stimulates teachers to look at how their material is presented and to try different strategies to improve outcomes doesn't it?

In his week 3 post on guidelines, Pradeep has selected two examples and I have suggested he add the word, How, to one of them so that he has more scope for his evaluation project. SD3 - [How] do students gain knowledge relevant to employment and/or current thinking in their field? You might like to bear this in mind for the guidelines you choose.

Catherine in her week 2 post has provided a good explanation about why evaluation is important to her, and mentions how complacency can easily creep into teaching and course design and leading to a loss of quality if evaluation is missed out. She mentions self-evaluation and peer discussion as areas of evaluation with which she is familiar, and makes a very good point about "critics sitting on the fence" ready to pass judgement, and in her statement also hits the mark: "No one will pass the comment.. "well what do you expect..it was delivered face-to-face!"

Her post illustrates very well how powerful a robust evaluation can be to assure the critics that all is well and that if it is not, the strengths can be built on and the weaknesses improved.

Debra M has written an excellent overview of her plans for evaluation and has critiqued some guidelines and her own "big picture" questions for her proposed evaluation. Several people have been leaving comments on her blog post and giving her feedback and encouragement. See what you think about Debra's ideas for making the pilot run of the new course interactive and for evaluating its success.

Kay has also been busy posting - week 3 and has chose two guidelines from the teaching area to address a lack of internal quality control procedures. Certainly a checklist for design and delivery is a helpful resource to have in eLearning. I wonder what others use? Adrienne has mentioned a resource she uses. Kay also posted her ideas around quality in her week 2 post and shows us how to connect the dots with others' posts. She mentions "specific indicators for measuring quality such as: assessment of student learning, feedback from students, peers and external reviewers and institutional accreditation procedures.

In contrast, Adrienne in her post has chosen guidelines from the student area with respect to choice, interaction and technologies. It is good to see you are back on board with a working computer.

A few people have found the cube of eLearning Guidelines a tad annoying to use. You can read their comments on Hervé's latest post. Hervé like Debra M and Craig has laid out some issues for his teaching examples and linked them to some guidelines.

Minhaaj on the other hand liked the cube and has put up a very thought-provoking post about diversity and online facilitation. He says, " [if] students are not clear about the expectations, processes, resources and outcomes of the course they are taking. From facilitator to students, it can be anyone's fault but in anycase it deprives the course of its spirit." It is so exciting to have so many of you participating who experienced the Facilitating Online Communities course. I have found that people are never the same once they have taken that course - we see such a change in the way they tackle the online environment. Challenge is a good thing, but supported challenge is essential otherwise the insecure learner will disappear into the cyber mire.

Krishan also liked the ELG cube for the guidelines. Krishan, similar to a few of you, has found a focus on student learning around motivation and technology issues to be important with his choice of the following guidelines -TD2:Do students get clearly defined learning objectives that assist them in focussing on their learning activities? - and TD5: Have a representative sample of students tested the e-learning materials, and any necessary modifications been made?

Iain in his week 3 post appears to be one of several people who are either designing new course material, or running courses which are currently face-to-face and thinking about how they might be presented online. If you fall into this category and will not have any online materials ready in time to evaluate them for this project, you might like it think about doing a needs analysis (front end analysis, needs assessment) evaluation which will help you to choose an optimal design for the subject and for your students. As part of the needs analysis you can look at the resources which are already available online, as well as working with stakeholders, e.g. students and lecturers, to find appropriate strategies for presenting content.

Rachel has done a good critique of her choice of guidelines. TD1 - Is the use of e-learning appropriate to the intended learning outcomes? This is particularly relevant as she is still considering using eLearning approaches for her teaching. Rachel is another candidate for a needs analysis approach. You might to let her know what you think.

Sam has posted a very interesting dilemma associated with the design of materials to provide practitioners with flexible and easy access to education about a new technique for assessing pressure areas on the skin. She has chosen two very appropriate and unique guidelines for evaluating delivery modes and use of technologies. See what you think.

Heather in her week two thoughts has mentioned the need for interaction online to ensure quality. I tend to agree, and although, some people advocate for regular interaction and feedback, others such as Terry Anderson (Athabasca) have written about the debate.

He says: "Contrary to popular belief, the major motivation for enrollment in distance education is not physical access, but rather, temporal freedom to move through a course of studies at a pace of the student's choice. Participation in a community of learners almost inevitably places constraints on this independence, even when the pressure of synchronous connection is eliminated by use of asynchronous communications tools." See Chapter 2, Toward a Theory of Online Learning - From: Anderson, T. (2004). Theory and practice of online learning. Athabasca University. Second edition (2008) also available.

Michelle in her week 3 post has stimulated some excellent discussion relevant to the whole area of designing learning for adult learners. The idea of evaluating whether to introduce an online Training Needs Analysis into an Induction package will fit with her proposed direction of conducting a needs analysis. Check out the info about the OTARA design model
mentioned by Michelle in the comments area.

So in conclusion, there are several people who have made decisions about their eLearning Guidelines and some who are still firming up their ideas. Some of you have also made some tentative decisions about the type of evaluation you will conduct, e.g. needs analysis, formative evaluation. Once you move through and decide on the evaluation paradigm and method you want to work under, and when you start writing your evaluation plan in weeks 7, 8 & 9, you may want to revisit the guidelines you have chosen to make sure they still fit. It is very heartening to see so many of you keeping up with the schedule, but do remember it is only a guide so if you feel as if you are falling behind the group, it is not a disaster. You have two weeks to explore the next topic: Weeks five and six: Evaluation methods, and there is two weeks holiday break from 6 to 17 April.

Week 4: Evaluation paradigms and models

2009-03-18T18:25:00.001+13:00

Image: Bedouin Jerry Can Band by Michael Flynn and the Taranaki Daily News

You can blame the lateness of this post on my pilgrimage to Womad. I see there has been a lot of action on your blogs while I was away, so tomorrow I will post a summary of week three activity. This week is probably the driest one with regard to evaluation theory. You do need to have an understanding about the main inquiry paradigms and models which currently influence evaluation. If you can see it as a set of terms regarding current beliefs about how evaluation is done, things will seem easier, though there are some very long terms to get your larynx around. There is no need to memorise the terms though as I will certainly not be testing you. Hopefully you will get this posting as an email.

Things to do:

Attend the optional fortnightly class meeting on Thursday 19 March 2009 - 7 to 8pm (NZST, +12hrs GMT)- web conference. This will be a discussion about quality and eLearning Guidelines and will be recorded.
Read the article on paradigms by Professor Tom Reeves.
Look at some information about ten different evaluation models.
Compare two different models in an article about evaluation models by Bronwyn Hegarty, 2003.

Week 4: Evaluation paradigms and models

2009-03-18T17:58:00.007+13:00

Things to do:

Attend the optional fortnightly class meeting on Thursday 19 March 2009 - 7 to 8pm (NZST, +12hrs GMT)- web conference. This will be a discussion about quality and eLearning Guidelines and will be recorded.
Read the article on paradigms by Professor Tom Reeves.
Look at some information about ten different evaluation models.
Compare two different models in an article about evaluation models by Bronwyn Hegarty, 2003.

Week 3: eLearning guidelines for quality

2009-03-09T22:08:00.008+13:00

Image: Lines & lights by tanakawho

This may be week 3, but if you are not up with the schedule do not worry. Most people have just completed the week one tasks of introducing themselves and setting up a blog. The best thing for everyone to do now is to follow the course weekly schedule. Please read important notice about my leave dates at the end of this post. Welcome to Pradeep who has joined us and is located at Polytechnic of Bahrain.

This week we are scheduled to look at some eLearning Guidelines which have been developed for New Zealand. These were used by 21 tertiary organisations as part of a Ministry of Education funded and sector wide project on implementing e-learning guidelines. This national project provided a mechanism for developing, introducing, trialling, evaluating, and sharing examples of e-Learning Guidelines. Check out the list of things to do for week 3. The eLearning Guidelines were developed specifically to address some of the issues around quality in online teaching and learning. Some of the factors impacting on quality in eLearning were discussed in the week 2 summary on the course blog.

Make sure you have read the information about the assessments and make contact with me if you have any questions. Remember my recommendation for keeping up with the course schedule is to spend approximately 5 hrs per week on it. Lots of people have found that marking dedicated time slots on the calendar works a treat and helps you fit your study into your busy weeks.

Optional fortnightly class meeting: 19 March 2009 - 7 to 8 pm. I hope you can attend the web conference. This will be a discussion about quality and eLearning Guidelines and will be recorded. If this time does not suit, please email times and days to the email group so we can find a more suitable time and perhaps add another day and time to the mix.

Process for Questions and comments
General questions and discussion can go to the email forum: evaluation-best-practiceATgooglegroupsDOTcom and personal email messages can be sent to bronwynDOThegartyATgmailDOTcom (letters are used on here to deter robots in case you are wondering). You can also generate discussions by using the comments area at the end of people's blog posts. It is alway great to get a comment when you have spent time and effort posting on your blog.

My Skype name is at the end of the group email, and if you would like me to Skype you please send me your user name and some suggestions for times and days to make contact. Generally, each week I will post a summary on the course blog with links you may wish to follow and this will help you pull some of the information together for the week. I also make an announcement at the beginning of each week regarding the main direction for the week.

I look forward to hearing about your ideas for evaluation and working on your projects with you. I am on leave from 12 to 17 March so will not be responding to emails over that time. Please contact Oriel Kelly, programme manager of the Grad Cert at MIT if anything comes up which just cannot wait.

Summary Week 2: Quality and evaluation

2009-03-09T21:51:00.010+13:00

Image: Butterfly My First by photogirl7
What a busy week you have all had. Everyone pretty much has their blog all set up and ready to go and several people have posted responses to the questions for Week 2: Quality and evaluation. Hence my summary has lots of substance and questions for you to ponder. I am very pleased with the progress everyone has made especially those of you for whom blogging is a new experience. It has been great to talk to several of you online or on the telephone.
The questions for your response in Week 2 were:
- Why is evaluation important to you and how do you define it;
- What sort of evaluations mentioned on the presentation are familiar to you already and why?
- Why is quality important in eLearning?
Answers to all these questions will vary depending on your personal views and perspectives and the context in which you work. For example, Adrienne's blog post is very relevant to her work, and she has provided us with a practical list of items which she considers when designing eLearning. There are some very useful things for you to look at when planning your evaluation projects. She also makes a very important point about the link between quality and student retention - a subject close to us all I reckon.

DebraM in her post has listed several of the evaluation items she was familiar with from the presentation and makes a good point about evaluation, "it can be used to 'pat yourself' on the back, as well as 'allowing room for improvement'. She has presented a very comprehensive discussion about quality - it is all in the "eye of the beholder" well the user's opinion anyway. And check out the first eLearning Guideline to be selected.

Sam in her post has offered a different slant on quality again and brings information about two different organisations and their views of quality to our attention - Concord Consortium versus Articulate. I love her discussion about the idea of rapid eLearning offerings and the demise of quality if we just "throw-up online" without taking time and care to provide a thoughtful product. Lots of ideas for debate in this post. See what you think.

Herve's post refers to the presentation as did DebraM, and he says the concepts about evaluation were all familiar. One thing I found useful from Herve's post was the perspective that evaluation in eLearning may be even more important than in a traditional classroom because the interactions are different. He and Joy are on a similar wavelength - do you agree with Herve and Joy?

Joy alludes to the way we evaluate constantly when teaching and others have mentioned this as well. She has also provided us with some definitions from the web as well as her own take on evaluation. Joy is a fan of the ADDIE model of design, and gives us a good overview of why it is a useful model to follow to ensure quality in eLearning. Joy, like Herve, also tells us how the different ways we interact online compared to face-to-face classrooms, can make it hard for lecturers to gauge how students are understanding or enjoying their learning. Therefore, evaluation is even more important in eLearning.

Iain like many of you, is focused on evaluating to ensure his students are getting the best learning experience possible. Yes we do want them to keep coming back for more don't we? I really like the description of the ways in which he uses learning outcomes as a focus at the end of each lesson. It encourages students to self-evaluate their understanding and also helps Iain measure how well the class is doing. Iain's company does a very comprehensive evaluation of student satisfaction and success and it sounds like evaluation is integrated into all areas of the student learning. Well done! This sounds like a great model. I will be following your trail with interest.

Rachel in her week 2 post, as well as some insightful statements about quality being necessary for a range of areas, e.g., needs assurance, cost effectiveness, student retention and positive outcomes for students, Rachel provides us with a thought provoking question - “Why is quality important in eLearning?’ – and ask is quality not important in all learning?, should there be a distinction." Good on you Rachel asking this extremely important question.

This has spurred me on to provide a lengthy reply. Quality as you mention Rachel has always been important and in all areas of education, not just eLearning. However, the reason quality became such a sticking point in eLearning harps back to the early days of online teaching. In the early days, people new to online teaching often took the pages and pages of lecture material they used in class and just "slapped" it online. Design principles were poorly adhered to (graphics, chunking of text, formats, navigation, instructions etc) and approaches were often inconsistent re tools, instructions, formats and layouts. Then the "bells and whistles" brigade came along and produced fancy multimedia which no-one could open (software issues) or you could only use with a Nasa-designed computer. :)

Now of course we are much better informed, but the dilemma in the online classroom which several people have mentioned, is the lack of personal interaction in an online classroom - lack of body language or facial expression, unable to observe students and answer queries immediately. And it is more difficult to teach unprepared online - how many of you experienced teachers have walked into a f2f classroom not as prepared as you would like, but comfortable in your knowledge and expertise on the subject and had a great session? I believe it is harder to do this online because people can get so lost and intimidated. They seem to like schedules, instructions, structure and clarity. Do you agree?

This means we have to be much more slick with the way we organise online courses. This is where evaluation in the design and development stages becomes so important - so we get the design of the online learning as good as possible before we impose it on students. And then we need to find out if the approach was effective. I guess it is not quality which is different but they way in which we ensure we are meeting particular standards or guidelines which help us produce quality learning.

Krishan's post also gave me plenty to write about. Krishan makes some very valid points. I agee that having materials with well written instructions is a very important aspect of quality. It is so easy to get lost online. I agree with him about the predominance of summative evaluation in education. In which case, the evaluation is far too late to keep at-risk students in a course. There is nothing more disruptive or soul destroying than a class of disgruntled students. Do you agree?

Krishan mentions his familiarity with evaluative observations - they certainly are an excellent technique for evaluating skills, however I still shake when thinking of being watched while I undertook nursing tasks.

In evaluation of multimedia including web-based courses, observation is a good way to evaluate the clarity of how materials are presented and laid out. By observing how students navigate through, and also how they use the materials, the bugs can quickly be located and fixed. What may seem to us to be an obvious pathway or icon to click is not always so to others. Do you agree?

Minhaaj in his week 2 post, mentions some very insightful ideas, and I certainly agree with his statement about the reason we need to evaluate: "without the evaluation process, learning will be restricted to a traditional non-interactive, one-way process which is not only monotonous but kills the enthusiasm and interest levels of participants." He mentions an evaluation rubric used in week 3 of a course development called Enhancing lessons, so participants could measure the quality of the materials they were creating. It is fair to say than some sort of structured approach to measuring quality is important if educational materials are to have credibility. I do wonder though whether a rigid evaluation framework stifle would creativity and innovation. Is there a balance and a compromise? I believe there is but it needs to be carefully considered - do you agree?

So in week 2 we certainly had a wide range of opinions about quality and evaluation. It was helpful that you shared your previous experience with evaluation and also the material you found when exploring the topic.

TIP: It is good to see references at the end of some posts - it is also good to have hyperlinks in body of the post especially when quoting or mentioning items you have obtained from your explorations and reading. Some of you are doing this very well.

Recordings of this week's meetings

2009-03-05T11:39:00.004+13:00

Image: Satellite Dishes by watchsmart

This week, we had a couple of good sessions on web conference using Elluminate and a good turnout to both sessions. Unfortunately it was not straightforward for some people who could not access Elluminate from their organisations. Hopefully the IT issues will be resolved by the next time we have a session in a fortnight's time. There were others who had other things come up, or could not get Elluminate to stabilise, but don't worry, the sessions are optional and recorded (not the same though is it), and I look forward to you all being able to connect at some time or other.

The Tuesday session was also pretty unfortunate for me as I kept getting booted off Elluminate and thankfully the group was able to keep conversations going without me. My Internet provider is looking into the problem. Apparently it has to be really bad in the analysis Telecom does on the line for them to actually do a lot. Looks like I might be investing in a satellite dish or six! :(

The recordings of this week's sessions are now available and are also on the wiki.
Tuesday 3 March at 3- 4pm - the recording of the meeting.
Wednesday 4 March at 7- 8pm - the recording of the meeting

It was good to clarify the assessment requirements and discuss people's backgrounds in elearning and their interests in evaluation. Some good ideas came through last night about quality in elearning and evaluation as well.

Last night's group asked for another session in a fortnight at the same time - 18 March 7-8pm. So hope you can make it. I will not be able to run the Tuesday session that week as I will be away, but suggest Tuesday 24 March 2- 3pm. The discussion topic will be announced on the blog on the week with the meeting link.

Also in about a month's time I will start the first in a series of presentations from previous participants in this course who have kindly agreed to talk about their evaluation projects using Elluminate. More to come about this...

Week 2: Quality and evaluation

2009-03-03T21:05:00.002+13:00

Image: Rowan by kevinzim
This week, we are taking a look at what constitutes quality in eLearning and discussing this topic. As part of this, you will be investigating the importance of quality, and establishing a definition of evaluation for eLearning which fits your professional context. There is a presentation to watch and some reading to do. The presentation will introduce some terms associated with evaluation, e.g. formative and summative, and hopefully give you another perspective about the subject.

There are three questions for you to ponder and post to your blog. A short email about your main points would be a bonus.

Why is evaluation important to you and how do you define it?
What sort of evaluations mentioned on the presentation are familiar to you already and why?
Why is quality important in eLearning?

Do check out Adrienne's post which will start the week off for you on the subject. Note the resource she has shared with us - this could be very useful when you are thinking about which aspects of eLearning you will evaluate. For example, will it be the course design or the support given to students or something else?
Do you agree with her reasons for ensuring quality in her courses and do you have the same issues?

Summary week One

2009-03-03T08:40:00.006+13:00

Hawthorn berries (cratageous monogyna) - Image by Dave-F almost retired

I just love this time of year when all the berries are appearing. We have some lovely juicy blackberries near us which my Briard and I love to eat. of course she prefers that I feed her the berries but the prickles don't seem to bother her too much.
There has been a good steady start to the course. Most people have set up their blogs and sent the links to the email group. Still missing a few so it would be great to get you all linked to the course blog. A few couple still to get connected on the email group.

Some of you may be waiting for the "real time" sessions to clarify some things about your blogs. I am looking forward to talking to everyone. If you can't make the sessions on Tuesday 3-4 pm and Wednesday 7-8pm, there will be a recording available. I will also be in touch by phone from time to time just to check in that everything is going ok.

Chocolate fish to the people who have set up blogs for the first time and others of you have re-activated your blogs for this course. A great effort!

We have a good mix of people from around the country - apologies if I have missed anyone. Two people from the South Island - Rachel (Dunedin) and Liz (Wanaka). Eight people from Auckland, four of whom are from Manukau Institute of Technology (MIT) - Michelle, Krishan, Joy, Herve - with Craig, Sam and Debra RW from private training organisations (PTO) in Auckland. Four people from UCOL in Palmerston North - Kay, Debra, Elaine, Heather (Bulls). One person from a PTO in Wanganui - Iain. And Adrienne from Kamo in Whangarei. We also have Minhaaj joining us from Pakistan.

I have learned something new today re Kamo - "The name is a Māori word meaning "eyelash",^[1] but has also been said to mean "to bubble up", referring to hot springs in the area." (Wikipedia.) Perhaps Adrienne will tell us more about her hometown on her blog. It will be great to get so many different perspectives from the mix of people from different backgrounds in the course. All links to people's blogs that I have so far are on the course blog and the snippet feed will update as you post - I hope.

back-up plan if you are unable to get into Elluminate

2009-02-24T09:35:00.001+13:00

Dear all
our back-up plan if you are unable to get into Elluminate - I hope you have tested it already. You need to do this well in advance in case you need to download Java on to your computer. If you cannot get in do the following:

call Help desk - 0800 765 948 - open 9am to 9pm
text my cell phone: 021735438
check the email group for updates - evaluation-best-practiceATgooglegroupsDOTcom
log into Skype - download if not already using this and invite me: Skype name - bronwynannh
get yourself a glass of wine!

Some tips for connecting to Elluminate

Once you get the Elluminate page, select 56kps even if on broadband - this gives a better quality sound.
Make sure your microphone is plugged in before you click the meeting link.
run the audio set up wizard via Tools/audio.

Student services helpdesk for Elluminate
Remember to check in with the student services helpdesk - 9am to 9pm. I will let them know today that we are using Elluminate for this course and the times of our two sessions this week.

Helpdesk for technical computing assistance is available by phone and email.

Phone: 0800 765 948
email: itssATtekotagoDOTacDOTnz

Ensure you tell them your name, my name, name of the course and a brief description of the problem.

This information is also on the course wiki under HELP.

Welcome to evaluation of eLearning for best practice

2009-02-10T10:19:00.006+13:00

It is my great pleasure to be able to facilitate this course again in 2009. The course runs from 23 February through to 3 July (apologies I was a week ahead of myself before with a start date of 16 Feb). We will use a variety of ways to communicate - this course blog for announcements, your blogs for communicating your ideas for your project to the class, and an email discussion forum for bouncing ideas off each other. More about this further on.

My priority in the course is to help you plan and carry out an evaluation project which is relevant to your work. I can't wait to get into some discussion about your ideas for evaluation, and am looking forward to assisting you to carry out your evaluation projects.

Evaluation is a key part of any design and development process as you can see in the diagram of the ADDIE model and this will provide us with a lynch pin to keep the course focussed.

This course blog will be used for announcements, and to summarise the weekly activities on your blogs and in the group discussions. The course calendar will be used to remind you about the weekly schedule. Please fill in the poll on the right so we know how many will be attending the first class meeting. This will be a low key get together to introduce yourselves and chat about the requirements for the course.

Course content: this is on the Evaluation of eLearning for Best Practice wiki (WikiEducator). All you need to know to get started is there.

Discussions: these will take place in an email forum this year and you should have recieved an invite to join the group. You are encouraged to also comment on specific items on each other's blogs.

Dates to notice: these will also be marked on the course calendar.

Holiday break - 6 April to 17 April - over Easter - after week 6.
Course ends 3 July.
Due date for final submission of assessments - 10 July.

Weeks fourteen, fifteen and sixteen: Write your evaluation report

2008-06-18T18:07:00.003+12:00

For some of you the next three weeks will be getting you closer to the finishing line. If you need an extension please let me know. Make sure you check out the Assessment and Marking Schedule for Part Four: Evaluation report, to make sure you have included the headings and material required. For those of you carrying on at your own pace, try and get your plans to me in the next few weeks.

There are links to example reports at: Online information literacy project. Some of the reports are brief e.g Prototype Evaluation Report, and others such as the final report for the project are more detailed. You will need to strike a balance.

You can either use the template provided or create your own. Follow the instructions on the wiki for submission but be sure and let me know when you have submitted your final report. I am happy to give feedback on your draft reports prior to submission.

Summary Weeks 12 and 13 - Analysis of data and results

2008-06-08T13:32:00.006+12:00

The picture for the summary this week is dedicated to Gordon who has managed to post some excellent insights and discussion about his preliminary analysis of results - even while on the road and 'riding the wild moose in Canada'.
Gordon's findings on his latest post illustrate how any evaluation the first time round, always finds some of the "fish hooks" in the way questions are posed and interpreted. This is always good particularly with a pilot evaluation, as it enables modification of the approach for next time.
It also looks like any further evaluation on the product might be best done as an effectiveness or impact evaluation once people have used the computer-based training resource. That way he will be able to find out how personnel are transferring their learning done with the CBT to real situations in the workplace is probably very important in the area of hazardous substances.

There has also been an interesting debate around Likert-type scales on Gordon's blog - his evaluation has illustrated the importance of the midline scale and brings up the issue of how to organise the rating scale. It seems more logical to have the positive responses as 4 & 5 though many studies use 1 & 2. Probably as this aligns with the importance associated with being 1st and second.

I am interested to see what others think about this idea.

Helga has been spending time extracting some meaning from her interview data, and asked for help on this. Unfortunately the tips from Bronwyn about how to go about analysing this kind of qualitative data arrived after Helga had sorted it using the website Yvonne suggested. She has also realised the importance of lots of small usability tests as you develop.

It is all about looking for common themes and patterns ie, making connections as Yvonne suggested. Themes are common words or phrases and usually match the research/evaluation questions and/or interview questions. In fact, in a well designed study the two areas will align nicely. For example a pattern in the data might be that all participants found the learning formats easy to navigate.

A table with columns is a good way to organise the themes. You can list the themes and place examples of the comments next to them. Helga used stickies very successfully. I also include one column which indicates how many times each theme (word or phrase) was mentioned. Taking some time to do this with your data will mean you can extract more meaning. If you summarise in a couple of sentences without working through the themes - you may miss some important points.

You also need to look for what is not mentioned as well. For example - if no-one mentions they enjoyed the experience you might begin to wonder was it too hard or were they pushed for time.

Helga in her previous post mentioned how she was having some fun with her formative, usability evaluation. She has demonstrated how important it is in usability testing to get feedback on early prototypes and to act on them reasonably quickly (Rapid Iterative Testing) - there may be several iterations of a prototype during usability testing before it goes live. Prototyping is outlined in Wikipedia.

When testing how usable a product is in real-use situations, Gordon asks how feasible it is to make changes on the spot. However, rapid iterative usability testing does do this. There is more about this on a site called: Learn about usability testing ......"You should be testing prototypes from early paper-based stages through fully functional later stages."

Rika has made some headway with a modified plan and is about to publish her presentation.

Weeks twelve and thirteen: Analysis of data and results

2008-06-03T09:10:00.006+12:00

The next two weeks in the course schedule have been allocated to the analysis of the data you have been collecting, and writing up your findings as results. For some of you this will be feasible, but for others data collection may just be beginning. And others have been in touch about extending the evaluation project. Flexibility is key in any project, so it is important to set yourself realistic timelines.

When you do get some data, remember to keep the analysis manageable, and use methods with which you are familiar. For example, calculating percentages (frequencies) and means can transform raw data into manageable chunks. For this project the key is to keep your data analysis simple - when you have more time you can explore the fancy stuff. Think about how you will present the data. Will it be in tables and/or graphs?

My suggestion is that you load your data into a spreadsheet so you can easily manipulate it, and create graphs if that is the way you wish to illustrate your findings. It is important to explain your findings to the reader whether they are presented in tables or graphs. The results section should have tables of data which has been manipulated e.g. means or frequencies. Tables which show basic data with little done to it, can be added to the appendices in your final report.

There are some examples of results and reports in the Resources area so you can see what others have done. Make sure you post your initial findings to your blog and ask for help if you need some direction. And do give feedback to at least two other people on their blogs; every bit helps.

Results of polls
It looks like people favour some synchronous sessions in the evening at 7.30 pm. There is a mix of preference for a question and answer session and a social get together, and the majority would like to hear from an expert. Keep your eyes on the blog for announcements.

Summary : Weeks ten and eleven: Conducting your evaluation

2008-05-29T10:20:00.008+12:00

Some very interesting posts this week with presentations of evaluation plans appearing. There are presentations from Hilary, Gordon and Yvonne. Gordon has posted some of the feedback he has been getting on his plan and his discussion around the suggestions makes for interesting reading.

Helga has emailed a Flash presentation to everyone so if you did not get it you had better let her know. Helga had some difficulty finding a website where she could upload a Flash file, and the one I suggested did not work. This demonstrates how it is not always easy to display one's work in an open environment, even though we would like to do this.

Hilary had some glitches with Myplick when uploading the presentation, and it was very time consuming doing the recording; finding the right tool for the job is always a tricky one. Hilary's finished plan is now available as well.

Rika has posted her second plan and good on her getting it adjusted so quickly after being let down by red tape. It is good to hear that Sue is still with us after being out of action for a while. Watch out for her updated plan which will be coming soon.

I have to say I am impressed by the variety of the evaluations being conducted by everyone, and the range of approaches. The support you are giving each other is fantastic and there are some wonderful ideas coming through. It is also heartening to see your perseverance in learning new tools to create and upload your presentations. For most of you the next couple of weeks will be busy with data collection. It would be good to see an update on how it is going on your blogs, and do not hesitate to ask for help. There are plenty of heads here to help you solve the problem.

Recording audio
I find Audacity is always a good bet and is available as a free download. It enables you to edit and export the file as an mp3 format - essential for ensuring everyone can access and listen to the recording one has spent so much time on.

Discussion about measuring grades in an evaluation - a synopsis
There has also been some interesting discussion in the comments area on Yvonne's blog around whether measuring grades is a good indication of the effectiveness of a course. In my opinion better grades unfortunately do not always indicate a deep level of learning - it depends on how the assessments are designed. So I am dubious about using grades as a key factor in measuring effectiveness.

In the early days of elearning researchers focussed on grade comparisons between f2f and online learning. They used scientific experimental models of research design with control groups. Little evidence was shown that one mode was better than the other. Nowadays it is often more important to find out if students are using the tools, how they are using them and if they are learning through using them. Enjoyment may be a big factor in learning as Yvonne is about to establish in her evaluation.

Weeks ten and eleven 19 May to 1 June: Conducting your evaluation

2008-05-18T13:14:00.006+12:00

Over the next two weeks, if you already have a plan in place, you will be conducting your evaluation with your target group.

There are a few things to do:

- Recruit participants - this may have been done previously.
- Collect data using the sampling tools you have stipulated in your plan.

It would be great to see some updates of how it is going in the field as you carry out your data collection. If you have not reached this point yet, keep working on your plan and post your ideas to your blog so others in the class can give you some feedback. Remember you need to negotiate your ideas for your plan with me.

Summary Week 9: Prepare and present an evaluation plan

2008-05-16T12:41:00.015+12:00

Thank you everyone who has responded to the polls. It looks like 3/7 would like to have a question and answer session on Elluminate in the next two weeks and 5/7 are not sure.

Suggestion: I will run another poll with some dates and times for an Ellminate session.

The presentations have been found to be helpful by 3/7 and 4/7 found them sort of helpful. Solution: I will now run a poll to find out what sort of synchronous sessions you would like.

Well this week has been steady with a few plans appearing in their final iterations, and Gordon is the first person to post a presentation for us to look at - excellent work Gordon. It is good to know that this course has been influential in getting him to conduct some evaluation into the use of a computer-based product. Now I have watched the presentation I am much clearer about how he is going to use the "blender application". It sounds like a useful tool to measure eLearning. I also liked his example of why he is going to do some cross-tabulation. That is the perceptions of senior staff versus less senior staff.

Gordon should get some excellent data to manipulate and the mixed methods approach will make up for the small sample. As a pilot evaluation it is excellent and can easily be tweaked if he finds it is necessary to evaluate the product further.

Gordon mentions some free software for audio recording - My Mp3 Recorder - and also Myplick for loading presentations and audio synching; this is the same facility used for the evaluation presentation you viewed in the second week of the course. Another free utility for audio is Audacity. I would be interested to know which one people find easier to use.

Helga is very happy as she has started gathering data for her evaluation, and is preparing a presentation for us to look at it. A public version of her final plan is yet to come. She also invites some discussion on her latest blog post about the use of surveys at the end of a course - How to motivate participants to fill out the survey (or happy sheet). I have moved her initial discussion post from the main WikiEducator discussion page to the course discussion page. There is already some great discussion happening on the topic on Helga's blog in the comments section.

Rika has got a few gremlins in the electronics and in her original plan due to organisational "hiccups", and she is working towards getting her presentation to us. How about some encouragement for her on her blog. In the meantime her plan is posted on her blog and ready for you to give her some feedback.

Yvonne's approach is original - she is asking about enjoyment and creativity in relation to the use of a Moodle learning Management System in a blended course. On her latest blog post, there is a link to her final plan and a very good one it is too. Refreshing to see this, as so often teachers are too focused on getting "down to business" and only want to find out how something was used - not how much enjoyment or creativity the students were exposed to as a result. I hope she manages to get the sample she would like. Good to see a mix of likert-type scales in use and open-ended questions.

This reminds me - if anyone is planning to use an online survey - Survey Monkey is free and easy to use.

Sue has her draft evaluation plan ready for people to view on her blog and is ready for you to give her feedback. As part of the evaluation, Sue is going to investigate the effectiveness of existing online formative assessment methods through a range of data collection methods involving students and tutors. Looking very good Sue. :)

So far we have six people marching onwards and getting close to collecting evaluation data with one already doing this. Others have some great ideas for their plans and their plans are coming. Others are still with us and trying to clear work commitments so they can get on with the course work. I will be in touch with those of you who have been very quiet.

Do not hesitate to get in touch if there is anything I can do to help you progress with your evaluation plans and projects. I am absolutely thrilled to see such a range of evaluation types and your ideas and plans are superb. It is encouraging to see how each of your plans, your ideas for evaluations are all very original and interesting and based on thorough research and thinking.

Week nine: Prepare and present an evaluation plan

2008-05-11T14:55:00.005+12:00

This week you will need to spend time preparing a digital presentation of your plan. This is so you can gather feedback on your ideas. Have a think about how you might do this. Perhaps a poll on your blog or an elluminate session? I can help you organise this. Make sure you have checked out the schedule.

Prepare your plan and present it according to the Assessment and Marking Schedule. (Remember you need to have have negotiated your plan with me first and posted it on your blog.) Provide feedback to at least two plans. Once you have incorporated the feedback you received, you are ready to submit your written evaluation plan (Part two A: Negotiate and write an evaluation plan) in the Digital Dropbox in the course Learning Management System or on your weblog or via email - but do email me even if you use the DDB. If you prefer you can put your plan in Google docs and post a link to it on your blog - remember publish it and give me access.

I will be running a poll this week to find out if anyone would like a question and answer session on Elluminate and also to find out if people have found the presentations to date helpful. Please get in touch if you are stuck on anything.

Summary Weeks 7 & 8: Negotiate and write an evaluation plan,

2008-05-11T14:53:00.000+12:00

The highlight this week was a presentation by John Milne, project manager of the eLearning Guidelines project at Massey University. You can view the recorded session on Elluminate here.

John took us through a very interactive session about the eLearning guidelines and tried out some of the features on Elluminate e.g. polling. I really liked the way he got us all to place our locations on the map at the start. People voted to keep the format of the guidelines as questions rather than a checklist - as working through the questions helps people to think more about what they are doing in an organisation. They can then develop their own guidelines once they have thought through the fish hooks.

John's session was great for clarifying how to use the guidelines - the way we have used them in this course as overarching questions is different and you could also use some of the guideline questions to develop a questionnaire or checklist.

As mentioned in my last summary, several people have evaluation plans underway. For example:

Rika has an evaluation plan shaping up for a needs analysis and wants to find out if an "existing marine programme should be developed into an electronically delivered programme". My suggestion was to undertake a needs analysis to determine how the existing marine programme could be re-developed using digital materials and electronic communication methods which will "best meet the needs of key stakeholders and the primary target audience".

The reason being, if the answer to the first part was no - then the eLearning needs analysis would not go ahead fully. Also sometimes people are not aware of eLearning options until an exploration is undertaken. In Rika's latest post she has asked for more feedback on her updated plan. I hope you can take the time to do this.

Gordon in this week's post is making some changes to his plan, following my feedback, to gather evaluation data more in line with a maintenance evaluation - this is where you investigate whether an existing resource or course is meeting the current needs of an organisation. He also mentions in his post that "a high scoring example from a previous course" would be more useful than the example currently provided - good suggestion. Only thing is the criteria have changed slightly.

And each year I find the standard seems to increase, and so far we have seen some excellent plans. I tend to push for excellence when I see potential as well, something I have seen in all plans submitted so far as part of the negotiation process. Thank you Gordon for your astute suggestions and attention to detail in your planning.

Gordon also makes the point that the assessment criteria give the best guide to what is required. Agreed. Perhaps I also need to develop a template specifically for this course which is more in line with the criteria. That is, insist on inclusion of a list of objectives and outcomes.

Jennifer has been doing some catching up in her latest post, and needs some help to decide - whether her formative evaluation is going to be about the usability of the new learning tool - expert review and student feedback, and peer review, or should she conduct a summative evaluation and look at the students reactions (level 1) and measure what they have learned (level 2) - kirkpatrick's model? Could Jennifer get away with adapting Kirkpatrick's model, level one for a formative evaluation?

She has presented some indepth critique about a needs analysis she conducted previously, and also about an evaluation conducted on some online information literacy modules.

Yvonne mentions in her post this week that she has her evaluation plan pretty much sorted, but is worried the potential participants might be disappearing. If this does happen to any of you, be prepared to move to plan B so it is a good idea to have several options for data collection in your plan.

Helga says in her latest post that she is about to conduct her evaluation and is going to provide us with a presentation in the near future. She is providing a high standard of role modeling in how to conduct an evaluation project - I am impressed. Helga mentions her dilemma about revealing who has developed the resource - my feeling is that although you want honest feedback you may also need to be transparent about the development. I found a similar situation occurred when observing users of the online information literacy modules.

What you are after is constructive feedback anyway, and people seem able to be able to provide this regardless.

There is also some great discussion going on around Hilary's plan as several people have added to the comments on her post. I am impressed with the way you are all helping each other.

Anupam in his latest post, mentions he has selected four guidelines to evaluate elearning from an organisational perspective - they are excellent; this is a different approach to others' plans where courses or resources are being investigated. These guidelines will be relevant in his situation as he is not involved directly in any eLearning. For example,MO17 - how does the institution monitor the impact and effectiveness of eLearning? I have made several suggestions about people in the organisation who might assist and participate in his evaluation.

Oneteachersview08 assures us in his last post that he is still with us and it was good to have him join John Milne's presentation this week. Like several of you, he has found work and other PD gets in the way.

For those people who have gone a bit quiet in the last few weeks there is still time to negotiate your plans and I hope you can have something in draft soon - even some very brief ideas for others to comment on will be helpful for you. And we would also like to know you are still with us.

If you take a look at Sue's latest post you will see how some "thinking out loud" about your ideas can help you clarify things. I have left Sue a very long comment, but she stimulated lots of suggestions from me. All her ideas have value and I was pleased I could help her untangle some of the areas she mentioned. It looks like Sue is heading towards a summative effectiveness evaluation. The guidelines will help you sort out the real purpose of your evaluation as they provide the over-arching questions. I am looking forward to her plan unfolding. Some really good thinking going on there.

I found this site, sent by a colleague quite useful and you may also find some of the tips helpful.
Tips on how to write better blog posts

To not have them this long is one tip, but what is a gal to do when you are doing such great work? :)

Week eight: Negotiate and write an evaluation plan

2008-05-04T20:28:00.009+12:00

Welcome back from the holiday break. This week we continue on with Negotiate and write an evaluation plan - see the post I made before the holidays.

A few people have been negotiating their plans with me and I have given them feedback. You can take a peek at the evaluation plans and some of my feedback for Yvonne, Helga, Hilary, Rika and Gordon. It will be good to see more plans this week and next.

On Thursday 8 May at 730-8.30 pm there will be a presentation by John Milne from Massey University (Wellington) about the eLearning Guidelines project he is leading. He will be able to give us some examples of evaluation projects around the country which are measuring against a range of eLearning Guidelines. It will be good to talk to you all again.

Click on the meeting link.

presentation - Sandra Elias - evaluation of online discussion in a midwifery course

2008-04-24T20:37:00.005+12:00

The third presentation by one of the Evaluation of eLearning in Best Practice alumni was by Sandra Elias from Vancouver. Sandra Elias conducted a summative evaluation of online discussion used for assessment in a nutrition course in a midwifery degree. The presentation eventually happened, but without any of the class present. This was due to a mix up in the difference in time zones - Canada versus NZ. Anyway we have Sandra's presentation with audio for you to listen to at your leisure.

Summative evaluation of online discussion in a midwifery course

You will find that Sandra's project is different again to the other projects we have had presented; she looked at the perceptions of students about assessment using an online discussion board. Whereas Bonnie and Suzy evaluated a DVD resource used in an online course, and Selena, Trish and Lois investigated the impact of a Learning Management System and the ways in which academic staff transferred their learning from one system to another. So we have two evaluations which were looking at effectiveness (summative) and one impact evaluation.

Holiday break

2008-04-20T15:18:00.005+12:00

There is a holiday break 21 April until 2 May. This means that you will have some catch up time. On 5 May, the course enters Week Eight, when you need to continue preparing your evaluation plan.

It would be good to see a draft outline up on your blog so others can give you feedback. You also need to negotiate your plan with Bronwyn; a good chance to make sure you are on the right path.

Weeks seven and eight: Negotiate and write an evaluation plan

2008-04-14T14:17:00.004+12:00

First up, I hope some of you can make the Elluminate presentation by Sandra Elias - live from Vancouver - Thursday 17 April 5.30 - 6.30 pm (http://tinyurl.com/2l6r7w). The presentation is about an evaluation of how online discussion was used in a midwifery nutrition course.

The schedule this week
This week if you are following the course schedule closely, you will need to firm up your ideas for your evaluation project. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment.

Things to do:
Decide on two eLearning guidelines against which you will evaluate - three is maximum.

Establish the paradigm and model for your evaluation.
Look at this example of an evaluation plan - Usability evaluation
Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
- You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
- Remember this can be modified to suit your situation.
Explore the Evaluation Cookbook to decide on some sampling methods for your project.
Prepare some draft ideas and post them on your blog.
Contact your lecturer to negotiate your plan.
Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Presentation - Selena Chan - Impact evaluation project

2008-04-08T15:46:00.004+12:00

There is another web conferencing presentation coming up this week. Selena Chan will be speaking about her impact evaluation project conducted when she was a student in the course a couple of years ago. A summary has been sent on email.

Title: E-Bees Study of tutorial staff preparedness for using a new learning management system. Selena Chan (with Lois Robertson and Trish Stewart).

Date and Time: Thursday 10 April 7.30-8.30 - meeting link - http://tinyurl.com/ywl8p8

Remember you will need a headset or stand-alone microphone but can access the presentation and listen and use chat only. The session will be recorded if you are unable to make it.

Results of the poll - hours per week spent on the course

2008-04-08T14:55:00.007+12:00

The poll recently carried out on the course blog was set up to find out how many hours per week participants in the course were spending studying in the course. There were nine (n=9) out of a possible 15 particpants who responded, a 60% response rate. The results to the question follow. For some reason, parts of the grid in the table have been lost. The table was organised in Word, and pasted into PaintShop Pro, then saved as a jpeg.

This is one way you can gather descriptive data and report on it as part of a formative evaluation during the running of a course.

Question - how many hours per week are you spending on average on the course?

I wonder if people who responded have any comments to make about the number of hours they are spending in study.

Is the amount of time spent adequate and/or acceptable to you? Please explain.
Is the time you are spending sufficient to help you understand the topics and keep up? Please expand.

Presentation of three implementation eLearning Guidelines (ELG) projects

2008-04-08T14:22:00.006+12:00

Dear all
You are invited to attend a presentation via computer web conferencing on Friday 11 April at 2-3.30 pm - about three implementation eLearning Guidelines (ELG) projects.

These projects are part of larger a tertiary wide project led by Massey University called ELG in which Otago Polytechnic and MIT and 16 other organisations are participating. The eLearning Guidelines newsletter has more information about the progress of the project.

The following people will be presenting their implementation projects which are part of the sector-wide ELG project. Bronwyn Hegarty, Otago Polytechnic, Oriel Kelly, Manukau Institute of Technogy (MIT) and Julie Mackey, University of Canterbury. More detail about the projects can be found on the project website.

bonnie and suzy evaluation best practice presentation

2008-04-04T10:20:00.008+13:00

Here is the recording of the Elluminate session - presentation and discussion, which ran last night with Suzy Poppe and Bonnie Schroyen from NorthTec. It was very interesting with eight people from the class attending and everyone asked great questions.

Evaluating the effectiveness of a DVD in the flexible mode of delivery in nursing education. Here is the link: Bonnie and Suzy DVD in nursing evaluation presentation.

Apologies the link was not working previously. I will write my reflections on my blog, and post the presentation and audio separately in the next few days.

Bronwyn

Weeks 5 & 6: Evaluation methods

2008-03-31T14:12:00.008+13:00

Over the next two weeks you will be exploring the different functions/types of evaluation i.e. methods you could use in your project. The method you choose will depend on the purpose of the evaluation. You have already heard the words formative and summative, needs analysis etc.

Well what does it all mean? Lots to look at on WikiEducator. The focus this time is to explore some projects in the literature and the types of evaluation which others have done. It is a good idea to choose projects in a discipline similar to your own.

You are now getting close to preparing an evaluation plan in preparation for carrying out your evaluation.

Poll about study hours each week
There is a poll running at the moment, as some of you have noticed, about the time you are spending each week. It would be great to find out how many hours you are all spending on average.

Presentation
Remember there is a presentation on Thursday night, 3 April 7.30-8.30 pm. Bonnie and Suzie from Northland Polytechnic will be speaking about the evaluation project they did for this course - DVD resources in nursing. There will be ample time for questions and discussion. Click on the meeting link to access the web conference on Elluminate.

It is best if you have a headset with microphone, and/or standing microphone so you can speak, but if this is not possible, then you can still listen and use the chat facility.

Make sure you test the link well in advance - it is open now, and if any problems contact me.

Summary Week Four: Evaluation paradigms and models

2008-03-28T12:29:00.011+13:00

Some steady activity has been happening over the Easter weeks. I am impressed that most of you are demonstrating your progress very well in your blogs and communicating with each other. Keep up the great work. If you prefer to move at your own pace that is okay as well. The course schedule is a guide for keeping pace with the group.

Rika has provided an excellent critique of the different paradigms with some examples of situations where they might be used - and has helped us to decipher the long words! I am very excited to see that she has actually used the Critical Theory Neomarxist post modern praxis paradigm for evaluating "education resources from a Treaty / Bicultural perspective".

Jennifer in her week 4 post questions the usefulness of the Constructivist-Heremeneutic-Interpretivist-Qualitative Paradigm; check out Rika's blog post - for an example of when it can be used: "Evaluating from this perspective I would immerse myself in the world of the subject and try and understand how they had learnt and how the course had affected their construction and interpretation of reality".

Jennifer also suggests how the Eclectic-Mixed Methods-Pragmatic Paradigm matches the 360◦ approach used in management/human resource surveys. Do you agree? There is also some very useful critiquing of different models in both Jennifer's post and Helga's post.

Helga has mentioned that Patton’s Qualitative Evaluation Model might be useful for her needs due to the use of a range of qualitative methods. I wonder which paradigms this model fits under? Any ideas everyone?

Hilary expresses the need for evaluation in relation to paradigms very well: "I can particularly relate to the ‘pragmatic’ aspect which reflects the fact that nothing is ever perfect in education but can be improved through design and recognition of possible flaws."

If you are looking for some easy way to make sense of all this paradigm stuff, Gordon had done some great paraphrasing in a table which he has linked on his blog. He has picked through the dense theoretical information and come out "alive" at the other end, armed with resources to make sense of it.

If people wish to read further about needs analysis and Stake's Responsive Evaluation Model there is a very informative article (Sleezer, C. M., Kelsey, K. D., & Wood, T. (2005). Assessing safety training needs at AVN 300: A case study) in the Resources area on the wiki.

Most people appear to favour the Eclectic mixed methods pragmatic paradigm, and can see the benefits of the rich data you can obtain from an evaluation under this paradigm. As Rika points out - it could be complicated to design, but well worth the effort.

Remember that through the use of mixed methods under the Eclectic-Mixed Methods-Pragmatic Paradigm , there is ample opportunity to incorporate both quantitative and qualitative approaches. In the real world it is also important to have versatile options and to investigate from several angles. Which is your paradigm and model of choice for your project?

It looks like some of you have got so tied up with the paradigms that you forgot about the models - it would be great to see some posts on possible models you might use in your projects.

You might also like to check out Oneteachersview08 - a creative take on the use of eLearning guidelines and competencies for secondary schools and a post about an elearning action plan. For example, competencies for "the ICT area and the Technology Curriculum and about being actively involved in communities. The guidelines he has chosen reflect this.

Wrap up
Paradigms and models did not mix well with chocolate eggs and rabbits for some people. I hope you manage to stay with the group and keep up with the material, but no worries if you wish to go at your own pace. Please try and post something short, at least weekly about your progress, in any case, so we know you are still with us.

If I have missed anyone out, please let me know here and I will add something about your post.

Summary Week Three: eLearning Guidelines

2008-03-21T13:17:00.017+13:00

Some great posts in Week three about two issues relating to eLearning and some eLearning Guidelines which relate to these. It looks like everyone has managed to find a diverse selection of guidelines and had no qualms about getting carried away. It is hard to choose just two or three. The following paragraphs outline some of the guidelines and issues you have all mentioned.

Yvonne has mentioned issues around blended learning and managed to find several guidelines which relate to this area in her post. She has also managed to find an educational situation perfect for evaluating. I wont give the game away entirely here, and you can read all about it on her blog. The eLearning guidelines Yvonne eventually settled on are interesting ones which can be used to really tease out what is important for making learning creative and effective where technology is involved. Here is one example - "ST7 Will the e-learning foster students’ curiosity and creativity?".

For Gordon in his post, there are issues around whether industry "are open to blended delivery options" for delivery of unit standards, the development of online communities, and also in relation to a suitable Learning Management system and staff training. For example, "TO1 Are staff development programmes for e-learning developed in consultation with teaching staff, in order to identify and meet staff needs?" An interesting discussion has started in the comments area.

Online discussion and timely feedback and retention rates are the areas selected by Hilary in her post and discussion about the associated issues and the relevant guidelines. For example, "TD10 Should students present work using online discussion tools?" The big issue here, which many of you will be able to relate to, is one about plagiarism and people taking stuff straight from the Internet.

And you may also be able to relate to this issue raised by Hilary, "..any structured course that contains several modules and/or is self paced, the highest drop-out will occur within the first few weeks," and the guideline associated with it. See for yourself and add to the discussion.

Lesley has chosen two guidelines which are very complimentary to each other, and could have very important implications for addressing the issues she has identified in her post. These relate to negotiated learning goals and completion of training modules, and getting honest and constructive feedback from trainees. Is it about motivation or about the quality of the materials? Again some great comments which are extending the discussion Lesley has commenced.

Helga has chosen several guidelines and in the first post about this topic she has discussed three main guidelines to be included in an evaluation as well as several subsidiary guidelines. For example, "SO7 Is it clear at the start of the course what support teaching staff will provide students?"

This is particularly important for an organisation which is offering online learning for the first time. The discussion of issues illustrates how it is probably best to pre-empt issues rather than always trying to put "bandaids on the gaping wounds" - have a look at Helga's post and see what you think.

In the part 2 post Helga mentions the issues around how the guidelines could be applied: "the guidelines can be applied at two levels. One is the whole computer training / induction framework, the other is the individual courses that are part of this framework." Helga is as yet undecided as to which approach will be taken, and has done a lot of preliminary thinking about the approach which will be most appropriate at this stage.

Barbara has three guidelines which compliment each other very well. TD1 - learner-centred and achievement of learning outcomes and TT6 - good practice and self assessment. TT13 also fits well as it is about meeting students' needs. I will be interested to read some more about the issues associated with these guidelines - perhaps some of you could chip in with your own experiences in this area.

Important - Describe the issues then select the guidelines
Some people have spent a lot of time outlining why the guidelines they have selected are important, however, they have not always described the issues which led them to select the guidelines or mentioned how the guidelines might help address the issues. It is important to look at the issues first, then choose the guidelines. The description about the issues will form the background in your evaluation plan so it is an important step in this process.

Article about an evaluation study
Also some of you have commented on the article of an evaluation study of podcasting methods introduced to dentistry students studying microbiology. It was conducted at the University of Michigan (U-M) School of Dentistry by Brittain, Glowacki, Van Ittersum and Johnson, 2006. This is a useful example to read as it has some description about the methodology used in the study.

Some advice
When honing down the number of guidelines (I recommend you end up with two or three) which will be used in your evaluation project, you all need to make a decision about the most important issues for your organisation at the present. Your evaluation project could help address these issues, so you need to choose the most appropriate guidelines against which you can evaluate and describe how they might address the issues. If you try and address too many issues and select too many guidelines, your project will become unmanageable, and time is limited as you are all well aware.

I now feel sort of caught up and hope most of you will get some time over Easter to do the same. Watch out for the Easter bunny.

Summary of Weeks One and Two: Introductions, evaluation, elearning and quality

2008-03-20T15:08:00.018+13:00

Great progress everyone. Like some of you I am playing catch-up having spent the last couple of weekends saturating myself with music and performance. More about that on my blog over the next few days.

Most people have a blog up and running and several people have posted responses to the weekly activities. There has also been a small group of you reading and responding to others' posts.

This aspect is very important as it is our main means of communication in the class. If you feel overloaded by the thought of this, it is best to select two or so different blogs each week to read and respond to and rotate until you get into the rhythm. You will soon find that there will be a cluster of blogs you prefer reading because they are posting about areas of eLearning you are most interested in. I will do my bit by summarising (more regularly from now on) each week or two's activities, and pointing you to people's posts where they have made statements and initiated discussion of note on the topic.

The range of opinion about the importance of evaluation is wide ranging and here is a sample of what people have said:

"..a comfort blanket" (Jennifer).
"..need to be sure that the e-learning is an effective use of the available resources" (Gordon- elearning solutions).
"..a means to measure effectiveness of learning. Also to check that goals are being achieved and if not, why?" (Hilary).
"..how do we improve on what we are doing or even know that something needs to be improved" (Lesley).
"..want to create WOW programmes and resources.." (Rika).

Defining evaluation was not so easy. Here are some I particularly like - you may have found others, if so please let us know here -

"..a thoughtful consideration of events which can inform future events for the better"(Yvonne ). "
"..define evaluation as a constant checking and fine tuning of all kinds of processes that happens most of the time unconscious and/or automatic and sometime in a planned way"(Helga).
"an assessment of the efficiency, effectiveness and quality of a learning experience, as viewed from the perspective of the relevant stakeholder"(Gordon).

Quality and eLearning
There has been a remarkable amount written about why quality is important in eLearning.

Jennifer's statement on her blog post about the importance of quality in eLearning is a goer - "I think quality is important in all learning. Is eLearning any different? " - and opens up an opportunity for debate. I would like people to comment on her blog and have a bit of discussion about this question. I have made a start there already. Yvonne has also mentioned her opinion about this "..to enable all students to learn effectively the teacher must be aware of what works and what doesn't." She also mentions costs and the need for skilled staff.

"Launch and Track' type e-learning" is described by Gordon who refers to learning without an instructor always present. The necessity for quality resources and learning experiences which engage and provide for remote learners and is cost-effective is also discussed - "more dependent on the material being clear and effective as the ability to engage synchronously with others, to clarify points and engage in spontaneous discussion, is limited."

Also these statements by Rika are very poignant about the importance of quality, "elearning resources have to be able to stand up by themselves." And "what's the point in putting energy into creating stuff that isn't any good and doesnt work for people?"

Hilary mentions the difficulties associated with transferring a f2f course to an online one and how necessary it is to overcome people's barriers to eLearning.

The presentation about evaluation has ended up providing an example-in-action of quality which you can all relate to as you had to watch it and experience it. There is more about this in the next section.

Quality of the evaluation presentation
Some people have also provided some very good feedback about the evaluation presentation. It seems several of you enjoyed it and found it useful. The 20 minute length caused some problems around concentration for that long in one sitting - I can relate to that. Luckily it could be paused for note-taking and the format enabled people to stop and start and return to it. The inclusion of humour (Flight of the Conchords) captured the interest and has been mentioned by a few of you. It would probably benefit from being broken up into three shorter sections - I might run a poll about this. Thank you all. :)

As well as the discussions about the importance of evaluation and quality and eLearning, several people have provided great examples of what they are already doing in this area in their organisations. A detailed example was posted by oneteachersview who has provided us with an excellent overview of how evaluation is conducted in a secondary school classroom using an "evaluation technique called SII (Strengths, Improvements, Insight)" - worth a look. I am interested to see what sort of guidelines will be appropriate for the secondary school context.

As you can see, weeks one and two have been busy and everyone who has posted has "waxed lyrical" about the need for evaluation, and what quality means for eLearning. It is great to see everyone is already familiar with evaluation of some kind and doing something. Good work team!

Week four: Evaluation paradigms and models

2008-03-18T14:37:00.008+13:00

Over the next two weeks we will be examining the different paradigms of evaluation so you will be able to decide on a theoretical umbrella under which your evaluation will sit. The design of your evaluation project will be based on a model which will determine the methods you will use to collect data.

Firstly , ask yourself whether you are interested in rich, descriptive information, or do you prefer to "crunch numbers"? Do you want to find out how people are rating certain features of your new eLearning system? Or do you want to find out how they are using something by hearing about it in their own words?

For example, under the analytic-empirical-quantitative paradigm, you would use an experimental model and quantitative methods e.g surveys (Likert-type scales) to collect numerical data. Whereas under a constructivist-hermeneutic-interpretivist-qualitative paradigm, you would select a constructivist (Guba and Lincoln or Pattons) model and collect descriptive data, using methods such as interviews and focus groups or content analysis of documents to gather data.

It is best not to get hung up on the big words. Read through the material and ask for clarification on the course blog if unclear about anything.

Week three: eLearning guidelines for quality

2008-03-10T21:01:00.009+13:00

This week you will be expanding your ideas about quality and investigating the eLearning Guidelines. These were developed for the tertiary sector and may not be absolutely suitable for private training organisations (PTOs) or secondary schools. They should, however, give you some ideas about how you can go about doing some checks and balances about quality in your workplaces. If anyone finds anything more relevant to PTOs we are keen to find out and you could add the information to the Resources page on the wiki and alert the class on the Discussion there.

As you blog, please tag/label each post with eval08 as well as any other tags/labels you want to add. That way you will be able to do a search in Google and find all the posts relating to this course.

Four speakers in April
Here is some great news. Four alumni from previous iterations of this course, have agreed to present their evaluation projects as 10 min presentations with discussion. All sessions will be added to the course calendar with the web conference link.

All presentations will be after Easter on the following dates and times - and will be recorded providing permission is given:

Bonnie and Suzy - 3 April, Thursday - 7:30-8:30pm - evaluating the effectiveness of a DVD in Nursing - meeting link open now for testing;
Selena - 10 April, Thursday - 7:30-8:30pm - title to be announced;
Sandra - 17 April, Thursday - 5:30-6:30pm - topic is on the effectiveness of online discussion in a midwifery nutrition course. Sandra is speaking from Vancouver, hence the earlier time.

The presentations are each week and we are deviating slightly from the fortnightly real time sessions, but this is so you can get the info early to help with your ideas.
Bronwyn

what happened in the web conference session 6 March

2008-03-07T15:59:00.000+13:00

On Thursday evening five people joined a web conference on Elluminate. We had a lively discussion about quality and evaluation in general, and the topics for week2 and week 3. Yvonne had to whisper for the first part so she did not wake hubby - well it was 6:30 am in the UK. Most people got the hang of the push-to-talk feature but there were a couple of hiccups until people realised you need to let the microphone button go.

Those of you may wish to have a look and listen to the recording. You will need to download Elluminate to access this so be patient. You will be prompted - just make sure you click the pop up window if it appears at the top of your browser.

An interesting point came up when Anupam asked for some suggestions about ways he can find out if his students find the online materials helpful. We talked about how it is hard for a lecturer to obtain honest answers when the students are worried about offending the lecturer; they may worry about their grades being affected, especially if it is a small class. It is often difficult for feedback to remain anonymous. Also we often forget to ask for feedback and carry on regardless.

There are ways to gather feedback during a course or when a new product is introduced, and also at the end of a course; the time when most people seem to ask for input. Helga made the point that it is often easier to get honest feedback on how usable something is, during the development phase rather when the product is finished; perhaps people do not feel as awkward being honest.

Hilary mentioned feeling a bit isolated not having a discussion on something like Blackboard as blog discussion did not seem as interactive. The group had a bit of discussion about this, and Bronwyn asked that people try getting involved in blog discussion by looking at each other's blogs (setting up a RSS feeds such as http://bloglines.com can help with this) and making sure you tick the email alert box.

Yvonne mentioned that it did take a while to get used to communicating in this way, and lesley explained why group email had been a bit problematic in her experience - nonprofessional use was made of it in some instances. Bronwyn said that some people found communication via list serv email groups hard to manage as it tended to flood their inbox - but there were ways to manage it.

Bronwyn said that if the email alert function is ticked for comments you make on blogs people will find that discussion will occur more naturally; people will be alerted as others post comments to their blogs and can respond. Otherwise people will have another place to have to go to post apart from their blogs.

Of course if people wish to have a discussion away from the blogs, there is also the Wiki discussion area and anyone can start a new forum anytime. At the moment we have introductions still happening on there. So maybe it is time for a discussion around Quality - but make sure you post to your blogs first each week as this is Assignment One and a progressive activity.

Have a great weekend.
bronwyn

Week 2: Quality and evaluation

2008-03-02T21:55:00.001+13:00

I am absolutely thrilled to have met several people already on teleconference, email and on blogs. Seven out of fifteen people have set up their blogs and introduced themselves and also notified the link on the wiki Discussion. Nine people have posted introductions on the wiki Discussion. This is a wonderful start. Don't forget to click the watch tab on the wiki so you get an alert when someone has posted. It is great if people can go back to the Discussion regularly and respond to people.

Also keep an eye on others' blogs so you can comment on what people are posting. Watching each others blogs is easier if you set up an account on Bloglines and add all the blog links there - then you only need to watch to see if the numbers change. Be sure and access some help on using a RSS reader such as Bloglines if you need it. I have decided only to have a link to participant blogs on the course blog, rather than a feed coming in there. Note: To read the writing on the image just click on it.

This week the topic is all about quality.
The presentation this week will give you an overview of the importance of evaluation and the relationship to quality. Hopefully it will stimulate some ideas for you to use in your evaluation project. There is also some light reading on the importance of evaluation, and some questions to respond to on your blog. If you look in the resources area there are several online journals which you might like to start exploring.

The first two weeks are light to allow people time to get set up and confident using the course wiki and blog, and to start posting on their own blogs. It is okay to move faster through the schedule if and when you have the time. You will find you will move through faster some weeks, and this is good, so when you need to take some time out, for whatever reason, you will not get left behind.

Link to try out computer conference Elluminate

2008-02-27T11:27:00.001+13:00

I would like to hold the next evening meeting on Thursday 6 March 730 - 830 pm (not 13 March as agreed with Tuesdays group) - I will be away on leave on 13 March and I am not going to be able to get a successful link up as the house where I will be staying has no Internet.

This will be an opportunity for people to try out computer conferencing and to have a chat and ask questions . Results so far from the blog poll indicate that the majority of people (n=8) have a preference for computer conferencing. To access Elluminate for computer conferencing you will need to make sure you have Java downloaded on your computer first - then elluminate should work from home but possibly not from your organisation.

I have set up a meeting link for you to try out. At: http://tinyurl.com/2apkug
this will stay open until 14 march and I will use it for the 6 March meeting as well. The meeting link is called: Evaluation best practice march meeting. You only need to enter your name NO password needed.

Please let me know how you get on - remember a microphone is needed if you are going to be able to speak.

The teleconference on Thursday 6 March at 1215 - 1pm is still a goer as well for people who are not free in the evening so you can say hello to me and others in the class. Contact details will be posted in Blackboard soon.
Bronwyn