Weeks 16 and 17: Evaluation reporting

http://jigsawpuzzlesgames.net/

In the last two weeks of the course, you are putting the puzzle together and preparing the evaluation report (Assessment Three). If waiting for feedback on Assessment Two, the Results, you can still work on the draft report, and also ensure your blog is up to date (Assessment Four). The final date for submitting these assessments is week 17 (11 December) - I can extend this to 14 December if you want these marked before Xmas. If this is not possible, please make contact and negotiate a date in the new year. If Assessments One and Two are outstanding, you need to make contact asap to negotiate submission dates.

Please fill out the poll about the weekly hours you have spent on this course.

Activity Thirteen - Evaluation reporting – preparation of a draft report

•  Access the Evaluation report template and add to your chosen document application - Word processing application, project wiki, or Google docs.
• Prepare the final sections of the report – conclusion and recommendations, executive summary and reference list - collaborate in your groups.
• Invite feedback - I suggest you email the lecturer and others in the class to alert them.
• Adjust sections of the Evaluation plan so they are suitable for the report - rewrite in third person, past tense, and include feedback from the marker. The sections are: Introduction, Background and rationale, Aim or Purpose and Evaluation questions, Decisions, Methodology.
• Adjust Results (Assessment Two) and add to the report.
• Make sure the Appendices are in order. 

Activity Fourteen -Finalise the evaluation report (Assessment Three), and your reflections about the evaluation process on your blog (Assessment Four).

• Submit the final evaluation report in pdf format. You can email this to the lecturer, and link to the report on your blog.
• Summarise your reflections about the evaluation process for Assessment Four on your blog.
• Notify the lecturer and the rest of the class when this is ready.
• Sit back and relax - you did it.

Week 13: Evaluation literature

Quill & Books by aurostar739

This week you are probably preparing the second assessment and writing up your findings. If you haven't got that far don't give up as I realise some groups have had a few delays. We are now up to reporting and information as well as activities to guide you in this phase is on the wiki. We begin with an activity which will help you to critically analyse your findings, and prepare the discussion section of your report. The discussion forum is set up on emit for you to discuss the information you are reading in the articles.

Activity 11
Discuss examples of evaluation literature that can be used in your report.

• Search for and select suitable articles or reports to discuss on the eMIT forum. (Hint: it is advisable to find evaluation studies that use a methodology similar to your project.)
• Annotate the articles - include enough information to be useful later on if you refer to the research in the Discussion section of your report.

The following prompts may assist:

• Which evaluation theories underpin the research?
• What methodologies were used - design, sampling methods, sample sizes?
• What were the main findings and recommendations?
• How similar or different to your results are the research findings in the articles?

Week Nine and Ten: Finalising plan and reflection on the group evaluation process

Spring Wellington by Newsbie Pix

Countdown this week - please get your draft plans ready for me to give feedback. I will be on leave from 1 to 17 October, and will mark the final plans on my return. However, to make sure you are ready to contact participants in readiness for collecting data after the course break - 10 to 21 October - I need to provide feedback on your evaluation design this week. I hope you can meet the deadline. I have included two activities - this week's and next.

Online meeting this week
The online meeting in Wimba will be on Thursday 19:00 to 20:00 for last minute queries about the planning process and collecting data.


Activity Eight- Finalise and submit the final plan.

• Incorporate feedback into the plan
• Organise collection of data – send out invitations to participate. 

Activity Nine- Reflect on the group evaluation processes you have undertaken to develop your plan.

• Check the criteria for assessment four in the course handbook.
• Update your blog according to the criteria.
• In particular, on your blog - report and reflect on the personal contributions you have made to the group process during the development of the evaluation plan - which sections were you responsible for, what did you contribute etc.
• Indicate what you learned about working in a group.

Week Eight: Planning evaluation –draft plan

Big sister watches the young uns

Yes these are my lambs - I couldn't resist. It was touch and go when the younger ones were born because it was freezing in Dunedin. Luckily they were born in the shed and didn't move out of there for several days.

It is excellent to see your plans evolving, and I understand there is lots of activity happening as you meet in groups to put your plans together. This is probably the hardest part of the process - getting the evaluation design just right. Be sure and organise a Skype or Wimba session with me if you need to discuss your plans.

Your blogs
A quick reminder that you need to contribute regularly to your blogs for Assessment Four - Reflect in an on-going manner about your contribution to the evaluation process, and post on your individual blog. Please read the criteria in the Course Handbook - they are posted at the end of this message. This is the main way you can indicate the extent of your contribution to the group process, and this assessment must be passed to pass the course. Tom is posting regularly and I am sure he would appreciate some comments - check out his wonderful photo taken as he flew over the motorway. :)

What should you be doing this week?
This week you need to continue developing your evaluation plan and request feedback. By the end of this week you should expect to have an easily, readable draft plan with all the sections completed on the project wiki, ready for others to give feedback. Make sure you  read the Analysis of data section on the wiki to sort out how you will analyse the data that you will collect so this can be included in the plan. This is also where the instructions for Activity Seven are located.

Activity Seven - Group work

• Prepare other sections of the plan to complete a draft plan – limitations, logistics and time line, budget, references and appendices.
• Finish writing the Introduction.
• Add to your project wiki and request feedback on the project forum. 

Criteria for Assessment Four - the blog

Demonstrates an understanding of evaluation (theoretical and practical) in an individual and professional context.

Produces evidence of learning and reflections about evaluation - including eLearning guidelines (or other benchmarks) and quality in a relevant context.

Reports and reflects, from an individual perspective, on the phases (plan, data collection, analysis and interpretation, and Results), progress and processes involved in the group evaluation project.

Reports and reflects on personal contributions to the group process, and what was learned about working in a group.

Demonstrates evidence of individual reflective learning about the process of evaluation.

Critiques the process used in one other group project, and describes the feedback provided at each stage.

Merit: Critical analysis of the evaluation process demonstrates evidence of reflective practice (effective reflection and professional learning[1]).

---

[1] Hegarty, B. (2011) A Framework to Guide Professional Learning and Reflective Practice. Unpublished Doctoral thesis. University of Wollongong, NSW.

Week Five: Planning evaluation - purpose and evaluation questions

New Zealand sheep by 111 emergency

Great news! Professor Tom Reeves has agreed to be a guest speaker and share some of his expertise about evaluation. I have scheduled him for Wednesday 7 September at 12:00 -13:00 NZST. I do hope lots of you can make it over the lunch hour. As usual we will meet in the Wimba classroom, and the session will be archived.  Do come ready with questions about evaluation. I will let him know beforehand some of the topics people from the previous online meeting asked to have explained.

Where should you be by now?
By now you should have decided on an evaluation project, and have prepared the background and rationale, goals and decisions. You should also have found a research article from the literature which relates to your plan. This will be useful in informing your evaluation design. I deliberately left writing the purpose of the evaluation (Aim) until you have had time to think about the type of evaluation you will do.  You need to choose an evaluation approach which fits the reasons for carrying out the evaluation and the audience involved. The audience is usually the participants as well as the managers and colleagues who have an interest in the outcomes of the evaluation. You can leave the Introduction for now, as this is best done once the rest of the plan is completed.


What is happening this week?
Continue with your evaluation plan. All the activities and resources related to Evaluation Planning are now located in the section called: Planning on WikiEducator (via Course Activities from eMIT). Other sections (Results, and Reporting) are not visible as yet as they are under development. (Evaluation processes has met its demise, and resources from there have moved.)

This week you need to decide clarify the purpose of the evaluation, and write the Aim and Evaluation questions. These are the 'big picture' questions. Some of you may have one evaluation question, and others may have two or three. Under each of the broad evaluation questions, you can list one or two sub-questions which are more specific.

Activity Five- Aim and Evaluation questions.
    * Prepare the aim and two or three evaluation questions, and sub-questions for your plan to fit the decisions and goals.
    * Add to the project wiki.
    * Use the project forum to ask for feedback from the class.

I hope you can provide feedback to each other. I will be checking the project wikis later on in the week, but do email me once you have a draft Aim and Evaluation questions ready.

Week Three: Methods of evaluation and projects

Snow at home in August 2011 by bronwynannh

Well it is easy enough to see that I will be playing in the snow this week. Luckily all the neighbours' wee lambs are tucked up safely in their big shed. My sheep are playing the hungry card, poor things.

Progress reminder
By now you should have your introduction on your blog and have contributed to the Activity two - evaluation and quality discussion forum.

What's on this week?
 This week we are looking at methods of evaluation - needs analysis, usability, effectiveness, impact and maintenance - described by Professor Tom Reeves and Professor John Hedberg in their 2003 book on evaluation - Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications - Chapter Three. Also, on Google books is a preview of the book. The information is summarised in the different Resources accompanying Activity Three.

Online meeting

 Evaluation methods will be discussed in the online meeting this week: 16 August - 19:00–20:00 on Wimba. Make sure you test the link well in advance of the session. Please take note of the telephone call number and student pin code so you can enter via your telephone if all else fails. Access via eMit - Online meetings/Wimba Classroom. Please text me if you have difficulties (contact details can be found via the Lecturer link on eMit).

Activity Three
The information about this week's activity and the resources accompanying it can be found via Course Activities/Evaluation Methods.

• Use the resources to find out about different types of evaluation.
• Explore the literature about evaluation which is relevant to your context. A lot of material is compiled in the References section of the wiki, and you may also wish to find your own.
• Post a summary of the key ideas to your blog.
• Examine the range of potential evaluation projects on offer (a link to them is available on eMit in the Projects discussion forum), and either choose one of these or advertise your own.
• Organise yourselves into groups for the evaluation projects - use the Projects discussion forum to join a project group and to negotiate your methods of working collaboratively.

I hope you find a project to suit you. If you have an idea for your own evaluation project, make sure you email me with the overview - using a similar structure to the one used for the list of potential projects. This information will be used to inform your evaluation plan. Have a great week, and I hope to meet lots of you on Tuesday evening.

Summary of week two activity

This week several people have contributed to the discussion about the ADDIE model and quality. Views about the usefulness of the ADDIE model of education design have been mixed, although consensus appears to be that it can provide some guidance for design and development. However, theory is believed to be only a starting point, and the ADDIE model is really just a "project management tool" (Alages and Tom), and other factors need to be considered, for example, time, quality and budget" (Rosanne). An excellent point was made that it is important not to get too restricted by one model such as the ADDIE as flexibility is needed when thinking about design of online systems (Mark).

Also, some people have used the model in their design work (Rosanne, Alages), or something similar to it (Sandra H, Mark and Sandra W), and that it is best not to become too linear oriented when designing online learning experiences as it is a dynamic process (Alages). Awareness about the challenges in undertaking evaluation are apparent in the discussions, and it is an area which is often left because of time constraints. Also, the need for evaluating during the design and development etc. phases is regarded as important and mentioned by most people.

I have referred people to the PADDIE model of design which we use at Otago Polytechnic, and as you will see it is still a work in progress - I am too busy doing the actual projects to finish the theoretical framework.  A case in point. The information about the ADDIE model for instructional design provided by Alages has more information about how to use the ADDIE model - for each phase details are provided about the type of evaluation you might undertake.

Only two people are familiar with the eLearning Guidelines, and everyone thought they would be useful although having time to explore them could be problematic. Some people mentioned they used other benchmarks but did not provide information. At Otago Polytechnic we use the Good practice checklist for benchmarking quality in Moodle courses. It is based on others' work as listed on the document. To access this file, you need to login as a Guest. A five stage model for measuring organisational quality in eLearning which some of you may be interested in is the eLearning Maturity Model for NZ.

• Maturity Model Getting Started 
• Marshall, S. (2006). E–learning Maturity Model Process Assessment Workbook. There are five main areas for assessing quality and benchmarking eLearning:

1. Learning Processes that directly impact on pedagogical aspects of e-learning; 2. Development Processes surrounding the creation and maintenance of e-learning resources; 3. Support Processes surrounding the oversight and management of e-learning; 4. Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle; 5. Organisation Processes associated with institutional planning and management.

 Additional benchmarks for measuring quality can be found on the Quality Principles site.  Are any of you these resources familiar to you?

Week Two: An overview of evaluation and quality - Activity two

Winter fog by bronwynannh

How is this for a photo from a basic cellphone? It takes a pretty good pic and certainly shows you what it has been like around my place lately. I thought this would be a good starting point to talk about quality and the reasons we need to evaluate. This week it is a good idea to start checking out the  Exemplars of previous students. I expect to have a list of potential projects available later in the week. If you have ideas already, please email me as soon as possible to discuss them.

Activity two
Detail about this week's activity (Activity two) can be found via Course activities in emit. You will end up looking at the WikiEducator page. The timetable and also up and coming activities will be listed on the Home page as a quick reminder. To find the information about this week's work, click on the tab called: Why evaluate?

This is where you will find the following instructions for Activity two,  plus a list of resources.

 Activity two

• Use the resources to develop your knowledge about the topic.
• Participate in the discussion about the evaluation of eLearning - on emit.

1. What do you think about the ADDIE model of educational design?

• How much do you know about it? Please tell us especially if you use it.
• How adequate is it for guiding effective eLearning design?

2. What type of benchmarks are you currently using to measure quality?

• Have you used the eLearning Guidelines? Do tell us more.
• What other benchmarks do you know about or use for evaluating teaching and learning?

 If you wish to contribute other resources for this topic, please share them on the discussion forum and also if you wish on your blog.

Summary of week one activity

Confused by Kristian D

The week has started well and eleven people have an individual blog up and running for activity one with an introduction, and there are some wonderful photos to look at. I am now getting a sense of who you all are, and where you work, and also have more understanding about your previous experience with evaluation. Look and see how many people are working in the Hair and Beauty Schools at MIT - I can feel a project coming on in that area. We also have people from Nursing and Health areas, and  lecturers in ESOL , electronics and mathematics, and business communications as well as a programme coordinator with a role in quality assurance and a distance learning manager. The blogs of seven others in the course are still to come, so it looks like we have a hearty number studying the course this year. It is fabulous to see that a number of you know each other already from other courses, and some of you are close to finishing the Graduate Certificate.

Most people are using the emit blog, and two people - Oonagh and SandraH have chosen to use external blogs. I will be interested in hearing about your preferences about the emit and external blog platforms once we have got underway. The links to peoples' external blogs are listed on the course blog - see to the right, and both Oonagh and Sandra have put a link in the individual blog area. Fred has also invited us to view his Personal Learning Environment.  

Online meeting archive
We also had the first online meeting, but there was a hiccup and one person could not get the technology working on the day - so make sure you run through the wizard on the Wimba classroom well in advance. The archive of the recording is available - click on Online meetings in emit, then Wimba Classroom, and once Wimba downloads scroll down the page past Enter Room and all the phone numbers until you see: Online classroom meetings - 08/04/2011 13:38 - future archives will appear there. 


Poll about a time for online meetings
Chocolate fish to the three people who responded to the Doodle poll. We also had a chat about times in the first online meeting. Please respond to the poll and indicate your choice of day and time for the regular fortnightly online meetings. It would be great to find a time which suits most of you. So far, Tuesday 5pm and Thursday 5 pm or 7pm have been mooted.

Are you confused?
Some of you may find the use of emit and external platforms for the Course blog and Course activities a tad confusing. The best plan is probably to enter the course via emit and use the course timetable to guide you with each week's activity. I will also post an overview on the Course blog each week and send a reminder on email. Once we get going you will hopefully get the hang of it. Make sure you ask questions if you are not sure what is expected - on the Questions and Answers discussion forum in emit so everyone can see, or on email to me, though brave people may wish to use the Comments area on here. ...Off now to prepare this week's blog post.