Showing posts with label eval08. Show all posts
Showing posts with label eval08. Show all posts

Sunday, June 8, 2008

Summary Weeks 12 and 13 - Analysis of data and results

The picture for the summary this week is dedicated to Gordon who has managed to post some excellent insights and discussion about his preliminary analysis of results - even while on the road and 'riding the wild moose in Canada'.
Gordon's findings on his latest post illustrate how any evaluation the first time round, always finds some of the "fish hooks" in the way questions are posed and interpreted. This is always good particularly with a pilot evaluation, as it enables modification of the approach for next time.
It also looks like any further evaluation on the product might be best done as an effectiveness or impact evaluation once people have used the computer-based training resource. That way he will be able to find out how personnel are transferring their learning done with the CBT to real situations in the workplace is probably very important in the area of hazardous substances.

There has also been an interesting debate around Likert-type scales on Gordon's blog - his evaluation has illustrated the importance of the midline scale and brings up the issue of how to organise the rating scale. It seems more logical to have the positive responses as 4 & 5 though many studies use 1 & 2. Probably as this aligns with the importance associated with being 1st and second.

I am interested to see what others think about this idea.

Helga has been spending time extracting some meaning from her interview data, and asked for help on this. Unfortunately the tips from Bronwyn about how to go about analysing this kind of qualitative data arrived after Helga had sorted it using the website Yvonne suggested. She has also realised the importance of lots of small usability tests as you develop.

It is all about looking for common themes and patterns ie, making connections as Yvonne suggested. Themes are common words or phrases and usually match the research/evaluation questions and/or interview questions. In fact, in a well designed study the two areas will align nicely. For example a pattern in the data might be that all participants found the learning formats easy to navigate.

A table with columns is a good way to organise the themes. You can list the themes and place examples of the comments next to them. Helga used stickies very successfully. I also include one column which indicates how many times each theme (word or phrase) was mentioned. Taking some time to do this with your data will mean you can extract more meaning. If you summarise in a couple of sentences without working through the themes - you may miss some important points.

You also need to look for what is not mentioned as well. For example - if no-one mentions they enjoyed the experience you might begin to wonder was it too hard or were they pushed for time.

Helga in her previous post mentioned how she was having some fun with her formative, usability evaluation. She has demonstrated how important it is in usability testing to get feedback on early prototypes and to act on them reasonably quickly (Rapid Iterative Testing) - there may be several iterations of a prototype during usability testing before it goes live. Prototyping is outlined in Wikipedia.

When testing how usable a product is in real-use situations, Gordon asks how feasible it is to make changes on the spot. However, rapid iterative usability testing does do this. There is more about this on a site called: Learn about usability testing ......"You should be testing prototypes from early paper-based stages through fully functional later stages."

Rika has made some headway with a modified plan and is about to publish her presentation.

Tuesday, June 3, 2008

Weeks twelve and thirteen: Analysis of data and results

The next two weeks in the course schedule have been allocated to the analysis of the data you have been collecting, and writing up your findings as results. For some of you this will be feasible, but for others data collection may just be beginning. And others have been in touch about extending the evaluation project. Flexibility is key in any project, so it is important to set yourself realistic timelines.

When you do get some data, remember to keep the analysis manageable, and use methods with which you are familiar. For example, calculating percentages (frequencies) and means can transform raw data into manageable chunks. For this project the key is to keep your data analysis simple - when you have more time you can explore the fancy stuff. Think about how you will present the data. Will it be in tables and/or graphs?

My suggestion is that you load your data into a spreadsheet so you can easily manipulate it, and create graphs if that is the way you wish to illustrate your findings. It is important to explain your findings to the reader whether they are presented in tables or graphs. The results section should have tables of data which has been manipulated e.g. means or frequencies. Tables which show basic data with little done to it, can be added to the appendices in your final report.

There are some examples of results and reports in the Resources area so you can see what others have done. Make sure you post your initial findings to your blog and ask for help if you need some direction. And do give feedback to at least two other people on their blogs; every bit helps.

Results of polls
It looks like people favour some synchronous sessions in the evening at 7.30 pm. There is a mix of preference for a question and answer session and a social get together, and the majority would like to hear from an expert. Keep your eyes on the blog for announcements.

Thursday, May 29, 2008

Summary : Weeks ten and eleven: Conducting your evaluation

Some very interesting posts this week with presentations of evaluation plans appearing. There are presentations from Hilary, Gordon and Yvonne. Gordon has posted some of the feedback he has been getting on his plan and his discussion around the suggestions makes for interesting reading.

Helga
has emailed a Flash presentation to everyone so if you did not get it you had better let her know. Helga had some difficulty finding a website where she could upload a Flash file, and the one I suggested did not work. This demonstrates how it is not always easy to display one's work in an open environment, even though we would like to do this.

Hilary had some glitches with Myplick when uploading the presentation, and it was very time consuming doing the recording; finding the right tool for the job is always a tricky one. Hilary's finished plan is now available as well.

Rika has posted her second plan and good on her getting it adjusted so quickly after being let down by red tape. It is good to hear that Sue is still with us after being out of action for a while. Watch out for her updated plan which will be coming soon.

I have to say I am impressed by the variety of the evaluations being conducted by everyone, and the range of approaches. The support you are giving each other is fantastic and there are some wonderful ideas coming through. It is also heartening to see your perseverance in learning new tools to create and upload your presentations. For most of you the next couple of weeks will be busy with data collection. It would be good to see an update on how it is going on your blogs, and do not hesitate to ask for help. There are plenty of heads here to help you solve the problem.

Recording audio
I find Audacity is always a good bet and is available as a free download. It enables you to edit and export the file as an mp3 format - essential for ensuring everyone can access and listen to the recording one has spent so much time on.

Discussion about measuring grades in an evaluation - a synopsis
There has also been some interesting discussion in the comments area on Yvonne's blog around whether measuring grades is a good indication of the effectiveness of a course. In my opinion better grades unfortunately do not always indicate a deep level of learning - it depends on how the assessments are designed. So I am dubious about using grades as a key factor in measuring effectiveness.

In the early days of elearning researchers focussed on grade comparisons between f2f and online learning. They used scientific experimental models of research design with control groups. Little evidence was shown that one mode was better than the other. Nowadays it is often more important to find out if students are using the tools, how they are using them and if they are learning through using them. Enjoyment may be a big factor in learning as Yvonne is about to establish in her evaluation.

Sunday, May 18, 2008

Weeks ten and eleven 19 May to 1 June: Conducting your evaluation

Over the next two weeks, if you already have a plan in place, you will be conducting your evaluation with your target group.

There are a few things to do:

- Recruit participants - this may have been done previously.
- Collect data using the sampling tools you have stipulated in your plan.

It would be great to see some updates of how it is going in the field as you carry out your data collection. If you have not reached this point yet, keep working on your plan and post your ideas to your blog so others in the class can give you some feedback. Remember you need to negotiate your ideas for your plan with me.

Friday, May 16, 2008

Summary Week 9: Prepare and present an evaluation plan

Thank you everyone who has responded to the polls. It looks like 3/7 would like to have a question and answer session on Elluminate in the next two weeks and 5/7 are not sure.

Suggestion: I will run another poll with some dates and times for an Ellminate session.

The presentations have been found to be helpful by 3/7 and 4/7 found them sort of helpful. Solution: I will now run a poll to find out what sort of synchronous sessions you would like.

Well this week has been steady with a few plans appearing in their final iterations, and Gordon is the first person to post a presentation for us to look at - excellent work Gordon. It is good to know that this course has been influential in getting him to conduct some evaluation into the use of a computer-based product. Now I have watched the presentation I am much clearer about how he is going to use the "blender application". It sounds like a useful tool to measure eLearning. I also liked his example of why he is going to do some cross-tabulation. That is the perceptions of senior staff versus less senior staff.

Gordon should get some excellent data to manipulate and the mixed methods approach will make up for the small sample. As a pilot evaluation it is excellent and can easily be tweaked if he finds it is necessary to evaluate the product further.

Gordon mentions some free software for audio recording - My Mp3 Recorder - and also Myplick for loading presentations and audio synching; this is the same facility used for the evaluation presentation you viewed in the second week of the course. Another free utility for audio is Audacity. I would be interested to know which one people find easier to use.

Helga is very happy as she has started gathering data for her evaluation, and is preparing a presentation for us to look at it. A public version of her final plan is yet to come. She also invites some discussion on her latest blog post about the use of surveys at the end of a course - How to motivate participants to fill out the survey (or happy sheet). I have moved her initial discussion post from the main WikiEducator discussion page to the course discussion page. There is already some great discussion happening on the topic on Helga's blog in the comments section.

Rika has got a few gremlins in the electronics and in her original plan due to organisational "hiccups", and she is working towards getting her presentation to us. How about some encouragement for her on her blog. In the meantime her plan is posted on her blog and ready for you to give her some feedback.

Yvonne's approach is original - she is asking about enjoyment and creativity in relation to the use of a Moodle learning Management System in a blended course. On her latest blog post, there is a link to her final plan and a very good one it is too. Refreshing to see this, as so often teachers are too focused on getting "down to business" and only want to find out how something was used - not how much enjoyment or creativity the students were exposed to as a result. I hope she manages to get the sample she would like. Good to see a mix of likert-type scales in use and open-ended questions.

This reminds me - if anyone is planning to use an online survey - Survey Monkey is free and easy to use.

Sue has her draft evaluation plan ready for people to view on her blog and is ready for you to give her feedback. As part of the evaluation, Sue is going to investigate the effectiveness of existing online formative assessment methods through a range of data collection methods involving students and tutors. Looking very good Sue. :)

So far we have six people marching onwards and getting close to collecting evaluation data with one already doing this. Others have some great ideas for their plans and their plans are coming. Others are still with us and trying to clear work commitments so they can get on with the course work. I will be in touch with those of you who have been very quiet.

Do not hesitate to get in touch if there is anything I can do to help you progress with your evaluation plans and projects. I am absolutely thrilled to see such a range of evaluation types and your ideas and plans are superb. It is encouraging to see how each of your plans, your ideas for evaluations are all very original and interesting and based on thorough research and thinking.

Sunday, May 11, 2008

Week nine: Prepare and present an evaluation plan

This week you will need to spend time preparing a digital presentation of your plan. This is so you can gather feedback on your ideas. Have a think about how you might do this. Perhaps a poll on your blog or an elluminate session? I can help you organise this. Make sure you have checked out the schedule.

Prepare your plan and present it according to the Assessment and Marking Schedule. (Remember you need to have have negotiated your plan with me first and posted it on your blog.) Provide feedback to at least two plans. Once you have incorporated the feedback you received, you are ready to submit your written evaluation plan (Part two A: Negotiate and write an evaluation plan) in the Digital Dropbox in the course Learning Management System or on your weblog or via email - but do email me even if you use the DDB. If you prefer you can put your plan in Google docs and post a link to it on your blog - remember publish it and give me access.

I will be running a poll this week to find out if anyone would like a question and answer session on Elluminate and also to find out if people have found the presentations to date helpful. Please get in touch if you are stuck on anything.

Summary Weeks 7 & 8: Negotiate and write an evaluation plan,

The highlight this week was a presentation by John Milne, project manager of the eLearning Guidelines project at Massey University. You can view the recorded session on Elluminate here.

John took us through a very interactive session about the eLearning guidelines and tried out some of the features on Elluminate e.g. polling. I really liked the way he got us all to place our locations on the map at the start. People voted to keep the format of the guidelines as questions rather than a checklist - as working through the questions helps people to think more about what they are doing in an organisation. They can then develop their own guidelines once they have thought through the fish hooks.

John's session was great for clarifying how to use the guidelines - the way we have used them in this course as overarching questions is different and you could also use some of the guideline questions to develop a questionnaire or checklist.

As mentioned in my last summary, several people have evaluation plans underway. For example:

Rika has an evaluation plan shaping up for a needs analysis and wants to find out if an "existing marine programme should be developed into an electronically delivered programme". My suggestion was to undertake a needs analysis to determine how the existing marine programme could be re-developed using digital materials and electronic communication methods which will "best meet the needs of key stakeholders and the primary target audience".

The reason being, if the answer to the first part was no - then the eLearning needs analysis would not go ahead fully. Also sometimes people are not aware of eLearning options until an exploration is undertaken. In Rika's latest post she has asked for more feedback on her updated plan. I hope you can take the time to do this.

Gordon in this week's post is making some changes to his plan, following my feedback, to gather evaluation data more in line with a maintenance evaluation - this is where you investigate whether an existing resource or course is meeting the current needs of an organisation. He also mentions in his post that "a high scoring example from a previous course" would be more useful than the example currently provided - good suggestion. Only thing is the criteria have changed slightly.

And each year I find the standard seems to increase, and so far we have seen some excellent plans. I tend to push for excellence when I see potential as well, something I have seen in all plans submitted so far as part of the negotiation process. Thank you Gordon for your astute suggestions and attention to detail in your planning.

Gordon also makes the point that the assessment criteria give the best guide to what is required. Agreed. Perhaps I also need to develop a template specifically for this course which is more in line with the criteria. That is, insist on inclusion of a list of objectives and outcomes.

Jennifer has been doing some catching up in her latest post, and needs some help to decide - whether her formative evaluation is going to be about the usability of the new learning tool - expert review and student feedback, and peer review, or should she conduct a summative evaluation and look at the students reactions (level 1) and measure what they have learned (level 2) - kirkpatrick's model? Could Jennifer get away with adapting Kirkpatrick's model, level one for a formative evaluation?

She has presented some indepth critique about a needs analysis she conducted previously, and also about an evaluation conducted on some online information literacy modules.

Yvonne mentions in her post this week that she has her evaluation plan pretty much sorted, but is worried the potential participants might be disappearing. If this does happen to any of you, be prepared to move to plan B so it is a good idea to have several options for data collection in your plan.

Helga says in her latest post that she is about to conduct her evaluation and is going to provide us with a presentation in the near future. She is providing a high standard of role modeling in how to conduct an evaluation project - I am impressed. Helga mentions her dilemma about revealing who has developed the resource - my feeling is that although you want honest feedback you may also need to be transparent about the development. I found a similar situation occurred when observing users of the online information literacy modules.

What you are after is constructive feedback anyway, and people seem able to be able to provide this regardless.

There is also some great discussion going on around Hilary's plan as several people have added to the comments on her post. I am impressed with the way you are all helping each other.

Anupam in his latest post, mentions he has selected four guidelines to evaluate elearning from an organisational perspective - they are excellent; this is a different approach to others' plans where courses or resources are being investigated. These guidelines will be relevant in his situation as he is not involved directly in any eLearning. For example,MO17 - how does the institution monitor the impact and effectiveness of eLearning? I have made several suggestions about people in the organisation who might assist and participate in his evaluation.

Oneteachersview08 assures us in his last post that he is still with us and it was good to have him join John Milne's presentation this week. Like several of you, he has found work and other PD gets in the way.

For those people who have gone a bit quiet in the last few weeks there is still time to negotiate your plans and I hope you can have something in draft soon - even some very brief ideas for others to comment on will be helpful for you. And we would also like to know you are still with us.

If you take a look at Sue's latest post you will see how some "thinking out loud" about your ideas can help you clarify things. I have left Sue a very long comment, but she stimulated lots of suggestions from me. All her ideas have value and I was pleased I could help her untangle some of the areas she mentioned. It looks like Sue is heading towards a summative effectiveness evaluation. The guidelines will help you sort out the real purpose of your evaluation as they provide the over-arching questions. I am looking forward to her plan unfolding. Some really good thinking going on there.

I found this site, sent by a colleague quite useful and you may also find some of the tips helpful.
Tips on how to write better blog posts

To not have them this long is one tip, but what is a gal to do when you are doing such great work? :)

Sunday, May 4, 2008

Week eight: Negotiate and write an evaluation plan

Welcome back from the holiday break. This week we continue on with Negotiate and write an evaluation plan - see the post I made before the holidays.

A few people have been negotiating their plans with me and I have given them feedback. You can take a peek at the evaluation plans and some of my feedback for Yvonne, Helga, Hilary, Rika and Gordon. It will be good to see more plans this week and next.

On Thursday 8 May at 730-8.30 pm there will be a presentation by John Milne from Massey University (Wellington) about the eLearning Guidelines project he is leading. He will be able to give us some examples of evaluation projects around the country which are measuring against a range of eLearning Guidelines. It will be good to talk to you all again.

Click on the meeting link.

Thursday, April 24, 2008

presentation - Sandra Elias - evaluation of online discussion in a midwifery course

The third presentation by one of the Evaluation of eLearning in Best Practice alumni was by Sandra Elias from Vancouver. Sandra Elias conducted a summative evaluation of online discussion used for assessment in a nutrition course in a midwifery degree. The presentation eventually happened, but without any of the class present. This was due to a mix up in the difference in time zones - Canada versus NZ. Anyway we have Sandra's presentation with audio for you to listen to at your leisure.



You will find that Sandra's project is different again to the other projects we have had presented; she looked at the perceptions of students about assessment using an online discussion board. Whereas Bonnie and Suzy evaluated a DVD resource used in an online course, and Selena, Trish and Lois investigated the impact of a Learning Management System and the ways in which academic staff transferred their learning from one system to another. So we have two evaluations which were looking at effectiveness (summative) and one impact evaluation.

Sunday, April 20, 2008

Holiday break

There is a holiday break 21 April until 2 May. This means that you will have some catch up time. On 5 May, the course enters Week Eight, when you need to continue preparing your evaluation plan.

It would be good to see a draft outline up on your blog so others can give you feedback. You also need to negotiate your plan with Bronwyn; a good chance to make sure you are on the right path.

Monday, April 14, 2008

Weeks seven and eight: Negotiate and write an evaluation plan

First up, I hope some of you can make the Elluminate presentation by Sandra Elias - live from Vancouver - Thursday 17 April 5.30 - 6.30 pm (http://tinyurl.com/2l6r7w). The presentation is about an evaluation of how online discussion was used in a midwifery nutrition course.

The schedule this week
This week if you are following the course schedule closely, you will need to firm up your ideas for your evaluation project. Remember to look at Assessment and Marking Schedule for the pointers on this part of the assessment.

Things to do:
Decide on two eLearning guidelines against which you will evaluate - three is maximum.

  • Establish the paradigm and model for your evaluation.
  • Look at this example of an evaluation plan - Usability evaluation
  • Look at the evaluation tools on the companion web site for the book by Reeves, Thomas, C. and John G. Hedberg (2003), Interactive Learning Systems Evaluation, Englewood Cliffs: Educational Technology Publications.
    • You can download a Microsoft Word Rich Text Format (rtf) version of an Evaluation Plan Template.
    • Remember this can be modified to suit your situation.
  • Explore the Evaluation Cookbook to decide on some sampling methods for your project.
  • Prepare some draft ideas and post them on your blog.
  • Contact your lecturer to negotiate your plan.
  • Write up your plan more fully using the suggested section listed in the Assessment and Marking Schedule

Note: a lot of the information in your plan can also be used in your final report about the project.

Tuesday, April 8, 2008

Results of the poll - hours per week spent on the course

The poll recently carried out on the course blog was set up to find out how many hours per week participants in the course were spending studying in the course. There were nine (n=9) out of a possible 15 particpants who responded, a 60% response rate. The results to the question follow. For some reason, parts of the grid in the table have been lost. The table was organised in Word, and pasted into PaintShop Pro, then saved as a jpeg.

This is one way you can gather descriptive data and report on it as part of a formative evaluation during the running of a course.

Question - how many hours per week are you spending on average on the course?

I wonder if people who responded have any comments to make about the number of hours they are spending in study.
  • Is the amount of time spent adequate and/or acceptable to you? Please explain.
  • Is the time you are spending sufficient to help you understand the topics and keep up? Please expand.





Presentation of three implementation eLearning Guidelines (ELG) projects

Dear all
You are invited to attend a presentation via computer web conferencing on Friday 11 April at 2-3.30 pm - about three implementation eLearning Guidelines (ELG) projects.

These projects are part of larger a tertiary wide project led by Massey University called ELG in which Otago Polytechnic and MIT and 16 other organisations are participating. The eLearning Guidelines newsletter has more information about the progress of the project.

The following people will be presenting their implementation projects which are part of the sector-wide ELG project. Bronwyn Hegarty, Otago Polytechnic, Oriel Kelly, Manukau Institute of Technogy (MIT) and Julie Mackey, University of Canterbury. More detail about the projects can be found on the project website.


Friday, March 28, 2008

Summary Week Four: Evaluation paradigms and models

Some steady activity has been happening over the Easter weeks. I am impressed that most of you are demonstrating your progress very well in your blogs and communicating with each other. Keep up the great work. If you prefer to move at your own pace that is okay as well. The course schedule is a guide for keeping pace with the group.

Rika has provided an excellent critique of the different paradigms with some examples of situations where they might be used - and has helped us to decipher the long words! I am very excited to see that she has actually used the Critical Theory Neomarxist post modern praxis paradigm for evaluating "education resources from a Treaty / Bicultural perspective".

Jennifer in her week 4 post questions the usefulness of the Constructivist-Heremeneutic-Interpretivist-Qualitative Paradigm; check out Rika's blog post - for an example of when it can be used: "Evaluating from this perspective I would immerse myself in the world of the subject and try and understand how they had learnt and how the course had affected their construction and interpretation of reality".

Jennifer also suggests how the Eclectic-Mixed Methods-Pragmatic Paradigm matches the 360◦ approach used in management/human resource surveys. Do you agree? There is also some very useful critiquing of different models in both Jennifer's post and Helga's post.

Helga has mentioned that Patton’s Qualitative Evaluation Model might be useful for her needs due to the use of a range of qualitative methods. I wonder which paradigms this model fits under? Any ideas everyone?

Hilary expresses the need for evaluation in relation to paradigms very well: "I can particularly relate to the ‘pragmatic’ aspect which reflects the fact that nothing is ever perfect in education but can be improved through design and recognition of possible flaws."

If you are looking for some easy way to make sense of all this paradigm stuff, Gordon had done some great paraphrasing in a table which he has linked on his blog. He has picked through the dense theoretical information and come out "alive" at the other end, armed with resources to make sense of it.

If people wish to read further about needs analysis and Stake's Responsive Evaluation Model there is a very informative article (Sleezer, C. M., Kelsey, K. D., & Wood, T. (2005). Assessing safety training needs at AVN 300: A case study) in the Resources area on the wiki.

Most people appear to favour the Eclectic mixed methods pragmatic paradigm, and can see the benefits of the rich data you can obtain from an evaluation under this paradigm. As Rika points out - it could be complicated to design, but well worth the effort.

Remember that through the use of mixed methods under the Eclectic-Mixed Methods-Pragmatic Paradigm , there is ample opportunity to incorporate both quantitative and qualitative approaches. In the real world it is also important to have versatile options and to investigate from several angles. Which is your paradigm and model of choice for your project?

It looks like some of you have got so tied up with the paradigms that you forgot about the models - it would be great to see some posts on possible models you might use in your projects.

You might also like to check out Oneteachersview08 - a creative take on the use of eLearning guidelines and competencies for secondary schools and a post about an elearning action plan. For example, competencies for "the ICT area and the Technology Curriculum and about being actively involved in communities. The guidelines he has chosen reflect this.

Wrap up
Paradigms and models did not mix well with chocolate eggs and rabbits for some people. I hope you manage to stay with the group and keep up with the material, but no worries if you wish to go at your own pace. Please try and post something short, at least weekly about your progress, in any case, so we know you are still with us.

If I have missed anyone out, please let me know here and I will add something about your post.

Thursday, March 20, 2008

Summary of Weeks One and Two: Introductions, evaluation, elearning and quality

Great progress everyone. Like some of you I am playing catch-up having spent the last couple of weekends saturating myself with music and performance. More about that on my blog over the next few days.

Most people have a blog up and running and several people have posted responses to the weekly activities. There has also been a small group of you reading and responding to others' posts.

This aspect is very important as it is our main means of communication in the class. If you feel overloaded by the thought of this, it is best to select two or so different blogs each week to read and respond to and rotate until you get into the rhythm. You will soon find that there will be a cluster of blogs you prefer reading because they are posting about areas of eLearning you are most interested in. I will do my bit by summarising (more regularly from now on) each week or two's activities, and pointing you to people's posts where they have made statements and initiated discussion of note on the topic.

The range of opinion about the importance of evaluation is wide ranging and here is a sample of what people have said:
  • "..a comfort blanket" (Jennifer).
  • "..need to be sure that the e-learning is an effective use of the available resources" (Gordon- elearning solutions).
  • "..a means to measure effectiveness of learning. Also to check that goals are being achieved and if not, why?" (Hilary).
  • "..how do we improve on what we are doing or even know that something needs to be improved" (Lesley).
  • "..want to create WOW programmes and resources.." (Rika).
Defining evaluation was not so easy. Here are some I particularly like - you may have found others, if so please let us know here -
  • "..a thoughtful consideration of events which can inform future events for the better"(Yvonne ). "
  • "..define evaluation as a constant checking and fine tuning of all kinds of processes that happens most of the time unconscious and/or automatic and sometime in a planned way"(Helga).
  • "an assessment of the efficiency, effectiveness and quality of a learning experience, as viewed from the perspective of the relevant stakeholder"(Gordon).
Quality and eLearning
There has been a remarkable amount written about why quality is important in eLearning.

Jennifer's statement on her blog post about the importance of quality in eLearning is a goer - "I think quality is important in all learning. Is eLearning any different? " - and opens up an opportunity for debate. I would like people to comment on her blog and have a bit of discussion about this question. I have made a start there already. Yvonne has also mentioned her opinion about this "..to enable all students to learn effectively the teacher must be aware of what works and what doesn't." She also mentions costs and the need for skilled staff.

"Launch and Track' type e-learning" is described by Gordon who refers to learning without an instructor always present. The necessity for quality resources and learning experiences which engage and provide for remote learners and is cost-effective is also discussed - "more dependent on the material being clear and effective as the ability to engage synchronously with others, to clarify points and engage in spontaneous discussion, is limited."
Rika elearning resources have to be able to stand up by themselves." And "what's the point in putting energy into creating stuff that isn't any good and doesnt work for people?"

Hilary mentions the difficulties associated with transferring a f2f course to an online one and how necessary it is to overcome people's barriers to eLearning.

The presentation about evaluation has ended up providing an example-in-action of quality which you can all relate to as you had to watch it and experience it. There is more about this in the next section.

Quality of the evaluation presentation
Some people have also provided some very good feedback about the evaluation presentation. It seems several of you enjoyed it and found it useful. The 20 minute length caused some problems around concentration for that long in one sitting - I can relate to that. Luckily it could be paused for note-taking and the format enabled people to stop and start and return to it. The inclusion of humour (Flight of the Conchords) captured the interest and has been mentioned by a few of you. It would probably benefit from being broken up into three shorter sections - I might run a poll about this. Thank you all. :)

As well as the discussions about the importance of evaluation and quality and eLearning, several people have provided great examples of what they are already doing in this area in their organisations. A detailed example was posted byan "evaluation technique called SII (Strengths, Improvements, Insight)" - worth a look. I am interested to see what sort of guidelines will be appropriate for the secondary school context.

As you can see, weeks one and two have been busy and everyone who has posted has "waxed lyrical" about the need for evaluation, and what quality means for eLearning. It is great to see everyone is already familiar with evaluation of some kind and doing something. Good work team!

Tuesday, March 18, 2008

Week four: Evaluation paradigms and models

Over the next two weeks we will be examining the different paradigms of evaluation so you will be able to decide on a theoretical umbrella under which your evaluation will sit. The design of your evaluation project will be based on a model which will determine the methods you will use to collect data.

Firstly , ask yourself whether you are interested in rich, descriptive information, or do you prefer to "crunch numbers"? Do you want to find out how people are rating certain features of your new eLearning system? Or do you want to find out how they are using something by hearing about it in their own words?

For example, under the analytic-empirical-quantitative paradigm, you would use an experimental model and quantitative methods e.g surveys (Likert-type scales) to collect numerical data. Whereas under a constructivist-hermeneutic-interpretivist-qualitative paradigm, you would select a constructivist (Guba and Lincoln or Pattons) model and collect descriptive data, using methods such as interviews and focus groups or content analysis of documents to gather data.

It is best not to get hung up on the big words. Read through the material and ask for clarification on the course blog if unclear about anything.

Tuesday, February 26, 2008

Class meeting today

Well two people have voted to meet today 4-5pm and I will be there to meet you. Teleconference details will be emailed to the class and posted on Blackboard. I will also have the Elluminate meeting link open as well (this is in the course calendar on this blog), but it looks like people have been having trouble accessing the meeting site from within their organisations.

Congratulations to the three people who are up and running with their blogs, and the three people who have found the Discussion area in the wiki and introduced themselves. I am looking forward to seeing more blogs and discussion pop up this week. I am adding links under participant blogs. You might like to add this course blog link to your blog - blogger's new layout features make this easy peasy to do and I am sure WordPress is as user friendly.

I have to find out how to add your blogs as an RSS feed and will do that later this week.
Bronwyn

Monday, February 25, 2008

First class meeting access

It would be great if people could try out the link to Elluminate - this will help decide if we can meet via computer conference for the first time. The link is open now until after the session on Tuesday. There may be a teleconference instead or as backup if people cannot get access.

Please let the facilitator know whether you can get in when you email to say you have joined the course. No password is needed and it is preferable if you have a headset with microphone - make sure you do the audio set up to check it all works. (http://tinyurl.com/2v63ch meeting link is also in the calendar)