Weeks fourteen, fifteen and sixteen: Write your evaluation report

For some of you the next three weeks will be getting you closer to the finishing line. If you need an extension please let me know. Make sure you check out the Assessment and Marking Schedule for Part Four: Evaluation report, to make sure you have included the headings and material required. For those of you carrying on at your own pace, try and get your plans to me in the next few weeks.

There are links to example reports at: Online information literacy project. Some of the reports are brief e.g Prototype Evaluation Report, and others such as the final report for the project are more detailed. You will need to strike a balance.

You can either use the template provided or create your own. Follow the instructions on the wiki for submission but be sure and let me know when you have submitted your final report. I am happy to give feedback on your draft reports prior to submission.

Summary Weeks 12 and 13 - Analysis of data and results

The picture for the summary this week is dedicated to Gordon who has managed to post some excellent insights and discussion about his preliminary analysis of results - even while on the road and 'riding the wild moose in Canada'.
Gordon's findings on his latest post illustrate how any evaluation the first time round, always finds some of the "fish hooks" in the way questions are posed and interpreted. This is always good particularly with a pilot evaluation, as it enables modification of the approach for next time.
It also looks like any further evaluation on the product might be best done as an effectiveness or impact evaluation once people have used the computer-based training resource. That way he will be able to find out how personnel are transferring their learning done with the CBT to real situations in the workplace is probably very important in the area of hazardous substances.

There has also been an interesting debate around Likert-type scales on Gordon's blog - his evaluation has illustrated the importance of the midline scale and brings up the issue of how to organise the rating scale. It seems more logical to have the positive responses as 4 & 5 though many studies use 1 & 2. Probably as this aligns with the importance associated with being 1st and second.

I am interested to see what others think about this idea.

Helga has been spending time extracting some meaning from her interview data, and asked for help on this. Unfortunately the tips from Bronwyn about how to go about analysing this kind of qualitative data arrived after Helga had sorted it using the website Yvonne suggested. She has also realised the importance of lots of small usability tests as you develop.

It is all about looking for common themes and patterns ie, making connections as Yvonne suggested. Themes are common words or phrases and usually match the research/evaluation questions and/or interview questions. In fact, in a well designed study the two areas will align nicely. For example a pattern in the data might be that all participants found the learning formats easy to navigate.

A table with columns is a good way to organise the themes. You can list the themes and place examples of the comments next to them. Helga used stickies very successfully. I also include one column which indicates how many times each theme (word or phrase) was mentioned. Taking some time to do this with your data will mean you can extract more meaning. If you summarise in a couple of sentences without working through the themes - you may miss some important points.

You also need to look for what is not mentioned as well. For example - if no-one mentions they enjoyed the experience you might begin to wonder was it too hard or were they pushed for time.

Helga in her previous post mentioned how she was having some fun with her formative, usability evaluation. She has demonstrated how important it is in usability testing to get feedback on early prototypes and to act on them reasonably quickly (Rapid Iterative Testing) - there may be several iterations of a prototype during usability testing before it goes live. Prototyping is outlined in Wikipedia.

When testing how usable a product is in real-use situations, Gordon asks how feasible it is to make changes on the spot. However, rapid iterative usability testing does do this. There is more about this on a site called: Learn about usability testing ......"You should be testing prototypes from early paper-based stages through fully functional later stages."

Rika has made some headway with a modified plan and is about to publish her presentation.

Weeks twelve and thirteen: Analysis of data and results

The next two weeks in the course schedule have been allocated to the analysis of the data you have been collecting, and writing up your findings as results. For some of you this will be feasible, but for others data collection may just be beginning. And others have been in touch about extending the evaluation project. Flexibility is key in any project, so it is important to set yourself realistic timelines.

When you do get some data, remember to keep the analysis manageable, and use methods with which you are familiar. For example, calculating percentages (frequencies) and means can transform raw data into manageable chunks. For this project the key is to keep your data analysis simple - when you have more time you can explore the fancy stuff. Think about how you will present the data. Will it be in tables and/or graphs?

My suggestion is that you load your data into a spreadsheet so you can easily manipulate it, and create graphs if that is the way you wish to illustrate your findings. It is important to explain your findings to the reader whether they are presented in tables or graphs. The results section should have tables of data which has been manipulated e.g. means or frequencies. Tables which show basic data with little done to it, can be added to the appendices in your final report.

There are some examples of results and reports in the Resources area so you can see what others have done. Make sure you post your initial findings to your blog and ask for help if you need some direction. And do give feedback to at least two other people on their blogs; every bit helps.

Results of polls
It looks like people favour some synchronous sessions in the evening at 7.30 pm. There is a mix of preference for a question and answer session and a social get together, and the majority would like to hear from an expert. Keep your eyes on the blog for announcements.