I am approaching the end of the course and it is time to reflect on the last few months.
I agree with Gordon that I learned heaps from being a participant in this online course as well as from the content.
Although we only carried out a relatively small evaluation, because we were able to ‘look in each others kitchen’ you realise how important it is to do evaluations in different stages and evaluate different aspects of it. The formative ones (like the one I did) are essential to ensure an optimal learning experience. For example not being able to hear the audio is more important than we first realised. Just providing captions is insufficient. The funny thing is that this is backed up by lots of research I found out recently (see Chapter 5:’Applying the modality principle’ in e-Learning and the science of instruction (Clark & Mayer, 2008)).
Gordon evaluated a module that has been created a while ago and it is encouraging to see that this module is still up to data and appreciated by staff and students. Yvonne on the other hand focussed on the introduction of a Moodle on a college. The e-learning guidelines prove to be very useful as it helped me to overview all (or at least many) aspects of e-learning. Besides the creating a good module, you also have to ensure good support and assistance, the technical environment, by-in from management. And although I could not personally include all these aspects during this course, I could ‘taste’ some of them through looking at what other participants did and are doing. I am looking forward to see their reports and also to pay more attention to the whole e-learning strategy in our own organisation. (I will start with reading ‘E-Learning: Strategies for Delivering Knowledge in the Digital Age’ (Rosenberg, 2000)).
I had a bit more trouble to get the grip on those paradigms. That was a bit too abstract to me. The models are already more practical and therefore more useful to me. That you use a model that uses triangulation to get a picture from more angles, that makes sense to me. It was a good exercise to spend some time on this because honestly, normally I would not have paid attention to this. But that is what you do a course for, to broaden your horizon!
I also had to find a balance between what is required for the organisation and what is required for the course. My project was/is a real time, real case example and I sometimes got a bit sidetracked because of that. For example, I had the comprehensive list of recommendations written and discussed with manager a while ago and proceed with the action and ‘forgot’ to write the rest of the report. Another example is that I forgot to add a “background about the organisation and why the web conferencing was useful enough to warrant a training programme to be created” as Yvonne suggested (but it is not too late for that I suppose).
I enjoyed the discussions we had. I will no longer use the four point Likert scale . It was great to test things out and to share ideas. I really wish we had more conferences or options to talk to each other. Maybe I should have started some of the other courses as well as I have still many questions. And there are so many people out there with all different knowledge and experience that I wish I had more opportunities to share ideas with those people.
To finish off, some feedback on the course delivery mode.
I really enjoyed being an online student. I appreciated the flexibility of doing this course in my own pace at a time that suits me. I realise that it must be a hard job being a facilitator for a course like this: motivating people, creating a sense of a group, being there with quick response to e-mails and contributions to blogs… Thanks heaps Bronwyn!!
The things that are not my preferences are: too many places to go to, to get all information. I sometimes just lost track of things. Besides that a couple of things were not that user friendly and frustrated me mostly. For example, I realised that I could not share pdf’s with you all after 36 attempts (would have been great if I had known that before hand), I struggled to contribute to the discussion page. (That was the reason I did not respond to Yvonne’s question around making organisation/people/courses anonymous. BTW I think you do not have to make the course name anonymous as well, especially when you have ‘generic’ courses. I reckon you have to judge this on a case by case basis; “Recruitment of new police officers” might give more away than “Health and safety”. ). But most of it is probably getting used to technology.
I do think that if we had had more conferences, we would have created more of a community.
And for the blogs, I realise that we all need to have some kind of way to express our thoughts and thinking on the subject but having to read all the blogs takes up quite some time. I sometime did not feel like responding or updating my own blog after reading the contributions of others. (This one is a great example, I might want to share this with you, but do you all want to know????).
Anyway, time to finish now. I will probably add some more to my blog but don’t count on it as I go on annual leave.
Thanks all for your contributions, your questions and comments kept me on my toes.
7/07/08
2/07/08
Draft version report ready
It took me quite a long time to complete the draft version of my report. I started with including everything (including all raw data) because I prefered to have it all in one document for future reference. But it became all too much.
With some suggestions from Bronwyn I have been able to reduce it quite a bit, so don't worry, you don't have to go through 86 page :-).
I will put it on Google docs and hope you all can access it. My earlier attempts failed, so i appologise for that.
i have to admit that these last few weeks have been rather lonely. I haven't updated my blog every week and looking at other blogs it seems that everybody is busy working on their own. I kind of miss the group. Maybe it is time for another conference?
By the way, although the report is now finished (in draft) that does not mean I have not moved on with the course in the meantime. It was pretty obvious to my manager and myself what needs to be done, so I have implemented quite a few of the recommendations already.
Good luck with the last tasks ahead of us!
Go to evaluation report
With some suggestions from Bronwyn I have been able to reduce it quite a bit, so don't worry, you don't have to go through 86 page :-).
I will put it on Google docs and hope you all can access it. My earlier attempts failed, so i appologise for that.
i have to admit that these last few weeks have been rather lonely. I haven't updated my blog every week and looking at other blogs it seems that everybody is busy working on their own. I kind of miss the group. Maybe it is time for another conference?
By the way, although the report is now finished (in draft) that does not mean I have not moved on with the course in the meantime. It was pretty obvious to my manager and myself what needs to be done, so I have implemented quite a few of the recommendations already.
Good luck with the last tasks ahead of us!
Go to evaluation report
10/06/08
Thanks for advice!
I have used the method Yvonne had suggested and I think it worked well for me.
In my plan I had added a table that mapped to which topics the interview would contribute. I used those topics as headers on a whiteboard. Then I listed the comments of the participants on sticky notes and put them under the applicable headings. I tried to group the answers as much as possible but without losing interesting comments either. I then listed the comments in tables and how often it was mentioned.
I did not compare the two sites with one another for each topic as Hilary suggested. As I knew audio could not be played at Citrix site I included Close Captions for them. I wanted to test if that was an issue for the participants or not. Apparently it is. So the next step is making sure that the delivery mode at Citrix site is different so they can play the audio. This relates to something I read about usability testing. It is better to test multiple times with smaller groups and change your design after each test, than do one big test at once. Some (bigger) faults first to be dealt with before (smaller) other faults can surface. As the audio was not an issue at the non-citrix sites, they releaved some issues that are probably issues for the citrix users but as there was a more annoying issue, most attention was drawn to that one.
In my plan I had added a table that mapped to which topics the interview would contribute. I used those topics as headers on a whiteboard. Then I listed the comments of the participants on sticky notes and put them under the applicable headings. I tried to group the answers as much as possible but without losing interesting comments either. I then listed the comments in tables and how often it was mentioned.
I did not compare the two sites with one another for each topic as Hilary suggested. As I knew audio could not be played at Citrix site I included Close Captions for them. I wanted to test if that was an issue for the participants or not. Apparently it is. So the next step is making sure that the delivery mode at Citrix site is different so they can play the audio. This relates to something I read about usability testing. It is better to test multiple times with smaller groups and change your design after each test, than do one big test at once. Some (bigger) faults first to be dealt with before (smaller) other faults can surface. As the audio was not an issue at the non-citrix sites, they releaved some issues that are probably issues for the citrix users but as there was a more annoying issue, most attention was drawn to that one.
8/06/08
HELP
I hope someone can offer me some help on the following.
I am trying to analyse the interviews. I have already listed the responses people gave in their own words but when I try to analyse it, I end up repeating the same thing in my own words if I do it in a detailed way.
Was wondering if I should just summarise in one or two sentences (per question) what the general outcome was.
I don't have a lot of experience in these sorts of things, so any hints would be useful!!
I am trying to analyse the interviews. I have already listed the responses people gave in their own words but when I try to analyse it, I end up repeating the same thing in my own words if I do it in a detailed way.
Was wondering if I should just summarise in one or two sentences (per question) what the general outcome was.
I don't have a lot of experience in these sorts of things, so any hints would be useful!!
5/06/08
Outcomes
Although I haven't completed my draft my report, here are already some outcomes.
Besides learning from the evaluation itself, our processes could also be polished up.
One of the managers on our organisation had a look at the course in advanced and was quite annoyed by the scroll bar that appeared on her screen. (Fair enough, I tried to avoid that, but apparently was not successful in all cases). So quickly I changed the design a bit, which meant I had less space for the navigation, etc. I made the tabs and the top and the navigation bar under the simulations smaller.
One of the main outcomes of the evaluation was that half of the participants missed the different tabs and the scroll bar! (Three of them skipped the instructions as well, but apparently the course is not as intuitive as I thought it would be). I think that if I go back to my earlier design it will improve the 'intuitively' and I would like to test that with another group.
Another thing was the audio. Some of our sites use Citrix to connect to the network. This means that there is no audio available, they use a screen within a screen (more scroll bars) and the connection is much slower. Although the download time of the simulations was pretty good (the individual file sizes of the sims are relatively small) the simulations did not play as fluent as they should. The other issue is the audio. Although you can display the spoken text on the screen, this means more reading and actually in reality most of them did not even bother to display the text and just looked at the screen and interpret it themselves. (A lot of the comments were around that they like the voice over and they don’t like a lot of reading). And we have the effective screen size. Only one person knew how to optimise the screen (although I expected that all would know as they deal with this environment all the time, another expectation, adding on the discussion at Hillary’s blog).
So now we are thinking of distributing the courses to those site on DVD, although there are some issues to sort out, like how to update our data in our Training Booking System and the ‘expiry date’ on the DVD. But that are just some technical issues we have to tackle. This shift is really big because my manager was absolutely against this idea to start with (for good reasons) but (as he joined me on the trip) realised there is only one option here.
Overall staff are very positive about online learning as it saves them heaps of time and they are happy that DOC jumped on this train. Of course there are issues, like not allocating time to it and not having a quiet dedicated spot. Reality is that if there is a whale stranding (just one reason), staff would have to skip a f2f course but the online course is still available to them!
Besides learning from the evaluation itself, our processes could also be polished up.
One of the managers on our organisation had a look at the course in advanced and was quite annoyed by the scroll bar that appeared on her screen. (Fair enough, I tried to avoid that, but apparently was not successful in all cases). So quickly I changed the design a bit, which meant I had less space for the navigation, etc. I made the tabs and the top and the navigation bar under the simulations smaller.
One of the main outcomes of the evaluation was that half of the participants missed the different tabs and the scroll bar! (Three of them skipped the instructions as well, but apparently the course is not as intuitive as I thought it would be). I think that if I go back to my earlier design it will improve the 'intuitively' and I would like to test that with another group.
Another thing was the audio. Some of our sites use Citrix to connect to the network. This means that there is no audio available, they use a screen within a screen (more scroll bars) and the connection is much slower. Although the download time of the simulations was pretty good (the individual file sizes of the sims are relatively small) the simulations did not play as fluent as they should. The other issue is the audio. Although you can display the spoken text on the screen, this means more reading and actually in reality most of them did not even bother to display the text and just looked at the screen and interpret it themselves. (A lot of the comments were around that they like the voice over and they don’t like a lot of reading). And we have the effective screen size. Only one person knew how to optimise the screen (although I expected that all would know as they deal with this environment all the time, another expectation, adding on the discussion at Hillary’s blog).
So now we are thinking of distributing the courses to those site on DVD, although there are some issues to sort out, like how to update our data in our Training Booking System and the ‘expiry date’ on the DVD. But that are just some technical issues we have to tackle. This shift is really big because my manager was absolutely against this idea to start with (for good reasons) but (as he joined me on the trip) realised there is only one option here.
Overall staff are very positive about online learning as it saves them heaps of time and they are happy that DOC jumped on this train. Of course there are issues, like not allocating time to it and not having a quiet dedicated spot. Reality is that if there is a whale stranding (just one reason), staff would have to skip a f2f course but the online course is still available to them!
20/05/08
Observations
Wow! That was useful!
After two long days (including 4 hours of driving on each day), I now have collected heaps of useful information.
It was interesting to see how people interact with the course. It is funny when you get so close to it while developing it, that you hardly can put yourself in someone elses position.
My manager joined me on the second day and I was glad that he did, so he could see with is own eyes some of the issues we have to deal with.
For example.
The course is quite hard to find on the Intranet. A link to the course sits under People and Organisation/Training and Development/Technical Training/Computer Skills.
But how figures that out. You can also go to the Training Booking System, but hardly anybody has heard of that.
And then how people interact with it. Although I created instructions how to use the course, more thatn half jumped straight into it, only to find out at the end that there was more to discover.
I ahve to admit that I put those instruction under Help, which is at the bottom of the menu. My husband pointed out that some people only ask for help if there are really really stuck (unless people like me who read the instruction manual of our new dvd recorder from start to end before touching the machine).
So lots of work needs to be done.
I also received quite a few evaluation forms, which is promising. Some of the same issues came out of it at first glance.
I will try to quickly put my evaluation report together and move on with the recommendations as we need to get this layout of the course sorted.
A tip for people who are planning to do observations and havent' done it yet, it is better to observer three to four people, adjust your design and test it again, and so on (if that is possible and applicable to your situation).
I tested two days in a row and of course things that came out of the first day, also came out of the second day.
Oh well, it is al about improving, even our evaluation methods, isn't it?
I have read the following book over the weekend:
Don't Make Me Think! A Common Sense Approach to Web Usability by Steve Krug
Altough it is about web design, lots of it is applicable to online modules I reckon.
The observations I did prooved that. For example, I had created a tab for resources. When people clicked on the tab they saw one line (which was a link) and you just could see the look at their faces "what is that". Some tried it out, others couldn't be bothered. Hence, it made them think instead of that it was obvious.
There is also a chapter about usability testing which prooved to be useful for my observations ("we are evaluation the system, not you, you can't do anything wrong").
It is an easy reading, so if you have some spare time (on your flight to Canada for example), I would recommend this book.
Apologies for not posting the presentation yet. Work is extremely busy, and I can just keep up with it. It is still on my to-do list.
Do you guys mind if I send you an email with a flash file? I still haven't figure out how to post a flash file on the internet.
That's all for now.
Helga
After two long days (including 4 hours of driving on each day), I now have collected heaps of useful information.
It was interesting to see how people interact with the course. It is funny when you get so close to it while developing it, that you hardly can put yourself in someone elses position.
My manager joined me on the second day and I was glad that he did, so he could see with is own eyes some of the issues we have to deal with.
For example.
The course is quite hard to find on the Intranet. A link to the course sits under People and Organisation/Training and Development/Technical Training/Computer Skills.
But how figures that out. You can also go to the Training Booking System, but hardly anybody has heard of that.
And then how people interact with it. Although I created instructions how to use the course, more thatn half jumped straight into it, only to find out at the end that there was more to discover.
I ahve to admit that I put those instruction under Help, which is at the bottom of the menu. My husband pointed out that some people only ask for help if there are really really stuck (unless people like me who read the instruction manual of our new dvd recorder from start to end before touching the machine).
So lots of work needs to be done.
I also received quite a few evaluation forms, which is promising. Some of the same issues came out of it at first glance.
I will try to quickly put my evaluation report together and move on with the recommendations as we need to get this layout of the course sorted.
A tip for people who are planning to do observations and havent' done it yet, it is better to observer three to four people, adjust your design and test it again, and so on (if that is possible and applicable to your situation).
I tested two days in a row and of course things that came out of the first day, also came out of the second day.
Oh well, it is al about improving, even our evaluation methods, isn't it?
I have read the following book over the weekend:
Don't Make Me Think! A Common Sense Approach to Web Usability by Steve Krug
Altough it is about web design, lots of it is applicable to online modules I reckon.
The observations I did prooved that. For example, I had created a tab for resources. When people clicked on the tab they saw one line (which was a link) and you just could see the look at their faces "what is that". Some tried it out, others couldn't be bothered. Hence, it made them think instead of that it was obvious.
There is also a chapter about usability testing which prooved to be useful for my observations ("we are evaluation the system, not you, you can't do anything wrong").
It is an easy reading, so if you have some spare time (on your flight to Canada for example), I would recommend this book.
Apologies for not posting the presentation yet. Work is extremely busy, and I can just keep up with it. It is still on my to-do list.
Do you guys mind if I send you an email with a flash file? I still haven't figure out how to post a flash file on the internet.
That's all for now.
Helga
12/05/08
Great day!
Just came back from a fabulous day trip.
I brought a visit to G. in his work environment and was really impressed by what he and his team are doing. His technical training development unit were developing really top level material and I came home inspired and with full of ideas.
Can't wait to start tomorrow with SCORM wrappers and what more....
BTW:
as the discussion pages on the wiki is rather empty, I have added a issue of which I would like to hear your opinion...
read discussion
Cheers, Helga
I brought a visit to G. in his work environment and was really impressed by what he and his team are doing. His technical training development unit were developing really top level material and I came home inspired and with full of ideas.
Can't wait to start tomorrow with SCORM wrappers and what more....
BTW:
as the discussion pages on the wiki is rather empty, I have added a issue of which I would like to hear your opinion...
read discussion
Cheers, Helga
Subscribe to:
Posts (Atom)
