Helga's blog

Some thoughts....

2008-07-07T22:38:00.000-07:00

I am approaching the end of the course and it is time to reflect on the last few months.
I agree with Gordon that I learned heaps from being a participant in this online course as well as from the content.

Although we only carried out a relatively small evaluation, because we were able to ‘look in each others kitchen’ you realise how important it is to do evaluations in different stages and evaluate different aspects of it. The formative ones (like the one I did) are essential to ensure an optimal learning experience. For example not being able to hear the audio is more important than we first realised. Just providing captions is insufficient. The funny thing is that this is backed up by lots of research I found out recently (see Chapter 5:’Applying the modality principle’ in e-Learning and the science of instruction (Clark & Mayer, 2008)).
Gordon evaluated a module that has been created a while ago and it is encouraging to see that this module is still up to data and appreciated by staff and students. Yvonne on the other hand focussed on the introduction of a Moodle on a college. The e-learning guidelines prove to be very useful as it helped me to overview all (or at least many) aspects of e-learning. Besides the creating a good module, you also have to ensure good support and assistance, the technical environment, by-in from management. And although I could not personally include all these aspects during this course, I could ‘taste’ some of them through looking at what other participants did and are doing. I am looking forward to see their reports and also to pay more attention to the whole e-learning strategy in our own organisation. (I will start with reading ‘E-Learning: Strategies for Delivering Knowledge in the Digital Age’ (Rosenberg, 2000)).

I had a bit more trouble to get the grip on those paradigms. That was a bit too abstract to me. The models are already more practical and therefore more useful to me. That you use a model that uses triangulation to get a picture from more angles, that makes sense to me. It was a good exercise to spend some time on this because honestly, normally I would not have paid attention to this. But that is what you do a course for, to broaden your horizon!

I also had to find a balance between what is required for the organisation and what is required for the course. My project was/is a real time, real case example and I sometimes got a bit sidetracked because of that. For example, I had the comprehensive list of recommendations written and discussed with manager a while ago and proceed with the action and ‘forgot’ to write the rest of the report. Another example is that I forgot to add a “background about the organisation and why the web conferencing was useful enough to warrant a training programme to be created” as Yvonne suggested (but it is not too late for that I suppose).

I enjoyed the discussions we had. I will no longer use the four point Likert scale . It was great to test things out and to share ideas. I really wish we had more conferences or options to talk to each other. Maybe I should have started some of the other courses as well as I have still many questions. And there are so many people out there with all different knowledge and experience that I wish I had more opportunities to share ideas with those people.

To finish off, some feedback on the course delivery mode.
I really enjoyed being an online student. I appreciated the flexibility of doing this course in my own pace at a time that suits me. I realise that it must be a hard job being a facilitator for a course like this: motivating people, creating a sense of a group, being there with quick response to e-mails and contributions to blogs… Thanks heaps Bronwyn!!
The things that are not my preferences are: too many places to go to, to get all information. I sometimes just lost track of things. Besides that a couple of things were not that user friendly and frustrated me mostly. For example, I realised that I could not share pdf’s with you all after 36 attempts (would have been great if I had known that before hand), I struggled to contribute to the discussion page. (That was the reason I did not respond to Yvonne’s question around making organisation/people/courses anonymous. BTW I think you do not have to make the course name anonymous as well, especially when you have ‘generic’ courses. I reckon you have to judge this on a case by case basis; “Recruitment of new police officers” might give more away than “Health and safety”. ). But most of it is probably getting used to technology.
I do think that if we had had more conferences, we would have created more of a community.
And for the blogs, I realise that we all need to have some kind of way to express our thoughts and thinking on the subject but having to read all the blogs takes up quite some time. I sometime did not feel like responding or updating my own blog after reading the contributions of others. (This one is a great example, I might want to share this with you, but do you all want to know????).

Anyway, time to finish now. I will probably add some more to my blog but don’t count on it as I go on annual leave.

Thanks all for your contributions, your questions and comments kept me on my toes.

Draft version report ready

2008-07-02T13:51:00.000-07:00

It took me quite a long time to complete the draft version of my report. I started with including everything (including all raw data) because I prefered to have it all in one document for future reference. But it became all too much.

With some suggestions from Bronwyn I have been able to reduce it quite a bit, so don't worry, you don't have to go through 86 page :-).

I will put it on Google docs and hope you all can access it. My earlier attempts failed, so i appologise for that.

i have to admit that these last few weeks have been rather lonely. I haven't updated my blog every week and looking at other blogs it seems that everybody is busy working on their own. I kind of miss the group. Maybe it is time for another conference?

By the way, although the report is now finished (in draft) that does not mean I have not moved on with the course in the meantime. It was pretty obvious to my manager and myself what needs to be done, so I have implemented quite a few of the recommendations already.

Good luck with the last tasks ahead of us!

Go to evaluation report

Thanks for advice!

2008-06-10T17:14:00.000-07:00

I have used the method Yvonne had suggested and I think it worked well for me.
In my plan I had added a table that mapped to which topics the interview would contribute. I used those topics as headers on a whiteboard. Then I listed the comments of the participants on sticky notes and put them under the applicable headings. I tried to group the answers as much as possible but without losing interesting comments either. I then listed the comments in tables and how often it was mentioned.

I did not compare the two sites with one another for each topic as Hilary suggested. As I knew audio could not be played at Citrix site I included Close Captions for them. I wanted to test if that was an issue for the participants or not. Apparently it is. So the next step is making sure that the delivery mode at Citrix site is different so they can play the audio. This relates to something I read about usability testing. It is better to test multiple times with smaller groups and change your design after each test, than do one big test at once. Some (bigger) faults first to be dealt with before (smaller) other faults can surface. As the audio was not an issue at the non-citrix sites, they releaved some issues that are probably issues for the citrix users but as there was a more annoying issue, most attention was drawn to that one.

HELP

2008-06-08T20:23:00.000-07:00

I hope someone can offer me some help on the following.
I am trying to analyse the interviews. I have already listed the responses people gave in their own words but when I try to analyse it, I end up repeating the same thing in my own words if I do it in a detailed way.

Was wondering if I should just summarise in one or two sentences (per question) what the general outcome was.

I don't have a lot of experience in these sorts of things, so any hints would be useful!!

Outcomes

2008-06-05T02:15:00.000-07:00

Although I haven't completed my draft my report, here are already some outcomes.
Besides learning from the evaluation itself, our processes could also be polished up.

One of the managers on our organisation had a look at the course in advanced and was quite annoyed by the scroll bar that appeared on her screen. (Fair enough, I tried to avoid that, but apparently was not successful in all cases). So quickly I changed the design a bit, which meant I had less space for the navigation, etc. I made the tabs and the top and the navigation bar under the simulations smaller.
One of the main outcomes of the evaluation was that half of the participants missed the different tabs and the scroll bar! (Three of them skipped the instructions as well, but apparently the course is not as intuitive as I thought it would be). I think that if I go back to my earlier design it will improve the 'intuitively' and I would like to test that with another group.

Another thing was the audio. Some of our sites use Citrix to connect to the network. This means that there is no audio available, they use a screen within a screen (more scroll bars) and the connection is much slower. Although the download time of the simulations was pretty good (the individual file sizes of the sims are relatively small) the simulations did not play as fluent as they should. The other issue is the audio. Although you can display the spoken text on the screen, this means more reading and actually in reality most of them did not even bother to display the text and just looked at the screen and interpret it themselves. (A lot of the comments were around that they like the voice over and they don’t like a lot of reading). And we have the effective screen size. Only one person knew how to optimise the screen (although I expected that all would know as they deal with this environment all the time, another expectation, adding on the discussion at Hillary’s blog).

So now we are thinking of distributing the courses to those site on DVD, although there are some issues to sort out, like how to update our data in our Training Booking System and the ‘expiry date’ on the DVD. But that are just some technical issues we have to tackle. This shift is really big because my manager was absolutely against this idea to start with (for good reasons) but (as he joined me on the trip) realised there is only one option here.

Overall staff are very positive about online learning as it saves them heaps of time and they are happy that DOC jumped on this train. Of course there are issues, like not allocating time to it and not having a quiet dedicated spot. Reality is that if there is a whale stranding (just one reason), staff would have to skip a f2f course but the online course is still available to them!

Observations

2008-05-20T02:18:00.000-07:00

Wow! That was useful!
After two long days (including 4 hours of driving on each day), I now have collected heaps of useful information.
It was interesting to see how people interact with the course. It is funny when you get so close to it while developing it, that you hardly can put yourself in someone elses position.
My manager joined me on the second day and I was glad that he did, so he could see with is own eyes some of the issues we have to deal with.
For example.
The course is quite hard to find on the Intranet. A link to the course sits under People and Organisation/Training and Development/Technical Training/Computer Skills.
But how figures that out. You can also go to the Training Booking System, but hardly anybody has heard of that.

And then how people interact with it. Although I created instructions how to use the course, more thatn half jumped straight into it, only to find out at the end that there was more to discover.
I ahve to admit that I put those instruction under Help, which is at the bottom of the menu. My husband pointed out that some people only ask for help if there are really really stuck (unless people like me who read the instruction manual of our new dvd recorder from start to end before touching the machine).

So lots of work needs to be done.

I also received quite a few evaluation forms, which is promising. Some of the same issues came out of it at first glance.

I will try to quickly put my evaluation report together and move on with the recommendations as we need to get this layout of the course sorted.

A tip for people who are planning to do observations and havent' done it yet, it is better to observer three to four people, adjust your design and test it again, and so on (if that is possible and applicable to your situation).
I tested two days in a row and of course things that came out of the first day, also came out of the second day.

Oh well, it is al about improving, even our evaluation methods, isn't it?

I have read the following book over the weekend:

Don't Make Me Think! A Common Sense Approach to Web Usability by Steve Krug

Altough it is about web design, lots of it is applicable to online modules I reckon.

The observations I did prooved that. For example, I had created a tab for resources. When people clicked on the tab they saw one line (which was a link) and you just could see the look at their faces "what is that". Some tried it out, others couldn't be bothered. Hence, it made them think instead of that it was obvious.

There is also a chapter about usability testing which prooved to be useful for my observations ("we are evaluation the system, not you, you can't do anything wrong").

It is an easy reading, so if you have some spare time (on your flight to Canada for example), I would recommend this book.

Apologies for not posting the presentation yet. Work is extremely busy, and I can just keep up with it. It is still on my to-do list.

Do you guys mind if I send you an email with a flash file? I still haven't figure out how to post a flash file on the internet.

That's all for now.

Helga

Great day!

2008-05-12T00:26:00.000-07:00

Just came back from a fabulous day trip.
I brought a visit to G. in his work environment and was really impressed by what he and his team are doing. His technical training development unit were developing really top level material and I came home inspired and with full of ideas.

Can't wait to start tomorrow with SCORM wrappers and what more....

BTW:
as the discussion pages on the wiki is rather empty, I have added a issue of which I would like to hear your opinion...

read discussion

Cheers, Helga

presenting and feedback

2008-05-08T02:11:00.000-07:00

Well, I still haven't managed to get my plan shared on google docs, but tomorrow it will become available.
I am thinking of creating a flash presentation for you to watch, which will be much better to consume than the long plan.

I know it is the idea that you would first commend on my plan and that I would adjust it, but next week (Friday 16 and Monday 19/05) I will conduct part of the evaluation and do the observations. Very exciting and I am much looking forward to it. I am expecting it to be a real eye-opener.

In one way it is better to have someone else evaluation the course that you have created but on the other hand, now I can experience myself how people interact (and I try to keep it quiet that I am the developer). Especially to see non computer literate people using it will probably tell me some things.

Anyway, I wil promise you to keep you informed how it all goes and what I have learned from doing the observations so you might be able to pick up some good ideas.

Time to read some other peoples plans now.

Cheers, Helga

Evaluation plan

2008-04-29T16:01:00.000-07:00

Well, I have added my plan to google docs.
Here is the link to my
evaluation plan.

PS I am not sure if the other one that I send you all an email about still exist. I can't find it anywhere myself.

If someone can inform tell me if you can still access it, if not i will add you to this new one.

Evaluation plan

2008-04-28T22:54:00.000-07:00

It looks like I have disappeared completely, but the opposite is true. I have been working on my evaluation plan and it is almost ready to be shared with all of you. I will try to finalise it and put it on google docs.
For if you don't have the time to read it all here is a short summary of what I am planning to do.
It is going to be a formative evaluation, based on the Eclectic-Mixed Methods-Pragmatic Paradigm and using Stake’s Responsive evaluation model.
Guidelines that I use are:
MO1 Can staff adn studenst easily use the learning technologies and online resources?
SD3 Do students gain knowledge relevant to employment and/or current thining in their field?
I will use the following methods.
I will observe eight staff in using the online training tool. Four in an office with normal network access and four in an office with remote access to the network. This is important as the 'remote sites' can not hear the audio and have to display the transcripts on the screen.
Afterwards I will interview those participants and they will do an exercise to assess their learning.
Besides that I will send out a questionnaire to twenty additinal staff who will complete the course. These people are again from different offices, so I will get a mix of people from different offices, roles, computer experiences.
I will also include any feedback from the stakeholders to the data I collect.

I am pretty excited about the evaluation and can't wait to start it. :o)

Course to be evaluated

2008-04-13T01:09:00.001-07:00

Evaluation plan
Subject of my evaluation will be the online course that I have developed with Dreamweaver CS3 and Captivate 3 (both Adobe products). The subject is the web conferencing tool that we use in our organisation.
The Department of Conservation is one of the six government agencies that will have to reduce their carbon footprint to become an example for others. The sustainability manager has a big toolbox that will help the department. Doing more meetings by web conference is one of them.
There are a few people in the department that will figure out quickly enough how to operate the web conference tool, but most prefer some kind of training before they start using it.
As the whole purpose of using the web conference tool is reducing the carbon footprint, I suggested that we shouldn’t do a tour around the whole country to train people, hence the online training tool.
The set up is as follows. The course has several chapters with topics (tasks). Each topic starts with an overview in which key point and business rules (if there are any) are explained. Participant can reveal the information by hovering with their mouse over boxes. The next tab is a Show Me tab, in which a virtual trainer demonstrates how to complete the task this tab contains audio. The participant can decide to complete the Let Me tab, in which the participant is in charge. There are cues provided, like arrows where to click and instruction boxes. There is also a Test Me tab. In this simulation the participant only gets a generic task description and has to complete the task without having any additional cues.
There is a help menu that explains how the course works and there are printable quick reference cards (both for how the course works and for the content of the course). Those documents can be found in the Resources tabs.
I will test and evaluate this course in the Wairarapa area office. I will go to the area office itself and most likely observe four to five individuals using the course. I will also go to Mount Bruce to do the same. The reason that I go to two offices is that one is a so called “Citrix” site, this office access the DOC network via Citrix. On Citrix sites is no audio available. Participants have to display the spoken text on the screen (closed captions). Therefore I would like to observe how this works (or not works) on Citrix site.
I have started working on an evaluation plan. As I have that at the office, I will later on post that on my blog.

Focus of my evaluation project

2008-04-03T00:47:00.000-07:00

It is likely that I will focus on one particular course that I am currently developing.
There are a couple of reasons for that.
First of all the subject of my evaluation is then not too big and overwhelming and it is more likely that I will get some results at the end of this course.
Second, the layout and design of this first course will be used in other courses that are yet to be developed. Finding out now what works and what not will prevent that we copy the same mistake.
The whole framework has yet to be approved and it is unlikely that I will be able to start a pilot in the next couple of weeks. I can use these weeks to design an evaluation plan for the pilot so we can do an effective pilot that will answer many questions.

Preferably I want to do an online survey to collect data from people in different roles and offices. On top of that I also want to also want to do some user testing. In other words be around while they test it to see how they use it. That is especially for the usability of the layout and set up. It is not practical to go through the country for that so I could try an office with mainly office staff and an office that is using a citrix environment and with people who are more out of the field so I have a range of that as well.

Tomorrow I will have another look at all the guidelines that I found useful to finally pick the two.

By the way, it was an interesting meeting tonight. I like that besides talking about evaluation we shared ideas about what works and what doesn’t from people experience (for example the expiry date tip was superb).

Evaluation models

2008-03-27T21:31:00.000-07:00

Some more information on evaluation models (from Chapter 2 of Interactive Learning Systems Evaluation (Thomas C. Reeves, 2003), and my reflection whether I can or want use them.

As with the guidelines, there are many models that are applicable and/or appealing to me. I am not sure if we have to pick one model, if so, I think the multiple methods evaluation model is the most useful for my situation, although it is a complex one.

Multiple Methods Evaluation Model
Key issues in Mark and Shotland’s approach are triangulation and bracketing. Triangulation involves using multiple measures to converge on a more accurate estimate of the ‘true’ value of a variable. Bracketing involves using multiple measures to provide a range of estimates of the value of a variable. Mark and Shotland maintain that the bracketing approach is more practical in evaluation because of the complexity and uncertainly involved in education. This one gives you a range of data from different perspectives. In my opinion is the triangulation very useful, especially in the formative phase when you are still finding out which road to follow.

Experimental Evaluation Model
If you want to compare e-learning with f2f learning, this means that you should have training available for both situation and then compare the two of them. This model is not practical for our situation. We don’t have the time and money to develop two packages.

Tylerian Objectives-Based Evaluation Model
In my opinion this focus is or should be included in all training evaluation. Training is only effective if the objectives are met. Besides that you can also evaluate if the method and approach used is the most efficient, appealing, cost effective, etc. one. So only focusing on the objectives is a bit limited.

Patton’s Qualitative Evaluation Model
Patton emphasizes the value of qualitative methods, especially observations, if the intent of an evaluation is formative, i.e. focused on improving a program or product (p. 39 Interactive Learning Systems Evaluation, Thomas C. Reeves, 2003). Methods that could be used are observations, case studies, interviews and document analysis.

As it is quite likely that the focus of my evaluation project is formative, this model might be useful to me.

Fourth Generation Evaluation Model - Guba and Lincoln
Guba and Lincoln don’t believe that evaluation leads to the ultimate resolution of social problems. They see evaluation as a continuous, recursive and divergent process with unpredictable outcome. They focus on local understanding rather than generalizations.

What I like the principle that evaluation have unpredictable outcomes. We should always try to keep an unbiased and neutral approach, but often we are involved in the design, delivery and evaluation so tend to look at outcomes that suit our needs.

Eisner’s Art Criticism Evaluation Model
Opponents of educational criticism claim that this model is too subjective but proponent respond that positivists are also subjective when they formulate research questions, choose instruments, and interpret data, but their subjectivity is hidden.

This links nicely to the commend I made before that you have to try to be neutral and unbiased. From this models’ point of view you should acknowledge that you are always subjective and that it is better to bring that to the surface to that others can judge the worth of your evaluation.

Postmodern Evaluation Model
From a postmodern perspective instructional designers should not automatically assume that their systems models and instructional technologies are the best methods for establishing conditions for teaching and learning. And they should constantly examine their motives and methods to ensure that minority perspective are included.

This relates to my commends I made on the Tylerian Objectives-Based Evaluation Model, that focusing on objectives only was a bit limited. This models says more or less that because the technology exist, it doesn’t mean it is the most effective way (Petrol cars are being used even though there are more effective alternatives like the electric car). And what I also want to keep in mind is that the minorities are included. In our organisation computer illiterate people are in the minority but if we don’t cater for them, the will miss out completely.

Stake’s Responsive Evaluation Model
This models has similarities with the Multiple Methods Evaluation Model as responsive evaluators will attempt to triangulate issues and findings using multiple methods. But it takes into account the stakeholders (including administrators, teachers, students, programme developers, community members, funders, evaluators).

Stakes also acknowledge the subjectivity of evaluation (as Eisner).

Positive feedback

2008-03-24T23:22:00.000-07:00

Just returned from an meeting with a colleague who gave me feedback on an online demonstration that I had created.

The in-house created Time Recording System was being upgraded, and during the TNA for computer training in general I heard the complaint that staff out in the field dislike the fact that Head Office ‘dumps’ an upgrade on them and they have to find out themselves what the new features are. So, I developed a 10 minute simulation/demonstration in Captivate, which would show them the new features. The business owner requested that in order to be able to install the new system, they first had to go through the demo (tricky! forcing everybody to do it).

But anyway, my colleague had mainly great feedback. He said that this was the smoothest upgrade of a system that he had during his career. He received the least queries! Not that I am saying that this is only due to the demo (although I like to say that!), but it was great to hear. Even the computer illiterate people did have problems going through it. The rangers in the field could easily relate to it (f.e. we use track markers as arrows). And we used plain English (sometimes being a foreigner and having a limited vocabulary is an advantage :-) ).

He gave me some other tips as well (for example testing the material at certain sites where there are technical limitations (e.g sound is not available at all sites, and the offices who work via satellite have slow network connection). Actually, I am considering doing a web-based survey to get more feedback on this demo, as we are still at the beginning setting up the training framework.

Anyone having experience in that or can someone give me some tips?

PS What was the label again that we should give our items on our blogs???

Issues and guidelines

2008-03-13T12:42:00.000-07:00

There are actually three guidelines that I definitely want to include in the evaluation.

MD1 Managers / Learning design / Good practice

What systems are in place for monitoring the quality of study material, including its periodic review and or redevelopment?

The answers for this one will probably not directly come out of the evaluation, but it is something that our organisation has to think about. At the moment we are not terribly good at the periodic review and redevelopment of course material (f2f courses) because new topics to develop training on constant arise and we hardly have time to look at the existing training material. It is quite likely that as it is not already something that we do for f2f courses, we won’t do it with the e-learning either. In my evaluation rapport I want to address this and come up with some possible solutions. Another guideline related to that is MO4 Is there an evaluation process to ensure further upgrades take into account user feedback for improvement?

TT4 Do students get prompt supportive comments on their progress in relation to learning expectations and goals?

All the material that we have created so far is computer simulations and in those simulations you can do it either right or wrong. An instant message comes up in case you have made a mistake and you will either get another change in the exercises, or you will get feedback what the correct step was in the practice assessments. I want to know after the pilot if the participants find this encouraging enough and informative enough. I also want to know how we should send out the final results. Is it desirable that the final result automatically appear because people would like to know how they have done it straight away, or is it more ‘human’ if the trainer contacts them with the results, discussed them and then the participant and trainer design a learning path together.

SO7 Is it clear at the start of the course what support teaching staff will provide students?

The material is designed that participants will get started with help of their manager, but after that is should be completely self paced and intuitive. During the pilot the developer of the material will be available for support and enquiries but what I want to find out during the pilot is what support we need to offer in the first place. Is this an email address, telephone, remote control, web conference? And what are the things we offer support on. Do the participants mainly have technical questions (e.g. audio not working) or questions about the content. And have we communicated well enough where participants should go with what question (or should there be one point of contact for all). Another guideline related to that is MO12 Are there clear lines of responsibility for e-learning technical and pedagogical issues and problems?

Scenario Department of Conservation

Up to now a paper checklist has been used to during the induction of new staff to find out current computer skill levels and introduce them to DOC’s specific essential systems. The assessment of skill level is not evidence based but simply a question can you or can you not. The manager is responsible for the induction of staff and therefore the quality of the introduction to new systems depends on time available, manager’s skills in using the systems, manager’s attitude in using the systems, customs in using the systems in that particular office, support from other staff in the office.

The idea is to replace the checklist with an online computer skill framework which concise of skill checks (evidence based assessment of skill level) and courses and demonstrations on the essential systems. The manager will only assist the new staff member with the start up and will explain the policies and procedures. After that the new staff member will work his way through all the items in his own pace.

Issues to consider and guideline that may assist are:

- Will all offices be able to use the framework? Are there technical restrictions preventing them to use it?

SO13 Do students get appropriate ICT support in a timely manner?

MO6 Does the organisation provide a reliable, secure and up-to-date IT infrastructure that meets the requirements of the course?

- Will the skill checks identify the minimum skill level set by the Department?

MT3 Does the institution review the effectiveness of its e-learning to assure alignment with institutional priorities and educational objectives?

TT11 Are online assessment tools for the course identified and evaluated?

TT2 Does course assessment relate to intended learning outcomes and to student learning needs and situations?

TD6 Does the teacher monitor the assessment to ensure effectiveness?

- Will the new staff member be able to complete the skill checks, demos and courses on his own?

MO5 Is an evaluation mechanism in place to allow staff to give feedback on support services?

MO1 Can staff and students easily use the learning technologies and online resources?

ST3 Do the students know at the start of the course what is expected of them?

SO12 Do students have access to content support in a timely manner?

Time to go to work now...
Helga

Elearning guidelines part 2

2008-03-08T23:45:00.000-08:00

E-learning guidelines

In our organisations the guidelines can be applied at two levels. One is the whole computer training / induction framework, the other is the individual courses that are part of this framework.

At this stage I haven’t decided to evaluate the whole framework or one particular course that I am developing (topic: use of the electronic document and records management system). The decision mainly depends on the decision making internally, when we are able to implement the framework. It is not likely that the framework will be in place for very long time at the end of the course, so I think that we can only do a summative evaluation for a couple of participants.

As I am still in the development phase, I will mainly focus on a formative evaluation.

TD5 Teaching staff/ Learning design

Has a representative sample of students tested the e-learning materials and any necessary modifications been made?

We have the idea to organise a pilot where about 6-8 people who have different roles and computer skill levels will be put together in a training room with laptops. They will have to go through the self paced frame work as if they were sitting at their desk. I will be there to observe and, respond to questions. At the end I will ask them to fill out a questionnaire/evaluation form. After that I want to do a group discussion about what they liked and didn’t like, if it was intuitive. Hopefully when talking about it, I get more information then just from the forms.

SD3 Students/ Learning design

Do students gain knowledge relevant to employment and/or current thinking in their field?

This guideline is extremely important as the whole framework that we are setting up is based on this assumption. It is not so much ‘knowledge’ but more skills and ‘finding way around’. Our assumption is that part of our staff who are primarily working in the field only need to have basic computer skills in certain applications. The rest of the staff who use the computer more often need to have a minimum level of skills to be able to work efficiently with the systems and databases in place. Therefore I have created a few ‘skill checks’ to find out if they meet these basic skills. If not, online training on those topics will be available and staff will have to complete it in a certain timeframe. We have carried out an extensive TNA which include interview with key people and analysing helpdesk calls and initiative that have taken place in the past. At the moment the organisation uses a checklist for new staff to find out if they are able to work on the DOC network. But on your first day, it is quite likely that you say “yes” to the question if you are able to use Outlook. Hopefully the skill checks will determine the real minimum level that people have.

SD8 Students / Learning design

Is there a mechanism in place to asses how students perceive the value of the course?

At the moment this is not yet in place but we haven’t launched any of the online courses either. So we still have some time to organise this.

I would like to do the common end of course evaluation but also want to know what their managers and the organisation as a whole think about it.

For example it is quite likely that a person might think that a whole course on the document and recordkeeping management system is a bit too much (although they should know by the end of the course that public servants have a legal obligation to keep records), but for the organisation as a whole it is very important to have this course as a mandatory part of the induction.

ST2 Students / Teaching relationships

Do the students know at the start of the course what is expected of them?

I picked this guideline because it is something that I should not forget while developing the course. When we started developing the first course we created what I call the from a to z courses. You start at the beginning and take them by the hand until the very end. The students are not able to create their own learning path/order (we are talking about computer (simulation) courses). But now we have move to a more flexible approach in which the students can decide which topic to do first and if they want to do the ‘show me’, ‘let me’, or ‘test me’ mode (or do them all). Because the participant is now more in charge in what to do, it is especially important to point out what is expected. I have to make sure that at the beginning of the course I point this out.

2008-03-07T21:30:00.000-08:00

Why is evaluation important to you and how do you define it?

I would define evaluation as a constant checking and fine tuning of all kinds of processes that happens most of the time unconcious and/or automatic and sometime in a planned way. Most of it is done during the proces but generally we tend to evaluate afterwards. Often because we are already tied in with other programmes and don't take time to look back and learn from it.

To get more out of evaluation it should be approach in a planned and organised way so you do get information on the most important toppics/items and if you approach it in a planned why it becomes more and more a routine.

What sort of evaluations mentioned on the presentation are familiar to you already and why?

In our organisation we do more with the formative evaluation than with the summative evaluation. Expert review, checklists, userbility testing, pilots (with feedback) are formats that we use when developing courses. After a course we tend to do the final 'good old questionairre', but that is mainly it. I do believe that the constant checking and fine tuning during the development proces is extremely important, because that makes sure that the soup that you serve is actually good. But what also is important is how valuable it all is at the end of the day. Does the learning last is something that we should and could focus on much more.

One of the things that I did when developing the computer training framework was analysing hundreds of helpdesk calls, to find out where people have most often problems with. Although this gave us only an indication as most people will ask their colllegues first before calling the helpdesk, it was all the data we had. A couple of months after implementation of the framework we will again analyse the latest data to see if there is a shift. This should help us finding out at least the framework has some kind of impact. (Of course, this will not be the only way to evaluate the framework, but that plan will crystalise during this course :-).

Why is quality important in eLearning?

With f2f courses, as a trainer you can constantly fine tune during the course if the content is what the participants need and want (so part of evaluation is done observing the participants during the course, you pick things up, what they say, how they behave and respond). With eLearning you miss this possibility. So you have to make sure that the course is what people need. You only find this out
with evaluation. The trick is to get honest and constructive feedback from the participants. That is a real challange. So the next couple of weeks I will do some more thinking around this....

Time for dinner now. Will get back with the 4 elearning guidelines.

eLearning Guidebook

2008-03-04T01:00:00.000-08:00

Well, it looks like that I will be able to participate with elluminate on Thursday. At work they could not really help me because apparently the site of the Otago University had all kinds of trouble when businesses and organisations try to use elluminate. But at home I was able to install it.

After I had read chapter 8 of the eLearning Guidebook, I was interested in the rest as well and I have read most of the other chapters. Especially the chapter on what is eLearning, what and what not can you call eLearning was quite helpfull. I started to think that the courses I create were not really eLearning but luckily I now know better. At least I have something to evaluate.

Started reading about the guidelines as well and realised that a lot of them are applicable to our situation. Some of them, we already take into account. But others made me realise that we should do something about them as well. I try to focus this week on the ones that we should but not have catered for. Later this week I will do some more thinking around them.

Talk to you on Thursday.

Other blog

2008-02-26T00:49:00.000-08:00

By the way, the other blog that I provided I link to, is written in Dutch. But the photos (foto's) will speak for themselves (if you are interested).

Some parts of this blog will probably be displayed in Dutch as well. The default language was Dutch and I havent' figured out how to change it on every page.

Welcome to my blog

2008-02-26T00:06:00.001-08:00

I am Helga Wientjes (forget the surname).

I work for the Department of Conservation as a Programme Leader Training. Our team develops training for our staff (and also for communities) on conservation related topics (i.e. track maintenance, hut inspection, pest control, monitoring techniques). Up to know all these courses have been face to face.

I have joined the team about 9 months ago and my task is to develop computer training for both the standard products (MS Office, IE, Windows XP) as well as the in-house created or customised databases and system (our time recording system, the document and record management system).

My first task was to find out where to focus on and how this all should be shaped up. With the whole sustainability discussion, the fact that our organization is very dispersed and that I am the only one responsible for the development and delivery of computer training, it was worthwhile checking out if web based training. So here I am developing all kinds of thing with Adobe Captivate and Dreamweaver.

Although I am very new to this whole business, I am aware that web based training is not the same as the e-learning that we do with the Evaluation of eLearning for Best Practice. But maybe (or probably) we will follow that path in the future as well, and the web based training also needs to be evaluated.

I first want to clarify for myself if I want to evaluate the “technique” and compare the outcome of a face to face (traditional) course with the outcome of a web based / online course on the same topic. A lot of people within DOC are quite skeptical about the whole self pace web based training. But is this because it is new? Is this because they don’t want to sit at a computer (most of our staff is operational, working in the field)? Is it because they are scared? Is it because they think they can’t get as much out of it as a traditional course? Another way of focusing on the evaluation is: we have completed a training needs analysis and identified the audience, topics, and methods. And is the selected method sufficient to cater for those needs. Is the course in itself successful?

Well, you have probably figured out that I am a person that thinks while talks. I am sure that during the next couple of weeks my ideas and thoughts will change drastically. But for know this is probably a good start….