Forget About Level 4? Never!Not too many years ago I remember the words of an L&D VP to whom I reported. We were talking about measuring the effectiveness of a very expensive training program we just delivered.“Just focus on the first three. Forget about this Kirkpatrick level four,” he said. “It’s too hard and too expensive to figure out.”As a refresher, here are the four levels of Kirkpatrick’s evaluation model (I’m avoiding the argument about Level 5 on purpose):Reaction – what the learners thought about the courseLearning – what the learners remember as well as any new skills and attitudes.Behavior – How much the learners transfer knowledge, skills, and attitudes from the schoolplace to the workplaceResults – the final outcome, months down the road from the event, which was initiated by the course.The first 3 levels are relatively easy to measure. They include the smile sheets (Level 1), demonstrations of what was learned (Level 2) and improvements in performance back at work (Level 3). The first two can happen during the training event; the third can be reviewed and assessed by a learner’s manager.It’s Level 4 that’s more difficult, even though it’s the level that measures real learning. Let me back up a bit. Rote learning is what ‘skill and drill’ teaching gets you. It’s perfect for a Level 1 and 2 evaluations. You can even get by if the Level 3 evaluation is done soon enough after the course is finished.If no one checks in after that you will probably not get a “Pass” on Level 4, unless you have adopted what you do every day and adapt it under a constantly changing set of circumstances. Level 4 is gated by the idea that “Practice Makes Perfect”. So it’s the down the road assessment that really tells you if the learning has become a new part of the learner’s way of doing their job.Level 4 is a longitudinal study or assessment. It can be done at intervals that range up to one year from the learning event. It’s usually not done at all because it is the most costly and time consuming of the four. What’s changed is that new technology can make it easy.
LCMS Learning Objects to the Rescue.
The LCMS is usually thought of in terms of their ability to author learning objects. These objects can be stored in a repository and used to deliver a custom learning program. The learning objects are assembled by an individual learner who can tailor them into a personal learning path. On the other hand, a course that is SCORMed and developed as one-size-fits-many can be seen as one big learning object fixed in space.
When people are done with either a course or their personal learning path, it looks like the pellets flying out of a shotgun. All the learners go off in their own direction, and have separate and individual experiences. In short, they learn to adapt the knowledge and know-how they acquire in a multitude of different ways.
The course object can only measure the mean or average since it was designed for many people. Most Level 4 measures I’ve seen look at corporate data as if it was functionally related to what the learner knows or has learned to do. For example, an increase in employee retention can be the result of wage increases or an improved management style. Reduced waste is an old manufacturing metric that has little validity in today’s manufacturing processes. Increased customer satisfaction results from a constellation of factors. Fewer staff complaints in a tough economy are to be expected (add in increased retention as well). So the standard measures used at Level 4 are virtually useless in today’s workplace and economic environment.
Learning objects on the other hand can be turned around as a one-to-one assessment down the road because they were assembled by each learner who proscribed their own learning path. Learning objects that state “What I need to learn” can be flipped to ask “Did you learn what you needed?” Turn a learning object around, add a question mark, and you have a Level 4 assessment. If the learner six months later has really learned a new skill or behavior, you can easily find out by assessing them on what they decided to learn. If the learner is struggling with what they tried to learn, you can determine that as well and provide whatever support is required.
Learning technology changes the equation. In the same way that elearning removed the barriers of time, space and the four walls of the traditional classroom, LCMS can provide an assessment of a learning event ‘down the road’, and really start get to that formerly unobtainable Level 4. It can measure the degree to which the learning has been adopted and is being adapted.
No comments:
Post a Comment