Make It Meaningful
The irony of data being almost immeasurable when it comes to proving the impact of your L&D is not something that’s lost on us here at Kineo. The secret to getting the most out of your data is being able to prove and measure what value has been added.
How Do You Measure Impact?
If you’ve been in L&D long enough, you’ve probably been confronted with questions about the impact of learning solutions. Your internal voice may say, “yes, of course we make a difference,” but you probably also have a sense of unease thinking about your ability to prove it. You’re not alone.
Brandon Hall Group’s 2020 Learning Measurement Study found that fewer than 16% of organisations are very effectively able to identify and track a series of metrics, including participation, satisfaction, knowledge transfer, behavior change, and business impact for any of their learning. In our experience, one of the main reasons for this shortfall is that most of the energy around measurement is spent debating the best way to measure the impact of a specific course or program. While doing so makes sense in some cases, it also takes a significant commitment from the business. On the other hand, we see too little energy being put into a broader learning analytics approach.
A Model For Evaluation
Kirkpatrick’s model of evaluation, first pioneered by Dr Don Kirkpatrick in 1993, is still one of the most popular training models used in evaluating training programs today. It consists of four training evaluation levels, with each level building on the previous one. These include reaction, learning, behavior, and results.
The levels outlined in the Kirkpatrick model of evaluation set clear defined objectives to achieve productive metrics. This is one of the golden rules when it comes to getting the most effective results and understanding your business outcomes. Know how the data you’re collecting will help you—will it inform design decisions, help you learn something about the training or audience, or help prove impact?
“One in two L&D leaders is being asked to prove that they’re adding more value. That’s a pretty strong stat, with only 6% saying they were under less pressure to prove more value” – David Wilson, Fosway
7 Steps To Smarter Data
Here are 7 approaches to data gathering for smarter measurement.
1. Business Impact
Business impact is about trying to establish a direct correlation between training and a business metric. A common example is making a connection between a new sales training program and the success of your salespeople. It can be one of the trickiest data points to measure, but it can be done by thinking creatively about how learning maps to business outcomes, or by conducting a control test, for example.
2. Behaviour Change
This approach starts with building a behavioral model as part of your needs assessment. The behavioral model identifies both positive behaviors—those we want our audience to continue or increase—and negative behaviors—those we want to decrease or eliminate. The measurement strategy revolves around different methods to quantify the frequency of these behaviors before and after the training.
3. Application
Assessing application is about creating different experiences in which the skills and knowledge covered in the training need to be used. The most common approach is a scenario-based assessment that provides the opportunity to evaluate learners using hypothetical or expected work situations. This approach is especially useful for assessing learners on things where mistakes and/or failure on the job has great consequences.
4. Knowledge Retention
Knowledge assessments are ubiquitous in corporate learning events or courses to measure a learner’s ability to recall facts and terminology. Most often, these assessments appear at the end of a course or module as knowledge checks or quizzes, and at the end of the course as final assessments.
5. Confidence
Confidence ratings are metacognitive (that means thinking about your own thinking!), requiring the learners to report their own awareness of their thinking. It is the learner’s self-assessment of their own confidence about a choice or decision, usually given retrospectively after the choice has been made. Learners answer a question and then rate their confidence in their answer.
6. Engagement
Unlike the categories above, engagement data isn’t about the content being taught. Instead, it’s about measuring activity. The most common data in this category includes registrations or starts to a course, completions, and time spent.
7. Reaction
Reaction data is typically collected via “smile sheets.” This data reflects the learner’s opinion about or reaction to learning. Questions can range from generic satisfaction (“did you like it”) to gauging helpfulness to the learner’s job, to likeliness to improve performance.
Originally published at kineo.com.
