Measuring training understanding Kirkpatrick Level 2 Learning

Training has long worried about justifying its existence. How do we measure training success? What are the critical outcomes of training? How do we know if the training is any good and if it is making any impact? In the second in this series, let’s turn our attention to Kirkpatrick Level 2 Learning analysis on the actual understanding and learning by participants.

Most learning and development professionals will be familiar with The Kirkpatrick Model of evaluating the effectiveness of training. It’s a great model and serves as inspiration for the built-in reporting in the Tribal Habits platform. The model suggests four levels of training measurement.

1. Reaction: The degree to which participants find the training favourable, engaging and relevant to their jobs.
2. Learning: The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training.
3. Behaviour: The degree to which participants apply what they learned during training when they are back on the job.
4. Results: The degree to which targeted outcomes occur as a result of the training and the support and accountability package.

Over a series of four articles, let’s examine The Kirkpatrick Model and how Tribal Habits can help any organisation with reporting on all four levels – automatically! We’ll see how organisations can select the appropriate level of reporting and utilise the information at each level to improve its training topics. And certainly, the ability to gather high-quality data which can demonstrate the learning and understanding outcomes from training is critical to supporting training budgets and initiatives.

What is Kirkpatrick Level 2 Learning trying to measure?

I think from the easiest way to think about Kirkpatrick Level 2 is simply ‘Has the participant actually acquired new knowledge?’. Ideally, they have learnt knowledge directly related to the course content and its objectives. However, if the participant encountered additional learning on other topics, or the content triggered insights into other areas, then we should acknowledge that too. Training should stimulate all learning.

Associated with the concept of learning is understanding. A more robust analysis at Level 2 would be the degree to which the participant actually understood, not just acquired, the knowledge in the training activity. Understanding is pivotal to the participant’s ability to improvise, adapt, combine or extend the knowledge in on-the-job application. It’s one thing to remember the training content, but another thing to fundamentally understand it.

Kirkpatrick Level 2 is a key measure of success for the majority of corporate training. It’s not the only measure, but it’s critical to achieve success at Level 2 in order to have success at other levels.

How is Kirkpatrick Level 2 Learning typically measured?

If the measure is learning and understanding, then some sort of demonstration of that learning is required by the participant. That often means assessment – a demonstration of technique, a submission of an application, or passing an exam. Let’s examine a few different ways to obtain a demonstration of learning in your training programs, along with how Tribal Habits can automate or assist in Level 2 measurement.

It is also worth noting, that a Level 2 measure is not always required. For some training, a simple acknowledgement of completion may be enough. For other topics, demonstration of behavioural change is more critical (Level 3).

Measuring Kirkpatrick Level 2 with quizzes

Let’s start with the easiest and most popular measure – a quiz. Quizzes work well as both the training creator and training participant immediately understand the concept. So there should not be confusion about the application of a quiz – and don’t underestimate the benefit of that, compared to other more complex types of assessments.

Quizzes typically test for three levels of learning.

1. Recall. Can the participant remember critical information? Tests of recall should focus on information which the participant would likely need ‘in the moment’ of their job or would be expected to know by heart. If someone doesn’t really need instant recall of that knowledge, then a recall quiz is a poor choice. That being said, recall quizzes can also be used simply to test if a participant was paying attention, but that’s not really testing for learning in the spirit of Level 2 analysis. Examples…
   1. What is the correct order to start this machine?
   2. What is the only exception for use of this product?
2. Understanding. Did the participant understand any information which may be complex or misinterpreted? These quizzes are actually used to help calibrate the learning – to check if the participant correctly understood the content. In this respect, these quizzes sometimes actually help with the transfer of knowledge, rather than measure the success of that transfer. Several quick quizzes after new content can be very effective as a learning tool, as well as later being used to verify understanding. Examples…
   1. Select all of the incorrect uses of this service from this list.
   2. Select the true statement from the list below.
3. Application. Can the participant correct apply the knowledge to given scenarios? These quizzes are now testing the ability to use the knowledge. These are typically ‘What if?’ quizzes, with participants having to interpret scenarios and apply the right knowledge. These are excellent quizzes for Level 2 and would typically come at the end of a training event as a real measure of success.
   1. What would be the correct response to this situation?
   2. Why is the following response incorrect?

I have used the word ‘quiz’ quite broadly here. Quizzes can be anything from true/false, multiple choice, hot-spot, short answer, drag and drop, ranking or ordering, or long answer. We can dive into different quiz types in another topic – for now, let’s keep the focus on the objective of quizzes as discussed above.

Quizzes in Tribal Habits

There are lots of different ways to create your quizzes in Tribal Habits.

First, creators can use the Quiz element to create list-based quizzes anywhere in their topic – multiple choice, single choice, true/false, scenarios and so on. Creators can use a series of quizzes at the end of an idea to quickly check understanding and calibrate the learning. Quizzes take moments to create and includes several templates to get you started.

Second, creators can enable a dedicated assessment module in their topic, with a specified number of quizzes and a minimum passmark. Tribal Habits will then sweep your topic and select quizzes to create a different assessment for each explorer. This means you can use the same quiz within the topic and again in the assessment with no additional work. Alternatively, you can create new quizzes which are tagged only for the assessment, with the ability to create those quizzes anywhere in the topic where you are inspired by the content.

Measuring Kirkpatrick Level 2 Learning: Reflection

An alternative Level 2 measure is to ask participants to complete self-reflection exercises. The results from these exercises need to be shared with other participants and/or reviewed by a training facilitator.

Self-reflection is often a very revealing measure of learning. Participants who engaged with, and understood, the training usually find self-reflection very easy. They can readily summarise key lessons and how they will apply them. Participants who didn’t engage or understand the content, typically struggle with self-reflection. The lack of learning is revealed in shallow or generic self-reflection.

Self-reflection exercises can be completed at the end of a training session as an excellent way for participants to both embed new knowledge, but also reveal to the facilitator the success of learning. A robust self-reflection discussion or activity will clearly highlight Level 2 success.

Reflection can also occur post-training in a 1:1 debrief with a manager, coach or key stakeholder.

Reflection in Tribal Habits

Let’s provide an example of this idea within the Tribal Habits system. Creators can enable built-in ‘Reflection questions’ in any topic. Tribal Habits then automatically creates regular reflection interactions for explorers through-out your topic.

Explorers will be asked to answer a variety of different types of reflection questions, with their answers copied into their journal for review by their manager or other stakeholder. Answers from explorers can easily be reviewed by topic creators or admins, as well as exported for additional reporting as a detailed qualitative measure of Level 2 success.

From a learning point of view, explorers can also review the reflections of their colleagues. This provides another learning moment, as they compare and contrast reflections.

Measuring Kirkpatrick Level 2 Learning: Activities

A final and very sophisticated demonstration of learning would be an activity. This is essentially an assessment which requires a human review of a demonstrated application of the learning. For example, in learning how to use a piece of software, participants might be required to then use that software to complete an example task. Their work would need to be reviewed by a manager or facilitator to finalise the activity.

The use of case studies, templates, or examples is typical for this demonstration of learning. This means this level of success is the most time consuming to both create – as it requires stimulus for the explorer – as well as to review. But it will give the highest possible demonstration of learning. The more critical or complex the topic, the greater the need for this level of demonstration. You can’t get an open drivers license just with a written exam for example!

Activities in Tribal Habits

Activities like this can be difficult to coordinate and organise, especially for large groups or on-going training. But that is something Tribal Habits can help with!

Any topic in Tribal Habits can include one or more tracked activities. Creators can simply add a new activity with just a few pieces of information. Activities can include files (case studies, templates) and deadlines (which will enable reminders). Activities might be to complete a detailed assessment, use a piece of software, respond to a case study, draft a document, or conduct a role-play with a manager.

Activities include built-in quantitative and qualitative reporting. Notifications can be enabled to inform managers, facilitators and/or related stakeholders when each explorer starts an activity, is overdue in completing an activity, or completes the activity. In combination, these options allow for a series of automated activities, including data capture, providing the ultimate measure of level 2 success! Almost all of Tribal Habits Marketplace topics have 1-3 built-in activities to automate this detailed Level 2 measurement.

Good Kirkpatrick Level 2 Learning reporting allows for training improvement

Kirkpatrick Level 2 Learning measurement is an interesting one. For many topics, Level 3 Behavior will be more important. However, Level 2 Learning will be critical for topics focused on process, regulations, safety, technical knowledge or product knowledge as it is the truest measure of quality for those topics. For if the explorer cannot demonstrate retention of the knowledge, then what was the point of the training?

The key is to pick the right degree of Level 2 measurement and then standardise that measurement in a way where data can be readily obtained. And certainly, that’s where Tribal Habits can help you out. Our Australian LMS can enable a wide range of tracking around participant understanding and learning, while enhancing the transfer of knowledge for those participants. You’ll get more feedback, more valuable data, and more demonstrations of success than ever before.