Measuring training outcomes and Kirkpatrick Level 4 Results

November 12, 2018 in Industry Trends



Measuring training outcomes and Kirkpatrick Level 4 Results

Training has long worried about justifying its existence. How do we measure training success? What are the critical outcomes of training? How do we know if the training is any good and if it is making any impact? In the second in this series, let’s turn our attention to Kirkpatrick Level 2 Learning analysis on the actual understanding and learning by participants.

Most learning and development professionals will be familiar with The Kirkpatrick Model of evaluating the effectiveness of training. It’s a great model and serves as inspiration for the built-in reporting in the Tribal Habits platform. The model suggests four levels of training measurement.

  1. Reaction: The degree to which participants find the training favourable, engaging and relevant to their jobs.
  2. Learning: The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training.
  3. Behaviour: The degree to which participants apply what they learned during training when they are back on the job.
  4. Results: The degree to which targeted outcomes occur as a result of the training and the support and accountability package.

Over a series of four articles, let’s examine The Kirkpatrick Model and how Tribal Habits can help any organisation with reporting on all four levels – automatically! We’ll see how organisations can select the appropriate level of reporting and utilise the information at each level to improve its training topics. And certainly, for professional services training, the ability to gather high-quality data which can demonstrate the learning and understanding outcomes from training is critical to supporting training budgets and initiatives.

What is Kirkpatrick Level 4 Results trying to measure?

The ultimate test of most training is the measurement of its impact on business results. It is true that some training – perhaps on technical knowledge – may only require a simpler analysis, such as tests of understanding or learning, to be considered a success.

However, professional services firms often invest in training for the purpose of improving the firm’s results. Through better skills, staff help the firm achieve its strategic goals. This leads to the final form of training measurement – level 4 on the Kirkpatrick scale – measuring the impact on results.

The ultimate measurement requires the ultimate commitment

To be fair, tracking Kirkpatrick Level 4 results can be extremely difficult.

Isolating the impact of training on the overall performance of a professional services firm can be almost impossible. Results may come from changes in competitors, the economy or services which occur at the same time as training. Every topic also has a different impact on a firm, in terms of both the areas it improves and the level of impact or improvement.

Measuring the business impact of training also requires planning. Appropriate business metrics must be selected and measured before the training occurs. Without this benchmark, level 4 analysis cannot be completed. Put simply – you can’t decide later to measure business impact from training. You must decide this before the training begins.

The selected business metrics must then be measured over an acceptable period. Sometimes the business impact may not occur for 6-12 months. Training on business development topics are a good example – it can take months for business results to be impacted. And the longer the time period, the harder it becomes to isolate the impact of training on those metrics.

Strict measurement versus anecdotal measurement

Nevertheless, we can make efforts to measure the impact on business results, even if to get a ‘sense’ of the impact or to create proxies for measuring results. Depending on the selected metric, a trend of change may be enough to demonstrate impact.

Anecdotal measurement, or observational feedback, can also act as a business impact measurement. For example, if managers feel that their professionals are agreeing to client contracts with fewer rounds of legal reviews, then that might be enough to suggest the negotiation training has resulted in a positive business impact. Further attempts to quantify the degree of positive impact may be futile.

With all that in mind, let’s examine how Tribal Habits measures impacts on business results from training – a level of measurement which is often far beyond traditional learning management systems or learning experience platforms.

Measuring results by goal achievement

The first way Tribal Habits tackles the impact on business results is through the use of learning goals.

Each topic in Tribal Habits can enable the capture of participant learning goals. Once enabled, explorers are asked to define what they would like to discover as part of exploring the topic. On its own, that’s great data for topic creators and the firm to determine if this topic is addressing staff needs and/or to consider the development of additional topics.

However, we can go further. In each topic, creators can also enable a poll which asks participants if they found the discoveries they were seeking.

To be fair, these stated goals are not always the same as an outcome goal, but this process (used in the right topics) often sets a participant’s mind to what they want to do differently. Indeed, explorers often think in terms of outcomes when setting their discovery goals. So tracking the achievement of this goal provides not only a measure of the effectiveness of the topic, but one proxy for overall business results.

If participants know their KPIs and what the business wants them to achieve, they can usually define what they need a particular training event to provide for them. If that information – skills, process, tool – if provided, then its likely to have a positive impact on the underlying business goal.

Measuring results through common criteria

Another useful way to measure business impact from training is to ask participants to define what aspects of their role they feel the training assist them with.

This allows the business to see if its selection of training topics is aligning with its business goals. You can also examine your overall portfolio of training topics to see if there is balance in the training being provided. Once again, if you can measure improvements in your key business metrics, and there is feedback from participants that their training is related to those metrics, then you have a good link established between training and business outcomes.

Within Tribal Habits, topic creators can simply enable a poll to measure the impact of any topic on six pre-defined business goals for professional services firms. This provides consistent tracking across topics as well as making the process of tracking this data as simple as possible.

Measuring results through participant observation

In every Tribal Habits topic, participants create a journal. Many parts of their journal are automatically created as the platform captures key activities, contributions and milestones. Explorers can also add their own notes to their journal. The journal can be emailed not only to explorers, but also to managers or other specified email addresses (HR or IT staff for example). Notification to those stakeholders can be automated at various points in the topic too.

This notification function creates accountability with the participant when they know their manager will have access to the journal and its outcomes. When tied to on-the-job activities (see measurement ideas for behavioural change in the previous chapter), it can trigger offline intervention from their manager – on-the-job observation, 1:1 debriefs, personal coaching, team success sharing.

It also encourages managers, or other stakeholders, to conduct personal observation with participants to confirm certain business outcomes are being achieved. This anecdotal observation can actually be quite accurate. Managers are typically very aware of the various factors influencing business outcomes and can often determine the extent to which training played a role. By providing managers with insight into the decisions and actions of their staff during the training, they can better align the business results they are seeing with the training they now understood to have occurred.

Measuring the impact on results ultimately measures RoI

Kirkpatrick Level 4 results measurement provides a professional services firm with demonstrated proof that its investment in training is justified. It brings full circle the purpose of training. It connects online and offline training, knowledge with application, behaviours with outcomes.

It provides the truest measure of any training topic.

Even though it may be a proxy or estimated impact, if you can capture even the ‘sense’ of the impact of training on key business goals, then you will have more data than 99% of most other training. Within Tribal Habits, this level of reporting is available for every topic with simple checkboxes. It doesn’t get any easier than that to demonstrate the value of the training you are providing.