I am still scratching my head after reading two articles on training measurement. The first was by James and Wendy Kirkpatrick in the November 2011 issue of T&D, describing a method for demonstrating the value of training by measuring “ROE,” or “return on expectations.” Building on Donald Kirkpatrick’s four-level framework, James (Donald’s son) and Wendy advocate for beginning with Level 4 results in mind and then identifying how to achieve those results by following this process:
- Focus on the organizational mission (Kirkpatrick Level 4: Results)
- Identify leading indicators
- Determine required drivers (Kirkpatrick Level 3: Behavior)
- Design learning (Kirkpatrick Level 2: Learning)
- Monitor and adjust
According to James and Wendy Kirkpatrick, clarifying this sequence of actions with stakeholders—and then taking the actions in implementing training and systems that support behavior change—“cannot fail,” because it “is built on a platform of business partnership and agreement from start to finish.” And, while the influence of training versus that of other factors (support systems, removing barriers) cannot be isolated, stakeholders are likely to rate the overall initiative as successful because it meets their expectations.
So far, so good.
Then I picked up the November/December 2011 issue of Training and read an editorial by Jack and Patti Phillips called “The Myths of Return on Expectation.” The Phillipses (the leaders in developing return on investment models) argue that introducing a term like ROE is confusing and unnecessary. They point out that it is unclear whether ROE is a number (say ROE=85.2), a concept (stakeholder satisfaction), or an objective (any of the four Kirkpatrick levels). ROE has the additional disadvantage, they say, of being confused with actual financial metrics (think return on equity). The Phillipses recommend sticking to the original (Donald) Kirkpatrick framework and quantifying results at the four levels, versus introducing a new term.
I think the Phillipses make a good point about introducing unnecessary terminology. But I also think James and Wendy Kirkpatrick’ systems approach is a powerful way to engage stakeholders in achieving business results.
The key points I take from all these thinkers:
- Begin by identifying your stakeholders and their expectations
- Make sure you have a clear “line of sight” between the training initiative and the company’s most important results
- Identify and address factors beyond training that influence performance (such as management support)
- Collect evidence (not necessarily “proof”) of impact at each step
- Use data to engage stakeholders in a dialogue about how training drives results
In addition, while a lot of training value is “backward looking”—that is, demonstrating the value of prior investments—I think learning professionals can add more value by being “forward-looking”—helping their stakeholders to make decisions that increase value. In the words of one executive, “Don’t tell me how much value I got from my last million dollar investment. Help me decide how to invest the next million!”
For more on measurement, check out our piece on the six critical measurement mistakes, and how to avoid them.