ROI
Return on Investment or Return on Illusion?
By Kevin J. Fleming, Ph.D.

A CEO calls in his chief engineer and asks him, “Can you tell me what two plus two is?” The engineer promptly replies, “It is exactly four.” Then the CEO has the corporate legal counsel come in and the CEO asks him: “Can you tell me what two plus two is?” The lawyer replies, “Well, it is somewhere between three and five.” Then the CEO calls in the corporate accountant and asks him, “Can you tell me what two plus two is?”

The accountant goes over to the window, draws down the shade and says, “What do you want it to be?”

When it comes to assessing ROI in the complex realm of training and development initiatives, the complexity of human nature rears its ugly head. And just when you thought the state of affairs was challenging enough, our own past comes back to haunt us–those days when we all fell asleep in statistics classes thinking, “This stuff doesn’t matter.”

Well, I am here to tell you that it does matter. You can’t have serious numbers without a serious science of numbers as well.

When it comes to evaluating my interventions, I offer serious science as a way of validating and measuring my work. For instance, I helped a client see that their pre-/post-methodology of assessing change after a communications workshop was flawed statistically and therefore in the bottom line analysis financially. How? Well, imagine if you were a participant at “time one” having to assess your current level of empathy with your current workers on a scale from one to 10. And say you felt you were an average “seven”—not great, but not horrible either; you then go through your training and development work withyour consultant and are ready to take your post-test at “time two.”

At this point, months later, you rate yourself a seven—not because you didn’t think anything changed, but because you realized you were really a “two” at “time one.” This phenomenon is called a “response shift bias” and is a silent killer to ROI calculations using the most common change method out there—pre- and post-assessments. The fix? Measure “pre” and “post” together at “time two.”

And since the best consulting comes from asking the right questions, not giving the right answers, here are some necessary questions to ask in your company when measuring the effectiveness of your consultants: What is the organization’s true position on training—a cost or an investment? Do the decision makers really need to have training evaluated by traditional ROI models? Understand the “break even” point. This is not a particularly sophisticated analytical technique but the lesson is clear. When is training not worth it? And most important, ask what matters most: the calculation or getting clear on who is accountable? Many times, building accountability and taking it seriously amongst your leaders fixes things more then any razzle-dazzle.

Dr. Fleming is a regular columnist on Transformation Insights for the high end executive magazine, Executive Decision.