How do you measure the success of your training programs? Yeah, that is a pretty complicated question for many learning professionals. However, we all know that it is critical to our programs that we show how and why they are working. The big gaps seem to be the mystery around how exactly we get that information.
On the first Train Like You Listen podcast of 2021, Brian talks to David Vance and Peggy Parskey from the Center for Talent Reporting and authors of Measurement Demystified, Creating Your L&D Measurement, Analytics and Reporting Strategy to find out more about how to measure training programs.
Transcript of the Conversation with David Vance and Peggy Parskey
Brian Washburn: Welcome, everyone, to Train Like You Listen, a weekly podcast about all things learning & development in bite-sized chunks. I’m Brian Washburn, the Co-Founder and CEO of Endurance Learning and the Train Like You Listen podcast is brought to you by Soapbox, the world’s first and only rapid-authoring tool for instructor-led training. It’s a little bit like an instant pot for training design, so throw a few ingredients in there, the number of people who are attending, the amount of time that you have, your learning objectives – and out pops a lesson plan. If you want to check it out it’s soapboxify.com.
I’m very excited to be joined by David Vance and Peggy Parskey, who both are from the Center for Talent Reporting. They have a new book out called Measurement Demystified, Creating Your L&D Measurement, Analytics and Reporting Strategy. David and Peggy, thank you so much for joining me today.
Peggy Parskey: Happy to.
David Vance: Happy to be here.
6-Words on Measurement of Learning
Brian Washburn: And so as we typically do with all of our guests, we ask our guests to introduce themselves using just 6 words in a sentence. So I’d love to hear a little bit more about your story.
Today we’re going to be talking about this idea of measurement demystified. If I had to introduce myself only using 6 words and this topic of measurement, you know, oftentimes I think of feedback and evaluation forms and so my 6-word biography would be “I don’t like reading negative feedback”. How about you, David, how would you introduce yourself in 6 words?
David Vance: Six words, I would say “I’m passionate about running learning like a business”.
Brian Washburn: And, Peggy, how about you?
Peggy Parskey: I would say “I’m building sustainable measurement security for organizations”.
Brian Washburn: I like it. Now this whole idea of measurement, it’s a really big topic and it’s a tough nut to crack. Perhaps the most fundamental question when it comes to measuring learning initiatives is why should L&D professionals measure in the first place?
Why Should Learning & Development Professionals Measure Learning Initiatives?
David Vance: Brian, I’ll start. I think there are 4 basic reasons we talk about in our book. Reasons like “to inform”, like to answer someone’s question “how many courses did you have? how many participants?” “Monitor” which to us means you have an opinion now about how you want that measure’s value to be. Like, you want to be sure that level 1 participants’ satisfaction scores remain above 75%. That’s your threshold. Reason #3 would be “program evaluation” which is the traditional reason for measuring. You want to see how successful the program was. But we add on another reason to measure, is to “manage”. And this is the one that we think is so important. So managing means that you set specific measurable goals at the start and you want to manage throughout the program or throughout the course of the year to make sure you deliver whatever you promised to begin with.
Brian Washburn: Ok. And so, Peggy, do you have anything to add in terms of “why”? I think it’s really important to set that, and we’ll talk about how in a second, but “why”? Why do we need to measure?
Peggy Parskey: Yeah, I mean I would add that we’re investing a lot in measurement. And it’s not just the direct costs of what we’re measuring but it’s also the opportunity costs of people attending measurement. And it’s really incumbent on us to be able to identify if it’s working and if there’s any return on that investment. And then, also, I think from an organizational perspective, it creates accountability and we should not get a pass on that because we’re in L&D and we’re in that, sort of, soft business. And so we really need to be taking that on as one of our key functions within L&D.
Brian Washburn: It goes down to value, right? So what value are we providing and how are we providing that value and how are we defining value? It is probably the holy grail of Learning & Development – measuring and showing results. It’s a really tough nut to crack. It’s really hard. Why? Why do you think it’s so hard to measure learning?
Why Is It Difficult to Measure Learning?
Peggy Parskey: I think part of it is that for many L&D practitioners it’s not something they were brought up learning about. I think they didn’t necessarily have a background in math or statistics and so it’s a bit foreign. And sometimes, just by virtue of the language, it feels a bit arcane. And so I think it’s a bit off-putting for individuals. I think the other thing is that it also does imply, sort of back to the earlier point — it implies a level of accountability that I have found, unfortunately, that in some organizations people are a little bit loathe to take on. And so they’ve created, sort of, a mental barrier that it’s something bad, it’s something hard, it is something is that is going to make me look bad. So I think as a result there’s oftentimes a view that “this is tough and I’m not going to come out looking good as a result of it”. So I think it’s something that we’ve built up in our minds, as opposed to what is really the case which is it is as much of a methodology and a step-by-step as anything else.
Brian Washburn: Yeah, so fear is one of the things that I’m hearing. And also just lack of know-how. Almost like it’s one of those things where people say “oh, I know how to talk, so I can train people”. It’s almost one of those things where people say “oh, I know how to put together an evaluation form so I can measure”, right? But there’s more to it than that. David, do you have any other thoughts? Is it actually hard or are people just going about it the wrong way?
Is It Hard to Measure Learning or Are People Measuring Learning The Wrong Way?
David Vance: Well, I think most people would answer that they don’t have the resource, that they don’t have the staff. Peggy and I think that is a little bit of an excuse for many to be using. And the reasons Peggy gave get more to it.
No, I think if you provide people the framework, with better step-by-step guidance, that it is very doable. There are a few aspects, like isolating the impact of training from other factors which may have also led to results. That can be a little trickier. But other than that it’s not inherently difficult. So it’s just a matter of setting a plan to do it, and getting the support you need to do it. And unfortunately we don’t have a lot of leaders role-modeling the use or importance of measures. Well, that holds us back too.
Brian Washburn: And sometimes it can be a question of what our mission, or are we measuring the right things. When you think of – or when I think of – measuring learning experiences, my mind always goes to Kirkpatrick and the famous “four levels of evaluation”. Is there more to measuring learning initiatives than the Kirkpatrick model or is that what people should be basing their foundation on?
There Is More to Measuring Learning Initiatives Than Kirkpatrick and Phillips
David Vance: Well, we’d say there is! And we’d say to expand your thinking along those lines. Kirkpatrick is designed to help you measure a particular program. And that’s great. And Phillips does the same thing and carries it a step further with ROI. But those are all program-focused. So we incorporate that into our TDRP approach in the book we wrote. But actually there are many other measures out there besides program measures. For instance, measures about your department, how many courses, what’s the utilization rate of instructors and rooms, costs for your entire department. So the Kirkpatrick approach focuses just on programs, but there are many other measures and many other reasons to measure. Remember in those four broad measures I talked about a minute ago, only one of them related to program evaluation. That’s where Kirkpatrick and Phillips comes in. The other three they don’t address! But we need to measure the others as well.
Peggy Parskey: I would add, one of the things I do with clients is that whenever we’re looking at a measurement plan, we often-times – even when it’s program-related – we use a method called logic-modeling. And in logic-modeling, you’re looking at inputs, activities, outputs, short-term impacts, and long-term outcomes. And so the whole range of measures of the things that Dave was talking about the need to play in. So it’s not just the effectiveness measures and some of the outcome measures, which are Kirkpatrick and Phillips, but also these efficiency measures that Dave was talking about. And so even for program evaluation, those things need to be part of it and it’s really critical people get that holistic view of what they’re doing, and not just focus in on those effectiveness measures for these programs.
David Vance: And to Peggy’s point, we have about 170 measures in our library for just L&D. Only 20 of those are the effectiveness measures that people generally think of in Kirkpatrick and Phillips. The other 150 are efficiency measures, so there’s a lot more to measurement than Kirkpatrick and Phillips.
Brian Washburn: Yeah, which I think is a really great and really important point because I think that anybody who has any sort of exposure to the world of Learning & Development and has been trained in L&D immediately defaults to Kirkpatrick, those four levels. Or maybe 5, if you throw in ROI. And so, there’s a lot, and I know that your book also emphasizes the idea of starting small and then growing your measurement. So if somebody’s listening right now and thinking “I want to start, but I’m not sure how…and suddenly it goes beyond Kirkpatrick, so now what?” Where can somebody start? How do you start small?
How Do I Start Small With Learning Measurement?
David Vance: Well I’d say 2 places. One, if you do want to measure the success of a program, then start with one good program, hopefully strategically aligned to an important goal in your company – like increasing sales, reducing costs, something like that, where the CEO cares about it. That would be a great place to start. And put a little time into it.
But what we would emphasize is the importance of meeting with the goal-owner upfront, not at the end of your program or project. But set mutual goals for success. Upfront, set roles and responsibilities. Set the values of all of the contributing measures to that, upfront. And then manage that throughout the life of the program so it is successful and delivers the values. Then you’re all on the same page throughout the entire program. And measurement isn’t an add-on at the end. No, you had to do it to manage it from day one. So it’s not seen as a separate thing out there.
And if you’d rather not start with a program, then talk with your CLO and find out what some of the key department measures are going to be for the coming year. “I want to reach ‘so many’ participants. I want to improve our utilization rate, reduce our cycle time, reduce our cost.” Whatever is important to the CLO, you could start with those. But, again, follow the same methodology to get agreement upfront, on what the goal is in specific form, what it’s going to take to achieve it, and then manage throughout the year, using a report which shows you that.
Peggy Parskey: I think I would add to it — I agree 100% with everything Dave just said — is that I’ve seen organizations who say “oh, I can do more. I can do it all.” And what I find with those organizations is they get into it, they realize there’s more to it than they thought, and then they abandon it because it was “too hard”. So I do think it’s really important for organizations that are just dipping their toe into this, or just realizing they need to do it differently, not to take on too much. And to really get some successes in very specific areas, such as the ones that Dave just mentioned.
Brian Washburn: I really appreciate you all taking some time to talk about this.
Get To Know David Vance and Peggy Parskey
Brian Washburn: Before we leave, we do have a few of the speed-round questions. Are you ready for being on the hotseat of the speed-round?
David Vance: Ready!
Brian Washburn: (CHUCKLING) Excellent. So, Peggy, what is your go-to pre-presentation food?
Peggy Parskey: This is going to sound super boring but it’s usually tea, because if I eat too much before I present, I always regret it.
Brian Washburn: I’m with you. A lot of times people will say coffee, but I like it, you threw a little curveball in there and go with tea. How about you, Dave?
David Vance: Mountain Dew.
Brian Washburn: Mountain Dew (LAUGHING) I love that answer. What’s a piece of training tech that you can’t live without?
David Vance: Well in our business it would be something like a learning management system or it’s equivalent in a company, where you’re going to find a lot of the data.
Brian Washburn: Mm-hmm. How about you, Peggy?
Peggy Parskey: Well I took it at a more personal level, so I do a lot of facilitating with groups and in facilitating I couldn’t do without my mind-mapping software. And I find it absolutely invaluable for engaging everybody and really getting some really good output from group interaction.
Brian Washburn: Is there a specific mind-mapping tool that you use?
Peggy Parskey: Yeah, I use Mind Jet and I just upgraded to it’s latest version and it’s fabulous.
Brian Washburn: Nice, nice. I know that listeners are always looking for tools like that so that’s really helpful. When you think of a book or a podcast that learning folks should be paying attention to what comes to mind?
Peggy Parskey: I’m going to go to a blast from the past, which is a book that I have been recommending to people for a long time. It’s called Understanding Variation by Don Wheeler. And I think it’s really important for people to understand this concept of variation within organizations and it should be on everybody’s bookshelf.
Brian Washburn: Nice. How about you, Dave?
David Vance: Well, Brian, I would say our book Measurement Demystified. That’s what you should read right now. And if that’s too self-serving, I just saw that Tamar Elkeles came out with a book 2 days ago called Forward-Focused Learning and it looks really good. So if you’ve already finished reading our book, then take a look at hers.
Brian Washburn: (CHUCKLING) I like it. And that was going to be my final question. Do you have any shameless plugs? You know, Measurement Demystified, Creating Your L&D Measurement, Analytics and Reporting Strategy, is out now. You can find it at TD.org or on Amazon. Peggy, do you have any other shameless plugs that might be helpful for people to hear?
Peggy Parskey: Yeah, I mean I think it’s really a book that should be accessible to everybody in the L&D field and so it isn’t geared towards just beginners or to more senior people so it’s got a really wide range. So we’d encourage everybody in the profession to read it and I think they’ll walk away with new “ah-ha’s” regardless of how long they’ve been doing this.
Brian Washburn: And I think that’s so important. It’s a serious topic that’s accessible for people. So I think that’s a really, really helpful way to end this conversation.
So, Dave, Peggy, thank you so much for joining me and thank you all for listening to this episode of Train Like You Listen. It’s a weekly podcast of all things L&D in bite-sized chunks. If you’d like to subscribe you can find us on Spotify, on Apple, on iHeartRadio, or wherever you get your podcasts. If you like what you hear, go ahead and give us a rating, and that’s how other people will hear as well. So, until next time, happy training everyone.
This week’s podcast is sponsored by Soapbox. Sign up today for a free demo at soapboxify.com.
