what are the biggest mistakes we make when measuring learning

Do you actually know if your training worked, or are you just hoping it did?

In this episode of Learning for Good, I break down the five biggest mistakes nonprofit learning and development professionals make when measuring training impact.

Most of us know we should be measuring our training, and we want to measure it, but somewhere between good intentions and actual practice, we get stuck. We settle for data that is easy to collect instead of data that tells us the truth. Good measurement isn't about checking a box, it's about using what you learn to make real decisions and show that the work actually matters.

▶️ Key Points:

00:00 Why Measuring Training Matters

03:00 The Kirkpatrick Model of Training Evaluation

04:23 The 5 Biggest Mistakes L&D Professionals Make When Measuring Learning

17:17 Turning Your Learning Dashboard into a Strategic Tool

Why Training Measurement Matters

Picture your car’s dashboard. Fuel gauge, speedometer, temperature, check engine light. You have a handful of signals telling you whether you’re going to make it where you’re headed. Now picture driving your organization’s entire training strategy while only glancing at two of those gauges. That’s what most nonprofit learning and development measurement looks like today.

The Association for Talent Development surveyed 222 L&D professionals specifically about how organizations measure learning, and less than half rated their own organization as good or excellent at it.

Most of us use the Kirkpatrick model. 71% of organizations report using the model. We know Kirkpatrick’s four levels of evaluation — reaction, learning, behavior, results. And 92% of organizations report collecting their data through surveys. So we have a framework. We have a method. And we’re still stuck.

But we can’t stay stuck.

Measurement is not just how you prove training worked; it’s how you prove your work matters. So on episode 201 of the Learning for Good podcast, I shared five specific, fixable mistakes standing between the dashboard you have and the one you actually need.

What Should We Measure Related to Training?

Mistake 1: You’re Measuring What’s Easy Instead of What’s Aligned to the Business.

L&D leaders often control the Learning Management System (LMS), so some data is just easier to capture. We can pull attendance data and completion rates quickly and easily.

We are responsible for the training itself, so satisfaction data also feels safe to promise. And, it’s pretty quick and easily captured with surveys.

But what happens 6 months after the training ends? That feels much harder to control and much harder to measure.

But we need to measure more than just completion and satisfaction.

Another reason we don’t always align to the business is we simply don’t have access to the organizational data we need. Staff retention rates, program metrics, donor data—those are all things that often live with other teams. If we don’t have the trust and credibility in place, asking finance or operations for their numbers can be intimidating.

But again, we need to do it.

We need to align ourselves with the business.

If a stakeholder asks for feedback training for people managers, and you design it well, run it, and even confirm behavior changed, you haven’t proven your work mattered. At least not yet. What if that stakeholder requested feedback training because they wanted to see higher staff retention rates? If you’ve measured completion rates, satisfaction, and behavior changed, you’ve proven the training worked, not that it mattered. It’s not until you look at turnover and retention rates that you can align with the business outcomes and prove the value of learning.

When Do We Decide What to Measure in Learning?

Mistake 2: You Decide What to Measure after the Training Is Built.

Many L&D pros wait until the end of the project to decide what and how to measure the training. It’s an easy trap to fall into; the training feels like the “real work.”

However, we need to know what success looks like from the beginning of the project. It can’t just be “managers need feedback training.” We have to get to the root of the request and define success.

Let’s say you’re working on that feedback training. What does success look like? It’s not just that managers learn a model. It might be the frequency with which they give feedback. Or how effective they are. And you can define what you mean by effective and identify the specific behaviors managers will need.

If you know this before you develop the training, you have a better chance of achieving success. And you’ll know how and what to measure. It won’t just be “they attended; they liked it; they learned the model.” It will be “behavior changed; here’s how.”

Are Training Completion Rates and Satisfaction Scores Enough?

Mistake 3: You Stop at Training Completion Rates and Satisfaction Scores.

Completion and satisfaction data are only part of the picture. They can be useful. They are typically immediate. But they aren’t the whole picture.

It’s tempting to close the project with just the immediate data, but that data isn’t proof that the training worked or that it made a difference in the organization.

How Do We Increase Training Survey Participation?

Mistake 4: You Never Get Buy-In for Measurement.

If you have ever run a survey, you know participation can be a challenge. And if you’re measuring behavior change and impact, you may be asking for more effort than just a survey completion.

People need to know why you’re asking them to participate, how you’ll use the data, what’s in it for them. We have to sell them on it.

It’s important because low participation isn’t just less data. It’s less reliable data.

And we are trying to prove our training efforts were valuable.

How Do You Tell the Story of Training Data?

Mistake 5: You Collect the Data and Do Nothing with It.

It’s tempting to create the report or dashboard and consider it done, but the dashboard isn’t the finish line. We have to use the data.

Training data can be used to tell a story of impact, inform a decision, or improve the next iteration.

Just saying “we had 93% satisfaction” doesn’t tell a story. But sharing “managers who completed the training are giving feedback twice as often, and we’re seeing early movement on retention” does.

To learn more about these 5 mistakes and how to avoid them, tune into episode 201 of the Learning for Good podcast.

Additional Resources Just for You

Other Helpful Podcast Episodes:

Subscribe to the Nonprofit L&D Insiders.

Join the Nonprofit L&D Collective.

Support the Learning for Good Podcast

Become a sponsor.

Ask a question or suggest a topic.

We love hearing from listeners!