CMP Series

The Trouble With Evals

The feedback form is such a fixture in the meetings industry that you would think it would have been perfected by now. But as a new Convene survey suggests, it's a tool that is ripe for redesign.

In a recent Convene survey, nearly 200 meeting organizers weighed in on the difficulties that feedback forms pose, from crafting the kinds of questions and response choices that will elicit actionable insights, to finding the format (paper or digital), timing (immediately after each session or once the conference concludes), and weighing the use of incentives to increase participation.

Among the conclusions that can be drawn from their responses is this one: It may be time to rethink how they are designed and delivered. Only half of the respondents said that they were satisfied with their current survey practices, but only 9 percent were actively looking for ways to improve them.

Will Thalheimer, Ph.D.
Will Thalheimer, Ph.D.

Convene asked Will Thalheimer, Ph.D., founder of Work-Learning Research, and author of Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, to help us construct survey questions and responses, and then to walk us through the results. We’ve shared a few top takeaways below, but you can find the full survey results in our digital edition.

1. Use specific-answer options.

Our respondents overwhelmingly — 87 percent — use the familiar Likert Scale, which asks users to rank responses to questions along a continuum that ranges from “strongly agree” to “strongly disagree,” or other numeric scales. That can lead to feedback that is less than clear, Thalheimer says. Better, he said, to offer a range of specific answers to questions for users to choose from. (For examples, see “Surveys That Make People Think.”)

2. Make it real-time.

More meeting organizers are collecting feedback after sessions (64 percent), rather than at the end of individual sessions, either on paper forms (37 percent) or digitally (32 percent). “This move away from immediate feedback means that attendees are probably unable to give specific feedback,” he said. “They’re also unlikely to be giving as accurate feedback. I think this points to the fact that mostly people are doing full-conference feedback.”

Approximately 80 percent of respondents collect feedback on individual sessions, compared to 95 percent who collect feedback on the entire conference. Yet evaluations of individual sessions are most important from a learning standpoint, according to Thalheimer.

3. Align your feedback questions with your meeting goals.

Eighty-two percent of respondents said they review the data from event surveys and look for meaningful findings — which Thalheimer found troubling. “I think this kind of non-specific review enables bias, doesn’t rely on standards, and is non-directive about what to do,” he said. Fewer than half (42 percent) use the data to determine if the organization has met its predetermined meeting goals.

Full survey results.

Michelle Russell

Michelle Russell is editor in chief of Convene.