CMP Series

The Trouble With Evals

The feedback form is such a fixture in the meetings industry that you would think it would have been perfected by now. But as a new Convene survey suggests, it's a tool that is ripe for redesign.

In a recent Convene survey, nearly 200 meeting organizers weighed in on the difficulties that feedback forms pose, from crafting the kinds of questions and response choices that will elicit actionable insights, to finding the format (paper or digital), timing (immediately after each session or once the conference concludes), and weighing the use of incentives to increase participation.

Among the conclusions that can be drawn from their responses is this one: It may be time to rethink how they are designed and delivered. Only half of the respondents said that they were satisfied with their current survey practices, but only 9 percent were actively looking for ways to improve them.

Will Thalheimer, Ph.D.
Will Thalheimer, Ph.D.

Convene asked Will Thalheimer, Ph.D., founder of Work-Learning Research, and author of Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, to help us construct survey questions and responses, and then to walk us through the results. We’ve shared a few top takeaways below, but you can find the full survey results in our digital edition.

1. Use specific-answer options.

Our respondents overwhelmingly — 87 percent — use the familiar Likert Scale, which asks users to rank responses to questions along a continuum that ranges from “strongly agree” to “strongly disagree,” or other numeric scales. That can lead to feedback that is less than clear, Thalheimer says. Better, he said, to offer a range of specific answers to questions for users to choose from. (For examples, see “Surveys That Make People Think.”)

2. Make it real-time.

More meeting organizers are collecting feedback after sessions (64 percent), rather than at the end of individual sessions, either on paper forms (37 percent) or digitally (32 percent). “This move away from immediate feedback means that attendees are probably unable to give specific feedback,” he said. “They’re also unlikely to be giving as accurate feedback. I think this points to the fact that mostly people are doing full-conference feedback.”

Approximately 80 percent of respondents collect feedback on individual sessions, compared to 95 percent who collect feedback on the entire conference. Yet evaluations of individual sessions are most important from a learning standpoint, according to Thalheimer.

3. Align your feedback questions with your meeting goals.

Eighty-two percent of respondents said they review the data from event surveys and look for meaningful findings — which Thalheimer found troubling. “I think this kind of non-specific review enables bias, doesn’t rely on standards, and is non-directive about what to do,” he said. Fewer than half (42 percent) use the data to determine if the organization has met its predetermined meeting goals.

Full survey results.

Michelle Russell

Michelle Russell is editor in chief of Convene.

  • Ed Bernacki

    As a speaker at many conferences I found this feedback rather useless. I have little interest if people ‘like’ me. I would rather discover if they learned something that prompts them to act in some way. This is a difficult issue to measure.
    As a university lecturer I discovered something interesting. For two years, an admin person came into the room on the final day and students did a review of the course. 100% participation. In year three we used “innovation” to do online evaluations. I was deeply concerned when just 60% of students responded. I called admin to ask why it was so low. I was told to be very happy as this is twice the average. Are we sure digital is the answer? It is easy but is it effective?
    As a speaker, when I can use my evaluation I flip the issue. This gets very interesting results (I speak on innovation and creativity). I ask…..
    1. How effectively did you participate in this session?
    2. Did you listen effectively to capture enough ideas to feel you got value from this presentation?
    3. Do you have any intention to act on your ideas?
    4. If not, why?
    5. If yes, how will you start.

    This has produced some fascinating results. In particular question 4. Some people have said..”there is no point. My boss kills every idea I come up with….”

  • Muzik & Muzik, LLC

    #3 here is particularly important — the thing for all of us as organizers and education providers to remember is that if the quality of learning is not high enough, attendance will drop off and leave us with nothing but crickets. Technology is enabling some awesome ways to assess and report on the quality of attendee learning, but nothing will ever replace the power of well-written questions and mindful engagement of our learners before, during, and after our meetings.