Research

Surveys That Make People Think

There are better alternatives to the overwhelmingly popular "strongly agree" to "strongly disagree" rankings of the Likert scale, one learning expert argues.

There is “a ton of room for improvement” in the typical conference feedback form, says learning expert Will Thalheimer, Ph.D., especially those using the popular Likert scale. Favored by 87 percent of Convene’s survey respondents, Likert-like scales provide a continuum of answer options, such as “strongly disagree” to “strongly agree,” or 1 to 5.

But Thalheimer’s research indicates that people seem to prefer questions that have more discernible answer choices as opposed to general Likert options. “I think when people see the answer choices are more concrete, they know that the questions are better,” he said. “You see a typical smile sheet, and people circle the same numbers all the way down. They’re not really thinking and then they’re not really giving you good data.”

Thalheimer offered this question as an example: “How effective was the session in helping you comprehend the concepts taught?” Respondents, he said, will have an easier time choosing between specifically worded options, such as:

  • I still have significant confusion and/or blind spots.
  • I gained a basic familiarity with the concepts taught.
  • I gained a solid understanding of the concepts taught.
  • I gained a deep and comprehensive understanding of the concepts taught.

When survey respondents are only given the option to choose along the Likert continuum, “there’s a little bit of fuzzy thinking going on,” Thalheimer said. Providing specific answer choices lends clarity to their thinking — and also sends the message that their responses are more likely to be taken seriously.

Moreover, the answers to concrete questions offer more insight to conference stakeholders. “When you get the results back,” Thalheimer said, “you can see where you stand and say, ‘Okay, if basic familiarity was what we wanted and we got it, that’s good. If that’s not enough — we really wanted people to have a solid understanding — then how can we get there?

Although the Likert scale’s numeric tabulations make it easy to rate sessions, Thalheimer questions the value of those numbers. “If you get a 4.1 between ‘strongly agree’ and ‘agree,’ you don’t know what to do with that, really,” he said. “Is it good? Is it bad? If you have questions for which the answer choice is distinguished better, you can actually decide on a set of [session] standards before you get the data.”

Concrete-answer choices can also help speakers. By offering them a chance to review the survey questions and response options well ahead of time, Thalheimer said, you’ll help give them a better understanding of the outcomes you’re seeking — and therefore prepare session content that is more aligned with learners’ needs.

This story is part of Convene’s CMP Series; more about feedback forms and surveys is available here.

Michelle Russell

Michelle Russell is editor in chief of Convene.