Tony Karrer's eLearning Blog on e-Learning Trends eLearning 2.0 Personal Learning Informal Learning eLearning Design Authoring Tools Rapid e-Learning Tools Blended e-Learning e-Learning Tools Learning Management Systems (LMS) e-Learning ROI and Metrics

Monday, August 27, 2007

Instant Session/Speaker Ratings at Conferences?

Great question posed by Mark Oehlert -
I will say that it is interesting that in addition to the slides you can also rate each session and leave comments about each. Total transparency. As we look for ways to improve conferences, this is a point we should ponder on.

How would you feel if the ratings for a session you presented say at ASTD or the eLearning Guild were instantly available for public consumption? I know how I feel about this but I'd like to hear from some people - instant, visible ratings...boon or bane?
I personally would love to see this happen. I often find myself in the speaker room at conferences trying to get a look at my session evaluations. I'm trying to find out what went right or wrong. It's often the case that conference organizers don't get evaluations back to you for 3-4 months. So, having real-time feedback is great.

I'm not sure logistically how you can get this to happen (as most attendees use a paper form at the end of the session). But if you could have someone putting these online immediately, it would be great.

I'll be curious to see what the arguments are against this?


Downes said...

I'd rather have feedback than ratings.

If I were boring, I'd rather be told I was boring, not merely receive an uninformative 2/10.

Tony Karrer said...

Stephen, that's a great point. I too want to have the feedback more so than the rating. However, ratings have value in terms of showing the general trend in text comments.

However, that said - I've had situations where someone said - "best presentation ever" and I got a 1 rating (1 was worst) - my guess would be that they didn't understand the scale.

Cammy Bean said...

Well, you could distribute Clickers to all participants and do instantaneous polling. Perhaps one of the questions could be, "is the speaker boring you right now?"

Tony Karrer said...

Cammy - that's a great idea!

In fact, I've often lamented that when I do online presentations I can offer polls and people can write in questions/thoughts that are not as easy to do real-time. Likely we could take advantage of these tools as a presenter in interesting ways.

Of course, you can't actually ask "is the speaker boring right now" because if they are boring, then you likely aren't paying attention. :)

Dave Ferguson said...

For years, ISPI used the no-carbon-required, two-part feedback forms. A volunteer at your session collected the forms and separated them; one stack for you, one for the conference organizers. So you got both the numbers (for what they were worth) and the handwritten reactions within 15 minutes of finishing.

My theory is that the handwritten comments tend to represent the passionate (pro or con), though there's no such thing as non-skewed.

Given current (and near-term future) technology, the NCR forms are a lot easier than some chip-and-WiFi dependent system (astonishingly, not everyone has a laptop yet), to say nothing of less burden on conference organizers.

I have mixed feelings about the value of making all the ratings public. Would it stop the "entitled" people from presenting? I doubt it. It might also intimidate some who get all the critique or criticism they need on a day-to-day basis.

Tony Karrer said...

Dave - what a novel concept. Dual copies so the speaker can get feedback!


When presenting it's great to get that instant feedback. If you are presenting several times during the day then it would allow you to tweak a your talk as needed. If you are doing one full day you can also often adjust your presentation to accommodate something that the participants are asking for in addition.

As for the ratings to be public...hmmm....I'm not so sure. I could see one presenter at 8:00am before morning coffee and think what the heck is wrong with him...snooze. Someone else could see the same presenter later that day and after him reading his instant feedback, takes by 3 espressos and pulls off the presentation of a lifetime. Okay that's extreme, but if other's in the conference rate him on my feedback, they may never get to see someone that was just a little off their game.

Tony Karrer said...

Tracy, while there's some truth to being "off" - my guess is that it's not going to make that much of a difference in ratings. Someone who is getting a 3, doubtful that an afternoon session they'll be a 7. A five maybe. But more likely still a 3. If the session is happening more than once, I'd definitely want to see the ratings.

Mark said... would be better of course - I think that is why the Adaptive Path site also allows people to leave comments.

I'm not so sure on instant feedback since it might take a while for people to latch onto the main ida and maybe that is a road I feel they need to travel to get to the heart of the matter. I do like systems that would allow for live questioning of a speaker...especially a keynoter.

I am in favor of transparency in the ratings. I also think that the ability for everyone to leave ratings and comments will even out those who really hate and those who really love the session. Honestly, I think it would be great for some of these people who have been doing the same presentation for years to see how people feel about them (for good or ill).