Tony Karrer's eLearning Blog on e-Learning Trends eLearning 2.0 Personal Learning Informal Learning eLearning Design Authoring Tools Rapid e-Learning Tools Blended e-Learning e-Learning Tools Learning Management Systems (LMS) e-Learning ROI and Metrics

Friday, May 19, 2006

Elves, Measuring Results and Informal Learning

Brent and I have been having a nice blog discussion. Our previous posts discuss what should be measured: Intermediate Factors in Learning, Intermediate Factors - Impact Many Measure One. And we finally seem to be agreeing with one exception. And this exception relates closely to my earlier concern eLearning Technology: Informal Learning is Too Important to Leave to Chance.

This discussion makes me think back to a question that I used to ask in my Computer Science Project class (based on something I read, but I now forget who it was - Fred Brooks maybe):
If an elf appeared and offer to give you a program that met your spec, how happy would you be?

After the initial jubilation wears off, the class realizes that there is some real concern on whether the program actually works as intended and without having insight into the guts of what's going on, it feels very uncomfortable. How do I know it works? How do I know what it's limitations might be?

It turns out that you really want more than just a program. You want one that you know how and why it works.

So, back to learning and measuring results. I actually don't want "just the outputs." I want to know how and why it's working. In the case of improving customer satisfaction based on knowledge of Store Layout and Product Knowledge, I want to know whether we've increased their knowledge and whether customer satisfaction has improved. While my client only cares about customer satisfaction, if it remains the same while knowledge increases, this is an important data point. It tells us about the system.

So, on to informal learning. Brent said in From Product Focus to Audience Focus:

The process is continuous and if our “training solution” is organic, dynamic, and flexible, it is very difficult to measure using the current method of measuring learning products. My point is “who cares”. If we have set up environments that help people collaborate, and support their informal learning, we should see output improvements.

And this is our only point of remaining disagreement. I would be much more comfortable if you can explain the internals of this system, how you know it works, when it will work and when it won't. As I said in Intermediate Factors in Learning ...
If you create an "organic, dynamic, flexible" learning solution but can't explain how it impacts the end numbers, then: (a) you won't get credit, (b) you won't know if you can repeat it successfully, and (c) you won't know if its really working.

Keywords: eLearning Trends, Informal Learning

3 comments:

Anonymous said...

This makes me think of two situations from my own life - in my head they tie up with what you're illustrating, although I don't know if anyone else will be able to see the connection!

1 At school, I occasionally had the ability to look at an algebra or (less frequently) geometry problem and know what the answer was. I had no idea why that was the correct answer, just that it was. If I wrote down that answer in an exam situation, I would score 1 point for the question, which may have been worth 10 points. The other 9 were reserved for proving that I understood, which (on a conscious level) I apparently did not.

2 I am a bit of a sudoku addict. Occasionally, my husband gets to one before me and, by the time I get it, it will have some numbers filled in. No use whatsover, since I have to go back over everything to see how he has come to each conclusion and to make sure he has recorded each fact that I know to be true.

So yes, having just the outputs isn't enough.

Tony Karrer said...

Great stuff Karyn and I think many people will appreciate the thinking.

On a personal level, I play a bit of Sudoku as well (not quite an addict). In Sudoku, there's nothing worse than making a mistake along the way, especially on tough puzzles. It makes you want to cry when you have to basically erase the puzzle and start over because of an early mistake! So, I agree. Getting a half-done puzzle is "no use."

Anonymous said...

Hmm. I think you have the makings of a fine sudoku addict, Tony ;-)