Elves, Measuring Results and Informal Learning

Brent and I have been having a nice blog discussion. Our previous posts discuss what should be measured: Intermediate Factors in Learning, Intermediate Factors - Impact Many Measure One. And we finally seem to be agreeing with one exception. And this exception relates closely to my earlier concern eLearning Technology: Informal Learning is Too Important to Leave to Chance.

This discussion makes me think back to a question that I used to ask in my Computer Science Project class (based on something I read, but I now forget who it was - Fred Brooks maybe):

If an elf appeared and offer to give you a program that met your spec, how happy would you be?

After the initial jubilation wears off, the class realizes that there is some real concern on whether the program actually works as intended and without having insight into the guts of what's going on, it feels very uncomfortable. How do I know it works? How do I know what it's limitations might be?

It turns out that you really want more than just a program. You want one that you know how and why it works.

So, back to learning and measuring results. I actually don't want "just the outputs." I want to know how and why it's working. In the case of improving customer satisfaction based on knowledge of Store Layout and Product Knowledge, I want to know whether we've increased their knowledge and whether customer satisfaction has improved. While my client only cares about customer satisfaction, if it remains the same while knowledge increases, this is an important data point. It tells us about the system.

So, on to informal learning. Brent said in From Product Focus to Audience Focus:

The process is continuous and if our “training solution” is organic, dynamic, and flexible, it is very difficult to measure using the current method of measuring learning products. My point is “who cares”. If we have set up environments that help people collaborate, and support their informal learning, we should see output improvements.

And this is our only point of remaining disagreement. I would be much more comfortable if you can explain the internals of this system, how you know it works, when it will work and when it won't. As I said in Intermediate Factors in Learning ...

If you create an "organic, dynamic, flexible" learning solution but can't explain how it impacts the end numbers, then: (a) you won't get credit, (b) you won't know if you can repeat it successfully, and (c) you won't know if its really working.

Keywords: eLearning Trends, Informal Learning