Tony Karrer's eLearning Blog on e-Learning Trends eLearning 2.0 Personal Learning Informal Learning eLearning Design Authoring Tools Rapid e-Learning Tools Blended e-Learning e-Learning Tools Learning Management Systems (LMS) e-Learning ROI and Metrics

Thursday, September 20, 2007

LMS Satisfaction Features and Barriers

Update, Oct. 2010. The data below is getting just a bit old at this point. It still might give you an idea of what's out there, but I'd be careful relying on it too much in your LMS selection process.

Here are some more recent resources such as my series on Selecting a Learning Management System (LMS):
Additional resources from my blog that are more recent around LMS:
  1. LMS and Social Learning- eLearning Technology, March 31, 2009
  2. Open Source LMS- eLearning Technology, December 10, 2009
  3. Test SCORM Courses with an LMS- eLearning Technology, January 13, 2008
  4. LMS Team Size and Time - Wow 23 Months!- eLearning Technology, October 31, 2007
  5. How long does it take to select an LMS?- eLearning Technology, January 8, 2009
  6. Test LMS- eLearning Technology, September 10, 2008
  7. Communities / Social Networking and LMS Merger- eLearning Technology, December 6, 2007
  8. What Goes in the LMS?- eLearning Technology, February 5, 2009
  9. Rapid Learning Management Systems- eLearning Technology, October 20, 2009
  10. What Makes an LMS Easy to Use?- eLearning Technology, February 11, 2010
  11. Leading with an LMS - Harmful to Your Health (or Skipping Stages in Bersin's Four Stage Model)- eLearning Technology, June 6, 2006
  12. LMS RFP- eLearning Technology, October 25, 2007
  13. Larger LMS Audience Means Lower Satisfaction- eLearning Technology, April 18, 2008
  14. LMS - Questions- eLearning Technology, November 5, 2007
  15. Moodle in Corporations- eLearning Technology, April 4, 2007
  16. Custom LMS Anymore?- eLearning Technology, January 5, 2010
  17. LMS Tracking of Podcasts and Video Casts- eLearning Technology, February 9, 2010
  18. One Week to Select an LMS – No Way- eLearning Technology, February 16, 2010
  19. LMS Solution for Simple Partner Compliance Training
  20. Low Cost LMS
And you should check out: Learning Management Systems (LMS) for a more extensive list.

I thought I had previously wrote about this, but I realized when I was posting Learning Management Systems (LMS) Gotchas that I'd not previously talked about the results from participating in the eLearningGuild's Learning Management System research report. You can get the abstract for the report and sign-up for a webinar on Sept. 26th on that same page. I was one of several authors who collaborated on the survey questions and wrote different sections of the report.

The survey data itself I find to be pretty interesting and useful for some important aspects of selection. The survey asks demographic and company questions of all the participants so that you know things like the industry, size of company, number of learners, their role, etc. Then it asks questions such as satisfaction level, satisfaction with particular features, are you planning to replace, primary barriers, importance of features, etc.

Overall Satisfaction in Large Corporations (Count >= 3)

LMS Satisfaction Large Corporations

Overall Satisfaction in Small and Medium Corporations (Count >= 3)

Learning Management System Satisfaction

Overall Satisfaction in Education and Goverment (Count >= 3)

LMS Satisfaction Education and Government

And to show some of the power of the slicing, I've got a graph of overall satisfaction for Manufacturing and Biotech/Pharma for companies with more than 1,000 employees and 1,000 learners with a count of 1 or more.

LMS Satisfaction Manufacturing

Some notes on these numbers...
  • These are being reported out of the larger eLearningGuild population and I would claim that they are likely more accurate than survey research that goes through the vendors themselves. However, when you get low counts, the numbers are highly suspect.
  • Several of the LMS vendors appear more than once, such as Oracle. Most of the time (as is the case with Oracle) these are different products. Sometimes it's how they were referred to by the survey respondents and it hasn't been cleaned up in the data.
Some comments on the satisfaction numbers themselves...
  • The dissatisfaction numbers are high in general. And this is not being reported only in this research. I mentioned in previously in LMS Dissatisfaction on the Rise. An LMS is a big, expensive tool that takes quite a bit of work and is generally harder than you think it's going to be when you start out. It's why I often try to convince people to not Lead with an LMS.
  • To me, it's interesting to see news like CornerStone OnDemand Raises $32M - Sept. 17 2007 - when their satisfaction score is being reported so low. That seems like a disconnect. I know the folks from CornerStone and their product does some really interesting things. They are moving towards a broader suite that focuses on other aspects of Talent Management. But given the number of respondents who've rated them low on satisfaction, it would give me pause during a selection process (so that makes me worried about sales), it also would have worried me as a potential investor. Maybe it's good I didn't post this until today.
  • Moodle scores very high in satisfaction, but we need to qualify that result a bit. I personally feel that Moodle is good at the limited stuff it does. It's free which improves satisfaction. But it's not really an enterprise LMS and has some very serious deficiencies when it comes to many of the needs of corporate training departements. However, I would be concerned if I was a starter LMS vendor because Moodle is going to cause you grief in the bottom of the market. If nothing else, it causes the perception that there's a free competitor.
  • SkillPort scores very high as well on satisfaction. However, like Moodle, it only addresses particular needs and so most often it's a starter LMS. I do highly recommend using a starter LMS if you are new and you don't necessarily know what you need later on. Note: this represents a lot of the market whether they admit it or not. Of course, if you are consciously choosing something as a starter LMS, make sure that everyone knows that its a starter and that you plan to move in a few years.
  • Oracle's (Learning Management and PeopleSoft Enterprise Learning Management but less so it's iLearning) and SAP Enterprise Learning score better than I would have expected. For the past several years during selection I've found that they trail Saba and SumTotal on needed capabilities and they were hard to use for particular tasks that were important to the clients involved. Based on the satisfaction numbers, it appears that they are catching up and it will be interesting to see where they are during the next evaluation. As these systems catch up, this will make the market really hard for Saba and SumTotal. Over the past few years, they've been able to fend of Oracle, PeopleSoft and SAP through superior products. If the playing field is more level, then it becomes harder to argue why you don't want an integrated solution. So large companies with Oracle or SAP implementations are going to start going more towards using those packages.
  • The numbers displayed above don't show this, but you can slice the numbers according to the level in the organization and by role. No surprise, but the people who have to work with the LMS day-to-day give a much lower satisfaction number than the managers and directors who get the results and the reports. Performing tasks (use cases) in an LMS are harder than they seem like they should be and it results in lower satisfaction.
  • I wouldn't read too much into any of these numbers. They can definitely be useful to see what LMS products are being used in companies similar to yours. They can be used to see how LMS vendors stack up against particular features. But you still need to look at differentiating use cases to choose a vendor that will work for you.

Other data in the report points to particular barriers and the importance of particular features during Learning Management System (LMS) Selection.

Barriers



LMS Barriers


Importance of Features



LMS Features


Some notes on Barriers and Feature Importance:
  • Good news for me - problem with 3rd party consultant scores low as a barrier. :)
  • A lot of the other factors are pretty significant barriers, e.g., cost, customization, flexibility.
  • On the features importance, looking at the aggregate numbers is not very interesting. However, looking at numbers for your particular type of industry, size, etc. can yield more interesting results.
Some other posts in my blog around Learning Management Systems:


LMS Systems:

LearnFlex Operitel Corporation 1 10.00
Extention LMS Acadia HCS 1 9.60
NetDimensions
Enterprise .. NetDimensions 1 9.30
OutStart Evolution LMS OutStart 2 8.40
Moodle Moodle 6 8.33
Oracle iLearning Oracle 4 8.25
TopClass e-Learning Suite WBT Systems 2 8.25
SkillSoft SkillPort Skillsoft 12 8.03
CourseMill Trivantis 1 8.00
Compliance LMS Pro-ductivity Systems 1 7.50
Oracle LMS Oracle 14 7.48
ResultsOnDemand SumTotal Systems Inc. 3 7.33
LearnCenter Learn.com 5 7.18
WBT Manager Integrity eLearning 1 6.90
Articulate Online Articulate 1 6.60
Saba Learning Suite Saba 9 •::: 6.37
TotalLMS SumTotal Systems Inc. 27 I 6.29
GeoMaestro LMS GeoLearning 6 6.27
Plateau Learning Managem.. Plateau Systems, LTD 22 6.24
Blackboard Academic Suite Blackboard, Inc. 8 6.19
PeopleSoft Enterprise Lear.. Oracle 2 6.15
Saba Enterprise Saba 23 5.97
SAP Learning Solution SAP 10 5.97
Virtual Learning System Plateau Systems, LTD 1 5.90
Cornerstone OnDemand Cornerstone 4 5.80
Training PartnerTM Learning.. GeoMetrix Data Systems Inc. 4 I 5.68
IBM Lotus Learning Manag.. IBM 1 5.30
KnowledgePlanet Learning KnowledgePlanet 3 4.50
ViewCentral ViewCentral 3 4.43
KnowledgeNet Platform LMS Thomson NETg 2 4.10
GeoConnect GeoLearning 1 3.90
Enterprise Knowledge Man.. Generation 21 Learning Syste.. 1 3.00
IntraLearn 5.0 IntraLearn Software Corporati.. 2 2.70
Pinnacle Learning Manage.. Learnframe, Inc. 1 2.50
TM SIGAL® Technomedia Training Inc 1 2.30
Enterprise Knowledge Asse.. Generation 21 Learning Syste.. 1 2.10
Meridian KSI Knowledge Ce.. Meridian KSI 1 1.60
Vuepoint CertPoint CertPointSystems VLS
Adobe

LMS Barriers:

The Cost
IT Support
Customization
Integration with other
systems (content. HR, ERP,..
Legacy system integration
Mind set to move learning
online
Problems with vendor
Support from management
Flexibility for future
requirements
Support from stakeholders
Clear business goals
Security
Support from learners
Tool/vendor selection
Administration
Problems with third-party consultant
Compliance

LMS Features:

Tracking, reporting, and
measurement
Content delivery
Assessment and testing
Asynchronous e-Learning
Blended learning
Standards (SCORM and AICC)
Training history
The ability to support different
models and sequences of blended..
Ability to create an index so that
people can find a particular topic
Security
User and group management
Registration
The integration with single sign-on so logging in is not required
Competency and skills
Synchronous e-Learning
Regulatory Compliance
Collaborative learning
Certification
Catalog
Instructor-led training management
The ability to support specific and complex business process models

13 comments:

Anonymous said...

Tony,

Wow! That's some post. Why didn't you just print the whole report? ;)

There's some stuff I want to share on this, but it includes some graphics so I've posted some musings over at the Guild Research blog.

http://www.elearningguild.net/research/?p=36

Steve Wexler
Director of research and emerging technologies
The eLearning Guild

Tony Karrer said...

Steve - it's funny to have you give that comment. You and I are always the first people to say - "don't use this information without going into depth about your particular needs, your size, your industry, etc."

The comparative pictures you did by feature are great!

Hey Steve - when are you going to make ALL the general data available? That would be cool!

Anonymous said...

Tony,

You wrote:

"Hey Steve - when are you going to make ALL the general data available? That would be cool!"

To what do you refer? With a handful for exceptions, anything I can access others can access. Some stuff is free and others are available for a fee.

Is there something we're sitting on that we should not be?

The comparables model is one of several Direct Data Access portfolios that comprise the LMS report. The screen shot shows a comparison of certain features, but you can also get into cost per learner, time to implement, etc.

Steve

Tony Karrer said...

Steve - I was thinking that only some of the overall data was available for free and the rest was for fee. Maybe that's a misunderstanding.

Steve said...

As a Moodle user and administrator, I was not surprised in the least to see that it ranked in the top five in all categories, and in the top two of four of those... So I was surprised to see your comments below the charts. You stated that it has some serious deficiencies, but did not clarify what you meant. Could you follow up with more details?
Regardless, users obviously like it as is, as even with its "very serious deficiencies" it outscored most of the pack. ;^)

Tony Karrer said...

On Moodle's deficiencies, how you would roll out training requirements to a large corporation where the requirements are based on job function and you need reporting and reminders up through management levels on compliance with the programs?

As you begin to run down the list of features and functions required in any particular corporate environment, in my experience Moodle has come up way short. As have many commercial LMS products.

On the other hand, I have seen several really good departmental or starter installations of Moodle in corporate environments that focus on a limited set of functionality and they meet with good success - as is shown in the research results.

Conrad said...

I would like to know what this group thinks an LMS will look like in the future.

I see two trends: The bare bones LMS that just delivers learning and teh complete talent management suite of tool that an an LMS is just a part of.

By the way it was Toni at the ASTD conference in San Antonio who inspired me to start blogging.

http://elearningprojectmanager.blogspot.com/

Unknown said...

LMS is also one of the feature in coming Oracle fusion, Demos are also present on http://peoplesoftsupport.blogspot.com/, check this out.

Anonymous said...

Our team at the Gilfus Education Group just released this white paper to provide critical insights to practitioners while clarifying "Social Learning" as a concept.

Social Learning Buzz Masks Deeper Dimensions Mitigating the confusion surrounding “Social Learning” (Download Here)

It is our hope that by leveraging socially based technologies the education industry can shape a new educational technology paradigm that realizes the promises of true “Social Learning”.

By understanding its applications we can create a unique opportunity to improve student engagement, student retention, academic success and overall educational outcomes.

– Stephen Gilfus, Gilfus Education Group

Jason said...

Hi Steve,

I ran across your research information and find it very interesting/useful. Do you plan on releasing similar statistics that are more current as they relate to 2010? Also, have you acquired any information for Intellum, another LMS provider who has seemingly become a viable option recently in the e-learning industry? Keep up the good work!

-Jason

Anonymous said...

Please correct me if I am wrong but do those numbers represent sample size (the number of respondents)?

If so, these graphs are mostly meaningless. One cannot draw conclusions based on such low sample sizes.

Just saying.....

BTW, I found this site because I am unhappy with our current hosted LMS (Inquisiq). It is too limited to meet our requirements and I am looking for a replacement.

We are currently paying $500 per month for ~ 240 learners. Can someone point me to a discussion of LMS's in a similar price range. We have spoke with Learn.com and GeoLearning - seems like great products but their set up fees are the killer! Moodle has been considered but my education department is staffed by only 3 people.

Tony Karrer said...

Yes, that's sample size. This was (and is) one of the largest surveys out there. So, you are unlikely to get much larger sample size. But it's also getting older now so I wouldn't read too much into it.

You might want to also look at:

http://elearningtech.blogspot.com/2009/02/low-cost-lms-help-needed.html

http://elearningtech.blogspot.com/2010/03/lms-solution-for-simple-partner.html

http://elearningtech.blogspot.com/2009/10/rapid-learning-management-systems.html

Tom Burns said...

Great post! I happen to think that some LMSs are turning off modern learners, specifically that they are 'anti-social' (see my article: http://tfburns.com/blog/?p=11).

What do you think?