You’re looking to buy off-the-shelf elearning and you’ve managed to get access to some demo content. But how do you actually review and evaluate it? You might get links, access to an LMS, or simply some videos. If you’re doing this on your own you might have a process, but you might also have a team of people who want to be involved in the buying and reviewing process. Elearning reviews don’t need to be difficult to manage, but they can quickly get out of control if you don’t have a good process in place.
Have you ever worked on projects where you’ve got to version 20 of something that doesn’t ever seem to stop? The same thing can happen when you’re evaluating courses. You send the demo content out and comments come flying in via email, Slack, and even voice notes. This doesn’t help. How can you be sure that you’ve delivered quality content into your LMS or LXP if the review process is chaotic?
It’s important to agree with your team upfront how many review rounds you need and how you’re going to manage your stakeholders. As part of the procurement process you need to be sure you’ve looked at everything and have a consistent scoring approach. Don’t be afraid to push back on suppliers if the process isn’t working. The benefit of a good process is that your scoring will be consistent across different providers.
Lack of Process Causes Problems
Often the reason a review process takes so long is the lack of structure and no clear idea of how to score. You’ll get feedback from multiple sources in different formats, and it becomes very difficult to collate.
A simple Google Sheet or Excel file is a good way to capture scores and notes across all providers. You can then use AI to run analysis on the scores and produce a summary report. Google Docs works well for freeform notes. If your supplier has provided links via Articulate Review 360, that’s an excellent tool for capturing detailed feedback and exporting it. Use whatever works, but using nothing is always a problem.
The successful review involves controlling where feedback comes in from. Have you ever sent something out for review and had comments arrive via email, text, WhatsApp, voice note, and in-person conversation all at once? You’ll miss things, your process looks out of control, and it takes far longer than it needs to.
Give your reviewers an instruction sheet. Tell them specifically what you want them to look at. Also think about getting your team to test on different devices, browsers, and times of day. The speed and performance of a course can vary significantly depending on how and where it’s being accessed.
Build a Scoring Matrix
The problem is everyone has an opinion, feedback comes from everywhere, and it can be hard to evaluate content properly. The answer is a scoring matrix agreed before you start, so that everyone is looking at the same things in the same way.
Getting the scoring matrix right is important, and testing does take time. Think carefully about what you want to include. Remember: you are not buying for yourself, you are buying for your organisation. You need to remove personal bias from the process.
Things to look for include:
- Production quality: does the content actually look like something people will use?
- Metadata quality: is the course data going to work in your LMS?
- Loading speed and performance across different devices
- Spelling and accuracy
- Course length: does it match what you’re looking for?
- Content and interaction style: is it just a wall of text?
Get a couple of people involved in the scoring, including someone from outside L&D. And make sure you’re seeing more than the vendor’s standard showcase. Ask for courses from your own shortlist, not just their top five.
Use this matrix to score each supplier’s demo content consistently. Score each criterion 1 to 5 (1 = poor, 5 = excellent) and total the scores to compare providers objectively.
| Criteria | What to look for | Score (1–5) | Notes |
|---|---|---|---|
| Production quality | Does the content look professional? Would your learners engage with it? | ||
| Content accuracy | Is the content factually correct, up to date, and free from spelling errors? | ||
| Accessibility | Does the course meet accessibility standards? Can it be used with a screen reader or keyboard navigation? | ||
| Course length | Does the length match your requirements? Is it appropriate for your audience? | ||
| Interaction and format | Is it genuinely interactive or just a wall of text? Does the format suit the topic? | ||
| Technical compatibility | Is it SCORM or xAPI? Does it work on your LMS, on mobile, across different browsers? | ||
| Loading speed and performance | How does it perform on different devices and connection speeds? | ||
| Metadata quality | Is the course data complete and LMS-ready? Are thumbnails and descriptions included? | ||
| Relevance to your audience | Does it speak to your learners’ context and role? Remove personal bias when scoring this. | ||
| Value for money | Does the price reflect the quality? What is the total cost of ownership? | ||
| Total | Maximum score: 50 | /50 |
Score each course from each supplier using the same criteria. Use the Notes column to capture specific observations. Compare total scores across providers to support an objective procurement decision.
Using the Right Tools
Work out how your reviewers will log, capture, and incorporate feedback. Listening to your customers is the key to better elearning courses, and your customer here might be internal or external. Either way, you need to capture their feedback properly.
If suppliers have provided Articulate Review 360 links, use them. Everyone involved can see the content, capture comments in one place, view previous versions, and export to CSV. It’s a solid review tool when it’s available.
Not everyone likes a review platform, and that’s fine. You might need to help new users get started and bring reluctant reviewers along. Initially some people will want to carry on using email. You need to share with them that they are one part of the project and that everyone needs to collaborate, share updates, and work in real time. Email simply isn’t the right tool for collating feedback and moving a procurement process forward.
If not, a shared Google Sheet or cloud-based spreadsheet works well. You can control the template, see feedback coming in real time, and avoid the version control nightmare of files being emailed back and forth. The key is getting everyone to capture feedback in one place, whatever that place is.
The value of using a tool properly shows at the end of the process, where you can show your audit trail and explain clearly how you managed each round of feedback. That’s useful both internally and when going back to suppliers with consolidated change requests.
People Solve Problems, Not Systems
Critically, people solve problems, not systems. You’ll still need to work through the comments, manage the process, and make sure everything is collated properly. Don’t rely on software alone to manage your review process.
One practical benefit of collating everything in one place: you’ll often find that several reviewers have raised the same issue. You can identify that quickly, save time, and present a consolidated set of requests rather than a chaotic list from multiple channels.
If you start capturing feedback consistently you might find patterns emerging across your process — are you missing something, is there something you could do differently? Is there something a supplier does that you consistently score highly, or something your reviewers always flag? Feedback is a leader’s secret weapon. Feedback and reviews are an opportunity to look at ways you can improve your process and develop, not just to make a buying decision.
Common Mistakes to Avoid
- Only seeing the vendor’s top five courses: ask to choose your own
- One-person selection teams: get other perspectives involved
- Judging by personal preference: buy for your audience, not yourself
- Only looking at suppliers you’ve worked with before
- Not having a detailed scoring criteria before you start
Frequently Asked Questions
How do you evaluate off-the-shelf elearning courses?
Build a scoring matrix before you start, agree the criteria with your team, and make sure reviewers are testing on different devices and browsers. Score each supplier’s demo content consistently across the same criteria so you can compare objectively. Don’t rely on personal preference — buy for your audience, not yourself.
What should be included in an elearning review process?
A clear scoring matrix, an agreed number of review rounds, a single channel for capturing all feedback, and an instruction sheet for reviewers. Include people from outside L&D in the process and make sure you’re testing the content on the devices and platforms your learners will actually use.
What tools can you use to manage elearning feedback?
Articulate Review 360 is excellent if suppliers provide links in that format. A shared Google Sheet or Excel file works well for scoring across multiple providers. The key is centralising all feedback in one place and avoiding comments arriving via email, WhatsApp, voice note, and in-person conversation simultaneously.
How many people should be involved in reviewing elearning content?
More than one, and ideally including someone from outside L&D. A one-person selection team is one of the most common procurement mistakes. Getting different perspectives helps remove personal bias and ensures the content works for a broader range of learners, not just the person making the decision.
Scott Hewitt
Scott Hewitt is the founder of Real Projects, an off-the-shelf elearning content library trusted by organisations including HowNow, OpenSesame, Ticketmaster, and easyJet. He has built a library of over 800 courses across nine languages, with a focus on practical workplace training that’s ready to deploy on any major LMS.
Scott has spent over 25 years working in elearning as a buyer, supplier, and creator, and has helped organisations build review and procurement processes that actually work.
Connect on LinkedIn
