Quality assurance (QA) is the testing activity performed after development of courseware is complete and before putting the courseware into production, right? Wrong! Done well, QA begins early in the development project and continues indefinitely while the courseware is in production. “Evaluation” is how ADDIE refers to QA and, as you can see in this model, it is ongoing throughout the development project and beyond.
Creating a test plan
The earliest QA activity is creating a test plan (I dedicated an entire blog post to this activity). It’s critical to control the quality of elearning content as early in development as possible because it’s easier to correct content in development formats. The first quality control (QC) is implemented in the design phase. The test plan documents all the testing conducted during development in detail. The QA team references it throughout the project for guidance on what testing to perform and how to do it. This remainder of this article presents the QCs the test plan covers.
Proofreading and editing
For textual content, the QC is proofreading and editing. This includes both the copy in the courseware and audio scripts. The instructional designer or subject matter expert authors as much of the copy and scripts as possible in the storyboards and other instructional design (ID) documentation. The ideal format is Microsoft Word because its Track Changes function is integral to proofreading and editing well. Editing has the greatest QC when done according to a formal editorial style guide for consistency from course to course. Text in supplementary materials goes through the same QC. Even text used in assessment tests is subjected to this QC. It’s much easier to proofread and edit copy, as well as to make the corrections, in Word than it is in a tool like Adobe Captivate or a production format like HTML and Flash (e.g. I authored and edited this post in Word, then pasted it into the blog, rather than authoring it in WordPress).
For elearning, content is not the only thing that requires QC. The functionality of the courseware also needs to be tested to ensure that the learner can effectively operate it. There are four QCs for functionality in the development of the courseware.
Very early in the project, the team develops a prototype of the courseware as the first step in designing the functionality of the courseware. The prototype is minimally functional courseware with no real content in it. It emulates the user interface (UI) so that it gives the customer a sense of how the finished courseware would function. After reviewing the prototype, the customer provides feedback on the changes they want made to the courseware. Because prototyping is done early in the design phase, the changes are easier to implement than they would be deep into the development phase.
As the development of each unit of courseware is completed, the unit is tested independently. A unit is typically delineated by a lesson or, in the case of SCORM courseware, by a developed “sharable content object” (SCO). If the units are SCOs, this is the first point at which to use a SCORM Conformance Test Suite. The unit testing covers every UI and content element for interacting with the courseware to verify that they all function correctly. The test also includes verifying every hyperlink. As shown in the documentation validation chain, each functional specification in the ID documentation has a corresponding QC in the test plan. Unit testing also confirms that every graphic, multimedia element, and audio is accurate and presented to the learner as designed. Accessibility features such as closed captions and screen reader output is tested to be sure it functions correctly.
When the entire course is developed, the alpha test is performed on the courseware as a whole. The alpha test ensures that the components of the courseware were all assembled correctly and that the courseware functions as intended from end-to-end. If the full course was compiled from multiple SCOs as a SCORM package, SCORM conformance testing is again performed. This time, it’s tested on the entire curriculum to verify that all the SCOs in it interoperate with the LMS correctly. Alpha testing is also the point at which to test any functional specifications that could not be tested in the unit testing.
After publishing the courseware and any supplemental materials, system testing is performed on the full production environment. When implementing the curriculum, it needs to be comprehensively tested in conjunction with all of the materials and related services as a whole system. They include learning management services, online testing engines, certification generation, and any other systems integrated with the courseware. The test plan prescribes how the entire system should operate. The test script ensures that all ancillary services and integrations will function as intended when implemented for actual learners. After the curriculum passes system testing, it is ready to be put into production.
Although it’s a QA activity, beta testing is done in production. The beta test is conducted using the full course just as delivered to paying customers. Beta testers should be representative of the target audience for the course, so the QA team or others involved in developing the course are typically not appropriate beta testers. More often, a small group of employees from the target audience are used (in which case, it might be called a “pilot”) or a key customer might be willing to provide beta testers in exchange for getting an early look at the course for free. The beta testers study the course as actual learners would, then they complete a questionnaire. The main purpose is to determine how effectively the course helps the learner meet the learning objectives but other feedback can also be elicited in the questionnaire. Beta testing is the first feedback from learners rather than from the development team, so it provides a unique perspective on quality. Final changes are made to the course based on the findings of the beta test or pilot.
QA in production
In spite of all the QCs used during development, no courseware will be perfect once it’s put into production. The QA team will invariably miss quality issues that need to be addressed. However, actual learners can be the best source of quality feedback simply because of the large number of people studying the course. Provide learners with a form to report erroneous content they discover and they will identify things like misspellings, transposed figures, dead hyperlinks, and other minor errors that are easily overlooked. Also elicit enhancement requests from production learners to learn what improvements they would like to be made to the course. Archive the learners’ feedback in a database for later. Although the development project has already ended by this point, these quality improvements can be implemented in the subsequent minor or major revision to the course.
Developing new versions of the same course is an important aspect of QA. All of the aforementioned QCs are again applied when revising the course. QA is a never-ending process. For the highest quality courseware, QA must begin early in development, performed at various times and in various ways during the project, and continue the entire time the course is in production until it is finally retired.