Corporate Profile: Continuous improvement based on valid measurements
Analytics helps education organization Kaplan go to the head of the class.
By Bror Saxberg (Left) and David Niemi (right)
“Continuous improvement” is a commitment to evidence-based improvement, but it is at least inefficient if not risky to make decisions based on inaccurate evidence. For important decisions, it’s worth some investment in understanding the validity of the evidence you are using — understanding the accuracy and reliability of the evidence, particularly the metrics, for the decision you are making.
To illustrate we draw on examples from Kaplan, Inc., a world-wide educational organization with more than a million students per year at all stages in life, and many levels of learning complexity, from middle-school math tutoring to law degrees, MBAs and more.
Complex, sophisticated analyses are readily available through the Web, through open source tools like R or through packaged software. Analyses drive decisions, but too often the quality of the data that go into the analyses is not checked.
In our experience, many users tend to think quantitative metrics or data are likely valid simply because they are quantitative: Objective-appearing numbers inspire confidence for many managers, and the more numbers the better — indeed, the more complex the analysis, the more the conclusion looks good. This is not right. The mere existence of data, even an immense amount of data, says nothing about whether it can be used with confidence for important decisions. The essence of validity is to collect additional evidence to show that inferences, decisions and actions based on the data or metrics reflect reality.
The good news is there are long-standing, extensively tested practices and principles of measurement validation to draw on, often developed through work in complex domains in the sciences, where much work has been done on validating evidence to avoid falling prey to faddish, capricious, popular or simply wrong decisions based on deficient information.
To accelerate evidence-based continuous improvement efforts worldwide, Kaplan has set up a Learning Innovations group (KLI) with expertise in evaluation, measurement, instructional design, training and analysis of expert performance. Initially, KLI identified a set of key quality indicators and performance metrics linked to them. KLI identified an early set of performance metrics for which initial validation effort was simply to check that the numbers are right; e.g., for enrollment numbers, withdrawal rates and graduation rates. Limited validation resources could then be concentrated on other key performance metrics more challenging to validate, including: learning; the effectiveness of instruction, curricula and other program components; student satisfaction; employer satisfaction; and career success. We’ve started with learning metrics, which are difficult to design and validate, but we expect the validation strategies we developed to apply to many important decision domains and the data that drive them.
|About Kaplan, Inc.
|Kaplan, Inc. is a leading international provider of educational and career services for individuals, schools and businesses. Kaplan serves students of all ages through a wide array of offerings including higher education, test preparation, professional training and programs for kids in grades K through 12. Kaplan is a subsidiary of The Washington Post Company and its largest division. For more information, visit http://www.kaplan.com.|
An index example of this initial work is the validation of learning metrics undertaken by Kaplan University (KU), one of Kaplan’s major business units. Unlike many higher education institutions, KU has been working for some time on measures of learning that focus systematically on mastery of learning objectives across its 1,000 or so course offerings. KU’s group of assessment professionals, the Office of Instructional Effectiveness, early on determined that typical grades are noisy and incomplete measures of mastery: It is not clear what objectives different students with a “B” grade have mastered, and it is also not clear if these students were graded in comparison to each other rather than based on actual mastery of a certain number of objectives. The goal then became to have a measure of learning that was as objective as possible, and to gather evidence about whether the measure had sufficient validity to justify its use in evaluating and improving teaching, and the design and implementation of curriculum and instruction, as well as student learning. A more valid learning metric could also be a great help in communicating with stakeholders: learners, faculty, regulators and prospective employers.
To address this need, Kaplan University staff developed and administered a system of Course Level Assessments (CLAs) across 1,000 courses. Typically, there are four to six CLAs in a course, each focused on an important course-learning objective, such as “Discuss typical neurobiological and behavioral responses to stress and their implications for physical and mental functioning.” Some CLAs are performance assessments, involving, for example, student analysis of a complex problem or realistic situation; scoring rubrics were developed to assist teachers in evaluating these performances. In other cases, CLAs consist of easily scored multiple choice quizzes or tests.
In all cases, however, there has been a concerted and sustained effort to validate the information obtained from the CLAs. Assessment experts reviewed the quality of the CLA tasks and the rubrics, as well as the alignment of the rubrics with the course learning objectives they were intended to assess. The results of these expert reviews are currently driving the refinement of the assessment tasks and rubrics.
Empirical analysis is another critical part of any validation effort. We have investigated, for instance, frequency distributions of CLA scores and the patterns of correlations between CLAs scores and other important metrics such as grades, teaching evaluations, and student retention, failure and withdrawal rates. Results have prompted further investigations and efforts to revise tasks or rubrics. We have also found, as a side benefit, that fluctuations in grades and CLAs scores may enable us to better predict which students are likely to fail or leave programs, and to intervene to help them.
Finally, since teacher scoring of CLAs is crucial, we have designed a series of studies of the reliability of that scoring. Discovering issues here can lead to improvements in scoring rubrics and how we train teachers. Another bonus will be that students get more consistent and valid feedback on their learning, which should improve their rate of learning (as extensive research has demonstrated).
Valid measures of learning are critical to Kaplan’s students and the whole enterprise, even though learning is not easy to measure well. Indeed, reliable learning data gives us a tool to make decisions about new approaches to learning, including multimedia, new hardware platforms and other online and offline techniques.
We expect in the near future to tackle validations of other critical metrics, determining in each case what the right levels and types of evidence are. We look forward to benefits (both expected and unexpected) as this work allows us to make better decisions to bring value to learners and our business.
Bror Saxberg (http://www.kaplan.com/brorsblog) is the chief learning officer and David Niemi is vice president, measurement & evaluation, at Kaplan, Inc.