Warning Lights
By Lawrence M. Butler
The annual “Trends & Indicators” issue of CONNECTION paints a statistical picture of the social, demographic and educational landscape that New England colleges and universities inhabit. We can even glimpse in these data some of the forces shaping the terrain in the years ahead. But how well do New England’s college presidents and trustees navigate that landscape? How well are their “vehicles” performing? In addition to the external view, shouldn’t they have their own internal “Trends & Indicators” issue – their own sets of institutional metrics to gauge progress toward student enrollment and retention goals, for example, or to alert key decision-makers to pending problems in fundraising or academic quality.
So-called dashboard reports – like an automobile’s instrument panel – present quick, comprehensible overviews of the institution’s status and direction. Instead of speed, RPM and engine temperature, dashboard reports display comparable measures of organizational performance and mission effectiveness. These key performance indicators (KPIs) are presented in consistent formats that enable institutional leaders to readily spot significant changes and trends. Like an automobile dashboard, these reports often display the equivalent of warning lights that flash on only when there is an impending problem or when certain variables stray outside of predetermined limits. In this way, the dashboard can serve as an early warning device alerting the board and senior administration when it might be important to dig deeper for greater insight.
Dashboard Styles
Among styles of dashboard reporting, the “scorecard dashboard,” which first gained currency in the for-profit sector, has become increasingly common in nonprofits, including colleges and universities. Figure 1 is an example of one college’s scorecard style of dashboard.
On a single page, 29 KPIs are listed along with their current values and their lowest and highest values during the previous five years. The direction of change of the current value for each KPI in relation to the most recent, previously reported value is indicated by an up or down arrow icon or a square for no change. The importance of that change (from a strategic, financial or mission perspective) is expressed as better, worse or neutral and indicated by the color of the icon (red, green, or gray respectively). This scorecard dashboard sits on top of a set of pages that briefly discuss each of the 29 indicators, adding detail as required.
Instead of the five-year historical perspective shown in Figure 1, the scorecard style is often used to compare current values of a set of indicators to previously established goals or benchmarks. These goals or benchmarks can be externally derived – for example, an industry norm or standard or a “best practice” performance level achieved by peer institutions. Or they may be internally derived based on the organization’s own historical performance, budget projections, vision targets or mission-based aspirations.
All the elements of the scorecard dashboard –selecting appropriate KPIs, determining benchmarks, calibrating how the icons indicate better, worse or neutral performance – should be thought through by senior executives in consultation with trustees during the dashboard design process. With that prior understanding, the president or trustees can quickly review the report and know which aspects of organizational performance are under control and which require deeper probing. The scorecard dashboard is a powerful data presentation format not only because it employs compelling, visual metaphors (like traffic light colors, arrows, meters and gauges) to direct attention to critical issues, but also because it rests upon this foundation of prior analytical and collaborative effort.
A more graphic style of dashboard reporting is illustrated by Figure 2. In this example, one page of a set of “vision dashboards” displays student performance data. Other pages in the set show enrollment, student body characteristics, academic quality, financial and development data. Five years of actual values for each indicator are presented along with a target value set for a vision year five years into the future. A well designed dashboard of this type combines the judicious use of bar, line and pie charts with numerical data tables and, if greater guidance is needed, brief narrative bullet points. Some users prefer graphic dashboards such as this, because they convey at a glance various patterns, relative proportions and relationships among the data that the scorecard style does not. The two styles can be combined in a hybrid format that displays a series of key indicators in scorecard fashion with a few selected indicators displayed as line or bar charts.
Of these two basic styles of dashboard reporting, the scorecard approach works well not only in hard copy, but online where the color-coded icons can be used as links allowing the user to jump to another page with greater explanatory detail. The graphic style with its multiple charts, numbers and words arranged on a single page, tends to work better in hard copy. Online versions of such dashboards often require scrolling to clearly see all the detail, which can undercut the user’s ability to take in the entire page at a glance and spot patterns and relationships.
What to Measure?
The real value of dashboards lies not in their compelling formats but in their ability to highlight key performance metrics. So clearly the most important step in designing any dashboard report is determining what to measure. A considerable amount of study has been devoted to this subject over the years. So much so that, when the Association of Governing Boards of Universities and Colleges (AGB) surveyed the literature on higher education performance indicators, it uncovered more than 200 assorted ratios, variables and indices used to gauge the effectiveness, efficiency and impact of colleges and universities. There are plenty of metrics from which to choose, but college officials should resist the temptation to display a comprehensive array. The trick is to select a small subset of indicators most meaningful to a given institution.
Figure 3 offers a set of possible indicators that help answer a series of questions in one category of institutional performance – in this instance, enrollment. Other “life-cycle” categories might include recruitment, retention, and alumni engagement. Questions in each category could address institutional reputation, followed by operational performance and finally resource adequacy and consumption. Similarly, indicators can be developed to respond to questions about academic quality and outcomes. Many financial indicators are available from sources like AGB, the National Association of College and University Business Officers, and the Integrated Postsecondary Education Data System of the National Center for Educational Statistics.
Benefits Beyond Reports
Dashboards are user-friendly tools for displaying performance measures. These measures are not the end product of organizational or program evaluation but rather the top layer – the high-level view that points institutional leaders to where they might want to drill down into a more detailed, refined understanding of organizational and program effectiveness. In fact, the value of the dashboard design process can be as important as the insights gained from the reports themselves. By investing the time to identify in a thoughtful, collaborative way what is most important to measure – those key indicators that reveal the most salient aspects of institutional performance – dashboards can help clarify mission and build a shared understanding of institutional vision and strategy.
This article was published in the Spring 2007 issue of CONNECTION: The Journal of the New England Board of Higher Education.