Accountability and learning assessment in
the future of higher education
Geri Hockfield Malandra
Geri Hockfield Malandra is
Vice Chancellor for
Strategic Management at
the University of Texas
System, Austin, Texas,
USA.
This paper was originally
presented as a keynote
address at the 7th Annual Texas
A&M University Assessment
Conference, February. 22-23,
2007. It has been updated with
more recent information.
DOI 10.1108/10748120810874478
Abstract
Purpose – The purpose of this paper is to broadly describe and provide insight into the national
dialogue in the USA concerning accountability for educational results in academic institutions.
Design/methodology/approach – The approach takes the form of a thorough survey of the key
questions, current issues, and organizational players in this national dialogue.
Findings – Policy makers and educators should use the present opportunity to work together, focusing
not on narrow one-size-fits-all measures but on the improvement of a new generation of complementary
approaches.
Originality/value – The paper presents a clear articulation and discussion of the key questions that are
driving the national dialogue on this topic.
Keywords Higher education, Management accountability, Accreditation of prior learning, Assessment,
Tests and testing
Paper type Viewpoint
Introduction: Questioning accountability and assessment in higher education
It used to be a hard sell to get non-specialists to notice, let alone take the time to
understand accountability and assessment of student outcomes in higher education,
despite the fact that across the nation, every institution that seeks regional accreditation
has been required for some time to develop methods to assess and report student
learning outcomes. While these efforts vary considerably from institution to institution and
among the many regional, national, and professional accrediting associations, higher
education policy groups and colleges and universities have developed a number of
significant approaches to help demonstrate their accountability and to understand
student outcomes[1].
What has occurred recently to shift more public and professional attention in this
direction? What are some of the key policy questions about accountability, assessment,
and testing in higher education? Where may we be heading nationally if this momentum
continues?
Today, accountability and assessment have become prominent topics of institutional and
public discussion. Serious critiques like Bok’s (2006) Our Underachieving Colleges and
Hersh and Merrow’s (2005) Declining by Degrees have added weight to the debate and to
attention-grabbing, if not entirely accurate headlines: ‘‘Should government take a yardstick
to college?’’ ‘‘US in a testing frenzy.’’ ‘‘Colleges enter testing debate.’’ ‘‘A warning on
measuring learning outcomes.’’ ‘‘Governor calls for exit testing.’’ ‘‘UT system applauds
standardized testing.’’ ‘‘No gr_d_ate left behind.’’ ‘‘Is your college a lemon?’’ ‘‘A year later,
spellings report still makes ripples.’’
VOL. 16 NO. 2 2008, pp. 57-71, Q Emerald Group Publishing Limited, ISSN 1074-8121 j ON THE HORIZON j PAGE 57
 The national dialogue has raised important policy questions:
1. Why is it, despite the mountains of evidence derived from 20 years of documentation
through accreditation reviews, that the academy’s ability and commitment to measure
and demonstrate its outcomes are still being called into question?
2. Assuming a different focus on accountability is needed, what should the units of analysis
be? Student? Program? Institution? State? Country? What is it we do not know? Is it the
process, or the results, or the use of the information that need improvement? Why, for
example, would it not be sufficient for policy makers to show through census data that a
baccalaureate degree confers nearly double the annual earning power of a high school
diploma?
3. What should be the role of testing in assessment of learning outcomes within this new
accountability framework? Should tests at the postsecondary level ever be mandated?
How should they be designed and implemented? How should the results be used? The
idea of testing, per se, is hard to defend as aberrant. We accept tests readily, and not just
in schools. We are naturally competitive – and anyone with kids knows this starts early.
The ancient Greeks created the Marathon. The ancient Chinese administered civil service
exams that were, indeed, high stakes for government careers. From the Olympics to a
myriad of ‘‘top ten’’ lists to competitive reality television shows like American Idol,
Americans in particular seem to indulge in a culture of comparison, competition, and
rankings. In the USA, the college admissions testing competition has become a kind of
fetish. This was poignantly illustrated in a New Yorker cartoon: a man standing at the
gates of heaven exclaims to Saint Peter, ‘‘No kidding! You count SATs?’’ (Twohy, 1991).
4. How can and should the decentralized and highly segmented ‘‘system’’ of
postsecondary education in this country be affirmed – a system that may be regulated
but is not run by state or federal governments? According to David Brooks, the essential
quality of this ‘‘system’’ is its inherent competitiveness, inherited through its ‘‘expansionist
genes.’’ American higher education has never been state-dominated or state-run, as is
the case in most of the rest of the world. ‘‘The competitive American universities not only
became the best in the world – eight out of the ten top universities are American – they
also remained ambitious and dynamic. They are much more responsive to community
needs’’ (Brooks, 2006). Our strength is that the states and the federal government have
supported autonomy and private creativity.
5. And, most critical, once these questions are resolved and changes have been made, how
will higher education be different – for students, faculty, graduates, governing boards,
policy makers, employers, and citizens?
There are no simple, final answers to these questions; they provide a framework for the
following consideration of national, state-based, and institutional work on accountability,
accreditation, assessment, and testing.

National trends
Since 2005, the national dialogue on the future of higher education initiated by US
Department of Education Secretary Margaret Spellings amplified attention on learning
outcomes. The Commission report, A Test of Leadership: Charting the Future of U.S. Higher
Education, called for a much stronger emphasis on assessing student learning, and higher
education associations, systems, institutions, and others have already begun to anticipate
and develop a response (US Department of Education, 2006).
The Spellings Commission on the Future of Higher Education focused attention on a broad
series of goals to improve student preparation and access, cost and financial aid,
transparency and accountability, innovation, life-long learning, and global leadership. These
are being pursued through a 30-item action agenda in which accreditation and assessment
loom large.
In discussing learning assessment, the Spellings Commission report recommended that
educators and policy makers act to:

j j
PAGE 58 ON THE HORIZON VOL. 16 NO. 2 2008
B Measure learning with quality-assessment data from tests like the Collegiate Learning
Assessment (CLA) and Measure of Academic Proficiency and Progress (MAPP).
B Provide incentives to states and others for interoperable outcomes-focused
accountability systems.
B Collect data in ways that allow meaningful state comparisons.
B Ensure that faculty are at forefront in defining educational objectives and developing
measures.
B Make learning assessment results, including value-added measures, available to
students and public.
B Administer National Assessment of Adult Literacy more frequently (every five years) and
broaden the sample to include graduates of 2- and 4-year colleges and universities.
The ‘‘hook’’ to get much of this done was embedded in the report’s accreditation
recommendations: agencies should make performance outcomes, including completion
rates and student learning, the core of their assessment. The accreditation framework
should:
B allow comparisons among institutions on learning outcomes;
B encourage innovation and continuous improvement;
B require institutions/programs to move toward world-class quality;
B report measurable progress in relationship to national and international peers; and
B make the accreditation process more open and accessible to public.
The US Department of Education began actively pursuing this aspect of the action agenda in
fall 2006. The Secretary hosted a national forum on accreditation in late November (www.ed.
gov/about/bdscomm/list/hiedfuture/index.html), and it was a primary theme of the action
agenda for the summit on higher education convened by the Department of Education on
March 22, 2007. Possible adjustments to accreditation regulations were considered through
four negotiated rule making sessions in February through May, 2007 (www.ed.gov/news/
pressreleases/2006/11/11292006.html).
Through these activities and ensuing national press coverage, the Commission did succeed
in building more awareness of these issues. But, even before the Commission was formed,
higher education associations had been working on new approaches to assessing learning,
in response to accreditation requirements for structures, process, and evidence of more
student learning assessment. This has been a serious concern for the past two decades and
there has been some progress. For example, among regional accrediting agencies, the
Middle States Commission on Higher Education’s guidelines for learning assessment and
institutional effectiveness provide a good example of the progress and clear communication
about frameworks for assessing student learning (Middle States Commission on Higher
Education, 2005). The quality of this Commission’s approach to assessment was attested in
the recent recommendation by the Department of Education committee that oversees
accreditation to extend the Middle States Commission recognition for a period of five
years[2].
Policy makers, however, do not seem to focus on the success stories. Instead, what they
seem to remember is that educators have been highly resistant to assessment. ‘‘Our sector
was dragged into outcomes measurement kicking and screaming. No one wanted to do it,
no one knew how to do it,’’ recalled Elise Scanlon, executive director of the Accrediting
Commission of Career Schools and Colleges of Technology[3]. While her statement referred
to national accreditors and proprietary schools, resistance pervaded all sectors of
postsecondary education. This resistance has a long history – fifty years ago, Rourke and
Brooks (1966, p. 8) described the belief that ‘‘educational outputs cannot be measured, and
that any attempt to do so is ludicrous if not actually subversive of the purposes for which
academic institutions exist’’.

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 59
 Many policy makers today believe we are still kicking and screaming, despite the mounting
evidence of thousands of accreditation reviews, national meetings, publications, news
stories, and use of assessment at institutions across the country. There is a big perception
and communication gap, and this begs the question – even if the policy makers believed all
colleges and universities were assessing outcomes – would what we are doing satisfy them?
Some critics of higher education accountability insist, for example, that accountability is a
‘‘myth,’’ if it does not include actions and consequences based on results (Carey, 2007).
The mantra in accreditation-focused assessment has been, ‘‘grades are not enough; tests
are not enough.’’ Other evidence – learning portfolios, capstone courses, and other
embedded activities – was to be included. Through the 1990s, the American Association of
Higher Education partnered with the North Central Association, running national seminars
and working with institutions, all on a voluntary basis. The focus was on individual institutions
or programs, that each would design its own goals and measures.
What was missing? Vincent Tinto, professor and chair of the Higher Education Program at
Syracuse University, said at the November 2006 National Symposium on Postsecondary
Student Success, ‘‘with all due respect to my colleagues, one might argue that we already
have sufficient research on student success . . . .what is missing in our view is the ability to
transform the knowledge that we have into practical knowledge’’ (Redden, 2006).
Margaret Miller, director of the Center for the Study of Higher Education at the University of
Virginia’s Curry School of Education, said at the January 2007 meeting of Council for Higher
Education Accountability that much of this work was good but failed to ask the question,
evidence ‘‘compared to what?’’ She has been persuaded that after two decades of trying to
assess college (as contrasted with student) performance, it is not sufficient to gauge a
college’s success based only on information about itself. ‘‘The answer to the question, ‘How
well are we doing?’ really depends on an answer to prior questions: ‘Compared to what?
Compared to whom?’’’ (Lederman and Redden, 2007).
Some groups anticipated this shift. For example, the Council for Aid to Education partnered
with higher education associations and institutions to develop and pilot test a ‘‘value-added’’
direct approach to learning assessment, to measure the higher-level, integrated cognitive
skills that should improve between the time of college matriculation and graduation. The
result of this effort is the Collegiate Learning Assessment (CLA), which has been
administered for the past three years (www.cae.org/content/pro_collegiate.htm).
But, following the Spellings Commission, activity has increased. The American Council on
Education (ACE) sent a letter to members in September 2006, on behalf of a consortium of
associations representing most of postsecondary education in the USA, essentially
accepting the challenges posed by the Commission and proposing to address the renewed
need for change. Its purpose was to reaffirm independence of approaches – to do all of this
voluntarily but with more coordination and communication (www.acenet.edu/AM/Template.
cfm?Section ¼ Search&template ¼ /CM/HTMLDisplay.cfm&ContentID ¼ 19857).
The National Association of State Universities and Land Grant Colleges (NASULGC) began
just around the time the commission was formed to develop its own Voluntary System of
Accountability (VSA). Supported in part by the Lumina Foundation, the VSA project has
grown into a national effort in partnership with the American Association of State Colleges
and Universities (AASCU), with five working groups on student/parent information; core
educational outcomes; student engagement; technical work on learning outcomes; and
technical work on student growth outcomes. These groups engaged in an in-depth analysis
of indicators of success, of available learning assessment instruments, and of the potential
for some instruments to be modified by test-makers for broad use by hundreds of institutions
in this accountability framework.
For example, the VSA core educational outcomes task force charge was ambitious: to focus
on how the academy can develop useful and credible evidence of the value that higher
education adds for its students in terms of core educational learning outcomes and student
growth outcomes. The purpose is to provide evidence that:

j j
PAGE 60 ON THE HORIZON VOL. 16 NO. 2 2008
B convinces the current skeptical audiences that what is measured really amounts to the
addition of important value for the student;
B facilitates comparison among institutions; and
B is reliably useful in helping higher education institutions go about making programmatic
improvement.
The product, now called the ‘‘College portrait,’’ is a template that provides specific
indicators of student outcomes, including elements such as graduation rates and
post-graduation education or employment plans, as well use of a choice of three specific
learning assessment tests (www.nasulgc.org/NetCommunity/Page.aspx?pid ¼
280&srcid ¼ 236). Completed in summer 2007, the template was approved by the
NASULGC and AASCU membership in fall 2007.
This progress is viewed as just a beginning. In September 2007, the US Department of
Education announced a $2.4 million award to NASULGC, AASCU, and the Association of
American Colleges and Universities (AAC&U), to investigate in depth the validity of the
measures used to assess critical thinking and writing by the three instruments used in the
VSA project, plus two others. The grant will also support development of rubrics to use
student portfolios to assess student learning, and to develop surveys that assess
non-cognitive growth during the college (www.nasulgc.org/NetCommunity/Page.
aspx?pid ¼ 280&srcid ¼ 236). And, as with James Traub’s September 2007 story, ‘‘No
gr_d_ate left behind,’’ the national press is beginning to report success stories about use of
these tests and the growth of a culture of assessment (Traub, 2007).
Other groups are creating their own approaches. In September 2007, a coalition of on-line
colleges announced a ‘‘Transparency by design’’ project to assess student outcomes at the
program level, focusing on ‘‘principles of good practice’’ (see Lederman, 2007a). Also in
September, AAC&U announced the launch of its new VALUE initiative, to document, assess,
and strengthen student achieving of essential learning outcomes[4].
Points of stickiness, if not full-blown resistance, remain. For example, the American
Association of Universities, which represents the nation’s most elite research universities,
agreed in spring 2007 to collect and aggregate data on undergraduate performance.
However, the AAU project will leave to individual institutions the decision to publish their
unique data (see Lederman, 2007b). In June, the National Association of Independent
Colleges and Universities (NAICU) announced the creation of its own accountability web
site, which will publish profiles of hundreds of member colleges (www.naicu.edu/
member_center/id.669,Secure.False/memberNews_detail.asp).
These approaches make consumer information more readily available in a consistent format,
but side step the issue of comparing institutions, and neither initiative includes data on
assessment of students’ learning. David L. Warren, president of NAICU, has been an
outspoken critic of ‘‘the very underlying concept of comparability, that the Spellings
Commission proposed,’’ because missions of the private colleges are ‘‘too varied and too
complex to be captured by any broad-based tests’’ (Basken, 2007). And George Kuh (2007)
labeled the VSA project a ‘‘grand experiment.’’ He warns that ‘‘above, all, we need to try
different approaches, do no harm, and refuse to allow pundits to declare winners and losers.
There is too much at stake’’.

Improving accreditation
In order to talk more specifically about learning assessment and testing, it is necessary to
address accreditation. Accreditation is considered by many academics to be one of the
most expensive and least value-added processes that universities are required to engage
in. It may be largely invisible to policy makers and to prospective and enrolled students most
of the time, but it looms as a large burden for many university and college administrators and
faculty involved in assessment work.
For many programs and institutions, the return on investment in accreditation is the quality
seal of approval that accreditation confers. Yet, accreditation, whether regional, national, or

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 61
 professionally specialized, is perceived as expensive and schizophrenic – focused
simultaneously on regulatory compliance (to prevent fraud in use of federal financial aid) and
on quality assurance and improvement.
What is broken? The Spellings Commission and other critics point out the following
concerns[5]:
B Lack of comprehensible standards.
B Lack of consistency in applying standards within accreditation associations and at the
federal level.
B Bureaucratization and politicization of the accreditation process.
B Barriers to competition caused by overly-rigid standards.
B Undercapitalization – scarce resources of accreditors and institutions to improve the
focus on student learning.
B Lack of public information about performance.
B Internal focus on individual institutions – no comparability or ability to examine progress
‘‘compared to what?’’
B Lack of objective data (of learning, of valuable skill sets, that the learning was obtained).
B Focus on inputs (resources and processes) more than outcomes of higher education.
B Unquestioned role of accreditation as a gatekeeper for federal financial aid and, by
extension, the viability of many postsecondary institutions.
For many, the key underlying policy question is: who should and will control quality in higher
education? Again, quoting Margaret Miller:

If we can. . . look at ourselves carefully and rigorously, I think there’s a very good chance that we
will be able to control the terms in which this question is answered. If we can keep this question
within our own control, we will do something that K-12 was unable to do, to everybody’s great
sadness (Lederman and Redden, 2007).

To date, that has been the position of the Department of Education. For example, at the
annual meeting of ACE in February 2006, Undersecretary of Education Sara Martinez Tucker
told members that they are the professionals, ‘‘probably in the best position to understand
what’s necessary to ensure that we have more access and better results’’ (Lederman,
2007c).
And, through negotiated rule making in February through May 2007, the Department of
Education and a group of non-federal negotiators from postsecondary institutions and
accrediting associations looked at ways within the existing policy and legal framework to
improve – to focus, clarify, align, and make more consistent – the application and
implementation of the accreditation standards. For example, a frequent observation about
accreditation is that the standards already have a lot that can be used to focus on student
learning and to establish benchmarks of performance, but that the standards are not used
fully.
Initially, the Department hoped that successfully negotiated rules would have provided a
head start to making improvements within the parameters of existing legislation. However,
the negotiators failed to reach consensus. The Secretary could have proceeded to issue
new rules even without consensus, but members of the key House and Senate committees
that oversee appropriations and authorization for education programs pushed back. In early
June 2007, the House passed a budget bill covering the Department of Education that
included a provision that would prohibit the department from promulgating, implementing, or
enforcing new federal regulations related to accreditation (www.insidehighereducation.com/
news/2007/06/08/accredit). And, later in June, 18 of the 21 members of the Senate Higher
Education Labor and Pensions Committee wrote to Secretary Spellings, asking her to
postpone issuing any new regulations until after the reauthorization of the Higher Education

j j
PAGE 62 ON THE HORIZON VOL. 16 NO. 2 2008
Act is completed (www.insidehighereducation.com/news/2007/06/18/senators). The
Secretary responded that she would delay as requested[6].
Throughout these activities at the national level, no one had been advocating a
‘‘one-size-fits-all’’ approach, or comparing everyone to everyone else. There has been no
consideration of federally mandated exit testing, or high stakes testing, or of federalizing
higher education, despite the headlines[7]. Secretary Spellings emphasized this position –
supporting variety by insisting on more and better information – in remarks before the
National Committee on Institutional Quality and Integrity in December 2007, where she said
that she has:
. . .been proud to lead delegations of university presidents to Asia and Latin America. Everywhere
we go, we talk about the diversity of American higher education – diversity that makes us the
envy of others around the globe.
We at the US Department of Education have never, and would never, work to undermine that
diversity or excellence. I attended the University of Houston. It would be nearly impossible to
equate my experience with yours at Boston College, Crystal, or with the different experiences of
many other students nationwide.
Every institution has its own unique strengths and attributes. But on behalf of consumers, be they
students, families, or institutions, we have the right and the responsibility to ask for more and
better information (www.ed.gov/news/pressreleases/2007/12/12182007.html).

It is clear that the Secretary of Education continues to believe it is important to measure

student learning and communicate outcomes to consumers, and is considering
incentives for colleges, states, and consortia that are willing to report on it. And, there
seems to be growing agreement that academia’s current approach to assessing student
learning is highly elaborate, labor intensive, and inward looking and does not really
serve internal or external interests in communicating to the public how colleges and
universities add value.

The testing debate

Certainly, there is not yet agreement that standardized tests must be used in all student
outcome frameworks, although this is a strategy that has been more broadly discussed.
However, resistance has remained high among groups that view the department’s actions as
a signal and a strategy to shift the ‘‘authority for defining and judging academic
quality. . .from institutions and accreditors to the federal government’’ (Eaton, 2007).
Despite the mantra that ‘‘tests are not enough,’’ and the fact that most admissions offices
and faculty at most institutions use tests like the ACT, SAT, GRE, LSAT, etc., to help evaluate
prospective students, much of the national and state debate has focused lately on
standardized testing and comparability. Resistance has been strong to using the same
types of tests to evaluate what happens once students are in the institution.
Margaret Miller, in ‘‘The legitimacy of assessment,’’ distinguishes three overlapping
purposes of testing and assessment (Miller, 2006):
1. To certify student achievement – assessing individual student achievement and potential
so that the student is accountable and pays, for example, in taking the SAT or the LSAT.
2. To improve programs – this focus increased in the 1980s through federal and state
accreditation – asking are academic content and quality acceptable? In this framework,
faculty and institutions are accountable – this is largely the focus of assessment experts
like you; it is largely internally focused, and the institution pays, although external
instruments like licensure exam results taken by individual students can be used to inform
the analysis.
3. To improve and assess institutions as a whole – focusing on holding institutions
accountable to constituencies and asking: is the investment paying off? And, the
institution pays.
Now we are experiencing a shift to a fourth level, to improve the outcomes of postsecondary
education at the state and national level – the accountability is owned by policy makers

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 63
 asking: is progress taking place at the state level? At the national level? Are we competitive
globally? At this level, data collected for other purposes may be selected and aggregated,
and it requires methodological care to ensure the validity of conclusions. Various
instruments, like the National Assessment of Adult Literacy or Organization for Economic
Co-operation and Development global rankings, may be useful individually in showing
comparative trends in American basic literacy and numeracy skills, but their samples, data,
and methodologies are quite distinct. The studies have garnered headlines and the attention
of the Spellings Commission, because American trends do not appear to be keeping up with
other OECD nations (for a recent example, see Petrie, 2007). More attention will be needed
to create and validate nationally and globally valid and meaningful measures.
How would testing fit into these frameworks? Richard Shavelson, one of the architects of the
Collegiate Learning Assessment, offers a positive approach in ‘‘Assessing student learning
responsibly: from history to an audacious proposal’’ (Shavelson, 2007). He emphasizes that
there is no single test that can be used to assess all of the important things we hope a
student gains during college. And, standardized tests are not a magic bullet; they do not
pinpoint problems; at best, they signal where a problem may exist. External assessments
must, according to the CLA framework, be used in the context of a range of
institution-specific assessments and other processes. At the same time, one of the
attractive features of the CLA is its capacity to provide an indicator of value added – what the
difference in results from freshman to senior test-takers suggests about the value the
campus adds to its learners.
As with any instrument, the CLA has limitations. It is a sampled approach; it does not
necessarily test the same cohorts; because it focus on broad cognitive abilities and is not
domain-specific, it does not directly relate to the college curriculum; and it still comparatively
new and needs to be replicated over several cycles to verify trends[8]. But it is among the
most promising tools available to get at this change and make comparisons across
institutions. Shavelson’s fundamental conclusion from a review of 35 years of attempts to
assess learning is that the ‘‘cultures of evidence do not lead to educational improvement if
what counts as evidence does not count as education or only counts as part of what we
expect from a college education.’’ To address this concern, he concludes by proposing
‘‘audaciously’’ that the CLA approach be combined with assessment of specific academic
programs, to tap knowledge and reasoning within a discipline or to get at individual and
social responsibility outcomes as content within the critical reading, writing, and
problem-solving tasks.
By contrast, Trudy Banta (2007), a doyenne of the assessment research movement, issued
‘‘A warning on measuring learning outcomes’’, based on her experience over the past 25
years and research going back to the 1960s. She warned:

A substantial and credible body of measurement research tells us that standardized tests of
general intellectual skills cannot furnish meaningful information on the value added by a college
education nor can they provide a sound basis for inter-institutional comparisons. In fact, the use
of test scores to make comparisons can lead to a number of negative consequences, not the least
of which is homogenization of educational experiences and institutions.

For Banta, this potential consequence calls into question the greatest strengths of American
higher education – its variety of opportunity and lack of central control. She concludes that,
while tests can be helpful in initiating faculty conversations about assessment, her research
casts serious doubt on the validity of using standardized tests of general intellectual skills for
assessing individual students, then aggregating their scores for the purpose of comparing
institutions.
A recent white paper by Steve Chatman on the difference between institutional and
academic discipline measures of student experience, based on a study of nearly 60,000
respondents, urges that assessments take into account disciplinary differences. Chatman
argues that aggregate measures (as with the CLA) disguise important distinctions in
learning and instruction among different disciplines (Chatman, 2007).

j j
PAGE 64 ON THE HORIZON VOL. 16 NO. 2 2008
My own ‘‘audacious proposal’’ is that these points of views should not be allowed to cancel
one another out; that serious researchers continue to develop and refine instruments; that
institutions continue to use them and objectively evaluate how effective they are in answering
the policy questions and contributing to institutional improvement; and that accreditors and
policy makers create incentives and practice some patience while this work goes on.
Perfection will certainly be the enemy of making progress in this debate.

At the state level: public policy and institutional initiatives

Higher education accountability is an ongoing concern for state officials and policy makers
who must deal with pressures to control the growth in state budgets while appropriating
sufficient funding for institutions to be accessible, to grow, and to compete. Recent
statements include the State Higher Education Executive Officers (SHEEO) National
Commission on Accountability in Higher Education and its 2005 report, Accountability for
Better Results: A National Imperative for Higher Education (www.sheeo.org/account/
comm-home.htm). But, the states have been developing systems of accountability for
postsecondary education for many years. A recent study by Peter Ewell and Marianne
Boeke found that 40 states administer some version of student unit record data systems
covering public institutions, covering an estimated 81 percent of the nation’s total enrolled
students (Ewell and Boeke, 2007).
Following on recommendations of the Spellings Commission, the National Conference of
State Legislatures issued a report in November 2006, Transforming Higher Education:
National Imperative – State Responsibility, recommending that ‘‘state legislatures must take
a leading role in reforming higher education, to make college more accessible and to better
meet state needs in a competitive global economy’’ (Fischer, 2006). Commenting on the
report’s findings, Texas Representative (and chair of the House committee on higher
education) Geanie Morrison said:
Too often, higher education policy is made in reaction to the latest budget crisis rather than on the
basis of long-term strategic goals. The Texas legislature is trying to do the latter: set clear
expectations and develop data collection systems . . . .And base appropriations on progress
toward goals – retention, graduation, increase accessibility and affordability.

The Texas case demonstrates that at the institution and state level, attention on
accountability has been high, and that progress can be achieved even in the midst of
ongoing policy debates.
In 2001, The University of Texas System first began development of an approach to learning
assessment and, in 2002 system decided voluntarily to include some information about
outcomes in its new, accountability report. By 2003, it had made the decision to include the
results of assessment surveys and tests in its accountability framework. This framework
extends to the nine universities and six health institutions in the System, with a total of over
190,000 students. Since the U.T. System has used a form of testing – through sampling only
of undergraduates at the nine universities in the System – for three years for institutional
accountability, the nervousness about the idea is familiar. The experience and transparency
have, however, been positive and has gained national attention for U.T. System institutions
whose test results show performance above expected levels (Carey, 2006; Kantrowitz, 2006;
Ewell, 2006).
In The University of Texas System accountability framework, student outcomes are a major
focus, with trend data displayed by institution on the following measures (The University of
Texas System, 2006-2007):
B 1st year persistence;
B 4-, 5-, and 6- year graduation rates (in addition to the accountability data, the Board gets
quarterly updates on progress, and a separate, in-depth annual analysis of the progress
toward graduation rate goals set for each of the nine universities in the system);
B graduation rates of community college students who enter UT System institutions with at
least 30 credits to transfer;

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 65
 B composite persistence and graduation rates for students who matriculate at a UT System
institution but go elsewhere in Texas to complete their education;
B licensure exam pass rates for key fields (teaching, engineering, nursing);
B National Survey of Student Engagement (NSSE) data (an in-depth report is made
separately, as well, to the UT System Board of Regents. For the accountability report,
measures are extracted regarding satisfaction with advising, the students’ overall
assessment of their college experience, and whether they would ‘‘attend again’’ if given
the chance);
B results of Collegiate Learning Assessment – this is administered to random samples of
100 freshmen and 100 seniors. Results are summarized, displayed and discussed in the
accountability report; the Board receives a separate, in-depth report as well; and
B data on postgraduation experience – employment or enrollment in a
graduate/professional program after graduating, provided by the Texas Higher
Education Coordinating Board.
After piloting other approaches, the U.T. System selected the CLA as the System-wide tool
because it provides a window on students’ development of higher order cognitive skills in a
holistic framework that transcends specific academic disciplines, and benchmarks results in
comparison with similar institutions (Millet et al., 2007). While there continues to be debate
about the concept, methodology, and validity of this instrument, as noted above, the UT
System experience has been largely positive and helpful, and suggests that it is worth
having the policy discussion and clarifying the purpose and use of this approach (Haurwitz,
2007).
The University of Texas System-level accountability framework is constant, but the
consideration of trends is not ‘‘one-size-fits-all’’; instead, progress is evaluated in related to
each campus’s individual mission, regional location, and types of students who attend. The
data are made readily available to institutions and the public: the accountability report is
posted on the Web and sent to legislators and hundreds of other people in Texas and
nationally. Institutions use the data in their internal planning and improvement efforts,
amplified by more in-depth campus-based assessments which can cover domain-specific
knowledge, soft skills, or other topics, in addition to the individual quality enhancement plans
required by Southern Association of College and Schools (SACS) for regional accreditation.
This is the level at which faculty can and must become engaged. For some institutions, the
results of the CLA tests are embedded in these broader institutional assessment plans. The
UT System also looks at the trends when it evaluates presidents and when proposals are
considered by the Administration and the Board about specific investments in the
campuses. In other words, the information is used for strategic decision making and
institutional improvement.
This accountability and assessment work connects, in addition, with the accountability
system that the Governor of Texas initiated in 2004 for all public postsecondary institutions
which included a goal of linking accountability data to state incentive funding (www.
txhighereddata.org/Interactive/Accountability). During the 80th Texas legislative session in
2007, Texas Governor Rick Perry put forward a bold but ultimately unsuccessful proposal to
set aside incentive funding for production of degrees, weighted for at-risk students and
critical fields[9]. This framework would have included performance on exit – but not high
stakes – exams that every graduating student would have to take. The results would have
been aggregated (weighted for a number of students’ ‘‘at risk’’ factors), and institutions
would have received incentives based on the overall institutional performance compared to
national standards, signaling a high bar for all.
The policy issue this proposal was attempting to address – how efficiently and effectively do
particular degree programs prepare students compared to the same degree programs at
other schools – mixed individual, program, and institutional accountability. Thus, the focus
was to be on discipline-specific tests but would also have included licensure exams and
others that could be designated – perhaps GREs, LSATs, etc. Existing, nationally- or
state-normed tests would be used; none would be created for this purpose. Each student

j j
PAGE 66 ON THE HORIZON VOL. 16 NO. 2 2008
would just have to take one test for this purpose. The institution would pay for the exams. The
key was in the details, too much remained to be worked out, and the proposal was not
included in the final versions of higher education legislation passed this year.
Over the same period, the Texas Council of Public University Presidents and Chancellors
(CPUPC) developed a proposal for a different kind of incentive program. The key operating
principles were that, first, base funding, growth, and cost-of-living increases would be
covered with new incentive funding above that level. Then, based on a survey of all
institutions, the proposal was to consider a small group among standard performance
indicators: graduation and persistence rates; enrollment, including diversity; administrative
efficiencies; community college transfers; faculty instruction of lower-division students; and
research measures[10]. Comparisons would be to prior individual institutional performance
– not in comparison to other institutions. A simpler version of this approach was adopted by
the legislature, with responsibility given to the Texas Higher Education Coordinating Board to
work out the details of the incentive plan. By October, 2007, working groups had devised,
and revised draft formulas to allocate among all public universities in Texas $20 million in
incentive funds, based on a formula focusing on measures of increased success over the
previous three years of ‘‘at risk’’ college students in overall degrees, and degrees in critical
fields[11].

Final questions and observations

What do these national and state examples suggest about the future of higher education
accountability and learning assessment? What will be different about postsecondary
education if the changes being considered now take hold? What will be the intended
consequences? What about un-intended consequences?

What role will institutions play in helping states and the nation become more competitive? For
example, in its October 2006 survey of global higher education, The Economist predicted
that the meritocracy gap will grow, not get smaller, between those with good educations and
those without (Wooldridge, 2006). Where should the bar be set, in terms of state-level and
national policy? Should we aim to get more or even most people to some mean standard? Or
should we aim to prepare more people to excel? In this same survey, The Economist
reported that ‘‘most people in America will accept inequalities if they are coupled with
equality of opportunity’’. Do we agree? Is this the right policy? The Economist concluded that
‘‘the rise of a global meritocracy offers all sorts of benefits from higher growth in productivity
to faster scientific progress.’’ And, if policy makers do not think they can afford both equity
and excellence – and neither of these is free – how do they choose? What is the best return
on the public investment? Only if those choices are understood, can we decide what would
be the most critical and useful assessment and outcome goals and measures. If, instead, we
just keep assessing and testing because it seems to be a good idea generally or a
convenient policy tactic, I predict continued dissatisfaction with the process and its utility.

What about testing? If regional and specialized accreditation continue along current
directions, current approaches are likely to remain useful on some level, because they are
the best available now. We might, however, expect to see more radical changes in the
business of testing and assessment – and this is a big business. For example, the National
Center on Education and the Economy predicted in its 2006 report Tough Choices or Tough
Times that moving from how we test now to capture other qualities (creativity, innovation,
facility with ideas and abstractions, self-discipline, organization to manage one’s work,
ability to work in teams, etc.) ‘‘will entail a major overhaul of the American testing
industry. . .with revised curricula, syllabi, teacher training . . . But it all starts with the
standards and assessment’’ [emphasis mine] (National Center on Education and the
Economy, 2006).
Answers to these questions also depend on who the customer is for the information – is it the
student, the institution, policy makers, the public? What kinds of assessments and tests can
serve more than one purpose and more than one customer group? The emphasis on
benchmarking and broad-scale comparisons will grow. The focus on institutional, as

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 67
 contrasted with individual, accountability will increase. We might see development of more
interrelated tests along the continuum of milestone events in postsecondary education:
two-year college ready; need for remediation; four-year college ready; transfer ready;
employment ready; graduate/professional school ready; career-change ready. . . A related
policy question is to what extent should assessments be incorporated into the regular
curriculum, within capstone courses, on transcripts, etc., and linked to the institutional
accreditation reports?
Can we benefit from our experiences and successes? Policy makers and educators should
focus on gathering, critically analyzing, and improving the best practices, based on the 20
years of experience in assessment[12].
Educators and policy makers should resist the urge to leap to a single, common measure or
instrument. Although it is tempting to simplify, particularly for public audiences, there is no
‘‘magic bullet’’ test; there is no ‘‘sleep number’’ that will give a single good housekeeping
seal on an institution[13]. Multiple measures are critical, and need to be collected and
discussed in relation to one another. Measuring everything in a disaggregated way is not
much more useful than measuring nothing. Let us figure out how to combine critical
indicators and integrate data into a clear framework that is meaningful to students, faculty,
administrators, policy maker to help know what kind of results a campus is getting, and to
help improve those results.
We must resist the urge for perfection – the next generation of accountability, assessment,
and testing must focus on analysis and improvement, not punishment, and not high stakes.
People within institutions will use the information and imbed it at many levels – in academic
programs, institutional planning, accountability reports, in deciding about rewards or other
consequences, and to communicate externally. The more the findings are used, the more the
methodologies, instruments, and reporting formats will be improved.
Although we must honestly evaluate the quality of the approaches being used, we must also
resist the urge to be hyper-critical, to focus on what will not work and why not. Let us
embrace opportunities to figure out what we already have that can be improved and what
might be added or changed to produce useful, meaningful information at every level of
analysis – for the student, the program, the institution, the states, and the nation as a whole.
And, lets reformulate the debate from one framed as a black-and-white, either-or dichotomy,
to thoughtfully evaluate suites of linked, complementary assessment approaches and tools.
Finally, now that the words accountability and assessment are nearly household terms, as
experts, I hope higher education professionals will seize opportunities to be transparent, to
share and publish data, take credit for success, and be forthright about addressing areas of
weakness. As one university president said in a speech to his faculty assembly, ‘‘the public
has every right to expect results. Our challenge is to meet these expectations head on and
do a much better job explaining our value’’ (Spaniolo, 2006). We need to participate actively
in the national dialogue, not defensively. The future of accountability and assessment is in
our hands.

Notes
1. For an in-depth study of higher education accountability and assessment from a number of
important policy perspectives, see Burke and Associates (2005).
2. National Advisory Committee on Institutional Quality and Integrity, May 30-31, 2007 meeting notes.
3. A statement made during the January 2007 Council for Higher Education Accountability (CHEA)
annual meeting. See Lederman and Redden (2007).
4. ‘‘AAC&U Takes Assessment to the Next Level with Launch of New Initiative, VALUE—Valid
Assessment of Learning in Undergraduate Education,’’ September 5, 2007 press release. http://
www.aacu.org/press_room/press_releases/2007/value.cfm
5. A recent critique is the report Why Accreditation Doesn’t Work and What Policymakers Can Do about
It, by the American Council of Trustees and Alumni, July 2007. http://www.goacta.org/publications/
Reports/Accreditation2007Final.pdf Addressing ‘‘seven deadly sins’’ of accreditation, the report

j j
PAGE 68 ON THE HORIZON VOL. 16 NO. 2 2008
 was timed as was an earlier and similar report by the same organization in 2002 (Can College
Accreditation Live Up to Its Promise?) to influence the Higher Education Act reauthorization
process.
6. Copy of letter to Senator Kennedy distributed as personal communication to nonfederal negotiators,
June 22, 2007.
7. Secretary Spellings has repeatedly countered these claims, as in her remarks at the June 2007
regional summit on higher education held in Boston: ‘‘Over the past year, my Department has
worked with the higher education community to inject more transparency into the system.
Unfortunately, there’s been a lot of misinformation about our work... some of which you’ve probably
heard like – We’re federalizing higher education, we’re creating a "one-size-fits-all" measure of
quality, we’re trying to circumvent Congress and create new laws. None of this is true, but there’s a
lot of confusion around our proposals.’’ http://www.ed.gov/news/pressreleases/2007/06/
06142007a.html

8. An accessible discussion of the logic behind the CLA is provided in a white paper by Roger
Benjamin, Mark Chun, and Richard Shavelson, ‘‘Holistic Scores in a Sub-score World: The
Diagnostic Logic of the Collegiate Learning Assessment (CLA),’’ Council for Aid to Education 2007.
http://www.cae.org/content/pdf/WhitePaperHolisticTests.pdf. In Steven Klein, Richard Shavelson,
Roger Benjamin, and Roger Bolus, ‘‘The Collegiate Learning Assessment: Facts and Fantasies,’’
CAE 2007, the authors respond to the main critiques of the CLA. http://www.cae.org/content/pdf/
CLA.Facts.n.Fantasies.pdf
9. A summary of the failed legislation and arguments pro and con may be found at: http://www.hro.
house.state.tx.us/focus/major80.pdf, p. 127.
10. Private correspondence, CPUPC.
11. The funds would be allocated for FY 2009; variants of the formula’s components and weightings are
still under discussion by policy makers.
12. My recommendations align in many ways with those that Lee Shulman has discussed. see Shulman
(2007)
13. Caution about narrow definitions of student success abound. A recent discussion and summary of
these challenges appears in papers prepared for the 2006 National Symposium on Postsecondary
Student Success. See Ewell and Wellman (2007)

References
Banta, T.W. (2007), ‘‘A warning on measuring student outcomes’’, Inside Higher Education, January 26,
available at: www.insidehighereducation.com/views/2007/01/26/banta
Basken, P. (2007), ‘‘A year later, Spellings report still makes ripples’’, The Chronicle of Higher Education,
28 September, available at: http://chronicle.com/weekly/v54/i05/05a00101.htm
Bok, D. (2006), Our Underachieving Colleges: A Candid Look at How Much Students Learn and Why
They Should Be Learning More, Princeton University Press, Princeton, NJ.
Brooks, D. (2006), ‘‘Our world cup edge’’, The New York Times, June 22, available at: http://select.
nytimes.com/2006/06/22/opinion/22Brooks.html?_r ¼ 1&hp&oref ¼ slogin
Burke and Associates, J.C. (2005), Achieving Accountability in Higher Education, Jossey-Bass,
San Francisco, CA.
Carey, K. (2006), ‘‘Is our students learning?’’, Washington Monthly, September, available at: www.
washingtonmonthly.com/features/2006/0609.carey.html
Carey, K. (2007), ‘‘Truth without action: the myth of higher-education accountability’’, Change,
September/October, pp. 24-9.
Chatman, S. (2007), ‘‘Institutional versus academic discipline measures of student experience: a matter
of relative validity’’, Center for Studies in Higher Education Research & Occasional Paper Series
CSHE.8.07, available at: http://cshe.berkeley.edu/publications/docs/ROP.Chatman.AcadDisciplines.6.
5.07.pdf
Eaton, J. (2007), ‘‘Institutions, accreditors, and the Federal Government: redefining their ‘appropriate
relationship’’’, Change, September/October, p. 19.

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 69
 Ewell, P.T. (2006), ‘‘From the States: smoke signals from the Hills of Texas’’, Assessment Update, Vol. 18
No. 3, pp. 12-13,15.

Ewell, P. and Boeke, M. (2007), Critical Connections: Linking States’ Unit Record Systems to Track
Student Progress, Lumina Foundation for Education New Agenda Series, January, available at: www.
luminafoundation.org/publications/Critical_Connections_Web.pdf

Ewell, P. and Wellman, J. (2007), Enhancing Student Success in Education: Summary Report of the
NPEC Initiative and National Symposium on Postsecondary Student Success, NPEC, Washington, DC.

Fischer, K. (2006), ‘‘Panel urges state lawmakers to take the lead in higher-education reform’’, The
Chronicle of Higher Education, December 8, available at: http://chronicle.com/weekly/v53/i16/
16a02202.htm

Haurwitz, R.K.M. (2007), ‘‘UT system applauds standardized testing’’, Austin American Statesman,
February 9, available at: www.statesman.com/news/content/news/stories/local/02/09/9test.html.

Hersh, R.H. and Merrow, J. (2005), Declining by Degrees: Higher Education at Risk, Palgrave
Macmillan, Basingstoke.

Kantrowitz, B. (2006), ‘‘Is your college a lemon?’’, Newsweek/MSNBC, October 9, available at: www.
msnbc.msn.com/id/15078635/site/newsweek/

Kuh, G.D. (2007), ‘‘Promises and pitfalls of institutional transparency’’, Change, September/October,
p. 35.

Lederman, D. (2007a), ‘‘College accountability movement moves online’’, Inside Higher Education,
17 September, available at: www.insidehighered.com/news/2007/09/17/adult

Lederman, D. (2007b), ‘‘AAU gets on board’’, Inside Higher Education, May 7, available at: http://
insidehighered.com/news/2007/05/07/aau

Lederman, D. (2007c), ‘‘Into the lamb’s den’’, Inside Higher Education, February 14, available at: www.
insidehighereducation.com/news/2007/02/14/tucker

Lederman, D. and Redden, E. (2007), ‘‘Accountability and comparability’’, Inside Higher Education,
January 31, available at: www.insidehighereducation.com/news/2007/01/31/compare

Middle States Commission on Higher Education (2005), ‘‘Assessing student learning and institutional
effectiveness: understanding middle states expectations’’, available at: www.msche.org/publications/
Assessment_Expectations051222081842.pdf

Miller, M.A. (2006), ‘‘The legitimacy of assessment’’, Chronicle of Higher Education, September 22,
available at: http://chronicle.com/weekly/v53/i05/05b02401.htm

Millet, C.M., Stickler, L.M., Payne, D.G. and Dwyer, C.A. (2007), ‘‘A culture of evidence: critical features
of assessments for postsecondary student learning, ETS 2007: provide a careful overview and analysis
of 12 assessment instruments currently available’’, available at: www.ets.org/Media/Resources_For/
Policy_Makers/pdf/cultureofevidence.pdf

National Center on Education and the Economy (2006), Tough Choices or Tough Times, report of the
New Commission on the Skills of the American Workforce, December, available at: www.
skillscommission.org/executive.htm

Petrie, M. (2007), ‘‘Report says United States lags in key education indicators’’, The Chronicle of Higher
Education, September 28, available at: http://chronicle.com/weekly/v54/i05/05a03102.htm

Redden, E. (2006), ‘‘Out of step’’, Inside Higher Education, November 2, available at: www.
insidehighereducation.com/news/2006/11/02/success;http://nces.ed.gov/npec/pdf/symp_agenda_
FINAL_10_06.pdf

Rourke, F.E. and Brooks, G.E. (1966), The Managerial Revolution in Higher Education, Johns Hopkins
Press, Baltimore, MD.

Shavelson, R.J. (2007), ‘‘Assessing student learning responsibly: from history to an audacious
proposal’’, Change, January/February, pp. 26-33.

Shulman, L.S. (2007), ‘‘Counting and recounting: assessment and the quest for accountability’’,
Change, January-February, pp. 20-5.

j j
PAGE 70 ON THE HORIZON VOL. 16 NO. 2 2008
Spaniolo, J.D. (2006), ‘‘Demonstrating achievement through assessment and accountability’’,
The University of Texas at Arlington, University Faculty and Associates Meeting, October 12,
available at: www.uta.edu/president/about/speeches/20061012.php

Traub, J. (2007), ‘‘No gr_du_te left behind’’, The New York Times, September 30, available at: www.
nytimes.com/2007/09/30/magazine/30grading-t.html

Twohy, M. (1991), The New Yorker, September 9, available at: www.cartoonbank.com/product_details.

asp?mscssid ¼ Q1HEA7X25W5B8MQ84P43D1BFJ5V92X20&sitetype ¼ 1&did ¼ 4&sid ¼ 39229&pid
¼ &keyword ¼ SATs&section ¼ all&title ¼ undefined&whichpage ¼ 1&sortBy ¼ popular

(The) University of Texas System (2006-2007), Accountability & Performance Report, The University of
Texas System, Austin, TX.

US Department of Education (2006), Washington, DC, available at: www.ed.gov/about/bdscomm/list/

hiedfuture/index.html

Wooldridge, A. (2006), ‘‘The battle for brainpower: a survey of talent’’, The Economist, October 7.

About the author

Dr Geri Hockfield Malandra is Vice Chancellor for Strategic Management at the University of
Texas System. She serves as Vice-Chair of the U.S. Department of Education’s National
Advisory Committee on Institutional Quality and Integrity. In 2007, she also served as a
non-federal negotiator for the US Department of Education on accreditation, and as a
member of the NASULGC Voluntary System of Accountability Task Force on Core Learning
Outcomes. Since 2002, she has led the development, implementation, and improvement of
the UT System’s first comprehensive accountability and performance reports as well as new
approaches to strategic planning at the board, System, and institutional levels. Earlier, as
Associate Vice Provost at the University of Minnesota, where she also held an adjunct
assistant professorship in history, she led the creation of Minnesota’s first comprehensive
accountability reporting system. She received her BA degree from Carleton College, and her
MA and PhD degrees from the University of Minnesota. She can be contacted at:
gmalandra@utsystem.edu

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints

j j
VOL. 16 NO. 2 2008 ON THE HORIZON PAGE 71