Contested Ground: Performance Accountability
Center for Literacy Studies, University of Tennessee
NCSALL REPORT #1
Adult basic education (ABE) has long been viewed by many educators and policymakers as a tool for addressing social and economic problems. Now, in a context of global economic restructuring, changes in work and employment, and the largest immigration to the USA since the early 1900s, ABE must demonstrate its success in terms of student and societal outcomes. In short, ABE is facing demands to be accountable for its performance.
A focus on results is new for adult education, and potentially transforming. Performance accountability offers the chance to re-focus on what learners accomplish by participating in ABE and to re-orient every aspect of the system to achieve the best results.
It is tempting to rush into developing measurement and reporting systems. Experiences with performance accountability show that this would be a mistake. Agreement on what to measure must be established first.
This policy paper lays out key issues in performance accountability and presents recommendations for policy and action. It is based on the literature from education, government, management, and other fields, and draws on interviews with researchers and adult education leaders at state and national levels. Its recommendations were informed by a discussion with a group of experienced adult education practitioners and researchers.
Why change? What is the problem?
Past policy is no longer a guide for action. The 1966 Adult Education Act, which has governed ABE for more than 30 years, provides little guidance in developing performance accountability. The act charges recipients of funding with enabling adults to "acquire the basic educational skills necessary for literate functioning." Definitions of literate functioning vary and are hotly debated. Over time, views of what literacy means have shifted from academic skills, such as decoding text, to functional skills, such as being able to perform certain tasks using literacy skills. Recent research has changed radically our understanding of literacy. Literacy is now described as multiple "literacies" rooted in particular social contexts. This change in definition shifts the focus even further from abstract skills to real-life practices.
This conceptual change requires a re-vision of what "performance" means. When literacy meant mastery of what was taught in schools, performance was testable. When literacy shifted to the notion of functional competency - being able to perform certain tasks using literacy skills - the issue of performance became more complex. Tests had to identify which real-life literacy tasks should be included and which not, with no theory to guide which to choose or how to create scales of difficulty. Test developers had to assume that performance on the test equated with how well the student performed the real-life equivalent task. The recent research on literacy in its social context has been carried out through careful observations of literacy events and activities which shed light on prevailing literacy practices. While it shifts the focus to performance in life, not in test situations, this new research has not yet been incorporated into practice, assessment, or policy.
Agreement on what "literate functioning" means is crucial to accountability. Is literacy a right or a necessity for the good life? Should performance be demonstrated in terms of literacy skills - the earlier view - or literacy practices, the current thinking?
The purpose of literacy is not defined. The Adult Education Act of 1966 had broad social intent. In addition to not strictly defining what literacy was, it did not define the purpose of literacy. If a performance accountability system is to measure only literacy gain, the task of developing such a system would be challenging enough. If a system is to measure the achievement of literacy for a predefined purpose, a lack of clear objectives makes accountability systems even more difficult to develop. The debate in this area centers around whether the purpose of literacy education is individual advancement or community development. Are the desired outcomes productive workers, good citizens, or merely more literate people? Once these questions are answered, the next set of questions arise: who defines whether these outcomes are achieved, and how much responsibility should the literacy program be asked to take for these outcomes?
Stakeholders are not mutually accountable. Another area of concern lies in the mutual responsibility for adult basic education. Many possible stakeholders - learners and teachers, administrators and policymakers, funders, employers, public school personnel, and taxpayers - may be said to have a legitimate concern with the outcomes of adult literacy education. All stakeholders are not, however, equal in terms of access to information or ability and power to hold the adult education system accountable. Learners, for example, often have limited information and little power to change the system. Congressional representatives stand for taxpayers in exercising accountability over the adult education system which is supported by public money. Legislators are often not held accountable by learners or educators for providing adequate resources and policy guidance to the system.
Capacity is weak. Adult basic education is struggling to create a national accountability system without a national service delivery system. It is difficult to have a management information system when there is no management system.
Research reveals a fragmented and incomplete system with multiple funding sources and reporting formats, diverse institutions, competing objectives, and missing or unreliable performance data. In most states, staff are part-time, and volunteers continue to have an essential role in student services. Per-student funding is low, and most programs are not able to meet client needs for childcare or transportation. While data on performance are fragmentary, what there are suggest that most learners do not stay long, make some initial learning gains, but may not make long-term skill gains.
The capacity to perform - to achieve desired goals - is linked with the capacity to be accountable - to document achievements and measure results. Some states are beginning to use program performance data successfully to improve program services. More reliable accountability data are collected when they are used at the program level to meet program needs.
Measurement tools are not up to the task. Learning is at the heart of ABE, and its measurement is of particular concern to performance accountability. Adult education cannot be accountable to learners or to policymakers without the ability to track learning of individuals, to demonstrate what has been learned, to compare learning across programs, and to judge learning against external standards.
Yet standardized tests, the most widely used tools for measuring learning, have been criticized both by researchers and practitioners because they do not demonstrate what has been learned. They are also incompatible with new research-based conceptions of literacy as social practices rather than isolated skills. Some programs are using various "authentic assessment" tools, such as portfolios, but so far these cannot compare learning between learners and across programs. Without external standards or criteria, authentic assessment will not meet policy needs.
What change is happening?
New initiatives are addressing performance accountability. The problems of accountability are well recognized and are beginning to be addressed at national and state levels.
Equipped for the Future: The National Institute for Literacy's Equipped for the Future (EFF) project is a broad-based system reform effort that has actively sought input from a wide range of stakeholders. These include stakeholders outside the adult education system (policymakers, employers, and civic leaders) as well as within it (adult learners, teachers, program administrators, and researchers). EFF has developed and validated a set of four purposes for adult education and lifelong learning: to have access to information, to give voice to ideas and opinions, to solve problems, and to be able to continue learning. EFF has related these purposes to the three adult roles of worker, citizen, and family member. With 25 development partners across the country, EFF is now testing a set of common activities and defined skills which will form the basis for future development of standards and performance measures. When fully articulated and validated, these could provide a framework for performance as the basis for an accountability system.
National Outcomes Reporting System: This project is funded by the U.S. Department of Education and has involved many state ABE directors. It is developing a common set of outcomes for adult basic education as a basis for data collection and reporting. However, the process has so far had limited input from the full range of stakeholders.
Other state and national level initiatives: A number of initiatives at state and national levels are bringing stakeholder groups into closer relationships. A National Summit, proposed by the National Center for the Study of Adult Learning and Literacy (NCSALL), in partnership with NIFL and the Department of Education, is an example of one way to ensure stakeholder involvement in deliberations about the future of ABE at the national level. Local and state adult learner organizations are forming, increasing learner engagement around issues like state and federal funding. A national adult learner organization, VALUE (Voices for Adult Literacy United for Education), has recently been formed and shows promise for building a strong and effective student voice.
State capacity-building: Over the last few years, many states have been focusing on building capacity for program delivery. Some have worked specifically on building capacity for accountability as well. The cases of Arkansas, Connecticut, and Pennsylvania, reviewed in this policy paper, as well as other states like Massachusetts, Iowa, and California, all demonstrate the importance of long-term and systematic investment in training, information, and technical assistance.
Lessons from the business world: High performance or "learning organizations" from the business world may prove useful models for adult education. These emphasize continuous improvement and learning in all parts of the organization, responsiveness to internal and external customers, participation in decisions, and shared responsibility at all levels. Traditional models of accountability are linear ¤ quality control inspectors check widgets at the end of the production line to see if they meet specifications. Mutual accountability engages members of the organization in developing common vision, determining goals and customer expectations, and designing effective means of monitoring, producing, and improving.
What next? A Framework for Action
The policy paper recommends four principles which would enable the field to perform effectively and to be held accountable for performance. This framework for future policy builds on what has been learned about performance accountability in past experiences, and provides guidelines for future action. The principles are:
1. Agree on
2. Build mutual accountability relationships.
3. Develop capacity both to perform and be accountable.
4. Create new tools to measure performance.
1. Agree on performances
Good performance - what needs to be measured - is not a technical question, but inevitably in the realm of values. The challenge is to come to agreement on performance as a "big tent" which can include the full diversity of purposes.
Experience from business and industry suggests that it is crucial that performance be defined neither too tightly nor too loosely. If performance is defined too tightly, a mis-match between system goals and individual goals may occur. For example, learners' purposes for entering literacy education may be to read to their children, but the system may only measure whether they get a GED. If performance is framed too loosely, no shared mission or common accountability measures can be developed. Getting it right requires a broad-based and inclusive process involving multiple stakeholders.
Lessons from the literature and experience in education and other fields suggest:
- Don't assume the question of what performance means can be skipped over or rushed. Without knowing what is important, measurement becomes an exercise in "gaming the numbers" to satisfy external demands, often with perverse results.
- Involve stakeholders and seek consensus. Without broad public debate it is difficult to frame performance goals which reflect the "big tent."
- Reflect newer understandings of literacy, and connect performance with real life. This is an opportunity for literacy research to connect with and support practice.
- Acknowledge multiple performances. Too narrow or tight a definition of goals will exclude learners and programs or force them to falsify their data.
2. Develop mutual accountability relationships
Reforming accountability in high performance terms requires a switch from one-way, top-down lines of accountability to a mutual web of accountability relationships. To participate, stakeholders need information and the ability or power to hold others accountable. This entails greater transparency in information, increased flows of information, and room at the table for groups who have not been there in the past.
Bring the full range of stakeholder groups into the process - including teachers and learners who often have not been at the table.
Provide support for stakeholders who have least access to information and power, including adult learner organizations at national and state levels.
Increase information flows among and between all stakeholders. Develop learning organizations at the program and state levels which would emphasize learning and continuous improvement, shared responsibility, and engagement in monitoring results.
3. Build capacity to perform and be accountable
State experiences with capacity-building reviewed for this policy paper indicate that the two kinds of capacity - to perform and to be accountable - are linked. By developing a learning organization approach in which there are continuous feedback loops, performance data can help programs improve performance.
Build the capacity to perform. Key elements include increased resources, focusing resources on quality rather than quantity, staff development and training, technical support, use of performance data for continuous improvement.
Build the capacity to be accountable. Key elements include accountability demands which are commensurate with resources and capacity, engagement of users in developing better measurement tools, staff training and support, timely information loops, rewards for improved performance.
4. Create new tools to measure performance
Accountability systems must meet the different information needs of different stakeholders. To do so, data users and data providers must be in communication so that the most appropriate measurement tools can be applied.
Accountability systems commonly use several types of indicators to track performance over time. Input indicators provide information about the capacity of the system and its programs. Process indicators track participation in programs to see whether different educational approaches produce different results. Output indicators are short-term measures of results, and outcome indicators are long-term measures of outcomes and impacts. No single indicator can suffice to measure performance, especially of an enterprise as complex as adult basic education.
ABE invested a great deal of work into developing indicators of program quality, but much less on performance measures. New approaches and tools for measurement are needed which are linked to performance.
Develop external standards or criteria against which individual student learning can be measured and through which program performance can be assessed.
Develop performance assessment tools for measuring learning. These directly assess learners' performance in terms of literacy practices rather than the indirect approaches of standardized tests which "stand for" real-life practices, usually inadequately. Initiatives in performance assessment in countries such as Britain and Australia may provide useful models for measuring and assessing learning.
Use the full range of potential of research, evaluation, and monitoring technologies to meet the needs of different stakeholders. These approaches to gathering, analyzing, and using information are based on different kinds of data and meet different purposes. Using them in appropriate ways, adult education can develop a dynamic system of information, analysis, and reporting.
This framework of principles for action acknowledges that there are no quick answers. To put the principles into action requires consultation with the field and with stakeholders. It will need meetings and taskforces, and it will take time. It requires learning lessons from elsewhere when appropriate, building on current initiatives when they are under way, and creating new tools when none exist. Policymakers have the capacity to set the stage, harness resources, and create a common agenda. Commitment to high performance requires the contributions of many players.
Full Report Available
To receive a
copy of the full report, send a request indicating the
report number or title
along with a check or money order in the amount of $10, payable to World Education, to:
44 Farnsworth Street
Boston, MA 02210
or call: (617) 482-9485