Volume 3, Issue B ::: June 1999
Translating Vision into Reality
by Agnes Precure
Oregon's accountability system is based on quality-of-life goals. But how does one measure "quality of life" and how does that translate into classroom practice?
In Oregon, accountability is led by a common vision known as Oregon Shines. Developed with input from a broad array of stakeholders, this blueprint for the state's future lays out three quality of life goals: (1) High-Quality Jobs for All Oregonians; (2) Safe, Caring and Engaged Communities; (3) Healthy Sustainable Surroundings. What sets Oregon Shines apart from other states' strategic plans is that it is supported by 92 benchmarks: measures of our progress toward the three quality of life goals. Each state agency is responsible for one or more benchmark. Data collected from the systems managed by the state agencies are fed to the Oregon Progress Board, a body created by the state legislature in 1989 to track benchmarks. The Progress Board then reports the results to the Governor, the legislature, and the citizens in a "report card." These benchmarks enable the government to provide a clear message to the people of Oregon about how state agencies and the systems they represent are doing in their efforts to reach the goals described in Oregon Shines.
The challenge has not been one of agreeing on the vision but on how to measure it. For benchmarks to become meaningful indicators of progress, they have to be measurable. So Oregon's benchmarks for adult basic education (ABE) are not framed in terms of quality of life; instead, they are the percentage of adults with a high school diploma or equivalent, and the percentage of adults reaching 'intermediate" levels of literacy. This means that our system outcomes come down to the types of things that ABE programs have been reporting to the federal government for more than 20 years: how many certificates of General Educational Development (GED) or Adult High School Diplomas earned, and how many students made skills gains.
Although this sounds simple enough, it's not. Decisions flowing from these benchmarks affect every aspect of our system. For example, to measure "intermediate literacy" Oregon's Office of Community College Services/Job Training Partnership Act Administration (OCCS/JTPA), the agency that administers ABE programs, had to define it better. We at OCCS/JTPA asked questions such as: Which skills define intermediate literacy? Reading? Writing? Math? All three? Do we want to know whether people have these skills or whether they know how to apply them? Apply them to what types of tasks? With the help of the Progress Board, we chose the 1990 Oregon Survey of Adult Literacy Survey (a derivative of the National Adult Literacy Survey) as our model. We defined intermediate literacy in terms of the ability to apply literacy skills to prose, document, and quantitative tasks. We set our baseline for those areas, then added two more categories: writing and oral communication (speaking and listening). We are still working with the Progress Board to determine how we will establish a baseline and measure progress for these additional measures.
So far, I have only described the policy around accountability. The answers to these policy questions, however, have a significant impact on our ABE programs. How we define the benchmarks determines how many people fall into the intermediate category, what has to be done to increase the number of adults at this level, and how we'll measure our progress. In other words, decisions about how to define and meet the Benchmarks translate directly into classroom issues such as recruitment, curriculum development, retention, assessment and reporting. Finally, in collaboration with Oregon's ABE program directors, we at OCCS/GTPA had to set targets for improvement, targets for which the programs in our system are responsible. This is the center of Oregon's ABE accountability system and serves as the focus for the ABE professional development system provided by the state.
For the remainder of this article, I will illustrate how accountability policy affects Oregon's teachers and professional development system by discussing two specific examples: the implementation of the Comprehensive Adult Student Assessment System (CASAS) and the implementation of the Tracking of Programs and Students (TOPS) data management system. Finally, I'd like to address some of the challenges raised by these choices.
To keep the broader vision of Oregon Shines in focus, Oregon's ABE programs adopted an instructional methodology that emphasizes teaching the basic skills of reading, writing, computing, listening, and speaking within the context of personal and career development and emphasizes the use of real life situations. In the late 1980s, we realized the need for an assessment system linked to this philosophy, to allow us to assess learner progress in the ability to apply basic skills in life or employability contexts. We chose CASAS, which also provided us with a core competency list that forms the basis for curriculum and instruction. For example, the life skills reading competency might require a learner to use reading skills to answer questions related to instructions for a telephone answering machine. For the employment skills competency, learners might answer questions related to an office supply catalogue. These competencies drive curriculum, instruction and assessment. They help learners see the link between the basic skills they are learning and the real life tasks to which they will apply these skills outside of the classroom. CASAS implementation totally changed the way Oregon's ABE programs delivered basic skills instruction.
Prior to CASAS implementation, instruction was almost entirely workbook based, focused on the remediation of discrete skills. Each program measured progress differently, and we had little sense of whether we were showing results as a system. Now Oregon's ABE programs use realia - real-life artifacts brought into the classroom -, conduct small group instruction, and talk about the types of things that our learners can do, rather than their grade levels. Teachers can clearly demonstrate to themselves and their learners when skills gains have been made. The result is that Oregon's ABE classrooms have become more vital and exciting as learners collaborate on meaningful projects, using workbooks for practice rather than as the core of instruction. Needless to say this, shift required a lot of training. The implementation of CASAS virtually defined our professional development efforts throughout the late 1980s and early 1990s. Training modules, ABE conferences, and curriculum development projects all centered around the implementation of competency-based instruction, group instruction, cooperative learning, and the application of life skills to real-life situations.
Measuring Learner Progress
The need to take accountability to the next step - that of looking at systems results and how they support our progress toward the Oregon Shines benchmarks - led to the implementation of the Tracking of Programs and Students (TOPS). TOPS is a computerized information management system we use to document individual learner progress. TOPS has replaced past approaches to accountability that relied entirely on program self-reports and a fair amount of guesswork at the end of each year. Using TOPS, we collect demographic information and learner gains for state and federal ABE reporting. Although TOPS is used primarily by the ABE and state and local Corrections systems, it has the capability to provide data matches with other agencies. These matches allow us to show how basic skills education relates to other learner outcomes related to employment, further education, and increased self-sufficiency. The TOPS system also includes a supplemental form that can be used to identify workforce readiness and workforce maturity skills.
Implementation of TOPS has not only helped us to better track skills gains, it has also allowed Oregon's ABE programs to gather the quality-of-life data that mirror the family work and community goals described in Oregon Shines. This is because instructors insisted that TOPS reflect those "other outcomes" that are typical of ABE classrooms. Progress in these family, work, community, and educational outcomes is linked directly to learner goals at program entry. This has meant a number of changes in the way that programs manage their intake processes. For TOPS to track progress effectively, programs have to collect more specific information about learner goals at intake. This information is causing some teacher to rethink what they are teaching and how their curriculum and instruction connects with learner goals.
As we become more able to collect reliable data using TOPS, we will be better able to plan for program improvement and new approaches to professional development. Professional development efforts focused initially on making sure that teachers knew how to put information into TOPS and how to get reports back. We've modified our assessment and instructional training modules to include TOPS information. The next step is to work with program directors and instructors to develop training on how to use the data for program improvement purposes.
Oregon's approach to ABE poses some special challenges. One is the challenge of using a contextualized approach to teaching those with rudimentary literacy skills. Teaching literacy skills in context is often confusing for emergent readers, who still need to master the basics. On the other hand, it is difficult to make beginning reading instruction meaningful and engaging. The trick lies in developing the right mix of discrete skill instruction and the application of skills. Oregon's programs are participating in a national study to examine these issues and develop approaches that will more effectively serve these emergent readers. This may lead Oregon programs to develop more specialized approaches to serving adults at lower literacy levels.
Another challenge is how to allocate time and resources. Better data collection is a time-intensive process. The implementation of TOPS has required increased time and resources for orientation and intake that have traditionally gone to instruction. This is likely to result in the restructuring of many Oregon programs. In addition, early results from TOPS data have pointed out some weaknesses in our current assessment practices, especially for limited English speakers. This means we will have to help programs to rethink how and when they currently assess students for placement and progress. Because TOPS collects detailed information about learner goals, questions about how better to meet those goals and show student progress are already being voiced by instructors. This too will lead to new professional development offerings as teachers work to become more skilled at meeting learners' needs.
The need to show outcomes is important not only for accountability to the Governor, legislature, and other stakeholders, but also for learners, instructors, and program administrators. All of Oregon's state and federally funded ABE programs are implementing TOPS. Everyone is rethinking how to allocate resources for intake and assessment. TOPS implementation has raised philosophical questions that challenge the very mission of ABE. We are examining whether it is more important to provide access to the greatest number of learners or to make sure that we are providing the best services possible for those we serve.
For me the greatest challenge lies in making sure that the vision of Oregon Shines that led to the benchmarks and increased accountability doesn't get lost in the implementation. How do we avoid losing the learner in the messages about accountability and systems reforms? This challenge and the ones above will be worth addressing. The answer to these questions will help reinforce the connection between "real life" outcomes and the work of Oregon's adult basic skills providers and their partner agencies.
Oregon Progress Board (1997). Oregon Shines II
About the Author
Agnes Precure is the Professional Development and Curriculum Specialist for the Office of Community College Services/Job Training Partnership Act Administration (OCCS/JTPA) in Salem, OR. Prior to her work for the state of Oregon, Ms. Precure was the National Literacy Project Director for Wider Opportunities for Women in Washington, DC.