printable version of page Printer-friendly page

Focus On Basics

Volume 3, Issue C ::: September 1999

Focus on Research:
Documenting Outcomes

An action research project led by Beth Bingman at The University of Tennessee is one of NCSALL's research activities that focuses on learner outcomes. Bingman and her colleagues are working with three adult literacy programs to develop methods for documenting the outcomes that occur in learners' lives as a result of participating in adult literacy programs.

"I originally thought we'd be doing something like collecting outcomes and come up with processes to do that," explains Bingman, "but it's more complicated than that." First they had to develop agreement, within and across the three participating programs, on what constitutes an outcome. They decided upon this definition: outcomes are changes in learners' lives as a result of participating in adult basic education (ABE). To establish a common language for outcomes assessment across the sites, the research team and program participants then developed what they call the "Inputs to Impacts Grid." This grid, shown on below is used by the teams to analyze their programs' components.

Each program team also compiled a list of what their program had been documenting before the project began. They examined who does the documentation, for whom it is done, how often, how is it used, and how is it reported. "This was a useful process for everybody," comments Bingman. "We discovered that there were very few places where outcomes were being documented." The teams uncovered duplication in the overall documentation that could be eliminated and came up with ways to streamline processes.

The project has three sites, one in Tennessee, one in Virginia, and the third in Kentucky. The Tennessee site is the literacy division of an urban ABE program. They offer day and night classes for literacy level students and have a family literacy program. The Virginia team is from one county in a seven-county rural program. The three teachers involved teach multilevel classes in a housing project, a jail, the local library, a vocational school, and a night class at a high school. The Kentucky team is the staff of a county adult learning center. They offer General Educational Development (GED) and literacy instruction in the center and at a family resource center in their rural mountain county.

Different States, Different Approaches
Federal policies allow states to design their own accountability systems. Each program in this research project is therefore taking a different approach to documenting outcomes, reflecting the different approaches their states take to performance accountability. While the procedures they are testing are different, many of the issues they face are the same. Tennessee provides one example: "In Tennessee," says Bingman, "we are working with four teachers, each of whom started by trying different ways to document outcomes. For instance, someone tried monthly taped interviews with two students about what had happened in their lives. Another teacher used a teacher checklist and a reflection log. Someone else kept portfolios of evidence. It was incredibly time intensive to get what they wanted. They were paid for the extra time they spent to do this. No one felt it was going to work program-wide." One big challenge across the country is developing processes to gather information on outcomes that can be used in a system where the majority of teachers are employed part-time or are volunteers.

Then there's the question of who gets to say that something is an outcome. If a learner says "Yes, I'm doing this," does it count as an outcome? Is it substantial or rigorous enough? Can the information be believed if it is reported by the learner? As Bingman points out, "one of the concerns is a tendency to report what people want to hear. But we believe that learners are the ultimate authority on changes in their lives."

On the other hand, if students are asked to report outcomes, the question becomes, Why should they bother? For example, the teachers in the Virginia program tried collecting outcomes by using in-class activities such as stem sentences, in which they provided the beginning of a sentence, "Now I can…" and learners completed them, and story circles, in which learners talked about changes in their lives. The teachers weren't satisfied and the learners felt their time was being wasted. So, after some discussion, the teachers developed a checklist of possible outcomes. After learners tried the list and suggested changes, the list was revised. The team now plans to use the form, which they call "Do, Set, Met," for initial goal setting as well as for documenting outcomes. They hope that having a variety of outcomes to choose from will help learners name specific goals that they can accomplish. The "Do, Set, Met" goals will tie into the state reporting system.

Outcomes
The outcomes of this research project, a set of processes teachers can use to document outcomes of student participation, will have to meet criteria that address all these challenges. And it will also have to fit with existing policies and requirements, such as the reporting requirement of the Workforce Investment Act or state-level requirements.

"People will have to feel like they can do it in a reasonable amount of time," says Bingman, "and that it's real evidence, collected systematically".

The project is slated for completion by the end of the year. A final report will include suggestions of how local programs can use the processes developed by the action research teams. The teams also plan to meet with state ABE staff to share what they have developed.

For more information, contact Beth Bingman at The University of Tennessee, 600 Henley Street, Suite 312, Knoxville, TN 37996-2135; telephone (423) 974-4109, or e-mail her at ncsall@utk.edu

click here for:
Documenting Outcomes Inputs to Impacts Grid

Updated 7/27/07 :: Copyright © 2005 NCSALL