Center for Justice & Economic Advancement
Normalizing Education Resource Center
Program Evaluation
Sandra Staklis, RTI International (Winter 2024)
An evaluation applies systematic methods to analyze whether a program or intervention achieves its goals and assess what works well and what could be improved. For education programs, evaluations can assess whether a program was implemented as intended and equitably, inform program development or improvements, and assess the effects of the program on student outcomes. For effective programs, evaluations can also provide convincing evidence of a program’s value to funders and policymakers. In the past two decades, multiple studies have contributed to a growing evidence base supporting the value of prison education programs for improving post-release employment rates and reducing recidivism (Bozick et al. 2018; Stickle & Schuster 2023) that helped build the case for Pell Grant reinstatement (Robinson & English 2017).
Evaluations can also examine the effectiveness of program components such as instructor training and student tutoring. In prison settings, an evaluation might assess the effect of work or housing assignments on student participation and success, or whether students continue their education after reentry. This brief describes the most common types of evaluations and factors to consider when planning an evaluation of a postsecondary education in prison program.
About JFF's Language Choices
Jobs for the Future strives to use equitable and inclusive language in all of our published content. Sometimes we share materials like these prepared by other organizations whose language choices differ from our own. In those cases, we use their terms to preserve accuracy. See our Language Matters Guide to learn more about the terms we use when we’re writing about education and employment programs for people with records of arrests, conviction, or incarceration.
Evaluation Types and Designs
The research design for an evaluation depends on the research questions—developed by program staff, funders, and, ideally, program participants—and on the program’s stage of development.
Process or formative evaluations assess whether a program is delivered as planned to the intended recipients and can also provide feedback to guide program development and improvement (Ross et al. 2018). During the development or implementation of a postsecondary education in prison program, an evaluation of the impact of the program on student outcomes is not feasible because the program may change significantly during implementation and enrollments may be too small to assess program effects. At this stage, a process or formative evaluation can provide actionable insights on implementation successes or challenges and identify equity gaps in design or delivery. For example, a 2020 evaluation of Second Chance Pell (SCP) pilot programs in Pennsylvania identified Pell Grant eligibility barriers among incarcerated individuals, such as state-imposed restrictions by conviction type, that contributed to lower-than-expected Pell Grant usage (Tahamont et al. 2022).
Summative evaluations (also referred to as outcome or impact evaluations) are conducted once a program has been implemented to assess a program’s success in achieving its stated outcomes and overall goals. Although the terms are often used interchangeably, outcome evaluations typically examine progress toward one or more of a program’s objectives, whereas impact evaluations seek to determine whether a program has achieved its longer-term goals and ultimate aims (GAO 2021; Lindquist & Martinez 2020; Rossi, Lipsey, & Freeman 2018). Depending on how a program’s outcomes and goals are defined, an outcome evaluation of a postsecondary education program in prison, for example, might analyze whether students continue their education after reentry and the types of students that are more likely to do so. An impact evaluation of the same program might examine whether the program increases participants’ reentry success, including degree attainment, employment, and recidivism. An impact study on individuals released from Minnesota prisons, for example, examined the effects of secondary and postsecondary degree attainment during incarceration on post-release employment and recidivism (Duwe & Clark 2014). The analysis found no effects for secondary degree attainment but found postsecondary degree attainment to improve employment outcomes and reduce recidivism. Outcome and summative evaluations can employ a variety of research designs and may include the collection and analysis of qualitative and quantitative data.
|
What is an Impact Evaluation? |
|
The term impact evaluation can be used in different ways. As described in this brief, impact evaluations might examine a program’s success in meeting its overall goals and objectives. In some contexts, impact evaluations refer to studies using causal research designs that include treatment and comparison groups, ideally selected through random assignment, to isolate the effects of a program on participants. The Institute of Education Sciences at the U.S. Department of Education, for example, encourages the use of causal or experimental research designs in evaluation studies (IES 2017), and some research organizations and funders set similar standards. Because impact evaluations can differ in approach, program staff should consult with funders and researchers to clarify expectations. |
Evaluation Readiness
Before undertaking either type of evaluation, researchers and program leaders should work together to determine the research questions that the evaluation will address and the research design and timeframe needed to answer them. A process or implementation evaluation can provide insights beginning at the early stages of program development.
Once a program is implemented, supportive conditions and infrastructure can enhance the effectiveness and rigor of a summative evaluation. In 2020, RTI International authored a resource brief for the U.S. Department of Justice, Bureau of Justice Administration on determining whether a reentry program is ready for a rigorous evaluation of its effects on participant outcomes. These five aspects of evaluation readiness are also relevant for assessing readiness for evaluations of other program types, such as postsecondary education in prison programs.
-
Established program
Creating or expanding postsecondary education programs takes time and experimentation to determine what is feasible and how best to respond to student needs. As noted above, evaluations of program outcomes or effectiveness are not feasible when a program is still being developed and implemented. If a program changes significantly while an evaluation is underway, researchers will not be able to determine which version of the program influenced student outcomes. During the implementation phase, an evaluation team might help the program development team document the implementation process and lay the groundwork for a future evaluation study.
-
Documented program model
Planning a summative evaluation typically includes the development of a logic model that describes the input, activities, and intended outcomes of the program to be evaluated. Logic models summarize how key program inputs and activities contribute to outputs and short- and long-term outcomes (McLaughlin & Jordan 2015). Logic models include detailed descriptions of program components such as funding and staffing, eligibility criteria, outputs (number of clients enrolled, number of sessions delivered, number of staff trained), goals and objectives, and participant and system-level outcomes.
-
Program size
Postsecondary programs in prison that serve a few dozen students a year may not have enough participants to support a rigorous evaluation of outcomes or impact. Small sample sizes make it difficult to determine whether program participants have better outcomes than people who do not receive the services. Many postsecondary programs in prison, however, anticipate increasing the number of students that they serve in the next few years with the reinstatement of Pell Grants. Although the number of participants needed depends on the evaluation design and type of program evaluated, an evaluator can help determine how many students are necessary to effectively measure outcomes.
-
Data availability
Evaluations can include the collection of data and the use of administrative data sources, such as data on the individual characteristics of the students participating in a program and the education and support services they received. These data may include demographic information collected by the department of corrections and the college, which can help identify equity gaps and provide information on students’ prior education, projected release date, course taking, and grades. For an evaluation to be successful, organizations that collect data will need to be willing to share the data with researchers, and the data must be of sufficient quality to support an analysis.
-
Leadership support
Evaluations may require extensive program information, staff time, and access to nonpublic data sources. Accessing the resources needed for an evaluation, therefore, requires leadership support. Do program and organizational directors understand the evaluation and what it entails? Has the evaluation been formally explained? Have program leaders been informed about the potential benefits of the evaluation? Have program leaders been informed about what must be in place for the evaluation to be successfully implemented (e.g., resources, staff time)?
-
Resources
The following resources provide more information about program evaluations, including equity-based approaches, factors to determine evaluation readiness, and the components and benefits of an evaluation.
This guidebook describes an equity-based approach that addresses the unequal power dynamics between researchers and the subjects of research, especially when the focus of the research is on marginalized communities. In addition to describing the ways that research can perpetuate inequities, the resource offers guidance for researchers (and evaluators) on recognizing unintended biases in their work and advice for community members on increasing their engagement and leadership in research and evaluation work.
Source: Chicago Beyond, 2018
Improving Evaluation Readiness for Reentry Programs
This brief describes the factors that program administrators and evaluators should consider when determining a program’s readiness for an outcome or impact evaluation. Although the examples in the brief are drawn from reentry programs, the brief highlights programmatic features, such as the size of the population served and data availability, that apply to a wide range of education and social service programming, including higher education in prison programs.
Source: C. Lindquist and Martinez, A., U.S. Department of Justice, Bureau of Justice Assistance, 2020
Reentry Program Evaluation Readiness Planning Guide
A partner publication to Improving Evaluation Readiness for Reentry Programs, this guide provides a tool for assessing a program’s readiness for an outcome or impact evaluation. The guide lists the five recommendations for evaluation readiness adapted for this brief and a set of questions that program administrators and evaluators can ask to determine whether each recommendation has been followed.
Source: C. Lindquist and Martinez, A., U.S. Department of Justice, Bureau of Justice Assistance, 2020
Demystifying Program Evaluation in Criminal Justice: A Guide for Practitioners
Developed by the Illinois Criminal Justice Information Authority and designed for criminal justice practitioners, this article provides an overview of key evaluation terms, concepts, and principles and practical tips for starting a program evaluation. The content describes how program evaluations can improve the efficacy and efficiency of programs and offers guidance on setting evaluation goals, developing logic models, and choosing a research design.
Source: Jessica Reichert and Alysson Gatens, 2019
This resource from the U.S. Department of Education offers education practitioners and evaluators practical guidance on conducting evaluations of local, state, and federal programs. The toolkit is composed of eight modules that include step-by-step guidance on different aspects of program evaluation, such as creating a logic model, developing evaluation questions, and reporting findings.
Source: J. Stewart, J. Joyce, M. Haines, D. Yanoski, D. Gagnon, K. Luke, C. Rhoads, and C. Germeroth, U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Central, 2021
-
Works Cited
- Bozick, R., Steele, J., Davis, L., & Turner, S. (2018). Does providing inmates with education improve post-release outcomes? A meta-analysis of correctional education programs in the United States. Journal of Experimental Criminology, 14(3), 389–428.
- Duwe, G., & Clark, V. (2014). The effects of prison-based educational programming on recidivism and employment. The Prison Journal, 94(4), 454–478.
- Institute of Education Sciences (IES). (2017). Evaluation Principles and Practices. National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/projects/pdf/IESEvaluationPrinciplesandPractices_011117.pdf
- Lindquist, C., & Martinez, A. (2020). Improving Evaluation Readiness for Reentry Programs. U.S. Department of Justice, Bureau of Justice Assistance. https://nationalreentryresourcecenter.org/sites/default/files/inline-files/evalReadinessBrief.pdf
- McLaughlin, J.A., & Jordan, G.B. (2004). Using logic models. In J.S. Whorley, H.P. Hatry, & K.E. Newcomer (Eds.), Handbook of Practical Program Evaluation (2nd ed., pp. 62–87). Jossey-Bass.
- Robinson, G., & English, E. (2017). The Second Chance Pell Pilot Program: A Historical Overview. American Enterprise Institute. https://www.aei.org/wp-content/uploads/2017/09/The-Second-Chance-Pell-Pilot-Program.pdf#page=2
- Rossi, P.H., Lipsey, M.W., & Freeman, H.E. (2018). Evaluation: A Systematic Approach. Sage Publications.
- Stickle, B., & Schuster, S.S. (2023). Are Schools in Prison Worth It? The Effects and Economic Returns of Prison Education. American Journal of Criminal Justice, 1–32.
- Tahamont, S., Hyatt, J., Pheasant, M., Lafferty, J., Bell, N., & Sheets, M. (2022). Ineligible anyway: Evidence on the barriers to Pell eligibility for prisoners in the Second Chance Pell pilot program in Pennsylvania prisons. Justice Quarterly, 39(2), 402–426.
- U.S. Government Accountability Office (GAO). (2021). Program Evaluation: Key Terms and Concepts. GAO-21-404SP. https://www.gao.gov/assets/gao-21-404sp.pdf