Implementing Outcomes-Based Quality Evaluation for Postsecondary Providers in Indiana

Revising EQOS’s Quality Assurance Framework to meet the needs of the Indiana Department of Workforce Development and piloting a new system of outcomes-based quality evaluation.

Introduction & Background

Across the country, leaders in postsecondary education and workforce development have leaned into strategies around data-driven decision-making to improve their programs and policies and increase learners’ and workers’ real-world success. The work being done in Indiana is a prime example of this trend. Representatives across state agencies have joined together, in conjunction with numerous national institutions, to identify available credentials of value, prioritize growth industries within the state, and collect relevant data on education and training providers that propel learners into gainful employment.

One aspect of this collaboration is a partnership between the Indiana Department of Workforce Development (DWD), the Education Quality Outcomes Standards (EQOS), and Jobs for the Future (JFF) focused on assessing and refining the collection of student outcomes data from education and training providers in order to facilitate results-driven decision-making related to determining eligibility for Indiana’s Eligible Training Provider List (ETPL). In this project, EQOS and JFF staff worked with state leaders and education and training providers to explore how the application of EQOS’s Quality Assurance Framework might allow for richer understanding of student outcomes, and might impact the way providers are selected for the ETPL. This report outlines the key findings from that research, and makes recommendations about how state policy leaders can continue to use data collection to improve ETPL criteria.

Background

The Indiana Pilot Program began in October 2020, and ultimately included representatives from EQOS and JFF, the Indiana Commission for Higher Education (CHE), the Governor’s Workforce Cabinet, and the DWD. The pilot program included four phases, outlined as follows:

Reviewed and standardized outcomes-based data collection processes for three quantitative metrics (completion, placement, and earnings) already included in Indiana’s ETPL requirements.
Developed updated definitions and measures of the five metrics contained in the EQOS Quality Assurance Framework (learning, completion, placement, earnings, and satisfaction) to better align with the needs of DWD and postsecondary training providers.
Piloted comprehensive outcomes data reporting processes with a cohort of four ETPL training providers:
Assessed results and provided recommendations for broader implementation of outcomes-based quality evaluation processes across the state.

The overall goal was to revise EQOS’s Quality Assurance Framework to better align with the needs of the DWD, training providers, and employers, as well as pilot a new system of outcomes-based quality evaluation to improve DWD’s ability to vet and intake eligible training providers.

Summary & Conclusion

Our research identified several components of the EQOS Framework where there were consistent challenges across providers in either generating the outcomes-based data internally or accessing that data from external sources. Those components include:

Assessment of Learning, where providers varied significantly in their definitions and uses of assessments, as well as their capacities for implementing assessments as effective measures of learning progress.
ROI, where providers used varying methods for calculating program ROI.
Survey-based data collection, which was used to gather measures on student and employer satisfaction, placement, and wage outcomes. All providers struggled to generate high levels of survey responses, which limited the collection of data in these measures of outcomes.

Broadly speaking, we found two categories of issues or challenges the providers faced in generating or securing this data:

Those where additional support and/or incentives are needed from the state for effective data collection across all providers, regardless of program type or scale (e.g., collecting consistent longitudinal data on employment and earnings over time).
Those that are tied to addressing specific capacity challenges within subcategories of providers (e.g., large, complex providers needing to build cross-departmental data collection processes, and smaller providers needing support in developing/maintaining contact with alumni).

The recommendations contained herein provide concrete actions and options that can be taken to further advance the outcomes-based data collection capacity for all providers, in alignment with the EQOS framework, as the DWD considers pathways for expanding this work across the state. We were encouraged by (and grateful for) the willingness and enthusiasm of the pilot providers to embrace this move toward outcomes-based measures and their openness and enthusiasm to begin using this data to identify areas for improvement in their programs on behalf of Indiana’s learners. It is a great foundation to build on for subsequent refinement and expansion as the DWD continues its work implementing outcomes-based standards for postsecondary providers in Indiana.