Project & Methodology

This pilot project was focused on developing a set of insights and recommendations relative to the data collection process itself, and on how to enhance and improve that process to be as successful as possible across a wide diversity of programs and providers across the state. It was not to use this initial data to evaluate or measure the specific programs or providers in any way, but to better understand how to develop and implement an outcomes-based data collection process at scale in IN. 

The pilot project has included the following phases:

Phase 1: Gap Analysis. 

EQOS and relevant state agencies performed a gap analysis of existing state processes and infrastructure for data collection relative to the five standards within the EQOS framework, and devised a scope of work to improve collecting/reporting outcomes data as defined by the framework. 

Phase 2: Subcommittee Working Group

Two subcommittee working groups were established to create a single set of definitions of outcomes for use in education and training programs across the state, in alignment with the EQOS Quality Assurance Framework for measuring student success.  

•  The first subcommittee working group was created to develop new measures on EQOS Metrics 1 and 5 (Learning and Satisfaction). 
• The second subcommittee working group was created to refine measures of EQOS metrics 2,3, and 4. (Completion, Placement, and Earnings)

The subcommittees developed a set of definitions that were used during the data collection phases of this pilot.

Phase 3: Recruiting Pilot Providers

EQOS and state agencies recruited and selected five training providers to collect outcomes data aligned with the five metrics of student success in the EQOS Quality Assurance Framework. The providers were selected in part to provide a wide range of program focus areas and provider size and complexity in order to be representative of the diversity of program providers across the state. The training providers included: 

• Eleven Fifty Academy (Coding/IT)
• Freedom Academy (Healthcare/Advanced Manufacturing) 
• Hardon Education (Healthcare)
• Hope Training Academy (IT)
• Ivy Tech Community College (State community college)

(Note: Eleven Fifty Academy dropped out of the pilot prior to the data collection phase due to internal operational challenges.)

Phase 4: Technical Assistance 

Through its partnership with JFF, EQOS provided technical assistance and guidance to the participating training providers to collect data across multiple cohorts of students and to report their student outcomes data according to the EQOS Quality Assurance Framework.

Phase 5: Data Collection

EQOS provided each training provider with an anonymized data reporting template to collect the target data across multiple cohorts of students in their programs via the following process:

1. Each provider selected two programs, and two cohorts of students who recently completed those programs, as the sources of data for this pilot. The goals were to provide:
1. Multiple sources of comparable data within each program.
2. A breadth of data collection experiences across programs and providers, to help understand and identify issues and challenges for data collection in a wide variety of program and provider contexts to inform data collection processes for future expansion of this work.
2. The providers then used two primary methods for collecting the data needed to complete the EQOS templates:
1. Data from existing provider systems/databases. This included data on student demographics, program completion and educational outcomes, credentialing, and ROI.  
2. Data gathered through surveys of alumni and employers. This included data on job placement, wages, and student and employer satisfaction.
3. EQOS and JFF met with each provider on a regular basis during the data collection process to identify issues, challenges, and successes with the process and to troubleshoot issues as needed in support of data collection.

Phase 6: Data Review

In addition to the meetings during the data collection process, EQOS conducted extensive reviews of the providers’ completed data templates as well as exit interviews to identify what worked well in the data collection process, and which areas or aspects of the data collection process need further refinement or modifications in preparation for potentially expanding the scale of these efforts across Indiana. 

Phase 7: Findings and Recommendations

With the goal of facilitating results-driven decision-making related to determining eligibility for Indiana’s Eligible Training Provider List (ETPL) front and center, EQOS then distilled the information gathered during the data collection and review phases into a set of findings and recommendations.  These findings and recommendations, as they relate to the data collection process for gathering outcomes-based data aligned to the EQOS Quality Assurance Framework, are summarized in Section 3 below.