Measuring and Validating Impact

When launching new technology partnerships, discussions on impact measurement and data collection should happen early to ensure that partner organizations are in alignment on data definitions, desired impacts, and existing data policies, sources, and processes. A useful framing exercise is to consider what outcomes you wish to achieve and what data you’ll need in order to tell a compelling success story. Then, determine which metrics each partner can provide and what types of data usage or data sharing agreements must be developed to ensure safe and secure use of the data. 

A good impact story is informed not just by program staff or participants, but by all of the stakeholders involved in an initiative. It requires a diverse collection of narratives and perspectives, and those narratives and perspectives must, ideally, be supported with reliable outcomes data. 

Once you’ve engaged your stakeholders in determining your desired impacts and the potential stories that may result from the initiative, you should coordinate with your data team and your partners’ data teams to determine whether or not you have the data needed to support these stories. It’s important to document this process along with the team’s anticipated data assets and needs because these should be included in any data use or sharing agreements required for the partnership. Once a project launches, it can be difficult to amend executed data agreements. Finally, consider the capacity and funding needs of each organization to ensure that there are adequate resources to support data management and a strong evaluative analysis of the initiative. 

Specifically, remember to do the following:

• Write data requirements into contracts with all parties
• Have data sharing agreements to make sure you know how and where data is flowing

To help shape your data story, consider some of the questions and sample answers provided here as examples of the foundations of impact stories that could potentially result from a tech-driven partnership:

1. Q: What is the story you want to tell? What are the most important narratives and how do they coalesce around certain perspectives?
   A: Understanding the supportive services required to help people overcome barriers to completing digital skills training. 
   
   
2. Q: Who is part of telling that story? What stakeholders should provide input?
   A: Community-based organizations that serve individuals who face significant barriers to educational access and persistence, workforce center case managers, local community colleges, and learners 
   
   
3. Q: How will you tell the story? What data points are important to gather? Through what mechanism will you collect them?
   A: Program persistence rate (source: training platform), barriers to employment (source: instructor/career coach, learner survey), number of learners receiving a service and levels of service (source: service providers/community-based organizations)

Keep in mind that these are strictly examples of potential questions and answers. Depending on the nature of your partnership, you may need to include additional questions to fully encompass the depth and breadth of your data needs. In addition, these questions may require multiple conversations and iterations before you arrive at a final decision that enables you to tell a story that accurately reflects the outcomes of the initiative and the impact it made in your community.