Is My Ed Tech Tool Making a Difference?

An Entrepreneur's Guide to Using Research to Improve Products and Measure Impact

Introduction

Every ed tech entrepreneur wants to develop an amazing tool that may one day become a household name. At NewSchools, we invest in ventures that also care deeply about impact. Through our ed tech accelerator NewSchools Ignite, we help entrepreneurs use research to generate data that can inform their product and business strategies.

Based on these experiences, we’ve developed a guide designed for entrepreneurs at any stage of their research journey. This web interface pulls together many of the main points for a quick read. The full guide is available to download as a PDF.

Our guide examines a range of research types, noting those that are appropriate at various stages of development, on small or large budgets, and within varying timeframes. We believe it will also offer useful information for educators and funders.

Key research practices explored in this guide:

  1. Define the intended impact and create a logic model. Identify the student outcomes you believe your product supports, document how use of your product supports these outcomes, and gather feedback on your ideas from a diverse group of stakeholders.
  2. Iterate based on feedback from usability and feasibility testing. Observe how the product is used in a real setting  by teachers and students, and refine product features and supports based on what you learn.
  3. Evaluate evidence of student outcomes. Develop research questions and timelines that align with your product and business roadmaps. Be mindful of key considerations including product stage, costs, and return on investment.
  4. Share what you learn about impact – both the celebratory insights and the tough lessons. Synthesize different types of evidence to describe how your product can support student outcomes, focusing on key audiences like educators and funders.

Many types of research have value, and the entrepreneur’s research journey is not linear. Research practices don’t always fall perfectly into a sequential order, and each can have value as an independent undertaking.

Getting Started

So, how can an ed tech entrepreneur check in and collect evidence of impact in the midst of an iterative product development cycle? Sequencing and timing are important. Conducting an efficacy study before having confidence about how a product is being used in classrooms is probably not the best plan. However, it’s crucial to keep student outcomes top of mind, even at an early stage.

Through our ed tech accelerator NewSchools Ignite, we invest in tools that support student learning and integrate research services into our investment strategy. We identified external research partners – WestEd, which conducted product reviews and small scale studies, and Empirical Education, which offered ventures student user demographic reports. We build these costs – an average of approximately $40,000 per investment – into our venture support. The content and learnings in this guide emerged in part from this work as well as conversations with ed tech researchers, entrepreneurs, and funders.

NewSchools Ignite launched six ed tech challenges from 2015 to 2018, focused on products addressing critical student needs in Science Learning, Middle and High School Math, English Language Learning, Special Education, Early Learning, and the Future of Work. Through these challenges we funded 84 small-scale research studies, conducted by WestEd, primarily focused on generating formative product feedback.

For ed tech ventures outside our portfolio, we recognize this level of early-stage research may be cost prohibitive. In recent years, several organizations have explored the potential value of “rapid-cycle evaluation,” designed to “quickly determine whether an intervention is effective” while also enabling “continuous improvement.” Yet even this type of research requires significant financial and human capital resources, so it’s important to consider its costs and benefits as part of a sustainable research strategy.

The four key research practices are outlined below.

Want to learn more? Download the full Research Guide here.

Define intended impact + logic model

The first step in the research process is defining the student outcomes you believe your product can support. Next, developers can begin to consider short- and long-term metrics related to these intended outcomes.

Across NewSchools Ignite’s first six challenges, we observed a range of intended impacts including various academic, social-emotional, and career and college-ready indicators.

After defining intended outcomes, ed tech developers can begin to outline how access to and use of the product is connected to these outcomes. By explicitly defining potential use cases, the developer makes clear what is required to access the product, and how it should be used in order to achieve the desired results.

NewSchools Ignite virtual convening: “Logic Models: Ensuring Your Product has Impact”
(Source: WestEd)

Considerations of access, use and outcomes can then be formally documented through a logic model, which describes a product’s “theory of change” through the lens of potential inputs, activities, outputs, outcomes and impact.

Consider the logic model a living document; it will evolve as the team learns more and will likely be repeated multiple times over the life cycle of the product.

Iterate on usability + feasibility

At this stage an ed tech entrepreneur should be working to understand whether the product can be used and implemented successfully within its intended learning context. Depending on a venture’s internal capacity, these tests can be conducted by team members or by an external research partner, and can be either self-funded or supported through investments from foundations or impact funders.

Research studies focused on formative feedback


* – Study costs vary widely and depend on a range of factors including: study length, sample size, types of data collected, measurement tools used, hardware costs, types of analysis performed, staffing needs, travel costs, and researcher salaries.

Usability and feasibility testing can provide valuable information about potential improvements, which can be integrated into a venture’s product roadmap and value proposition.

Compared to summative research, which focuses on measuring outcomes, formative research is meant to gather data that can inform product and business strategy, and is relatively inexpensive and low-risk.

Across studies of products funded through NewSchools Ignite, evidence suggests making products easier to use can have a positive impact on student learning.

Want to learn more? Download the full Research Guide here.

Evaluate student outcomes

As the product matures, if it is performing well, there should be positive indicators that suggest it is worth investing more time and money into the product, including more rigorous research. At NewSchools, we define “rigorous evidence” as a randomized controlled trial (RCT) or quasi-experimental design (QED) study, conducted by an external researcher, that demonstrates positive student outcomes. The federal K-12 education law, Every Student Succeeds Act (ESSA) also stipulates that studies meeting its evidence standards must be “well-designed and well-implemented,” which places additional requirements on the study design. That being said, many types of evidence have value.

Research continuum

Study designs range from formative (on the left) to summative (on the right), with both rigor and cost increasing as you move from left to right.
(Source: WestEd)

As rigor increases, so generally does the cost of research. While investment in ed tech continues to grow, there will likely be additional resources available for research, but entrepreneurs need to be able to make the case that research is aligned with product and business goals.

Research studies focused on measuring student outcomes


* – Study costs vary widely and depend on a range of factors including: study length, sample size, types of data collected, measurement tools used, hardware costs, types of analysis performed, staffing needs, travel costs, and researcher salaries.

The table above provides information about what an ed tech entrepreneur can expect from conducting various types of research and why it should be done, including costs and potential returns on investments.

Share your ed tech impact story

Once the evidence is synthesized, it’s time to communicate how the product is impacting student outcomes. With respect to ed tech impact, even mature well-resourced companies may not have a good sense of what they should be sharing, with whom or how best to share it.

Potential outcome statements and descriptions by study type


* – Actual outcome statements are dependent on study design and results.

Before sharing your impact story, always start by determining the objective and the audience. The distribution channels must also be tailored to match the audience. Regardless of where the company fits on this spectrum, here are some guidelines for communicating:

  • Avoid embellishment and be sure any statements or claims about the product are accurate and can be substantiated.
  • Make optimal use of the channels the company can control, such as Twitter, Facebook, LinkedIn, the company website and blogs. Publish issue briefs or white papers with tight overviews. Use analytics to track reach and ROI.
  • Use language that is accessible and friendly. Avoid excessive use of jargon, acronyms and complex language. Focus on top-line findings and keep it simple. Research need not be obtuse.
  • Remember the power of first-person testimonials and storytelling to bring the research story to life.
  • Share research findings at conferences about education research, ed tech or PreK-12 education – being sure to tailor the message for different audiences.
  • Use third-party validators such as other researchers and thought leaders in education to amplify the message.

Conclusion: A Call to Action for the Broader Education Community

Our investments to support research have surely created value for our ventures and their products’ users. We hope this guide – which distills much of what we’ve learned about ed tech research – extends our collective knowledge to entrepreneurs beyond our portfolio, as well as to educators, researchers and funders who are investing resources into this important work.

  • Entrepreneurs must make it a priority to collect evidence. Remember that regardless of your budget or the product’s maturity, many types of research can be valuable – and not just to you, so share what you are learning.
  • Educators need to think of themselves as partners, not just users or customers. They must be clear about what they and their students need, and consider providing critical feedback to help entrepreneurs refine their products.
  • Researchers should put themselves in the seat of ed tech developers and educators. The product development cycle is dynamic and fast paced, and classrooms are full of students who are literally changing and growing every day. Ed tech entrepreneurs need information that is timely and actionable.
  • Funders need to fund ed tech research, building it into their overall investment. Early-stage ed tech ventures have the double challenge of needing to show impact while having limited resources and capacity available to measure it.

Want to learn more? Download the full Research Guide here.

Developed in partnership with