Evidence & Evaluation
The Social Innovation Fund (SIF) is one of six federal tiered innovation and evidence initiatives that prioritizes evaluation and building evidence of effectiveness. The focus on evidence in SIF goes beyond confirming that the funded programs achieved their intended outcomes and impacts; it includes understanding how they are successful and how they can be improved. By developing a strong evaluation process, putting resources towards supporting quality evaluations and encouraging our grantmaker intermediaries and subgrantees to do the same, the SIF increases the evaluation capacity of all stakeholders and transparently offers best practices and lessons learned to the broader social sector.
A demonstrated track record of using evaluation to make programmatic decisions is a requirement for SIF grantmaker intermediaries. In addition, all funded programs being implemented in communities must complete a rigorous evaluation to strengthen their base of evidence and to document and assess whether their approach works more effectively, costs less, or leads to better results for our country’s communities. Through its investment in evaluation, the SIF will increase evidence on what works to support economic opportunity, boost youth development, and promote healthy futures.
What are Evidence Tiers?
The SIF relies on a framework that organizes evidence levels into three categories: preliminary, moderate, and strong. This framework is very similar to those used by other federal evidence-based programs such as the Investing in Innovation (i3) program at the Department of Education
Preliminary evidence means the model has evidence based on a reasonable hypothesis and supported by credible research findings. Examples of research that meet the standards include: 1) outcome studies that track participants through a program and measure participants’ responses at the end of the program; and 2) third-party pre- and post-test research that determines whether participants have improved on an intended outcome.
Moderate evidence means evidence from previous studies on the program, the designs of which can support causal conclusions (i.e., studies with high internal validity) but have limited generalizability (i.e., moderate external validity) or viceversa - studies that only support moderate causal conclusions but have broad general applicability. Examples of studies that would constitute moderate evidence include: (1) at least one well-designed and well-implemented experimental or quasiexperimental study supporting the effectiveness of the practice strategy, or program, with small sample sizes or other conditions of implementation or analysis that limit generalizability; or (2) correlational research with strong statistical controls for selection bias and for discerning the influence of internal factors.
Strong evidence means evidence from previous studies on the program, the designs of which can support causal conclusions (i.e., studies with high internal validity), and that, in total, include enough of the range of participants and settings to support scaling up to the state, regional, or national level (i.e., studies with high external validity). The following are examples of strong evidence: (1) more than one well-designed and well-implemented experimental study or well-designed and well-implemented quasi-experimental study that supports the effectiveness of the practice, strategy, or program; or (2) one large, well-designed and well-implemented randomized controlled, multisite trial that supports the effectiveness of the practice, strategy, or program.
How Does the SIF Apply these Evidence Tiers?
Preliminary evidence is the threshold for entry. All programs funded by SIF must demonstrate a minimum of preliminary evidence of effectiveness. SIF grantmaker intermediaries, who select programs through a competitive subgrant process, must ensure that the programs they fund also meet this minimum evidence requirement.
Once funded, programs must build on their level of evidence. A program must conduct a rigorous evaluation by partnering with an independent evaluation team that will help build the evidence supporting its effectiveness and potentially move it to a higher tier of evidence.
Programs with higher levels of evidence are prioritized for greater expansion. We expect programs with stronger levels of evidence will receive more financial support (i.e. larger grants) so that they can scale up their programs. Scale for organizations with preliminary levels of evidence will allow for limited expansion to support evaluation efforts by testing interventions with new populations or locations, whereas scale for organizations with moderate and strong levels of evidence will allow for more substantial growth to provide services to larger numbers of people in the current or new geographic area(s).
How does the SIF Support Evaluations?
The Social Innovation Fund provides grantmakers and their subgrantees with access to a team of evaluation experts. These experts work with each grantmaker to review and provide feedback on their evaluation design strategies to ensure that these designs meet the program’s evidence goals in a highly rigorous fashion and lead to the development and implementation of quality evaluations.
SIF and its evaluation contractors provide grantmakers and subgrantees with guidance documents and ongoing feedback as they develop their evaluation plans, implement them, and report on results. The SIF promotes a culture of learning and support among its grantmakers and shares learnings with the field. The SIF sees evaluation as essential for program improvement, not just for grantmakers but for the broader philanthropic and nonprofit field. That is why the SIF is committed to sharing information about its evaluation approach and outcomes, as well as lessons learned along the way.
What does SIF Evaluation Investment Look Like?
As of January 2014, SIF has invested over $59 million towards the design and implementation of 72 approved program evaluations. This evaluation program is managed by SIF and monitored by independent evaluation experts who check each SIF evaluation study for rigor and quality.
Type of Research Design
SIF evaluation designs can be placed into three categories: experimental, quasi-experimental, and non-experimental.
Experimental design studies using random control trials or RCTs assign program participants to two distinct groups (at random): the treatment group, which receives program services, and the control group, which does not. The control group is called the “counterfactual,” representing the condition in which the program or intervention is absent. Random assignment ensures that the treatment and control groups are initially similar and do not differ on background characteristics or other factors. Random assignment thus creates an evaluation design where any observed differences between the two groups after the program intervention takes place can be attributed to the intervention with a high degree of confidence.
Quasi-experimental design (QED)
A design that forms a counterfactual group by means other than random assignment. This approach is used for conducting impact evaluations where observed changes in the treatment group are compared with a comparison group (as a counterfactual representing an absence of intervention) to assess and estimate the impact of the program on participants. However groups formed in these designs typically differ for reasons other than chance, and these differences may influence the impact estimate. There are different types of approaches used in quasiexperimental designs such as those using Propensity Score Matching (PSM), Regression Discontinuity, Interrupted Time Series (ITS) and others.
The term is a catch all category that refers to a range of research and evaluation studies that do not fall under the experimental or quasi-experimental research designs. They include process and outcomes evaluations, case studies, cost effectiveness or cost benefit analysis, feasibility studies, rapid assessments, situational and contribution analysis, developmental evaluation, strategic learning, systems change studies, and others.
Targeted Level of Evidence by SIF-Supported Evaluations
By employing these different evaluation designs, SIF grantees aim to move to a higher level of evidence. If successful, three-quarters of these studies will allow programs to demonstrate a moderate or strong level of evidence.
Note: As of 2011 all SIF interventions are required to reach at least a moderate level of evidence by the end of their grant term.