1 Introduction

This report is a product of the collaboration between ideas42 and the Golub Capital Social Impact Lab to advance practical applications of machine learning to behavioral science policy and field experimentation.

The contents of this report are designed for any audience looking to learn more about how machine learning can add new techniques to behavioral design, causal inference, and experimentation. A big thank you to Schmidt Futures and The Alfred P. Sloan Foundation for their generous contributions to this innovative collaboration and research. This report is written in RMarkdown with bookdown.

1.1 Who should read this report?

• You’re an economist, researcher, policy designer, et cetera who uses evidence from randomized controlled trials or A/B tests to make decisions about what works.
• You’re a student of computer science who cares about social impact.
  
• You’re a behavioral scientist or practitioner who wants to understand how increasingly large data can be used for good.

1.2 Why did we pursue this research?

Behavioral science interventions have already generated widespread, positive social impact – but the field has yet to reach its full potential.

A vision for bigger, better impact

Typically, behavioral scientists design and test interventions to learn how they work on average, but we can achieve greater impact if we use data to learn about what works for whom. With new computational methods, we can uncover unexpected insights about who does (and does not) respond well to specific treatments. Equipped with predictions about how interventions are likely to perform across groups, we can decide who should receive which treatment and select policies that meet our goals in a given context (e.g. reduce cost, reduce disparities between groups, maximize the number of people who follow through on their intentions).

From vision to action

Our goal for this research was to translate the vision articulated above into actionable insights for practitioners. We applied cutting-edge machine learning methods (causal forests, optimal policy estimation, and bandit algorithms) to a set of real-world behavioral interventions – and in the process advanced the methods themselves. This report reviews the findings from our research, provides links to supplemental materials and tutorials, and offers suggestions for how practitioners can scope and design projects to incorporate these methods most effectively. As these methods continue to evolve, we hope you will apply them in new contexts and share your insights so we can continue to innovate and positively impact people’s lives.

1.3 Navigating the report

This report lives at the intersection of several fields (economics, psychology, and computer science) and applies to researchers and policy-makers alike. The cross-disciplinary nature of this work is part of what makes it so exciting. This also means that it may be useful to navigate directly to certain subsections, depending on what you already know and what you are hoping to learn. To navigate the report, read the overview below and use the table of contents (on the left) to access the sections most applicable to your work (or your curiosity).

1.3.1 Report overview

In the remainder of the introduction, we provide background information on the fields of behavioral science and machine learning. Use these to get up to speed on each discipline.

• Perhaps you are a computer scientist curious about applying your technical prowess to projects with social impact. Read “What is Behavioral Science and Design?”
• Perhaps you are a behavioral scientist or economist who wants to explore new ways to leverage data. Read “What is Machine Learning?”
• Perhaps you are a policy-maker or practitioner new to both machine learning and behavioral science. Read “What is Behavioral Science and Design?” and “What is Machine Learning?”
• The last section of the introduction, entitled “How can machine learning improve the quality of social impact research?” summarizes the potential benefits of this cross-disciplinary approach, and should be widely applicable.

Section 1 overviews ideas42’s methodology, and introduces two behavioral interventions to which we eventually applied innovative computational methods. But before introducing new computational methods (which we describe in section 2), we created and ran randomized controlled trials (RCTs) that helped us understand whether our interventions were effective on average. This section demonstrates the way we, as behavioral scientists, typically design interventions and measure impact. While sections 2-4 describe “where we are going,” section 1 provides an imperative perspective on “where we are coming from.”

Section 2 describes the methods themselves in detail, using the two examples from ideas42’s work as case studies. This section and its accompanying tutorial are especially useful for anyone seeking to apply these methods in their own work. Step-by-step, we cover how we used machine learning to identify heterogeneous treatment effects (variation in how individuals respond to treatment) in a behavioral intervention to increase financial aid renewal among community college students. We then illustrate how this helped us suggest “optimal policies” for future interventions. We also describe how adaptive experimentation can help us re-engineer the design of behavioral interventions. We again illustrate our approach using an example from ideas42’s work – in this case, an experiment that aimed to reduce vehicle booting and towing in New York City (and the accompanying financial burden for drivers).

Section 3 offers concrete recommendations for how to design experiments that can be evaluated using the methods explained in section 2. By applying these methods ourselves, we learned more about the criteria for using them successfully and identified the key characteristics that make a setting ripe for this approach. Read this section to learn how to scope projects that can incorporate these methods successfully. We describe what data you would need, how you can design in a way that allows you to find heterogeneity if it exists, and how you can ensure your work is fair and transparent.

Section 4 summarizes several takeaways from this work:

• When designing experiments, behavioral scientists typically converge on a few designs that are likely to work on average. In cases where we think different people might respond differently to an intervention, and we have the ability to learn about and implement more targeted policies, behavioral scientists should create multiple treatment arms and a set of diverse designs that may work for some groups but not others.
• It is not always the case that people respond differently to interventions. We are also likely to discover cases in which treatment effects are simply not heterogeneous. In such cases, there is less value added from methods for targeting, but that insight can also be useful to guide future innovation and testing.
• For those who do not yet have the data to explore these methods, improving the way you collect and organize data is a good first step.

Finally, the appendix links to additional resources you can use to get started with these methods.

1.4 About us

1.4.1 ideas42

We’re a non-profit using deep insights into human behavior—into why people do what they do—to help improve lives, build better systems, and drive social change. Working globally, we reinvent the practices of institutions and create better products and policies that can be scaled for maximum impact.

We also teach others, ultimately striving to generate lasting social impact and create a future where the universal application of behavioral science powers a world with optimal health, equitable wealth, and environments and systems that are sustainable and just for all.

For more than a decade, we’ve been at the forefront of applying behavioral science in the real world. And as we’ve developed our expertise, we’ve helped to define an entire field. Our efforts have so far extended to 40 countries as we’ve partnered with governments, foundations, NGOs, private enterprises, and a wide array of public institutions—in short, anyone who wants to make a positive difference in peoples’ lives.

Visit ideas42.org and follow us on Twitter to learn more about our work. Contact us at info@ideas42.org with questions.

1.4.2 The Golub Capital Social Impact Lab

We use digital technology and social science research to improve the effectiveness of leading social sector organizations. We drive measurable social impact through two key activities: collaborating on original research and sharing best practices.

Based out of Stanford Graduate School of Business, the lab is a research initiative of affiliated academics, staff, and student researchers who are passionate about conducting research that guides and improves the process of innovation, as well as applying and disseminating that research to achieve social impact at large scale. We collaborate with a wide range of organizations, including large firms and smaller startups, for-profits and nonprofits, as well as NGOs and governments.

Through research projects, the lab works to promote, develop, and apply innovative technology solutions and best-practice methodologies at social impact organizations. We create and disseminate the insights to students and practitioners, thus guiding future innovation in more productive directions and improving the efficiency of the innovation process.

1.4.3 Acknowledgements

We would like to thank Schmidt Futures and the Alfred P. Sloan Foundation for funding this research into the intersection of two innovative fields. We would also like to thank the following individuals for their invaluable support in launching and sustaining this effort.

• Our advisors: Guido Imbens, Jens Ludwig, Sendhil Mullainathan, Ted Robertson
• Our research team: Vitor Hadad, Niall Keleher, Matthew Schaelling, Henrike Steimer
• Our collaborators: Michael Stern, Charlie Connell, Carolyn Silverman, Gregory Stoddard, Everet Rummel, Nicole Bellettiere, Colin Chellman, Tom Tasche, Peter Polga-Hecimovich, Imanol Arrieta Ibarra