Adaptive Experimentation at Scale: Testing Benefit-Cost Transportability of Information Interventions in Colombia

Abstract

Students make high-stakes higher education choices with limited information, and they often underuse readily available data on program quality and labor-market returns. Colombia's national testing agency (ICFES) built an online portal with detailed information by major and institution, yet fewer than 5% of students accessed it. We partnered with ICFES to study a science-of-scaling question that arises when digital delivery is cheap: which intervention design should be scaled? We conducted two nationwide randomized experiments that delivered pop-up messages to 519,000 students as they accessed mandatory exam results. The interventions substantially increased information consumption: 15-25% of treated students clicked to view additional information. However, testing multiple variants revealed that more elaborate designs did not reliably outperform simpler prompts: in the 2017 cohort, a simple salient message generated 20.9% engagement versus 13.1% for a loss-framed alternative, and the same qualitative ranking appears in the 2018 cohort despite a different population. Despite large changes in information-seeking behavior, we find limited effects on tertiary enrollment over the subsequent three years. Our findings demonstrate both the promise and limits of digital nudges. Digital platforms enable rapid, policy-relevant experimentation that identifies which variants retain effectiveness in the natural implementation environment. The key advantage of platforms is thus not only cheap delivery, but cheap learning about which designs to scale and which outcomes scale with them.

@article{ icfes-nudge2search,
  title = "Adaptive Experimentation at Scale: Testing Benefit-Cost Transportability of Information Interventions in Colombia",
  author = "Bettinger, Eric and Kremer, Michael and Lizarazo, Felipe and Neilson, Christopher A. and Posso, Christian and Saavedra, Juan",
  journal = "",
  note = "wip",
  doi = ""
,
  abstract = "Students make high-stakes higher education choices with limited information, and they often underuse readily available data on program quality and labor-market returns. Colombia's national testing agency (ICFES) built an online portal with detailed information by major and institution, yet fewer than 5% of students accessed it. We partnered with ICFES to study a science-of-scaling question that arises when digital delivery is cheap: which intervention design should be scaled? We conducted two nationwide randomized experiments that delivered pop-up messages to 519,000 students as they accessed mandatory exam results. The interventions substantially increased information consumption: 15-25% of treated students clicked to view additional information. However, testing multiple variants revealed that more elaborate designs did not reliably outperform simpler prompts: in the 2017 cohort, a simple salient message generated 20.9% engagement versus 13.1% for a loss-framed alternative, and the same qualitative ranking appears in the 2018 cohort despite a different population. Despite large changes in information-seeking behavior, we find limited effects on tertiary enrollment over the subsequent three years. Our findings demonstrate both the promise and limits of digital nudges. Digital platforms enable rapid, policy-relevant experimentation that identifies which variants retain effectiveness in the natural implementation environment. The key advantage of platforms is thus not only cheap delivery, but cheap learning about which designs to scale and which outcomes scale with them."
,
  url_article = "https://christopherneilson.github.io/"
}

Current Draft

Work in progress.

Overview

This paper reports results from two sequential nationwide randomized experiments designed to systematically test information interventions about higher education returns at policy-relevant scale in Colombia. In partnership with ICFES, the Central Bank of Colombia, and Consilium Bots, we randomly displayed pop-up messages to students when they logged in to view their Saber 11 test scores. The intervention reached over 530,000 students across two cohorts in 2017 and 2018.

Key Findings

• Digital nudges successfully increased information consumption from 5% to 15-25% (p<0.001)
• Simple messages outperformed more elaborate behavioral frames (20.9% vs. 13.1% engagement for loss-framed messages)
• Despite large changes in information-seeking behavior, we find limited effects on tertiary enrollment over the subsequent three years
• The adaptive at-scale approach generated policy-relevant evidence in months rather than years

Keywords

Information interventions, higher education, behavioral economics, randomized controlled trials, Colombia, digital platforms, science of scaling, prospect theory, external validity

JEL Classification

I23, I24, C90, C93, D83, O15

• Coauthors: Eric Bettinger, Michael Kremer, Felipe Lizarazo, Christian Posso, Juan Saavedra
• Published:
• Date:
• Citations (Google Scholar):
• PDF