Informing Housing Policy through Web Automation: Lessons for Designing Programming Tools for Domain Experts

Housing costs have risen dramatically in the past decade, surpassing their pre-Recession levels, but the data that housing researchers and policymakers rely on to understand these dynamics remain subject to important limitations in their spatiotemporal granularity or methodological transparency. While these aspects of existing public and private data sources present barriers to understanding the geography of cost and availability in markets across the United States, web data about housing opportunities provide an important alternative—albeit one that demands technical skills that would-be data users may lack. This case study documents the experiences of a collaboration between social and computer scientists focused on using a novel programming-by-demonstration tool for web automation, Helena, to inform rental housing policy and inequalities in the United States. While this project was initially focused on collecting housing ads from a single site within the Seattle area, the capacity to scale our project to new sources and locations afforded by Helena’s human-centered design allowed a team of social scientists to progress to scraping data across the country and multiple platforms. Using this project as a case study, we discuss a.) important programming and research challenges that were encountered and b.) how Helena’s design helped us overcome these barriers to using scraped web data in basic research and policy analysis.