Analyze This!: Human rights group confronts abuses with data-driven evidence
By Vijay Mehrotra
For those who think that the analytics revolution is only about big companies in a few business verticals, the Human Rights Data Analysis Group (http://www.hrdag.org) and its parent organization Benetech (www.benetech.org) just might change your world view. Benetech and its HRDAG initiative are an important, intriguing and instructive success story about analytics and the digital revolution.
I often tell my students that data is the analyst’s water supply, so as soon as I heard about HRDAG I immediately asked, “Where does the data come from?” Dr. Megan Price, a HRDAG statistician and analyst, patiently explained that there were some things I first needed to understand about Benetech. Benetech’s tagline is, “Technology Serving Humanity,” and its goal is to nurture, find and/or integrate seemingly disparate projects that support this mission and help turn them into self-sustaining initiatives.
Human rights “data” often starts out as a set of narratives that are being captured on the ground in hot spots around the world, primarily by grassroots organizations that live under constant threats from not only weather and natural disasters but also hostile governments and angry armies. Launched by Benetech in 2003, the MARTUS initiative (www.martus.org) provides such organizations with software that is used to encrypt, upload and securely store its data on remote servers. This open source software is made freely available for download by the MARTUS team and is used by more than 100 different organizations worldwide. In addition, MARTUS offers Web-based and in-person product demos, technical support and training. Much of the data that HRDAG analyzes comes from the MARTUS platform.
HRDAG began as the brainchild of Dr. Patrick Ball, who has been working on the application of measurement and rational thinking to the field of human rights since 1991. After operating out of the American Academy for the Advancement of Science for several years, HRDAG joined forces with Benetech in 2004. It was a very natural fit, with HRDAG’s expertise with statistical modeling, data analysis and human rights experience serving to leverage the descriptive narratives stored through the MARTUS platform.
The process of converting text-based chronicles of human rights abuses into structured databases suitable for statistical analysis is complex, Price explained to me. The HRDAG team begins with the development of a controlled vocabulary that maps to their “Who Did What to Whom” data model, and then training individuals to use this vocabulary when coding the narratives into structured data. This process requires both statistical rigor (including things like inter-rater reliability scales) and legal precision (for example, carefully drawing distinctions between kidnappings and disappearances). Given all of this, the coding process is usually technically and emotionally intense, often bordering on surreal.
Once this has been completed, the data is ready for statistical analysis. To make valid statistical inferences, the HRDAG analysts carefully consider how the data was gathered and work hard to apply appropriate statistical tools. On its many projects around the world (including El Salvador, Chad, Kosovo, India, Liberia, Iran, Cambodia and many other countries), HRDAG’s analysts employ a broad array of tools ranging from simple descriptive statistics to sophisticated multiple system estimation methods  that allow them to make claims based on data from two or more sources gathered by different groups.
As hard as these statistical problems might be, these are not the most difficult part of what HRDAG does. For HRDAG, speaking truth to power means knowing everything they do will be scrutinized incredibly closely, so every team must intimately understand how the data used in their project was collected, coded and analyzed. There have also been skirmishes with other would-be experts about methods and conclusions . But the most difficult challenge is maintaining objectivity while knowing just how high the stakes and emotions are for the grassroots activists and families who have invested so much and taken such personal risks to record victims’ powerful and profound stories.
HRDAG has both limited resources and a sharp focus on conducting analyses that drive policy changes, legal justice, funding outcomes and other tangible outcomes. As such, the decision to engage in any particular project is significantly influenced by the answer to a simple and powerful question: “Does the truth matter?” That is, will the results of a statistical analysis have a chance to make a difference within the context in which the alleged human rights violations have taken place? Sadly, sometimes the answer is “no.”
In other cases, after the available data has been analyzed, the results of the analysis simply may not be able to support the allegations of systemic abuse, a challenge that Ball discussed quite candidly in a recent presentation . Indeed, there may be no better evidence of Ball’s commitment to the truth than his willingness to openly and honestly acknowledge such imperfections in past practices.
Twenty years in, HRDAG has emerged as a classical high-performing analytics organization. As advances in technology have enabled improved data quality and availability, HRDAG has successfully developed and deployed appropriate analysis techniques to address complex questions. Its leaders carefully select projects based on the potential for significant impact, and they have developed a culture of carefully examining every step in their information value chain, standing firmly behind what they have learned and acknowledging the limits of what is knowable from the available data.
There is no way for us to know how much of their success is due to Gretzkian creativity (“I skate to where the puck is going to be, not where it has been”) and how much is due to Goethe-inspired commitment (“The moment one definitely commits oneself, then providence moves too”). But certainly the world is a better place for having HRDAG and Benetech in it. ?
Vijay Mehrotra (firstname.lastname@example.org) is an associate professor, Department of Finance and Quantitative Analytics, School of Business and Professional Studies, University of San Francisco. He is also an experienced analytics consultant and entrepreneur and an angel investor in several successful analytics companies.
REFERENCES, NOTES & FURTHER READING
- These techniques were originally developed for use with fisheries and wildlife populations, and were later adapted for a wide variety of other applications. For more, see “Capture-Recapture and Multiple-Record Systems Estimation I: History and Theoretical Development,” 1995, American Journal of Epidemiology, Vol. 142, No. 10, pp. 1,047-1,058.
- See for example www.hrdag.org/resources/publications/Co-union-violence-paper-response.pdf.
- For more information about this presentation, see www.forbes.com/sites/mahaatal/2011/02/17/patrick-ball-on-the-perils-of-misusing-human-rights-data/.