An algorithm intended to reduce poverty might disqualify people in need
“The questions asked don’t reflect the reality we exist in,” says Abdelhamad, a father of two who makes 250 dinars ($353) a month and struggles to make ends meet, as quoted in the report.
Takaful also reinforces existing gender-based discrimination by relying on sexist legal codes. The cash assistance is provided to Jordanian citizens only, and one indicator the algorithm takes into account is the size of a household. Although Jordanian men who marry a noncitizen can pass on citizenship to their spouse, Jordanian women who do so cannot. For such women, this results in a lower reportable household size, making them less likely to receive assistance.
The report is based on 70 interviews conducted by Human Rights Watch over the last two years, not a quantitative assessment, because the World Bank and the government of Jordan have not publicly disclosed the list of 57 indicators, a breakdown of how the indicators are weighted, or comprehensive data about the algorithm’s decisions. The World Bank has not yet replied to our request for comment.
Amos Toh, an AI and human rights researcher for Human Rights Watch and an author of the report, says the findings point to the necessity of greater transparency into government programs that use algorithmic decision-making. Many of the families interviewed expressed distrust and confusion about the ranking methodology. “The onus is on the government of Jordan to provide that transparency,” Toh says.
Researchers on AI ethics and fairness are calling for more scrutiny around the increasing use of algorithms in welfare systems. “When you start building algorithms for this particular purpose, for overseeing access, what always happens is that people who need help get excluded,” says Meredith Broussard, professor at NYU and author of More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech.
“It seems like this is yet another example of a bad design that actually ends up restricting access to funds for people who need them the most,” she says.
The World Bank funded the program, which is managed by Jordan’s National Aid Fund, a social protection agency of the government. In response to the report, the World Bank said that it plans to release additional information about the Takaful program in July of 2023 and reiterated its “commitment to advancing the implementation of universal social protection [and] ensuring access to social protection for all persons.”
The organization has encouraged the use of data technology in cash transfer programs such as Takaful, saying it promotes cost-effectiveness and increased fairness in distribution. Governments have also used AI-enabled systems to guard against welfare fraud. An investigation last month into an algorithm the Dutch government uses to flag the benefit applications most likely to be fraudulent revealed systematic discrimination on the basis of race and gender.
