An Algorithm Told Police She Was Safe. Then Her Husband Killed Her.

In a small apartment outside Madrid on Jan. 11, 2022, an argument over household chores turned violent when Lobna Hemid’s husband smashed a wooden shoe rack and used one of the broken pieces to beat her. Her screams were heard by neighbors. Their four children, ages 6 to 12, were also home.

Ms. Hemid’s husband of more than a decade, Bouthaer el Banaisati, regularly punched and kicked her, she later told the police. He also called her a “whore,” “disgusting” and “worthless,” according to the police report.

Before Ms. Hemid left the station that night, the police had to determine if she was in danger of being attacked again and needed support. A police officer clicked through 35 yes or no questions — Was a weapon used? Were there economic problems? Has the aggressor shown controlling behaviors? — to feed into an algorithm called VioGén that would help generate an answer.

VioGén produced a score:

The police accepted the software’s judgment and Ms. Hemid went home with no further protection. Mr. el Banaisati, who was imprisoned that night, was released the next day. Seven weeks later, he fatally stabbed Ms. Hemid several times in the chest and abdomen before killing himself. She was 32 years old.

Spain has become dependent on an algorithm to combat gender violence, with the software so woven into law enforcement that it is hard to know where its recommendations end and human decision-making begins. At its best, the system has helped police protect vulnerable women and, overall, has reduced the number of repeat attacks in domestic violence cases. But the reliance on VioGén has also resulted in victims, whose risk levels are miscalculated, getting attacked again — sometimes leading to fatal consequences.

Spain now has 92,000 active cases of gender violence victims who were evaluated by VioGén, with most of them — 83 percent — classified as facing little risk of being hurt by their abuser again. Yet roughly 8 percent of women who the algorithm found to be at negligible risk and 14 percent at low risk have reported being harmed again, according to Spain’s Interior Ministry, which oversees the system.

At least 247 women have also been killed by their current or former partner since 2007 after being assessed by VioGén, according to government figures. While that is a tiny fraction of gender violence cases, it points to the algorithm’s flaws. The New York Times found that in a judicial review of 98 of those homicides, 55 of the slain women were scored by VioGén as negligible or low risk for repeat abuse.

Spanish police are trained to overrule VioGén’s recommendations depending on the evidence, but accept the risk scores about 95 percent of the time, officials said. Judges can also use the results when considering requests for restraining orders and other protective measures.

“Women are falling through the cracks,” said Susana Pavlou, director of the Mediterranean Institute of Gender Studies, who coauthored a European Union report about VioGén and other police efforts to fight violence against women. The algorithm “kind of absolves the police of any responsibility of assessing the situation and what the victim may need.”

Spain exemplifies how governments are turning to algorithms to make societal decisions, a global trend that is expected to grow with the rise of artificial intelligence. In the United States, algorithms help determine prison sentences, set police patrols and identify children at risk of abuse. In the Netherlands and Britain, authorities have experimented with algorithms to predict who may become criminals and to identify people who may be committing welfare fraud.

Few of the programs have such life or death consequences as VioGén. But victims interviewed by The Times rarely knew about the role the algorithm played in their cases. The government also has not released comprehensive data about the system’s effectiveness and has refused to make the algorithm available for outside audit.

VioGén was created to be an unbiased tool to aid police with limited resources identify and protect women most at risk of being assaulted again. The technology was meant to create efficiencies by helping police prioritize the most urgent cases, while focusing less on those calculated by the algorithm as lower risk. Victims classified as higher risk get more protection, including regular patrols by their home, access to a shelter and police monitoring of their abuser’s movements. Those with lower scores get less support.

In a statement, the Interior Ministry defended VioGén and said the government was the “first to carry out self-criticism” when mistakes occur. It said homicide was so rare that it was difficult to accurately predict, but added it was an “incontestable fact” that VioGén has helped reduce violence against women.

Since 2007, about 0.03 percent of Spain’s 814,000 reported victims of gender violence have been killed after being assessed by VioGén, the ministry said. During that time, repeat attacks have fallen to roughly 15 percent of all gender violence cases from 40 percent, according to government figures.

“If it weren’t for this, we would have more homicides and gender-based violence,” said Juan José López Ossorio, a psychologist who helped create VioGén and works for the Interior Ministry.

Yet victims and their families are grappling with the consequences when VioGén gets it wrong.

“Technology is fine, but sometimes it’s not and then it’s fatal,” said Jesús Melguizo, Ms. Hemid’s brother-in-law, who is a guardian for two of her children. “The computer has no heart.”

‘Effective but not perfect’

VioGén started with a question: Can police predict an assault before it happens?

After Spain passed a law in 2004 to address violence against women, the government assembled experts in statistics, psychology and other fields to find an answer. Their goal was to create a statistical model to identify women most at risk of abuse and to outline a standardized response to protect them.

“It would be a new guide for risk assessment in gender violence,” said Antonio Pueyo, a psychology professor at the University of Barcelona who later joined the effort.

The team took a similar approach to how insurance companies and banks predict the likelihood of future events, such as house fires or currency swings. They studied national crime statistics, police records and the work of researchers in Britain and Canada to find indicators that appeared to correlate with gender violence. Substance abuse, job loss and economic uncertainty were high on the list.

Then they came up with a questionnaire for victims so their answers could be compared with historical data. Police would fill in the answers after interviewing a victim, reviewing documentary evidence, speaking with witnesses and studying other information from government agencies. Answers to certain questions carried more weight than others, like if an abuser displayed suicidal tendencies or showed signs of jealousy.

The system produced a score for each victim: negligible risk, low risk, medium risk, high risk or extreme risk. A higher score would result in police patrols and the tracking of an aggressor’s movements. In extreme cases, police would assign 24-hour surveillance. Those with lower scores would receive fewer resources, mainly follow-up calls.

Predictive algorithms to address domestic violence have been used in parts of Britain, Canada, Germany and the United States, but not on such a national scale. In Spain, the Interior Ministry introduced VioGén everywhere but in the Catalonia region and Basque Country.

Law enforcement initially greeted the algorithm with skepticism, police and government officials told The Times, but it soon became a part of everyday police business.

Before VioGén, investigations were “based on the experience of the policeman,” said Mr. Pueyo, who remains affiliated with the program. “Now this is organized and guided by VioGén.”

VioGén is a source of impartial information, he said. If a woman attacked late at night was seen by a young police officer with little experience, VioGén could help detect the risk of future violence.

“It’s more efficient,” Mr. Pueyo said.

Over the years, VioGén has been refined and updated, including with metrics that are believed to better predict homicide. Police have also been required to conduct a follow-up risk assessment within 90 days of an attack.

But Spain’s faith in the system has surprised some experts. Juanjo Medina, a senior researcher at the University of Seville who has studied VioGén, said the system’s effectiveness remains unclear.

“We’re not good at forecasting the weather, let alone human behavior,” he said.

Francisco Javier Curto, a commander for the military police in Seville, said VioGén helps his teams prioritize, but requires close oversight. About 20 new cases of gender violence arrive every day, each requiring investigation. Providing police protection for every victim would be impossible given staff sizes and budgets.

“The system is effective but not perfect,” he said, adding that VioGén is “the best system that exists in the world right now.”

José Iniesta, a civil guard in Alicante, a southeastern port city, said not enough of the police are trained to keep up with growing case loads. A leader in the United Association of Civil Guards, a union representing officers in rural areas, he said that outside of big cities, the police often must choose between addressing violence against women or other crimes.

Sindicato Unificado de Policía, a union that represents national police officers, said even the most effective technology cannot make up for a lack of trained experts. In some places, a police officer is assigned to work with more than 100 victims.

“Agents in many provinces are overwhelmed,” the union said in a statement.

When attacks happen again

The women who have been killed after being assessed by VioGén can be found across Spain.

One was Stefany González Escarraman, a 26-year-old living near Seville. In 2016, she went to the police after her husband punched her in the face and choked her. He threw objects at her, including a kitchen ladle that hit their 3-year-old child. After police interviewed Ms. Escarraman for about five hours, VioGén determined she had a negligible risk of being abused again.

The next day, Ms. Escarraman, who had a swollen black eye, went to court for a restraining order against her husband. Judges can serve as a check on the VioGén system, with the ability to intervene in cases and provide protective measures. In Ms. Escarraman’s case, the judge denied a restraining order, citing VioGén’s risk score and her husband’s lack of criminal history.

About a month later, Ms. Escarraman was stabbed by her husband multiple times in the heart in front of their children. In 2020, her family won a verdict against the state for failing to adequately measure the level of risk and provide sufficient protection.

“If she had been given the help, maybe she would be alive,” said Williams Escarraman, Ms. Escarraman’s brother.

In 2021, Eva Jaular, who lived in Liaño in northern Spain, was slain by her former boyfriend after being classified as low risk by VioGén. He also killed their 11-month-old daughter. Six weeks earlier, he had jabbed a knife into a couch cushion next to where Ms. Jaular sat and said, “look how well it sticks,” according to a police report.

Since 2007, 247 of the 990 women killed in Spain by a current or former partner were previously scored by VioGén, according to the Interior Ministry. The other victims had not been previously reported to the police, so were not in the system. The ministry declined to disclose the VioGén risk scores of the 247 who were killed.

The Times instead analyzed reports from a Spanish judicial agency, released almost every year from 2010 to 2022, which included information about the risk scores of 98 women who were later killed. Of those, 55 had been classified as negligible risk or low risk.

In a statement, the Interior Ministry said that analyzing the risk scores of homicide victims doesn’t provide an accurate picture of VioGén’s effectiveness because some homicides happened more than a year after the first assessment, while others were committed by a different partner.

Why the algorithm incorrectly classifies some women varies and isn’t always clear, but one reason may be the poor quality of information fed into the system. VioGén is ideally suited for cases when a woman, in the moments after being attacked, can provide complete information to an experienced police officer who has time to fully investigate the incident.

That does not always happen. Fear, shame, economic dependency, immigration status and other factors can lead a victim to withhold information. Police are also often squeezed for time and may not fully investigate.

“If we already enter erroneous information into the system, how can we expect the system to give us a good result?” said Elisabeth, a victim who now works as a gender violence lawyer. She spoke on the condition her full name not be used, for fear of retaliation by her former partner.

Luz, a woman from a village in southern Spain, said she was repeatedly labeled low risk after attacks by her partner because she was afraid and ashamed to provide complete information to the police, some of whom she knew personally. She got her risk score increased to extreme only after working with a lawyer specializing in gender violence cases, leading to round-the-clock police protection.

“We women keep a lot of things silent not because we want to lie but out of fear,” said Luz, who spoke on the condition her full name not be used for fear of retaliation by her attacker, who was imprisoned. “VioGén would be good if there were qualified people who had all the necessary tools to carry it out.”

Victim groups said that psychologists or other trained specialists should lead the questioning of victims rather than the police. Some have urged the government to mandate that victims be allowed to be accompanied by somebody they trust to help ensure full information is given to authorities, something that is now not allowed in all areas.

“It’s not easy to report a person you’ve loved,” said María, a victim from Granada in southern Spain, who was labeled medium risk after her partner attacked her with a dumbbell. She asked that her full name not be published for fear of retaliation by him.

Ujué Agudo, a Spanish researcher studying the influence of artificial intelligence on human decisions, said technology has a role in solving societal problems. But it could reduce the responsibility of humans to approving the work of a machine, rather than conducting the necessary work themselves.

“If the system succeeds, it’s a success of the system. If the system fails, it’s a human error that they aren’t monitoring properly,” said Ms. Agudo, a co-director of Bikolabs, a Spanish civil society group. A better approach, she said, was for people “to say what their decision is before seeing what the A.I. thinks.”

Spanish officials are exploring incorporating A.I. into VioGén so it can pull data from different sources and learn more on its own. Mr. Ossorio, a creator of VioGén who works for the Interior Ministry, said the tools can be applied to other areas, including workplace harassment and hate crimes.

The systems will never be perfect, he said, but neither is human judgment. “Whatever we do, we always fail,” he said. “It’s unsolvable problems.”

This month, the Spanish government called an emergency meeting after three women were killed by former partners within a 24-hour span. One victim, a 30-year-old from central Spain, had been classified by VioGén as low risk.

At a news conference, Fernando Grande-Marlaska, the interior minister, said he still had “absolute confidence” in the system.

‘Always cheerful’

Ms. Hemid, who was killed outside Madrid in 2022, was born in rural Morocco. She was 14 when she was introduced at a family wedding to Mr. el Banaisati, who was 10 years older than her. She was 17 when they married. They later moved to Spain so he could pursue steadier work.

Ms. Hemid was outgoing and gregarious, often seen racing to get her children to school on time, friends said. She learned to speak Spanish and sometimes joined children playing soccer in the park.

“She was always cheerful,” said Amelia Franas, a friend whose children went to the same school as Ms. Hemid’s children.

Few knew that abuse was a fixture of Ms. Hemid’s marriage. She spoke little about her home life, friends said, and never called the police or reported Mr. el Banaisati before the January 2022 incident.

VioGén is intended to identify danger signs that humans may overlook, but in Ms. Hemid’s case, it appears that police missed some clues. Her neighbors told The Times they were not interviewed, nor were administrators at her children’s school, who said they had seen signs of trouble.

Family members said Mr. el Banaisati had a life-threatening form of cancer that made him behave erratically. Many blamed underlying discrimination in Spain’s criminal system that overlooks violence against immigrant women, especially Muslims.

Police haven’t released a copy of the assessment that produced Ms. Hemid’s low risk score from VioGén. A copy of a separate police report shared with The Times noted that Ms. Hemid was tired during questioning and wanted to end the interview to get home.

A few days after the January 2022 attack, Ms. Hemid won a restraining order against her husband. But Mr. el Banaisati largely ignored the order, family and friends said. He moved into an apartment less than 500 meters from where Ms. Hemid lived and continued threatening her.

Mr. Melguizo, her brother-in-law, said he appealed to Ms. Hemid’s assigned public lawyer for help, but was told the police “won’t do anything, it has a low risk score.”

The day after Ms. Hemid was stabbed to death, she had a court date scheduled to officially file for divorce.