When AI learns human bias

A study of more than 311,000 loan applications shows how human bias can flow into AI and how businesses can stop it upstream.

As banks and fintech companies increasingly turn to AI for lending decisions, what happens when machines inherit the biases of the people who trained them? The answer could determine who gets a loan, who gets turned away, and on what terms.

Bias in, bias out

New research led by Xiyang Hu, assistant professor of information systems, finds that human bias doesn't disappear when lenders roll out AI.

Instead, bias can become embedded in the historical decisions used to train the models. Those choices can skew the training data, passing human bias downstream to AI. For borrowers, that means discrimination can outlast the people who created it.

Hu and his co-authors, including Tian Lu, assistant professor of information systems, analyzed more than 311,000 loan applications from a leading Asian micro-lending platform over 33 months. Because many borrowers returned for multiple loans, the team could separate different forms of human bias and trace how they flowed into AI.

The study also found that AI is not doomed to repeat human mistakes. Even when trained on biased historical lending decisions, the machine-learning model reduced bias by about 43% compared with human loan officers.

To understand why, the research team focused on two forms of human bias: "preference-based," or simply favoring one group over another based on group identity, and "belief-based," an initial judgment based partly on group-level assumptions when little is known about the individual, which can be revised as more personal information becomes available.

The researchers then measured how those biases shaped lending decisions. In the study, fairness was measured by whether equally creditworthy men and women were approved for loans at the same rate.

A fairness fix with a financial upside

What happened when those human biases were removed before the AI learned from the resulting decisions? Reducing bias in the upstream human decisions produced more balanced training data and made the AI fairer. Among qualified applicants who ultimately repaid their loans, the gender gap in approval rates between men and women narrowed from 4.30% to 3.87% for first-time applicants and from 3.39% to 3.11% for repeat borrowers.

There was also a financial upside for lenders. The research found that reducing bias in loan decisions boosted profitability. By correcting bias against male applicants, lenders approved more men who ultimately repaid their loans.

The additional profits from those borrowers outweighed the losses from the relatively small number who later defaulted. For human loan officers, removing both forms of bias lifted expected profits by 6.7%. The pattern held for AI, too: removing those same biases from the training data lifted expected profits by an additional 2.5%.

Asked what business leaders should take away from the research, Lu says companies should be careful not to let data harden into stereotypes. Doing so can cause lenders to overlook creditworthy borrowers while backing weaker ones. "People generally believe female users will have higher credit," he says.

The findings point to another lesson for businesses rolling out AI: root out bias upstream, before it reaches the models. Less biased human decisions produce more balanced training data, which can lead to fairer AI lending decisions.

Lu also points to a practical way companies can curb employee bias. "Human bias can be mitigated by training and monitoring their behavior," he says. "If we reduce bias in human decisions, we also improve the data later used to train AI."

Why human oversight still matters

Although some banks are replacing human loan officers with AI, the paper suggests there is a role for both, depending on the applicant.

When both human and machine decisions were evaluated after removing the two identified forms of human bias, human loan officers made fairer lending decisions for first-time applicants. Machine learning pulled ahead for repeat applicants, once borrowers had accumulated repayment histories.

That conclusion surprised even the researchers. AI did not simply inherit human bias wholesale, Hu says. "Before the research, we did not know which biases could be inherited and which could not, and which would be inherited more severely."

"The pattern was actually quite interesting: Humans can outperform AI in reducing bias in some cases, and vice versa. That suggests there may be an optimal balance," Hu adds.

Beyond lending

The findings could also have implications far beyond lending, including criminal risk assessments, hiring, and other high-stakes decisions. "It can be applied to criminal evaluations and hiring and so forth," Lu says.