AI Bias, Displacement Data, and the People Who Get Left Out

Where bias comes from

Bias in AI is not a single thing. It is at least four overlapping problems that get bundled under one word and then talked about as though addressing any one of them addresses all of them. For humanitarian work, the distinctions matter.

Representation bias is the simplest. If a group is under-represented in the training data, the model will be worse at any task involving that group. A facial recognition system trained mostly on lighter-skinned faces is worse at recognising darker-skinned faces, a pattern documented in *Gender Shades* (Buolamwini and Gebru, 2018). A language model trained mostly on English text is worse at Pashto. A displacement forecasting model trained on cross-border registration data is worse at predicting movements of populations who avoid registration.

Measurement bias is more subtle. The proxies a dataset uses to capture a real-world phenomenon are imperfect, and the imperfection is rarely random. IDP counts based on camp registrations under-count urban IDPs, who are harder to enumerate. Casualty counts based on media reporting under-count incidents in regions with less press access. Each of these proxies is what the model learns. The gap between proxy and reality is what the model misses.

Aggregation bias is what happens when a single model is built to serve populations whose underlying distributions differ. A health-prediction model trained on a mixed-population dataset and then applied at the bedside performs worse on every sub-population than a separately calibrated model would. Applied to displacement, an aggregated model can perform acceptably on the population average while failing badly for the populations whose patterns differ most from the average.

Deployment bias is the gap between what a model was built to do and what it ends up being used for. A risk-scoring model built to triage cases for follow-up review becomes, in practice, a decision model when the follow-up review is under-resourced. The technical artefact has not changed. The role it plays has.

Why displacement data is especially prone to bias

The data that humanitarian models train on is shaped by who is counted, who is counted accurately, and who is allowed to be counted. None of those are neutral.

Registration data captures people who present themselves for registration. People who avoid registration, for reasons that include fear of authorities, lack of documentation, and active denial of access, are not captured. In some contexts the gap is large enough that the registered population is a minority of the displaced population. A model trained on registration data therefore learns a model of the registered population and applies it as though it were a model of the displaced population.

Camp data captures people in formal camps. Most internally displaced people do not live in camps. They live with host families, in informal urban settlements, or move between locations as conditions allow. A model trained on camp data is a model of camp-resident IDPs.

Survey data is shaped by who can be reached. In areas with security restrictions, weak telecommunications, or active conflict, sample frames are constructed from what is reachable. The people who are not reachable are correlated with the conditions that make them not reachable, which is to say the people in the worst conditions are systematically under-sampled.

Reporting data is shaped by language and media presence. Events in regions covered by international media generate more records than events in regions that are not. A conflict-event dataset is partly a measure of conflict and partly a measure of journalistic attention.

Each of these is a real, structural feature of the data. None of them is fixed by adding more data of the same kind. A model that does not account for the systematic gaps reproduces and often amplifies them. For the underlying definitions of who is counted as displaced and how, see our guide to understanding displacement and what counts as a humanitarian crisis.

Cases where this has gone wrong

Several well-documented cases are worth naming.

Biometric registration systems deployed in refugee operations have been independently audited and shown to produce different match rates across demographic groups. The gap has narrowed over successive system generations but has not closed. Where biometric verification is the gate to assistance, the gap is a denial-of-service against the populations the system performs worst on. The published audits typically recommend additional verification pathways for populations where match rates are lower, and the recommendations are implemented unevenly.

Asylum risk-scoring systems in several jurisdictions in Europe and North America have been shown to reproduce the patterns of past refusal decisions when trained on past refusal data. Where the model's output influences caseworker decisions, the patterns become self-reinforcing. The EU AI Act now classifies these systems as high-risk and requires transparency and contestability, but enforcement is still developing. The AI Incident Database tracks documented harms of this kind in close to real time.

Predictive targeting models for cash-based assistance, which rank households by predicted vulnerability, have been shown in published evaluations to under-rank some categories of vulnerable household, particularly female-headed households in contexts where past data on female-headed households was incomplete. The patterns are not always obvious from headline accuracy metrics; they appear only when accuracy is disaggregated by demographic.

Automated language-detection systems in chatbot deployments have been shown to misclassify under-resourced languages as the closest majority language and then translate accordingly. A speaker of a minority Sudanese language being served responses in Arabic is not getting help; they are getting a confidently irrelevant answer.

What responsible deployment looks like

There is no clever technical fix that removes bias from a model trained on biased data. The patterns that have emerged from sector practice are operational rather than algorithmic.

First, document what the dataset is and is not. Every training dataset used in production should have a written description of who is in the data, who is missing from it, and how it was collected. *Datasheets for Datasets* (Gebru et al., 2018) is the most-cited published framework, and adoption is slowly spreading. A dataset without a datasheet is a model whose limits cannot be reasoned about.

Second, audit by sub-group. Aggregate accuracy hides the patterns that matter most. Model performance should be reported separately for the demographic groups the model will affect, with the gaps published as part of the model's documentation. A model that cannot be audited this way should not be deployed where its mistakes will fall unevenly across groups.

Third, keep a human in the loop for terminal decisions. AI triage that flags cases for human review is useful. AI that closes cases without human review is dangerous. The human review needs to be resourced enough to function. A human-in-the-loop process that becomes a rubber stamp because the review queue is too long has not removed the risk; it has hidden it.

Fourth, make the redress path real. Every AI-mediated decision should have a path to appeal that does not require the affected person to understand the model. If the path requires the person to identify and contest a specific algorithmic factor, it does not function. If the appeal queue takes months and the underlying decision deprives someone of assistance now, the redress is theoretical.

Fifth, be willing to not deploy. The most under-used option in humanitarian AI is the option to conclude that the model is not good enough for the use case and to keep the existing process. The pressure to deploy comes from donors and from internal innovation mandates. The pressure to not deploy comes from the rights of the people the system will affect. Both are legitimate. The latter is currently under-represented in the decision-making.

How this connects to our work

Humanity Centered Data publishes analysis derived from displacement data drawn from UNHCR, IOM, IDMC, OCHA, ACLED, and several other primary sources. We use AI in the workflow, and that means we inherit the bias properties of the underlying datasets. We try to be honest about that.

The discipline we apply is the same discipline we recommend to any humanitarian user of AI. Document the source set for every page. Cite every numerical claim inline. Disclose AI-assisted pages with a visible banner. Audit our country coverage for who is missing, not only for who is present. And accept that there are claims the underlying data cannot support and decline to make them, even when a fluent model would happily produce a sentence that sounded as though it could.

The deeper point is that bias in humanitarian AI is not a bug in the models. It is the data, the proxies, and the choices about what to measure, all reproduced at the scale and speed AI permits. The technology amplifies what the sector chooses to count. The choice of what to count is older than the technology, and it is the choice that determines who gets seen. For the wider context, our piece on AI in humanitarian work catalogues the use cases this discipline applies to, and our long-form research library collects the country-level analysis that depends on getting these foundations right.

Sources

• Buolamwini and Gebru, *Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification*, PMLR 2018. Foundational audit on demographic accuracy gaps in facial-analysis systems.
• Gebru et al., *Datasheets for Datasets*, 2018. The dataset-documentation framework referenced in the responsible-deployment section.
• Mitchell et al., *Model Cards for Model Reporting*, 2018. The companion model-documentation framework used alongside datasheets.
• Suresh and Guttag, *A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle*, 2019. Source for the representation, measurement, aggregation, and deployment bias taxonomy used above.
• Access Now, *Uses of AI in migration and border control: A fundamental rights approach to the Artificial Intelligence Act*. Documentation of automated decision systems in refugee and migration contexts.
• European Union Agency for Fundamental Rights, *Bias in Algorithms: Artificial Intelligence and Discrimination*. EU reference report on algorithmic discrimination cited in the risk-scoring section.
• AI Incident Database. Ongoing register of documented AI harms across jurisdictions.
• UNHCR Refugee Data Finder, IOM Displacement Tracking Matrix, IDMC Global Internal Displacement Database, OCHA Humanitarian Data Exchange and ACLED. The primary displacement and conflict datasets whose structural biases are discussed throughout.