Humanitarian data API integration: merging UNHCR, ACLED, and OCHA sources for longitudinal research

1. Step 1.Decide what a single row in your final dataset means

   Before writing a request, define the unit of analysis: country-month, admin1-quarter, or camp-year. UNHCR reports stock figures by year and country of asylum, ACLED codes discrete events with a timestamp, and OCHA/HDX publishes mixed-frequency indicator tables. Almost every integration failure traces back to merging three different units of analysis into one table and treating the result as comparable.

2. Step 2.Pull UNHCR population statistics

   The UNHCR Refugee Data Finder API at api.unhcr.org/population/v1/population/ serves refugee, asylum-seeker, IDP, and stateless stock figures. Key parameters are yearFrom, yearTo, coo (country of origin), coa (country of asylum), and cf_type=ISO for three-letter codes. Responses are paginated; read the total field and loop pages until you have every row.

3. Step 3.Pull ACLED conflict events and collapse them to your unit

   ACLED returns one row per coded event. To join it against annual stock figures, aggregate first: group by country and period, then sum fatalities and count events per event_type. Keep the raw event pull cached on disk so you can re-aggregate at a different granularity without re-hitting the API.

4. Step 4.Pull OCHA and HDX indicator tables through CKAN

   HDX runs on CKAN. Use data.humdata.org/api/3/action/package_search with a query such as q=idp+sudan to find datasets, then read the resources array for a direct CSV or XLSX download URL. HDX resources are files rather than a query interface, so plan for schema drift: column names and admin codings change between vintages.

5. Step 5.Normalise geography with P-codes and ISO3

   Join on ISO3 country codes at national level and on OCHA P-codes at subnational level, never on place names. Names differ by transliteration, punctuation, and administrative reform. Where a source ships only names, map them once through a lookup table you maintain and version, and log every unmatched name rather than silently dropping it.

6. Step 6.Merge, validate, and record provenance

   Left-join onto a complete country-period spine so gaps stay visible as nulls instead of disappearing rows. Store the retrieval timestamp and source vintage alongside every value. A merged table without provenance columns is not citable, and reviewers will ask which UNHCR release a 2026 figure came from.

Python: pull UNHCR and ACLED, merge onto a spine

import requests
import pandas as pd

# 1. UNHCR population statistics (no key required)
def unhcr_population(coa_iso3: str, year_from: int, year_to: int) -> pd.DataFrame:
    rows, page = [], 1
    while True:
        r = requests.get(
            "https://api.unhcr.org/population/v1/population/",
            params={
                "coa": coa_iso3,
                "yearFrom": year_from,
                "yearTo": year_to,
                "cf_type": "ISO",
                "limit": 1000,
                "page": page,
            },
            timeout=60,
        )
        r.raise_for_status()
        batch = r.json().get("items", [])
        if not batch:
            break
        rows.extend(batch)
        page += 1
    return pd.DataFrame(rows)


# 2. ACLED events, aggregated to country-year before joining
def acled_country_year(country: str, key: str, email: str) -> pd.DataFrame:
    r = requests.get(
        "https://api.acleddata.com/acled/read",
        params={"key": key, "email": email, "country": country, "limit": 5000},
        timeout=60,
    )
    r.raise_for_status()
    ev = pd.DataFrame(r.json().get("data", []))
    ev["year"] = pd.to_numeric(ev["year"], errors="coerce")
    ev["fatalities"] = pd.to_numeric(ev["fatalities"], errors="coerce").fillna(0)
    return (
        ev.groupby("year")
          .agg(events=("event_id_cnty", "count"), fatalities=("fatalities", "sum"))
          .reset_index()
    )


# 3. Merge onto a complete spine so gaps stay visible
spine = pd.DataFrame({"year": range(2015, 2027)})

pop = unhcr_population("SDN", 2015, 2026)
pop = pop.groupby("year", as_index=False)["refugees"].sum()
pop["year"] = pd.to_numeric(pop["year"], errors="coerce")

conflict = acled_country_year("Sudan", "YOUR_KEY", "you@example.org")

panel = (
    spine.merge(pop, on="year", how="left")
         .merge(conflict, on="year", how="left")
)
panel["retrieved_at"] = pd.Timestamp.utcnow().date().isoformat()
panel["sources"] = "UNHCR Refugee Data Finder; ACLED"

print(panel)

R: the same pull with httr2 and dplyr

library(httr2)
library(dplyr)
library(purrr)

unhcr_population <- function(coa, year_from, year_to) {
  page <- 1
  out <- list()
  repeat {
    resp <- request("https://api.unhcr.org/population/v1/population/") |>
      req_url_query(coa = coa, yearFrom = year_from, yearTo = year_to,
                    cf_type = "ISO", limit = 1000, page = page) |>
      req_perform() |>
      resp_body_json(simplifyVector = TRUE)
    if (length(resp$items) == 0) break
    out[[page]] <- resp$items
    page <- page + 1
  }
  bind_rows(out)
}

panel <- unhcr_population("SDN", 2015, 2026) |>
  mutate(year = as.integer(year), refugees = as.numeric(refugees)) |>
  group_by(year) |>
  summarise(refugees = sum(refugees, na.rm = TRUE), .groups = "drop") |>
  right_join(tibble(year = 2015:2026), by = "year") |>
  arrange(year) |>
  mutate(retrieved_at = as.character(Sys.Date()),
         source = "UNHCR Refugee Data Finder")

panel

Discovering HDX resources through CKAN

curl -G "https://data.humdata.org/api/3/action/package_search" \
  --data-urlencode "q=internally displaced sudan" \
  --data-urlencode "rows=5" | jq '.result.results[] | {name, organization: .organization.title, resources: [.resources[] | {format, url}]}'

HDX returns dataset metadata with a resources array. Read the format and url fields, download the file, and record the dataset name and last-modified date in your provenance columns.

Related resources on Humanity Centered Data