nadaverse: Browse Microdata Catalogs Using NADA REST API

nadaverse gives researchers, policy analysts, and data enthusiasts streamlined, programmatic access to collections of global microdata.

Many national and international organizations—including the World Bank, IHSN, FAO, UNHCR, the WHO, and national statistics offices—use the National Data Archive (NADA) software to manage and disseminate their survey and census data. While these catalogs are rich sources of information, interacting with them often requires tedious manual browsing or complex API construction.

nadaverse cuts through that complexity. It provides a unified, reliable, and user-friendly interface to search, filter, and retrieve crucial metadata and documentation (such as file lists and data dictionaries) directly into your R environment.

Features

• Search across 9 NADA catalogs using a unified interface
  
• Retrieve study metadata, file inventories, and data dictionaries
  
• Filter by year, keywords, country, catalog, and more
  
• Built on httr2, with a 30-second timeout and up to three attempts per request
  
• Tidy-friendly: outputs are clean data frames ready for immediate use
  
• Includes convenient helper functions for codes, catalogs, and recent entries

Installation

Install the CRAN release:

install.packages("nadaverse")

Or install the development version from GitHub:

devtools::install_github("guturago/nadaverse")

Searching

1. Catalog Discovery

The catalogs() function is the starting point, providing a complete, current list of the supported NADA repositories, along with their unique identifiers required for subsequent queries.

library(nadaverse)
library(tidyverse)
library(knitr)

catalogs()
#> 
#> ── List of Supported Catalogs ──
#> 
#> ℹ name: Link to the catalog
#> • df: Data First (<https://www.datafirst.uct.ac.za>)
#> • erf: Economic Research Forum (<https://erfdataportal.com>)
#> • fao: Food and Agriculture Organization (<https://microdata.fao.org>)
#> • ihsn: International Household Survey Network (<https://catalog.ihsn.org>)
#> • kenya: Kenya National Bureau of Statistics
#> (<https://statistics.knbs.or.ke/nada/index.php>)
#> • unhcr: United Nations High Commissioner for Refugees
#> (<https://microdata.unhcr.org>)
#> • wb: The World Bank (<https://microdata.worldbank.org>)
#> • who: World Health Organization Health Information Systems
#> (<https://apps.who.int/healthinfo/systems/surveydata/index.php/catalog>)
#> • zambia: Zambia Statistics Agency (<https://nada.zamstats.gov.zm>)

2. Targeted Metadata Search

The search_catalog() function allows for granular control over the search space. Instead of relying on the catalog’s often limited web interface, users can programmatically search by catalog ID, keywords, publication date ranges, and more.

The output is a standardized data frame, simplifying cross-catalog comparisons. Here, we search the IHSN catalog (ihsn) for studies from 2023 to 2025:

search_catalog(
  catalog = "ihsn",
  from = 2023, 
  to = 2025,
  ps = 5
)

3. Deep Dive: File and Variable Metadata

Once a specific study is identified via its unique ID (e.g., 3110), nadaverse enables the retrieval of documentation critical for data preparation.

File Inventory (data_files): This function retrieves the list of data file assets, their size, and descriptions, allowing users to determine the exact resources needed for download.

ctl <- "wb"
data_files(ctl, 3110) |> 
  select(where(~ !all(. == "NULL"))) |> 
  kable(format = "pipe")

Data Dictionary (data_dictionary): Access to variable-level metadata is paramount for data quality checks and ethical use. This function retrieves the complete data dictionary, including variable names and labels, enabling preparation work before downloading large datasets.

data_dictionary(ctl, 3110) |>
  head(10) |> 
  select(where(~ !all(. == "NULL"))) |> 
  kable(format = "pipe")

Advanced Wrangling and Analysis Preparation

The design goal of nadaverse is to ensure its outputs are immediately “tidy” and ready for integration into analytical pipelines. This means the results can be piped directly into dplyr verbs for filtering, reshaping, and analysis preparation, as demonstrated by this example.

This transformation searches the FAO catalog, filters studies by keyword (“Food Insecurity”), and reshapes the resulting metadata into a concise matrix showing which countries conducted the survey in which years—a common preparatory step for cross-country comparative research.

search_catalog("fao", "Food Insecurity", ps = 10000) |>
  filter(grepl("Food Insecurity Experience Scale", title, TRUE)) |>
  select(nation, year_start) |>
  arrange(nation, year_start) |> 
  mutate(value = "Yes") |>
  pivot_wider(id_cols = nation,
              names_from = year_start,
              values_from = value,
              values_fill = "-") |>
  head(5) |> 
  kable(format = "pipe")

Helper Functions for Workflow Efficiency

To further streamline the research process, nadaverse includes several helper functions that provide necessary IDs and codes used as query parameters in NADA systems.

These utility functions assist in identifying necessary access codes, collection names, and country codes to use in search_catalog() queries.

access_codes("fao")
collections("wb")
country_codes("wb")
latest_entries("ihsn")

Request settings

Each request waits up to 30 seconds and is attempted up to three times before nadaverse reports an error. If a catalog returns something other than JSON, such as a block page for automated clients, you get a clear error message.

By default, requests send a browser user agent, because some catalogs, such as UNHCR, reject other clients. Use set_user_agent() to send your own:

set_user_agent("My research project (nadaverse)")

# Back to the default
set_user_agent(NULL)