---
title: "Working with studies"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Working with studies}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(httptest2)
start_vignette("std")
set_redactor(function(x) {
  gsub_response(x, "index.php/api/catalog/", "")
})
```

After a search, four functions let you look closely at a study before you
download anything. `metadata()` gives the description, `data_files()` lists the
data files, and `data_dictionary()` lists the variables. This guide uses
World Bank studies, and the output comes from saved responses.

```{r library}
library(nadaverse)
```

## Two ways to name a study

A study can be given as the number in the `id` column of a search or as the
text in the `idno` column. Both work in every function that takes a study.

```{r ids}
md_num <- metadata("wb", 8098)
md_txt <- metadata("wb", "ALB_2012_LSMS_v01_M_v01_A_PUF")
md_num$study_desc$title_statement$title
md_txt$study_desc$title_statement$title
```

These are two different studies, one named by number and one by text. For a
single study, both forms return the same result.

## What the metadata contains

`metadata()` returns a nested list. The top level separates the description of
the documentation from the description of the study.

```{r sections}
names(md_txt)
names(md_txt$study_desc)
```

Inside `study_desc`, `study_info` holds the scope and coverage of the study,
`method` describes the design and data collection, and `data_access` gives the
terms of use. You can follow the list down with `$`, or use `str()` to look at
one part:

```{r title}
str(md_txt$study_desc$title_statement)
```

```{r study-info}
names(md_txt$study_desc$study_info)
```

## Data files

`data_files()` has one row per file in the study. This section uses the
Ethiopia Socioeconomic Survey from 2015.

```{r files}
study <- "ETH_2015_ESS_v03_M"
files <- data_files("wb", study)
nrow(files)
```

Some columns of the table can hold lists, which print in an awkward way. If
that happens, `unlist()` each column you need:

```{r files-tidy}
files_tidy <- data.frame(
  file_id = unlist(files$file_id),
  file_name = unlist(files$file_name),
  vars = as.numeric(unlist(files$var_count))
)
head(files_tidy, 5)
```

## Data dictionaries

Without a `file_id`, `data_dictionary()` returns the variables of the whole
study.

```{r dictionary}
dict <- data_dictionary("wb", study)
nrow(dict)
head(dict[, c("vid", "name", "labl")], 5)
```

Large studies can have thousands of variables, so it often pays to ask for one
file at a time. Here the smallest file is chosen by its number of variables:

```{r one-file}
smallest <- files_tidy$file_id[which.min(files_tidy$vars)]
dict_file <- data_dictionary("wb", study, file_id = smallest)
nrow(dict_file)
```

## Finding variables

A dictionary is an ordinary data frame, so you can search it with base R. This
finds variables whose label mentions households:

```{r find}
hits <- dict[grepl("household", dict$labl, ignore.case = TRUE), ]
head(hits[, c("vid", "name", "labl")], 5)
```

Dictionaries can be large and slow to download. Fetch one once, keep it in an
object, and use `saveRDS()` if you want to reuse it in a later session.

```{r teardown, include = FALSE}
end_vignette()
```