---
title: "Catalog differences and troubleshooting"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Catalog differences and troubleshooting}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(httptest2)
start_vignette("hosts")
set_redactor(function(x) {
  gsub_response(x, "index.php/api/catalog/", "")
})
```

All the catalogs use the same NADA software, but they are run by different
organizations on different servers. This guide covers where they differ and what
to do when a request fails. The output comes from saved responses.

```{r library}
library(nadaverse)
```

`catalogs()` lists the supported catalogs with their names and links. The
codes are `df`, `erf`, `fao`, `ihsn`, `kenya`, `unhcr`, `wb`, `who` and
`zambia`.

## Size and columns

Catalogs differ a lot in size. A search with `ps = 1` and `rows = FALSE` is a
cheap way to see how many studies a catalog holds. Here it is run on four
catalogs:

```{r info}
codes <- c("wb", "fao", "ihsn", "kenya")
info <- lapply(codes, search_catalog, ps = 1, rows = FALSE)
names(info) <- codes
total <- function(x) if (is.null(x$total)) NA_real_ else as.numeric(x$total)
vapply(info, total, numeric(1))
```

The columns can also differ, because each catalog decides what to publish.
These are the columns that all four have in common:

```{r columns}
cols <- lapply(info, function(x) names(x$rows))
Reduce(intersect, cols)
```

If your code runs across several catalogs, rely on columns from a list like
this one, and check `names()` before you use any other column.

## Missing features

Some catalogs do not use every feature. IHSN, for example, has no collections.
`collections("ihsn")` warns that it returned 0 collections and gives an empty
result:

```{r ihsn-collections}
length(suppressWarnings(collections("ihsn")))
```

## Slow or unavailable catalogs

Every request waits up to 30 seconds and is attempted up to three times before
it fails, including after connection errors. A catalog that is down can
therefore take about 90 seconds to give up. The error that follows comes from
the connection and names the host.

Some catalogs answer with a web page instead of data, for example a block page
for automated clients. nadaverse notices this and stops with a message like:

```
Error: The catalog did not return JSON from <url>.
i Got content type "text/html" (HTTP 200).
i The catalog may be down or blocking automated requests.
```

When you see an error like this, open the address in a browser to check that
the site is up, wait a while, and try again. If a catalog keeps failing for a
long time, try another catalog or report it at the package's issue tracker.

## The user agent

Every request sends a user agent, a short text that tells the server what kind
of program is asking. By default nadaverse sends a browser-style user agent.
In our checks, UNHCR returned an error when the request did not look like it
came from a browser, while the other catalogs accepted both.

If you want to identify your own project instead, use `set_user_agent()`. It
returns the previous setting invisibly, so you can put it back:

```{r agent}
old <- set_user_agent("My research project (nadaverse)")
getOption("nadaverse.user_agent")
set_user_agent(old)
getOption("nadaverse.user_agent")
```

`NULL` means the default browser user agent is in use. If a catalog starts
rejecting requests after you change the setting, go back to the default with
`set_user_agent(NULL)`.

## Using catalogs considerately

The catalogs are public services. Keep `ps` small when you are exploring, save
results you will need again, and avoid loops that send thousands of requests.
Each catalog also has its own terms for using the data, which you can read on
its website.

```{r teardown, include = FALSE}
end_vignette()
```