---
title: "Information and Error-Rate Metrics"
subtitle: "EIG and PPV-Range for Lineup Decisions"
author:
- "Colin G. Tredoux"
- "Tamsyn M. Naylor"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Information and Error-Rate Metrics}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

> **Acknowledgement.** Large language models supplied by Anthropic and OpenAI
> were used in preparing this vignette, to assist with checking code, testing
> the veracity of the approaches described, and on occasion editing the text.
> Responsibility for the content rests with the authors.

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5,
  warning = FALSE,
  message = FALSE
)
```

## Introduction

This vignette demonstrates two complementary approaches for evaluating lineup evidence:

1. **Expected Information Gain (EIG)** following Starns et al., which quantifies how much a response reduces uncertainty about guilt.
2. **PPV-range estimation** following Fitzgerald, Tredoux, and Juncu, which reports how positive predictive value changes under different corrections for lineup size.

Both methods use the same data format as ROC/CAC analyses: a data frame with `target_present`, `identification`, and `confidence` columns.

```{r}
library(r4lineups)
data(lineup_example)
head(lineup_example)
```

## 1. Expected Information Gain (EIG)

### In plain terms

Before running a lineup, you are uncertain whether the suspect is guilty. After
the witness responds, you are — usually — less uncertain. EIG measures **how much
of that uncertainty the procedure removes, on average**.

The unit is the **bit**, which has a concrete meaning: one bit is the
uncertainty in a fair coin toss. If you begin thinking guilt is a coin toss and
end up certain, you have gained exactly 1 bit. An EIG of 0 means the responses
told you nothing; an EIG of 1 (with a 50% prior) would mean every response
resolved the case completely.

The virtue of this is that it grades *every* response, not just identifications.
A confident rejection is informative — it makes guilt less likely — and EIG
counts that, where a measure built only from suspect identifications cannot.

### The arithmetic

Uncertainty is Shannon entropy. For a probability \(p\) of guilt,

\[
H(p) = -p \log_2 p - (1-p) \log_2 (1-p)
\]

which is 1 bit at \(p = 0.5\) and 0 bits at \(p = 0\) or \(p = 1\).

For each response category \(x\) — a decision crossed with a confidence bin —
Bayes' theorem gives a posterior \(P(G \mid x)\), and the information that
response carried is the drop in entropy:

\[
\mathrm{IG}(x) = H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr)
\]

EIG is the average of those, weighted by how often each response occurs:

\[
\mathrm{EIG} = \sum_x P(x)\,\Bigl[H\bigl(P(G)\bigr) - H\bigl(P(G \mid x)\bigr)\Bigr]
\]

Two consequences follow. Individual responses can have **negative**
information gain — a response that moves belief towards 0.5 increases
uncertainty — but the expectation over all responses cannot be negative. And EIG
depends on the prior \(P(G)\), which is set by `prior_guilt` and is a property
of the investigation, not of the procedure.

```{r eig_compute}
eig_obj <- make_eig(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  show_plot = FALSE
)

eig_obj$eig
```

Against a prior entropy of `r round(eig_obj$prior_entropy, 2)` bit, that is
`r round(100 * eig_obj$eig / eig_obj$prior_entropy)`% of the uncertainty
removed — informative, but a long way from resolving the question.

### Response-level information gain

```{r eig_responses}
knitr::kable(
  head(as.data.frame(eig_obj$response_data), 8),
  digits = 4, row.names = FALSE,
  caption = "Per-response information gain and posterior probability of guilt, lineup_example."
)
```

### Plots

```{r plot_eig}
plot_eig(eig_obj)
```

```{r plot_eig_posteriors}
plot_eig_posteriors(eig_obj)
```

## 2. PPV-range estimation

Positive predictive value is the proportion of suspect identifications that are
correct: hits / (hits + false alarms). The difficulty is the denominator. In
field data you observe that *someone* was identified, but not whether a
target-absent identification landed on the suspect or on a foil — and only the
former is a wrongful identification.

The response is to *estimate* the innocent-suspect identifications from the
observed errors, and the estimate depends on an assumption about lineup size
(Fitzgerald, Tredoux & Juncu, 2023):

- **Nominal size.** Errors are spread evenly over all \(k\) members, so
  \(1/k\) of them fall on the innocent suspect. This shows the risk to an
  innocent suspect if the lineup had been fair (Fitzgerald et al., 2023), and
  it requires the lineup to be perfectly fair.
- **Effective size.** Errors are spread over the members that actually attract
  choices, estimated from the data. Fewer functioning alternatives means more
  errors land on the suspect. The correction assumes that the innocent suspect
  is one of the more plausible members of the lineup.
- **No correction.** Every mistaken identification is treated as an
  innocent-suspect identification. The worst case, and a genuine lower bound.

Reporting all three gives a *range* rather than a single figure resting on an
assumption that cannot be checked. Only the uncorrected figure rests on no
assumption about the suspect; the other two are conditional on where the
innocent suspect falls among the members.

The PPVs in this vignette also rest on a prior. With `base_rate = NULL`, the
default, PPV is a ratio of counts, so the implicit prior is the study's own
ratio of target-present to target-absent lineups. `lineup_example` has equal
numbers of each, so every PPV below assumes a prior of .5. Supply the
base rate that applies to the case at hand through the `base_rate` argument.

### When the three coincide, and why

`lineup_example` has a **designated innocent suspect**: its target-absent
lineups record whether the identification fell on the suspect or a foil. When
that is known, nothing needs estimating, and the corrections have no work to do:

```{r ppv_designated}
ppv_designated <- make_ppv_range(
  lineup_example,
  lineup_size = 6,
  show_plots = FALSE
)

knitr::kable(
  data.frame(
    Correction = c("Nominal size", "Effective size", "None"),
    `Overall PPV` = round(c(ppv_designated$ppv_nominal$overall_ppv,
                            ppv_designated$ppv_effective$overall_ppv,
                            ppv_designated$ppv_none$overall_ppv), 4),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "PPV under three corrections, lineup_example -- which designates an innocent suspect, so all three coincide."
)
```

All three are identical, and that is correct rather than a fault: the count
being corrected for is observed directly. `make_ppv_range()` says so when it
happens.

### When they diverge

The corrections are important when no innocent suspect is designated — the usual
situation with field data. We can create that case by relabelling the
target-absent suspect identifications as foil identifications **and** removing
the `suspect_pos` column, which is exactly the information loss that occurs when
the lineup did not nominate a suspect in advance. Both steps are needed: a
non-missing `suspect_pos` tells `make_ppv_range()` that a suspect was designated,
and the design would then stay designated with zero innocent-suspect
identifications, giving PPV = 1 under every correction.

```{r ppv_estimated}
no_designation <- lineup_example
ta_suspect <- !no_designation$target_present &
               no_designation$identification == "suspect"
no_designation$identification[ta_suspect] <- "filler"
no_designation$suspect_pos <- NULL

ppv_estimated <- make_ppv_range(
  no_designation,
  lineup_size = 6,
  innocent_suspect = "estimated",  # state the design rather than rely on "auto"
  show_plots = FALSE
)

knitr::kable(
  data.frame(
    Correction = c("Nominal size (fair lineup)",
                   "Effective size",
                   "None (worst case)"),
    `Overall PPV` = round(c(ppv_estimated$ppv_nominal$overall_ppv,
                            ppv_estimated$ppv_effective$overall_ppv,
                            ppv_estimated$ppv_none$overall_ppv), 4),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "PPV under three corrections, same data with the innocent-suspect designation removed. This is the range the method exists to report."
)
```

The spread is large, from about
`r round(ppv_estimated$ppv_none$overall_ppv, 2)` to about
`r round(ppv_estimated$ppv_nominal$overall_ppv, 2)`. An analysis reporting only the nominal-size figure would be asserting
that the lineup was perfectly fair, which is precisely what is unknown.

The effective-size correction sits between the two because the lineup's members
do not attract choices equally:

```{r effective_sizes}
knitr::kable(
  data.frame(
    `Confidence bin` = ppv_estimated$ppv_effective$ppv_data$confidence,
    `Effective size` = round(ppv_estimated$ppv_effective$ppv_data$effective_size, 2),
    `Error rate`     = round(ppv_estimated$ppv_effective$ppv_data$error_rate, 3),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "Effective size estimated within each confidence bin, against a nominal size of 6."
)
```

The error rate in this table is the proportion of target-absent responses in
the bin that are identifications. Rejections fall into few of these confidence
bins, so most bins contain only identifications and show 1.000. The estimated
number of innocent-suspect identifications, and so the PPV, does not depend on
that denominator, because the error rate is multiplied back by the same number
of target-absent responses.

### Plots

```{r plot_ppv_range}
plot_ppv_range(ppv_estimated)
```

```{r plot_effective_size}
plot_effective_size_conf(ppv_estimated)
```

## Interpretation guidance

- **EIG** grades every response category, including rejections, and reports how
  much uncertainty the procedure removes overall. Use it to compare procedures,
  or to identify which responses carry the evidence.
- **PPV range** answers the question a court asks — how often is an
  identification of this kind correct — and makes visible how much the answer
  depends on an untestable assumption about lineup size.

The two are complementary: EIG covers all responses and depends on the prior
through `prior_guilt`; PPV concerns identifications only, and is the quantity
with direct forensic meaning. PPV depends on the prior as well, through the
study's ratio of target-present to target-absent lineups or the `base_rate`
argument, so a court needs the base rate for the case at hand.

## Technical Details

### EIG

Computed over response categories formed by crossing decision with confidence
bin. For each category the posterior comes from Bayes' theorem using the
observed likelihoods \(P(x \mid G)\) and \(P(x \mid \neg G)\), estimated as
response proportions within the target-present and target-absent arms
respectively. When no innocent suspect is designated, `make_eig_data()` first
moves \(1/k\) of the target-absent filler identifications into the matching
suspect category, so EIG then assumes a fair lineup in the same way as the
nominal-size correction. Entropies are in bits (base-2 logarithms). The prior is set by
`prior_guilt`, defaulting to 0.5, at which the prior entropy is exactly 1 bit.

EIG inherits the binning issue discussed in
`vignette("calibration_decision_analysis")`: it is computed over discrete
categories, so a category with no observations in one arm gives an unbounded
likelihood ratio, and the number of bins is a researcher decision that affects
the result.

### PPV range

With a designated innocent suspect, innocent-suspect identifications are counted
directly and no correction is applied. Otherwise, writing \(e\) for the
target-absent error rate within a confidence bin:

\[
\hat{r}_{\text{nominal}} = e/k, \qquad
\hat{r}_{\text{effective}} = e/\hat{E}, \qquad
\hat{r}_{\text{none}} = e
\]

where \(\hat{E}\) is effective size estimated within the bin. PPV then follows
as hits / (hits + estimated innocent-suspect identifications).

### Assumptions

1. Target-present and target-absent lineups are independent samples.
2. The nominal-size correction assumes a perfectly fair lineup. It shows the
   risk to an innocent suspect if the lineup had been fair, not an estimate of
   the actual risk. The effective-size correction assumes that the innocent
   suspect is one of the more plausible members of the lineup.
3. Effective size is estimated per confidence bin, so bins with few observations
   give unstable corrections — check the counts before relying on a bin.
4. Effective size is itself a plug-in estimate with small-sample bias; see the
   "Small-sample bias" section of `?esize_T`. In near-fair lineups the bias is
   downward, which makes the correction more conservative, but its direction is
   not guaranteed: with strongly unequal choices the plug-in estimate can be
   biased upward. Bins with few target-absent choices deserve particular care.

## References

::: {.hanging-indent}

Fitzgerald, R. J., Tredoux, C. G., & Juncu, S. (2023). Estimation of eyewitness error rates in fair and biased lineups. *Law and Human Behavior, 47*(4), 463-483.

Starns, J. J., Cohen, A. L., & Rotello, C. M. (2023). A complete method for assessing the effectiveness of eyewitness identification procedures: Expected information gain. *Psychological Review, 130*(3), 677-719.

:::

## Further Reading

- `?make_eig`, `?make_ppv_range`
- `vignette("calibration_decision_analysis")` for the Bayesian information-gain
  curves, which report the same evidence on a per-response rather than expected
  basis