Policy Learning with Decision-Theoretic Bounds

Deniz Akdemir

2026-11-09

Introduction

This vignette demonstrates how to use causaldef for safe policy learning — making treatment decisions with quantified guarantees even when unobserved confounding exists.

The key insight is the policy regret transfer bound:

\[\text{Regret}_{do}(\pi) \leq \text{Regret}_{sim}(\pi) + 2|\mathcal{A}|\, M \cdot \delta\]

where: - \(\text{Regret}_{do}(\pi)\) = regret under the true interventional distribution - \(\text{Regret}_{sim}(\pi)\) = regret computed under the adjusted (simulated) interventional laws - \(|\mathcal{A}|\) = number of actions (2 for a binary treatment; the factor drops to 1 when comparing fixed actions) - \(M\) = utility range (max - min possible outcomes) - \(\delta\) = deficiency (quantifies confounding)

The Safety Floor Concept

policy_regret_bound() reports two complementary quantities:

(Versions up to 0.2.1 used \(M\delta\) and \((M/2)\delta\); both constants were corrected in 0.3.0.)

If \(\delta>0\), no algorithm can guarantee zero worst-case regret without stronger assumptions or randomized data.

Implications for AI/ML Safety

  1. No algorithm can beat the safety floor: Even infinite data doesn’t help if confounding exists
  2. Deficiency is the price of observational learning: To eliminate the safety floor, you need randomized experiments
  3. Confidence intervals aren’t enough: Standard ML uncertainty quantification doesn’t capture confounding bias

Practical Workflow

Step 1: Define the Causal Problem

Step 2: Estimate Deficiency

What \(\delta\) is here. estimate_deficiency() returns a propensity-score total-variation proxy, not an identified estimate of the exact deficiency. Bounds below that are stated in terms of \(\delta\) inherit that: they are only as good as the proxy standing in for it.

Step 3: Visualize Deficiency

Step 4: Compute Policy Regret Bounds

Step 5: Visualize the Safety Floor

Interpreting the Results

The Safety Floor Report

Sensitivity Analysis with Confounding Frontiers

What if there’s additional unmeasured confounding?

# Map the confounding frontier
frontier <- confounding_frontier(
  spec,
  alpha_range = c(-2, 2),
  gamma_range = c(-2, 2),
  grid_size = 30
)
#> ℹ Computing benchmarks for observed covariates...
#> ✔ Computed confounding frontier: 30x30 grid

# Find an illustrative low-risk region for this utility scale
safe_region <- subset(frontier$grid, delta < 0.1)
cat(sprintf(
  "Illustrative low-risk region covers %.1f%% of confounding space\n",
  100 * nrow(safe_region) / nrow(frontier$grid)
))
#> Illustrative low-risk region covers 100.0% of confounding space

Visualize the Frontier

Policy Learning with grf (Optional)

If you have the grf package installed, you can use causal forests for heterogeneous treatment effect estimation with deficiency bounds:

# Estimate deficiency using causal forests
if (requireNamespace("grf", quietly = TRUE)) {
  def_grf <- estimate_deficiency(
    spec,
    methods = c("aipw", "grf"),
    n_boot = 50
  )
  
  print(def_grf)
  
  # Get individual treatment effect predictions
  kernel_grf <- def_grf$kernel$grf
  if (!is.null(kernel_grf$tau_hat)) {
    cat("\nHeterogeneous Effects Detected:\n")
    cat(sprintf("ATE from forest: %.2f\n", kernel_grf$ate))
    cat(sprintf("CATE range: [%.2f, %.2f]\n", 
                min(kernel_grf$tau_hat), 
                max(kernel_grf$tau_hat)))
  }
}

Best Practices for Safe Deployment

Pre-Deployment Checklist

These cutoffs are illustrative for the 0–100 utility scale in this vignette. In practice, calibrate them to the decision stakes.

Check Illustrative heuristic Action if Failed
Transfer penalty < 5 points Small additional regret budget Deployment may be reasonable
Transfer penalty in [5, 10) points Material but possibly manageable residual risk Deploy only with active monitoring
Transfer penalty >= 10 points Large residual regret budget Consider pilot RCT or narrower deployment
NC diagnostic falsified Any Do not deploy without more data or stronger assumptions

Monitoring in Production

Mathematical Details

Policy Regret Transfer (Manuscript)

For any policy \(\pi\) and bounded utility function \(u \in [0, M]\):

\[\text{Regret}_{do}(\pi) \leq \text{Regret}_{sim}(\pi) + 2|\mathcal{A}|\,M\delta\]

Proof sketch: For each action \(a\), the deficiency \(\delta\) bounds the total variation distance between the simulated and true interventional laws. A policy’s value is a sum over actions of expectations of functions with values in \([0, M]\), so its simulated value is off by at most \(|\mathcal{A}|M\delta\); a regret is a difference of two values, which gives the factor 2. The factor 2 cannot be removed in general.

Why This Matters

Traditional ML focuses on: - Prediction error: How well does my model predict \(Y\)? - Generalization: Does performance hold on new data?

But for causal policy learning, we need: - Interventional validity: Does my policy work when deployed? - Confounding robustness: How much could unmeasured bias hurt me?

The safety floor answers these questions with formal guarantees.

Summary

Concept Definition Function
Transfer penalty \(2|\mathcal{A}|M\delta\) — additive regret inflation term $transfer_penalty
Minimax safety floor \((M/4)\delta\) — irreducible worst-case regret $minimax_floor
Regret bound simulated regret + transfer penalty $regret_bound
Deficiency Information gap between obs and do estimate_deficiency()
Confounding Frontier Deficiency as function of \((\alpha, \gamma)\) confounding_frontier()

Use these tools to make safe, accountable decisions from observational data.

References

  1. Akdemir, D. (2026). Constraints on Causal Inference as Experiment Comparison. DOI: 10.5281/zenodo.18367346. See thm:policy_regret (Policy Regret Transfer) and thm:safety_floor (Minimax Safety Floor).

  2. Athey, S., & Wager, S. (2021). Policy learning with observational data. Econometrica, 89(1), 133-161.

  3. Kallus, N. (2020). Confounding-robust policy evaluation in infinite-horizon reinforcement learning. NeurIPS.