---
title: "Chat and Agents"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Chat and Agents}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE, purl=FALSE}
# Every chunk needs a GGUF model (and usually a GPU), so this vignette is
# static: the code is shown but not run at build time.
knitr::opts_chunk$set(eval = FALSE, purl = FALSE)
```

llamaR turns a local GGUF model into a chat backend for the R ecosystem. You
can talk to it several ways, from lowest to highest level:

* **HTTP server** — `llama_serve_openai()` exposes an OpenAI-compatible API any
  client can hit (OpenCode, the `openai` SDK, `curl`).
* **Anthropic server** — `llama_serve_anthropic()` exposes an Anthropic Messages
  API so **Claude Code** runs against local inference (see section 7).
* **ellmer `Chat`** — `chat_llamar()` returns an `ellmer::Chat`, so the whole
  ellmer / ragnar toolchain works against local inference.
* **Command-line example** — `inst/examples/chat.R` wraps both for quick use.

Both servers share a tool-aware chat layer (`llama_chat_build()` /
`llama_chat_parse()`) so tool-calling models work end to end (section 6).

```{r, eval=FALSE, purl=FALSE}
library(llamaR)
```

---

## 1. The chat object: `chat_llamar()`

`chat_llamar()` returns an [ellmer](https://ellmer.tidyverse.org/) `Chat`. It
has two modes, picked by which argument you pass — the same DBI-style choice as
`DBI::dbConnect()` (connection parameters *or* a ready connection).

### Mode A — spawn a server for a model

Give it a model file and it starts `llama_serve_openai()` in a background
process (via the **callr** package), waits for it to come up, and points a
`Chat` at it. The server's lifetime is tied to the returned object: when it is
garbage-collected (or R exits) the process is killed.

```{r, eval=FALSE, purl=FALSE}
chat <- chat_llamar(model_path = "Ministral-3B-Instruct.gguf")

chat$chat("Why is the sky blue?")

chat_llamar_stop(chat)   # stop the spawned server (or just let GC do it)
```

Large models can take a while to load from disk; raise `timeout` (default 180s)
if a 14B at Q8 doesn't come up in time:

```{r, eval=FALSE, purl=FALSE}
chat <- chat_llamar(model_path = "Qwen3-14B-Q8_0.gguf", timeout = 300)
```

### Mode B — connect to a running server

If you already run a server (in another process, or a pool of them), pass its
URL. No process is spawned.

```{r, eval=FALSE, purl=FALSE}
# In another process / shell:
#   llama_serve_openai("model.gguf", port = 11434L)

chat <- chat_llamar(base_url = "http://127.0.0.1:11434/v1")
chat$chat("Hello!")
```

### System prompt

```{r, eval=FALSE, purl=FALSE}
chat <- chat_llamar(
  model_path    = "Ministral-3B-Instruct.gguf",
  system_prompt = "You are a concise assistant. Answer in one sentence."
)
chat$chat("What is R?")
```

> **Under the hood.** `chat_llamar()` wraps `ellmer::chat_vllm()`, which talks
> to the server's `/v1/chat/completions` endpoint — the de-facto standard our
> server implements. (ellmer's `chat_openai()` targets OpenAI's newer
> `/v1/responses` API, which the server does not implement.)

---

## 2. The server: `llama_serve_openai()`

`chat_llamar(model_path=)` is a convenience wrapper; you can run the server
directly for non-R clients. It needs the optional **drogonR** package for the
HTTP/SSE layer.

```{r, eval=FALSE, purl=FALSE}
llama_serve_openai("model.gguf", port = 11434L, n_ctx = 8192L)
```

It blocks, serving:

* `GET /v1/models`
* `POST /v1/chat/completions` (both blocking and `stream = true`)

Point any OpenAI client at `http://127.0.0.1:11434/v1`:

```bash
curl http://127.0.0.1:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"model","messages":[{"role":"user","content":"Hello"}]}'
```

A runnable launcher lives at `inst/examples/serve_openai.R`.

### Connecting OpenCode

Add an OpenAI-compatible provider in `opencode.json` (see the one in this repo)
with `baseURL` set to `http://127.0.0.1:11434/v1` and the model id matching what
`/v1/models` reports.

---

## 3. The command-line example

`inst/examples/chat.R` wraps both modes for the terminal:

```bash
# Spawn a server for the model and open an interactive prompt
Rscript inst/examples/chat.R model.gguf

# Positional [port] [n_ctx], plus flags
Rscript inst/examples/chat.R model.gguf 11434 8192 \
  --system "Be concise." --timeout 300

# One-shot: a trailing message prints a single reply and exits
Rscript inst/examples/chat.R model.gguf "Why is the sky blue?"

# Connect to a server you already started
Rscript inst/examples/chat.R --url http://127.0.0.1:11434/v1
```

In interactive mode, type a message and press Enter; a blank line or Ctrl-D
quits. A spawned server is stopped automatically on exit.

---

## 4. ragnar: retrieval-augmented chat

Because `chat_llamar()` returns a real `ellmer::Chat`, it plugs into
[ragnar](https://ragnar.tidyverse.org/). Pair it with `embed_llamar()` (see
`vignette("getting-started")`) for a fully local RAG stack: local embeddings
for the store, local generation for the chat.

```{r, eval=FALSE, purl=FALSE}
library(ragnar)

store <- ragnar_store_create(
  location = "store.duckdb",
  embed    = embed_llamar(model = "embedding-model.gguf")
)
ragnar_store_insert(store, documents)
ragnar_store_build_index(store)

chat <- chat_llamar(model_path = "Ministral-3B-Instruct.gguf")
ragnar_register_tool_retrieve(chat, store)
chat$chat("What do the documents say about X?")
```

> **Note.** Tool calling is mediated by the chat protocol. Both servers emit
> `tool_calls` (the tool-aware chat layer, see section 6), so a tool-calling
> model can autonomously invoke the registered retrieve tool. Tool-calling
> quality is model-dependent — capable models (e.g. Qwen3) drive retrieval
> reliably; very small models may need manual retrieval.

---

## 5. Tool calling: `llama_chat_build()` / `llama_chat_parse()`

The servers above call a lower-level, tool-aware chat layer you can also use
directly. `llama_chat_build()` applies the model's chat template to messages
**plus tool definitions** and returns everything needed to constrain and parse
a tool call; `llama_chat_parse()` turns the raw output back into structured
tool calls.

```{r, eval=FALSE, purl=FALSE}
tools <- list(list(
  type = "function",
  "function" = list(
    name        = "get_weather",
    description = "Get the current weather for a city.",
    parameters  = list(
      type       = "object",
      properties = list(city = list(type = "string")),
      required   = list("city")
    )
  )
))

messages <- list(list(role = "user", content = "What's the weather in Paris?"))

built <- llama_chat_build(model, messages, tools = tools)
# built$prompt   — the formatted prompt to feed the model
# built$grammar  — grammar that constrains tool-call output
# built$format   — format id to pass to llama_chat_parse()
# built$grammar_lazy, built$trigger_patterns, built$trigger_tokens,
# built$additional_stops, built$preserved_tokens, built$parser
```

Generate against the returned prompt and grammar, then parse. Pass the lazy
triggers **only when `built$grammar_lazy` is `TRUE`**:

```{r, eval=FALSE, purl=FALSE}
lazy <- isTRUE(built$grammar_lazy)
raw <- llama_generate(
  ctx, built$prompt,
  grammar          = built$grammar,
  trigger_patterns = if (lazy) built$trigger_patterns,
  trigger_tokens   = if (lazy) built$trigger_tokens
)

parsed <- llama_chat_parse(raw, format = built$format, parser = built$parser)
parsed$content        # assistant text (may be empty for a pure tool call)
parsed$tool_calls     # data frame: name, arguments (JSON string), id
```

Tool-calling quality is model-dependent: capable models (e.g. Qwen3) emit clean
tool calls; very small models are less reliable.

---

## 6. Claude Code on a local model: `llama_serve_anthropic()`

`llama_serve_anthropic()` exposes an **Anthropic Messages API** so Claude Code
(or any Anthropic SDK client) runs against local inference. It uses the
tool-aware layer from section 5 and streams the Anthropic SSE event sequence.
Like the OpenAI server it needs the optional **drogonR** package.

```{r, eval=FALSE, purl=FALSE}
llama_serve_anthropic("model.gguf", port = 11435L, n_ctx = 32768L)
```

It serves `POST /v1/messages` and `GET /v1/models`. Point Claude Code at it via
environment variables:

```bash
export ANTHROPIC_BASE_URL=http://127.0.0.1:11435
export ANTHROPIC_API_KEY=local   # any non-empty value
claude
```

`enable_thinking` (default `FALSE`) toggles the chat template's reasoning mode
for hybrid thinking models (Qwen3.5, etc.). It is off by default so Claude Code
gets direct answers and fast tool calls; set it `TRUE` (and raise `max_tokens`)
to keep the reasoning trace. A runnable launcher lives at
`inst/examples/claude_code_launcher.sh`.

---

## 7. Concurrency

The server is **single-sequence**: it handles one request at a time on the main
R thread. That is enough for a single local user or agent. For parallel
sessions, run a pool of servers on different ports and create one
`chat_llamar(base_url=)` per worker — the worker-pool architecture is described
in `TODO.md`.

```{r, eval=FALSE, purl=FALSE}
ports <- c(11434L, 11435L, 11436L)
chats <- lapply(ports, function(p)
  chat_llamar(base_url = sprintf("http://127.0.0.1:%d/v1", p)))
```

---

## See also

* `vignette("getting-started")` — the rest of the package.
* `?chat_llamar`, `?llama_serve_openai`, `?llama_serve_anthropic`
* `?llama_chat_build`, `?llama_chat_parse`
* `inst/examples/chat.R`, `inst/examples/serve_openai.R`,
  `inst/examples/claude_code_launcher.sh`