Guidance

Overview

Guidance (guidance-ai/guidance, version 0.3.x) constrains the token sampling itself, so output always matches a regex, a fixed list of options, a JSON schema or a context-free grammar. Model objects are immutable: lm += ... returns a new model with the text and any named captures. Backends: Transformers, LlamaCpp, OnnxRuntimeGenAI, OpenAI, Azure helpers (create_azure_openai_model, create_azure_aifoundry_model) and Mock for tests. Constraints are fully enforced only where the backend gives token-level control, i.e. local models; hosted APIs restrict what can be expressed.

Instructions

Install

pip install guidance          # backend packages (transformers, torch, llama-cpp-python) are installed separately

Chat models: roles, capture, regex, select

from guidance import system, user, assistant, gen, select
from guidance.models import Transformers

base = Transformers("microsoft/Phi-4-mini-instruct")   # downloads the model on first use

lm = base
with system():
    lm += "You classify product reviews."
with user():
    lm += 'Review: "The product arrived damaged but customer service was great"'
with assistant():
    lm += select(["positive", "negative", "mixed", "neutral"], name="sentiment")

print(lm["sentiment"])       # one of the four labels, nothing else is possible

gen(regex=r"0\.\d{2}", name="confidence") constrains a free-text field to a pattern; gen(max_tokens=100, stop="\n") bounds length. Read captures with lm["name"].

Guaranteed JSON from a Pydantic model or schema

import json
from pydantic import BaseModel
from guidance import json as gen_json

class ProductReview(BaseModel):
    product_name: str
    rating: int
    pros: list[str]
    cons: list[str]
    recommendation: bool

lm = base
with user():
    lm += "Extract data: The XPS 15 has an amazing display and battery life, but runs hot. Would buy again."
with assistant():
    lm += gen_json(name="review", schema=ProductReview)

review = ProductReview.model_validate_json(lm["review"])

schema accepts a Pydantic class or a JSON-schema dict. Use model_config = dict(extra="forbid") on the Pydantic class for strict objects.

Control flow and lists

Python if/for run between generations; f-strings can embed grammar pieces. one_or_more and zero_or_more (from guidance) add repetition to a grammar:

lm = base
with user():
    lm += "Classify this message: 'Please restart the staging database'"
with assistant():
    lm += f"Type: {select(['question', 'command', 'chitchat'], name='kind')}"

if lm["kind"] == "command":
    with user():
        lm += "Write the shell command only."
    with assistant():
        lm += gen(name="command", stop="\n", max_tokens=80)

Reusable grammar functions and offline testing

Mark functions with @guidance (add stateless=True to compose them into a grammar). Debug constraints without a model:

from guidance import gen, select
from guidance.models import Mock

grammar = "Sentiment: " + select(["positive", "negative", "mixed"], name="sentiment") + "\nScore: " + gen(regex=r"0\.\d{2}", name="score")
assert grammar.match("Sentiment: mixed\nScore: 0.82") is not None
assert grammar.match("Sentiment: great\nScore: 0.82") is None

lm = Mock(b"<s>Sentiment: mixed\nScore: 0.82") + grammar
print(lm["sentiment"], lm["score"])     # mixed 0.82

Examples

"Tag support tickets with a priority and a due-date format"

lm = base
with user():
    lm += "Ticket: 'Checkout page returns 500 for all EU customers since 14:00'"
with assistant():
    lm += "Priority: " + select(["P1", "P2", "P3"], name="priority")
    lm += "\nDue: " + gen(regex=r"20\d{2}-\d{2}-\d{2}", name="due")

Result: lm["priority"] is P1, P2 or P3, and lm["due"] always looks like 2026-10-21; no parsing or retry is needed.

"Extract an invoice into typed JSON"

class Invoice(BaseModel):
    vendor: str
    total_eur: float
    paid: bool

with user():
    lm += "Invoice text: Hetzner Online GmbH, total 41.65 EUR, status: paid"
with assistant():
    lm += gen_json(name="invoice", schema=Invoice)

Result: {"vendor": "Hetzner Online GmbH", "total_eur": 41.65, "paid": true}, valid against Invoice by construction (content correctness still depends on the model).

Guidelines

• Valid format does not mean correct content: constraints guarantee syntax, not facts. Validate semantics separately.
• Use select() for classification and regex= for IDs, dates and numbers; always give gen() a max_tokens or stop when the field is free text.
• Prefer a local backend for hard guarantees; with models.OpenAI only the constraints the API can honor are available, so test your grammar before relying on it.
• Use the chat roles (system(), user(), assistant()) with instruct models so the chat template is applied correctly; raw f-string prompts suit base models.
• Models are immutable: assign the result (lm = lm + ... or lm += ...) and start each independent request from the base model object.
• Test grammars offline with grammar.match(...) and Mock, which needs no GPU or API key.
• Overly tight constraints (a regex the model finds very unlikely) can produce odd text; loosen them or improve the prompt.