Haystack

Overview

Haystack models an application as a Pipeline of components connected by typed inputs and outputs. Typical building blocks are an InMemoryDocumentStore (or a vector database integration), embedders, retrievers, a ChatPromptBuilder, and a chat generator such as OpenAIChatGenerator. Since 3.0 (2026) the framework is leaner: the legacy text Generators are removed in favour of Chat Generators, AsyncPipeline is merged into Pipeline, and many local-model components live in separately installed integration packages. Code written for 2.x commonly breaks on these points, see the migration notes below.

Instructions

Install

pip install haystack-ai
export OPENAI_API_KEY=...       # for the OpenAI components
export HAYSTACK_TELEMETRY_ENABLED=False   # optional: disable anonymous telemetry

Integrations are separate packages (pip install qdrant-haystack, sentence-transformers-haystack, transformers-haystack, ...) imported from haystack_integrations.*.

Build a pipeline

1. Create a document store and an indexing Pipeline (converter -> DocumentSplitter -> document embedder -> DocumentWriter).
2. Create a query Pipeline: text embedder -> retriever -> ChatPromptBuilder -> chat generator. Connect with pipeline.connect("sender.output", "receiver.input").
3. Run with a dict keyed by component name: pipeline.run({"text_embedder": {"text": q}, "prompt_builder": {"question": q}}). Use include_outputs_from={"retriever"} to see intermediate results.
4. ChatPromptBuilder takes a list of ChatMessage objects with Jinja2 templates; the generator's output is result["llm"]["replies"][0], a ChatMessage whose text is .text.

Custom components

Decorate a class with @component, declare outputs with @component.output_types(...), and implement run. Build the class in a module you control; the loader needs to trust it (see Guidelines).

Agents

haystack.components.agents.Agent(chat_generator=..., tools=[...], system_prompt=...) runs the tool loop itself (the standalone ToolInvoker is gone in 3.x). Define tools with from haystack.tools import tool on a typed, documented function. Behavior hooks (before_tool, after_tool, on_exit ...) go in hooks={...}.

Save, test, stream

• pipeline.dumps() / Pipeline.loads(yaml_text), and dump/load for files.
• For tests without an API key, use MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder.
• Streaming: pass streaming_callback=print_streaming_chunk (from haystack.components.generators.utils) to the chat generator.
• Async serving: await pipeline.run_async(data) on the same Pipeline class.

Migrating from 2.x to 3.x

The full table is in MIGRATION.md of the repository.

Examples

Example 1: RAG over a handful of release notes

User request: "Index these notes and answer questions about them with OpenAI."

from haystack import Pipeline, Document
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.writers import DocumentWriter
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.document_stores.in_memory import InMemoryDocumentStore

store = InMemoryDocumentStore()
indexing = Pipeline()
indexing.add_component("embedder", OpenAIDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store=store))
indexing.connect("embedder.documents", "writer.documents")
indexing.run({"embedder": {"documents": [
    Document(content="Release 3.0 removed OpenAIGenerator in favour of OpenAIChatGenerator."),
    Document(content="Release 3.0 merged AsyncPipeline into Pipeline."),
]}})

template = [ChatMessage.from_user(
    "Answer from the documents only.\n"
    "{% for doc in documents %}- {{ doc.content }}\n{% endfor %}\nQuestion: {{ question }}")]

rag = Pipeline()
rag.add_component("embedder", OpenAITextEmbedder())
rag.add_component("retriever", InMemoryEmbeddingRetriever(document_store=store, top_k=2))
rag.add_component("prompt_builder", ChatPromptBuilder(template=template))
rag.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
rag.connect("embedder.embedding", "retriever.query_embedding")
rag.connect("retriever.documents", "prompt_builder.documents")
rag.connect("prompt_builder.prompt", "llm.messages")

q = "What happened to AsyncPipeline?"
out = rag.run({"embedder": {"text": q}, "prompt_builder": {"question": q}})
print(out["llm"]["replies"][0].text)

Result: a sentence saying AsyncPipeline was merged into Pipeline in 3.0, generated from the retrieved notes.

Example 2: Custom filter component, tested without an API key

User request: "Add a step that keeps only documents of one category, and test the pipeline offline."

from haystack import component, Document, Pipeline

@component
class CategoryFilter:
    @component.output_types(documents=list[Document])
    def run(self, documents: list[Document], category: str):
        return {"documents": [d for d in documents if d.meta.get("category") == category]}

p = Pipeline()
p.add_component("filter", CategoryFilter())
result = p.run({"filter": {"documents": [
    Document(content="Refund policy", meta={"category": "billing"}),
    Document(content="Reset password", meta={"category": "account"})],
    "category": "billing"}})
print([d.content for d in result["filter"]["documents"]])   # ['Refund policy']

Result: ['Refund policy']. For the full query pipeline in CI, use MockChatGenerator(), MockTextEmbedder() and MockDocumentEmbedder() (from haystack.components.generators.chat and haystack.components.embedders) instead of the OpenAI ones.

Guidelines

• Missing API keys now fail at warm_up() (the first run), not at construction.
• Pipeline loading is allowlisted: Pipeline.loads refuses classes from modules outside haystack and haystack_integrations. For your own components pass allowed_modules=["my_app.components"] or set HAYSTACK_DESERIALIZATION_ALLOWLIST; never use unsafe=True on YAML you did not write.
• InMemoryDocumentStore is for development; use a vector database integration (Qdrant, Pinecone, Weaviate, pgvector, ...) in production, with the same embedder at index and query time.
• Document.id hashing changed in 3.0 for documents with metadata, so re-indexing 2.x data can create duplicates; clear and rebuild the store.
• Pin haystack-ai and integration versions together; integrations release independently.
• Do not use Haystack for a single prompt-and-response call; the pipeline abstraction pays off with retrieval, branching or several models.