Vector

Overview

Vector reads data from sources, reshapes it with transforms and delivers it to sinks. Each component has a name and an inputs list that wires it to upstream components. It runs as an agent on every host (DaemonSet in Kubernetes) or as a central aggregator, and ships as a single static binary. Configuration can be TOML, YAML or JSON; this skill uses TOML.

Changes in recent releases that older tutorials get wrong:

• 0.57: environment-variable interpolation (${ES_PASSWORD}) in config files is off by default. Re-enable it with --dangerously-allow-env-var-interpolation or VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true, run the file through envsubst first, or use a secrets backend.
• 0.57: sinks that accept {{ field }} templates (S3, Elasticsearch, Kafka, HTTP, Loki, file and others) refuse templates with no literal prefix, such as topic = "{{ x }}". Write topic = "logs-{{ x }}", or set dangerously_allow_unconfined_template_resolution = true only if you accept the injection risk. vector validate reports these at startup.
• Elasticsearch auth needs auth.strategy = "basic"; without it validation fails with "missing field strategy".
• The http sink has no condition option; route errors to it through a filter transform, and keep errors out of sampling with exclude on sample.
• Latest release when checked: 0.58.0 (26 August 2026).

Instructions

Install

Prefer a package manager or container image:

brew install vector                       # macOS
helm repo add vector https://helm.vector.dev
helm install vector vector/vector         # Kubernetes
docker run --rm -v "$PWD/vector.toml:/etc/vector/vector.toml:ro" \
  timberio/vector:0.58.0-alpine --config /etc/vector/vector.toml

For apt, yum and archive installs follow https://vector.dev/docs/setup/installation/. If you download an archive from packages.timber.io, verify its published checksum before unpacking, and avoid piping the install script into a shell.

Configuration

# vector.toml — collect, filter, sample and route logs and metrics
data_dir = "/var/lib/vector"

# --- Sources ---
[sources.app_logs]
type = "file"
include = ["/var/log/app/*.log"]
read_from = "end"                     # "beginning" re-reads old files on first start

[sources.http_logs]                   # apps POST JSON here
type = "http_server"
address = "127.0.0.1:8686"
decoding.codec = "json"

[sources.host_metrics]
type = "host_metrics"
collectors = ["cpu", "memory", "disk", "network"]
scrape_interval_secs = 15

[sources.otel]                        # OpenTelemetry: outputs are otel.logs / otel.metrics / otel.traces
type = "opentelemetry"
grpc.address = "127.0.0.1:4317"
http.address = "127.0.0.1:4318"

# --- Transforms ---
[transforms.parse_json]
type = "remap"
inputs = ["app_logs"]
source = '''
  . = parse_json!(.message)
  .environment = get_env_var("ENVIRONMENT") ?? "production"
  .timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S%.fZ")
  if exists(.email) {
    .email = redact(string!(.email), filters: [r'\S+@\S+'])
  }
'''

[transforms.filter_noise]
type = "filter"
inputs = ["parse_json"]
condition = '''
  !includes(["GET /health", "GET /ready", "GET /metrics"], .request) && .level != "debug"
'''

[transforms.sample_info]              # keeps 1 in 10 events, never drops errors or warnings
type = "sample"
inputs = ["filter_noise"]
rate = 10
exclude = '.level == "error" || .level == "warn"'

[transforms.errors_only]
type = "filter"
inputs = ["filter_noise"]
condition = '.level == "error"'

[transforms.aggregate_metrics]
type = "aggregate"
inputs = ["host_metrics"]
interval_ms = 60000

# --- Sinks ---
[sinks.elasticsearch]
type = "elasticsearch"
inputs = ["sample_info"]
endpoints = ["https://es.internal.shopwave.io:9200"]
bulk.index = "logs-%Y-%m-%d"
auth.strategy = "basic"
auth.user = "${ES_USER}"              # needs env-var interpolation enabled (see Overview)
auth.password = "${ES_PASSWORD}"
compression = "gzip"
batch.max_bytes = 10485760
batch.timeout_secs = 5
buffer.type = "disk"                  # survives destination outages and restarts
buffer.max_size = 1073741824
buffer.when_full = "block"

[sinks.s3_archive]
type = "aws_s3"
inputs = ["sample_info"]
bucket = "shopwave-logs-archive"
region = "us-east-1"
key_prefix = "logs/{{ service }}/year=%Y/month=%m/day=%d/"   # literal prefix keeps 0.57+ confinement happy
compression = "gzip"
encoding.codec = "json"
batch.max_bytes = 104857600
batch.timeout_secs = 300

[sinks.prometheus]
type = "prometheus_exporter"
inputs = ["aggregate_metrics"]
address = "0.0.0.0:9598"

[sinks.slack_errors]
type = "http"
inputs = ["errors_only"]
uri = "${SLACK_WEBHOOK_URL}"
method = "post"
encoding.codec = "json"
batch.max_events = 1
request.rate_limit_duration_secs = 1
request.rate_limit_num = 5

Validate, then run:

vector validate --no-environment vector.toml      # add --dangerously-allow-env-var-interpolation if the file uses ${VAR}
vector --config vector.toml

--no-environment skips network health checks, so it works offline; drop it in CI to also check that destinations are reachable. Unused sources only produce warnings.

VRL (Vector Remap Language)

VRL is compiled and type-checked: fallible calls must be handled with ! (abort the event on error) or ?? default.

. = parse_json!(.message)

if starts_with(string!(.message), "AUDIT:") {
  .route = "audit"
} else if (to_int(.status_code) ?? 0) >= 500 {
  .route = "error"
} else {
  .route = "general"
}

.duration_ms = to_float(.duration_ms) ?? 0.0
.user_id = del(.user.id)
del(.user)

Try snippets interactively with vector vrl, or watch live events with vector tap and vector top against a running instance.

Unit tests

[[tests]]
name = "normalizes warning and redacts email"
[[tests.inputs]]
insert_at = "parse_json"
type = "log"
[tests.inputs.log_fields]
message = '{"level":"warning","status_code":"503","email":"dana@shopwave.io"}'
[[tests.outputs]]
extract_from = "parse_json"
[[tests.outputs.conditions]]
type = "vrl"
source = '''
  assert_eq!(.severity, "warn")
  assert!(!contains(string!(.email), "dana"))
'''

Put the [[tests]] next to the transforms they cover and run vector test vector.toml. Output: test normalizes warning and redacts email ... passed.

Examples

Example 1: Cut Elasticsearch cost for a Node.js API

User request: "Our Node API logs JSON to /var/log/api/*.log and Elasticsearch costs too much. Drop health checks and debug, keep all errors, sample the rest."

Create vector.toml from the configuration above, with include = ["/var/log/api/*.log"]. Run vector validate --no-environment vector.toml; it prints Validated for transforms and sinks. Start with vector --config vector.toml and confirm with vector top that filter_noise receives more events than sample_info emits. Errors and warnings all reach Elasticsearch; about one in ten info events does.

Example 2: Fix a failing VRL remap

User request: "Vector says unhandled fallible assignment on .timestamp = parse_timestamp(.ts, \"%+\")."

parse_timestamp can fail, so VRL refuses to compile it. Change the line to .timestamp = parse_timestamp!(.ts, format: "%+") to drop events with a bad timestamp, or parse_timestamp(.ts, format: "%+") ?? now() to keep them. Add a [[tests]] case with a malformed ts and run vector test vector.toml to lock in the behaviour.

Guidelines

• Pin the image or package version; upgrade one minor at a time and read the upgrade guide, since 0.57 changed env-var and template handling.
• Never put secrets in the config file. Use a secrets backend, or env vars with interpolation explicitly enabled, and keep the file out of Git.
• Filter noise and sample high-volume info logs before sinks; keep 100% of errors and warnings.
• Archive everything cheaply to S3 and send only recent or important data to Elasticsearch.
• Enable disk buffers on sinks that must not lose data during outages; the default buffer is in memory.
• Bind source addresses to 127.0.0.1 unless remote hosts must reach them; 0.0.0.0 on syslog or HTTP exposes the port.
• Prefer VRL over regex-heavy chains and test every remap with vector test before deploying.