📜 Loki Guide

Loki: a log database that only indexes the labels, so it stays small and cheap
logcli query '{job="nginx"} |= "404"' --since 1h
curl -s localhost:3100/ready

Loki stores log lines. Unlike a search engine, it does not index the text of every line. It indexes a small set of labels (job, host, status) and keeps the lines in compressed chunks. You find lines by choosing labels first, and then filter or parse the text of what is left. That makes Loki cheap to run and fast on the common question ("show me the errors from web1 in the last hour"), and it is why the label rules in this guide matter so much.

Loki does not collect logs itself. An agent on each machine reads the log files and pushes them in. This guide uses Grafana Alloy, Grafana's agent, for that. Grafana (the dashboard tool, see the Grafana guide) is the usual way to look at the logs, and logcli is the command-line way.

Every example below ran for real: Loki 3.7.8, logcli 3.7.8 and Alloy 1.20.1 (the current releases when this was written), installed from Grafana's apt repository on Ubuntu 24.04. Logs came from a real nginx on a second container; Grafana 13.2.4 and the Prometheus lab from the earlier guides were used for the Grafana and alerting parts. The containers have no systemd, so the programs were started by hand with the same command lines the services use; the systemd commands are marked as not run. Nothing was driven in a browser: Grafana was checked through its API.

Four programs, and two of them are optional:

  loki          the server: accepts pushed log lines, stores them, answers queries, evaluates alert rules (port 3100)
  alloy         the agent on each machine: reads log files (or the journal, or containers) and pushes them to Loki (its own status page is on port 12345)
  logcli        the command-line client for Loki queries
  Grafana       the web interface for exploring logs, dashboards and alerts (optional; see the Grafana guide)

The flow: an application writes a line to a file. Alloy tails the file, attaches labels (job="nginx", host="web1") and pushes batches to Loki. Loki files each line under its label set. A query names the labels first and then filters the text:

  {job="nginx", status="404"} |= "/admin"

A set of lines with the same labels is called a stream. Loki's job is to keep the number of streams small. That is the whole design, and the label section says how.

Loki is not a replacement for a metrics system (Prometheus counts things cheaply; Loki keeps the text) and it is not a full text search engine. It is a very good "what did the logs say around 03:12?" tool, and it can turn log lines into numbers when you need them.

All three come from the same repository as Grafana. If you did not add it for the Grafana guide, add the signing key and the repository now:

$ sudo apt-get install -y gnupg wget ca-certificates
$ sudo mkdir -p /etc/apt/keyrings
$ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
$ echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
$ sudo apt-get update
$ for p in loki alloy promtail logcli; do echo "$p: $(apt-cache policy $p | sed -n 3p)"; done
loki:   Candidate: 3.7.8
alloy:   Candidate: 1.20.1-1
promtail:   Candidate: 3.6.11
logcli:   Candidate: 3.7.8

(The key's fingerprint is shown in the Grafana guide: B53A E77B ADB6 30A6 8304 6005 963F A277 1045 8545. Check it.)

Promtail is the older agent. It is still in the repository, but Grafana points new installations at Alloy, so this guide uses Alloy. Check the Loki documentation for Promtail's support dates before you start a new setup with it (not run in the lab).

Install Loki and logcli on the log server, and Alloy on every machine whose logs you want:

$ sudo apt-get install -y loki logcli          (on the log server)
$ sudo apt-get install -y alloy                (on each machine that has logs)
$ loki --version | head -1
loki, version 3.7.8 (branch: release-3.7.x, revision: 09e6ce2f)
$ alloy --version | head -1
alloy, version v1.20.1 (branch: HEAD, revision: 95e12cf)

What the packages create:

  loki      user loki (no login), /etc/loki/config.yml, a systemd unit running: /usr/bin/loki -config.file /etc/loki/config.yml
  alloy     user alloy, /etc/alloy/config.alloy, settings in /etc/default/alloy; the alloy user is in the groups adm and systemd-journal, so it can read /var/log files owned by group adm and the journal

Neither service is started for you. On a real server (not run in the lab, which has no systemd):

$ sudo systemctl enable --now loki
$ sudo systemctl enable --now alloy
$ journalctl -u loki -f

The configuration the package installs is a development setup. Look at what it does before you trust it:

  path_prefix: /tmp/loki           the data goes in /tmp, which is emptied at reboot on many systems
  log_level: debug                 the lab's log grew by 1,342 lines in the first 12 seconds
  auth_enabled: false              anyone who can reach port 3100 can read and write all logs
  no retention                     old data is kept until the disk is full
  no listen address                the default is every network interface

The lab started it first, to see. It was not ready at once: the ready check answered "Ingester not ready: waiting for 15s after being ready" and the real answer came after about 40 seconds. Wait for ready before you push:

$ curl -s localhost:3100/ready
Ingester not ready: waiting for 15s after being ready
$ curl -s localhost:3100/ready
ready

Replace it with a real one. This is the lab's (the schema and ring settings are the single-server standard; the rest is what you decide):

# /etc/loki/config.yml
auth_enabled: false

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 3100
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 336h
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16
  max_query_series: 500
  volume_enabled: true

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

ruler:
  storage:
    type: local
    local:
      directory: /var/lib/loki/rules
  rule_path: /var/lib/loki/rules-temp
  alertmanager_url: http://prom:9093
  enable_api: true

analytics:
  reporting_enabled: false

What matters:
  http_listen_address 127.0.0.1   only this machine can reach Loki. Put a proxy in front for the network (see the security section). The lab first used 0.0.0.0 so that Alloy and Grafana on other machines could connect, then moved to a proxy
  path_prefix /var/lib/loki       the data directory: create it owned by loki
  schema_config                   the index format. tsdb and v13 are the current choice. The "from" date is when this schema starts; to change it later you add a new entry with a later date, you never edit the old one
  retention_period 336h           keep 14 days. Retention only works with compactor.retention_enabled (see the retention section)
  reject_old_samples_max_age      lines older than this are refused (7 days)
  volume_enabled                  lets Grafana show log volume
  ruler                           the part that evaluates alert rules written in LogQL and sends them to Alertmanager (the Alerts section)

$ sudo mkdir -p /var/lib/loki && sudo chown loki:nogroup /var/lib/loki && sudo chmod 750 /var/lib/loki
$ sudo -u loki loki -verify-config -config.file /etc/loki/config.yml
level=info ts=... caller=main.go:109 msg="config is valid"

Check before you start. -verify-config reads the file the way Loki will. A misspelt setting is reported with its line (the lab changed retention_period to retention_perid):

failed parsing config: /tmp/bad.yml: yaml: unmarshal errors:
  line 33: field retention_perid not found in type validation.plain.

Loki refuses to start on a config it cannot parse, so a typo fails loudly. The lab started Loki this way (the systemd unit runs the same command line) and checked the data directory filled in:

$ sudo systemctl restart loki                      (not run in the lab; it ran the command line above by hand)
$ ls /var/lib/loki
chunks  compactor  rules  tsdb-shipper-active  tsdb-shipper-cache  wal

Loki's front door is one HTTP call. Pushing a line by hand is the quickest way to check the server, and it shows what the data looks like. A push has a set of labels (the stream) and a list of [timestamp in nanoseconds, line]:

$ NOW=$(date +%s%N)
$ curl -s -o /dev/null -w "push: %{http_code}\n" -H "Content-Type: application/json" -X POST localhost:3100/loki/api/v1/push \
    -d "{\"streams\":[{\"stream\":{\"job\":\"manual\",\"host\":\"web1\"},\"values\":[[\"$NOW\",\"hello from curl\"],[\"$((NOW+1000000))\",\"level=error msg=\\\"disk almost full\\\"\"]]}]}"
push: 204

204 means accepted. Ask for the lines back with a label selector:

$ curl -s -G localhost:3100/loki/api/v1/query_range --data-urlencode 'query={job="manual"}' | jq -c '.data.result[] | {stream, values}'
{"stream":{"detected_level":"error","host":"web1","job":"manual","service_name":"manual"},"values":[["1792203975749451022","level=error msg=\"disk almost full\""]]}
{"stream":{"detected_level":"unknown","host":"web1","job":"manual","service_name":"manual"},"values":[["1792203975748451022","hello from curl"]]}

Two things Loki 3 added that you did not send: service_name (copied from the job label) and detected_level (Loki's guess from the line). The two lines came back as two separate result streams, with different detected_level values. The labels and their values:

$ curl -s localhost:3100/loki/api/v1/labels | jq -c .
{"status":"success","data":["host","job","service_name"]}
$ curl -s localhost:3100/loki/api/v1/label/host/values | jq -c .
{"status":"success","data":["web1"]}

What Loki refuses, as seen in the lab:

A line older than the limit (reject_old_samples_max_age, 7 days in the lab's real config; the default config's limit was the same):
  entry for stream '{job="manual", service_name="manual"}' has timestamp too old: 2026-10-07T02:26:16Z, oldest acceptable timestamp is: 2026-10-10T02:26:16Z      [HTTP 400]

A query with no real selector:
  $ curl -s -G localhost:3100/loki/api/v1/query_range --data-urlencode 'query={}'
  parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will      [HTTP 400]

A line that arrives a little out of order (the lab pushed one stamped 30 minutes in the past, into a stream that was already open) was accepted with 204. Loki 3 accepts out-of-order lines within a window, so agents that retry or send from several places do not lose data.

Every distinct set of label values is a separate stream, and every stream costs memory, index entries and a chunk. Few streams with lots of lines is cheap. Millions of tiny streams is the way Loki gets slow and expensive. So labels should describe where a line came from, with values from a short list that you know in advance:

  good labels      job, host, env, service, status (a few values), level
  bad labels       user id, request id, IP address, URL path, trace id, anything with thousands of different values

Put the details in the line and use a parser at query time (see LogQL) instead. The lab shows the cost on a small scale: 25 lines each with its own request_id label made 25 streams:

$ for i in $(seq 1 25); do curl -s -o /dev/null -H "Content-Type: application/json" -X POST localhost:3100/loki/api/v1/push -d "{\"streams\":[{\"stream\":{\"job\":\"bad\",\"request_id\":\"req-$i\"},\"values\":[[\"$((NOW+i))\",\"GET /thing 200\"]]}]}"; done
$ logcli series '{job="bad"}' --since 5m | grep -c request_id
25
$ curl -s localhost:3100/metrics | grep -E "^loki_ingester_memory_streams"
loki_ingester_memory_streams{tenant="fake"} 28

Twenty-five lines, twenty-five streams. A busy web server does that every second. (The tenant name "fake" is what Loki uses when auth_enabled is false.)

The limits Loki enforces, read from the running server (curl -s localhost:3100/config):

  max_label_names_per_series      15           a stream may have at most 15 labels
  max_global_streams_per_user     5000         total active streams (the number to watch)
  per_stream_rate_limit           3MB          per second, per stream
  ingestion_rate_mb               8            per second, per tenant (the lab set this)
  max_line_size                   256KB        longer lines are refused
  max_entries_limit_per_query     5000         the most lines one query returns
  query_timeout                   1m
  reject_old_samples_max_age      1w

When you cross one, pushes are refused with an error message that names the limit. Raise the limit only after you have checked that the cause is not a label that should not exist. Not run in the lab: hitting a limit such as max_global_streams_per_user (the only refusals the lab produced were the too-old line and the 401 below).

How to look at what you have: the series API lists every stream a selector matches, and the stream metric above counts them:

$ logcli series '{job="nginx"}' --since 72h
{filename="/var/log/nginx/access.log", host="tgt", job="nginx", service_name="nginx", status="200"}
{filename="/var/log/nginx/access.log", host="tgt", job="nginx", service_name="nginx", status="404"}

Two streams for one log file: the status label (three digits, a handful of values) is the kind of label that is fine. The same file with the URL path as a label would not be.

Alloy is configured with a file of components, each one named and wired to the next. A log pipeline has four parts: find the files (local.file_match), read them (loki.source.file), change them on the way (loki.process) and send them (loki.write). The lab's file for the nginx logs on the web server:

# /etc/alloy/config.alloy
logging {
	level = "warn"
}

local.file_match "nginx" {
	path_targets = [{
		"__path__" = "/var/log/nginx/*.log",
		"job"      = "nginx",
		"host"     = constants.hostname,
	}]
}

loki.source.file "nginx" {
	targets    = local.file_match.nginx.targets
	forward_to = [loki.process.nginx.receiver]
}

loki.process "nginx" {
	stage.regex {
		expression = `^(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) `
	}

	stage.timestamp {
		source = "time"
		format = "02/Jan/2006:15:04:05 -0700"
	}

	stage.labels {
		values = {
			status = "",
		}
	}
	forward_to = [loki.write.default.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://loki/loki/api/v1/push"

		basic_auth {
			username = "alloy"
			password = "labpass-loki1"
		}
	}
}

Reading it:
  local.file_match      lists the files and gives each the labels job and host. The path is a glob; "constants.hostname" is the machine's name
  loki.source.file      tails every file, remembering how far it got (in the positions file under /var/lib/alloy/data), so a restart does not repeat or lose lines
  stage.regex           pulls fields out of each nginx line into temporary values (ip, method, path, status, time)
  stage.timestamp       uses the time in the line as the line's time (see below)
  stage.labels          turns only the status value into a label. method and path stay in the text
  loki.write            sends batches to Loki; basic_auth is for the proxy in the security section. If Loki is open on port 3100 you do not need it

Check the file before you start. alloy validate parses it and checks every wire; alloy fmt -w rewrites it in the standard layout (tabs):

$ alloy validate /etc/alloy/config.alloy
$ alloy fmt -w /etc/alloy/config.alloy

A misspelt component in forward_to (the lab wrote loki.write.defualt):
Error: /tmp/bad.alloy:28:16: component "loki.write.defualt.receiver" does not exist or is out of scope

Permissions: nginx's logs are owned by www-data with group adm and mode 640. The alloy user is in the adm group, so it can read them with no change. For another log, add alloy to the group that owns it, or the file stays unread without an error you will notice (look at Alloy's own log and the positions file).

$ sudo systemctl enable --now alloy                    (not run in the lab)
$ sudo -u alloy alloy run /etc/alloy/config.alloy --storage.path=/var/lib/alloy/data --server.http.listen-addr=127.0.0.1:12345     (what the lab ran)
$ curl -s localhost:12345/-/ready
Alloy is ready.

What went wrong, and is worth knowing:

1. The timestamp. The first version of the config had no stage.timestamp. Alloy then stamps each line with the time it read it. The web server's access log already held 300 old lines (health checks from two days earlier); all 300 were stored as "now". A count of the last two hours showed 300 requests with status 200, although those lines were two days old. After adding stage.timestamp, the lab emptied Loki and Alloy's positions and re-read the file: the first line came back dated 2026-10-15T00:34:33Z, as in the file. Any file that has history needs the timestamp stage. For new lines it matters less, but it is still right.

2. A wrong password does not retry for ever. With the wrong password in loki.write the lab's Alloy logged
   level=error msg="final error sending batch, no retries left, dropping data" ... status=401 tenant="" error="server returned HTTP status 401 Unauthorized"
   and dropped the batch. A 401 is not retried: the lines are lost. Watch for that message after any change to the endpoint, user or password.

Not run in the lab: reading the systemd journal (loki.source.journal), Docker container logs (loki.source.docker), structured-metadata and JSON stages, and Alloy clustering.

LogQL has two halves. The stream selector (in braces) picks streams by label. The pipeline (after the selector) filters and parses the lines of those streams. Always start with the narrowest selector you can: the selector is cheap, filtering text is the expensive part.

$ export LOKI_ADDR=http://localhost:3100
$ logcli labels
filename
host
job
service_name
status
$ logcli labels status
200
404
$ logcli query '{job="nginx"}' --since 1h --limit 4 --quiet -o raw
172.22.0.3 - - [17/Oct/2026:02:28:19 +0000] "GET / HTTP/1.1" 200 615 "-" "Blackbox-Exporter/0.29.0"
172.22.0.3 - - [17/Oct/2026:02:28:15 +0000] "GET /nope HTTP/1.1" 404 162 "-" "curl/8.5.0"
172.22.0.3 - - [17/Oct/2026:02:28:15 +0000] "GET /index.html HTTP/1.1" 404 162 "-" "curl/8.5.0"
172.22.0.3 - - [17/Oct/2026:02:28:15 +0000] "GET /admin HTTP/1.1" 404 162 "-" "curl/8.5.0"

(-o raw prints just the lines; --quiet hides the URL logcli prints. logcli returns the newest lines first, so add --forward for oldest first.)

Selectors match with = (equals), != (not equal), =~ (regular expression) and !~ (not a match). Combine them with commas; all must hold:

  {job="nginx", status="404"}              equals
  {job="nginx", status=~"4.."}             any 4xx
  {job=~"nginx|app", host!="test1"}        regex and not-equal

The pipeline filters the text. Line filters are the fast ones:

  |= "text"      line contains text
  != "text"      does not contain
  |~ "regex"     matches the regular expression
  !~ "regex"     does not match

  {job="nginx"} |= "404"                   (the lab's search for 404 found the 4 requests it had just made)
  {job="nginx"} |= "/admin" != "health"    chained filters

Parsers turn the text into temporary fields you can test and print. The pattern parser is for fixed layouts such as access logs. The lab used it to pick out errors and rewrite the line:

$ logcli query '{job="nginx"} | pattern `<ip> - - [<_>] "<method> <path> <_>" <status> <size> "<_>" "<agent>"` | status >= 400 | line_format "{{.method}} {{.path}} {{.status}}"' --since 1h --limit 3 --quiet -o raw
GET /nope 404
GET /index.html 404
GET /admin 404

For JSON logs use | json, for key=value logs | logfmt. The lab pushed eight JSON lines for an app (job="app") and asked for the failed payments:

$ logcli query '{job="app"} | json | status >= 500 | line_format "{{.req}} {{.status}} {{.latency_ms}}ms"' --since 10m --quiet -o raw
POST /api/pay 500 1300ms
POST /api/pay 502 2100ms
POST /api/pay 500 1500ms
$ logcli query '{job="app"} | json | level="error" | latency_ms > 1400 | line_format "{{.level}} {{.req}} {{.latency_ms}}"' --since 10m --quiet -o raw
error POST /api/pay 2100
error POST /api/pay 1500

The sample line looked like {"level":"error","msg":"request done","req":"POST /api/pay","status":500,"latency_ms":1300}. After | json, every key is a field: level, req, status, latency_ms. Comparisons with numbers (status >= 500, latency_ms > 1400) work on the parsed values. The fields are not labels: they are not in the index, so this is a filter over the lines the selector chose. Narrow the selector and the time range first, or the query reads far more than it needs to.

Why the access log was parsed at query time and only status was made a label: the label is for the question you ask in nearly every query (the 404s); the method and path vary and are better left in the text.

Wrap a log query in a range function and the answer is a number per label set. This is how you graph errors per minute without any second system. The two you will use most are count_over_time (how many lines) and rate (lines per second):

$ logcli instant-query 'sum by (status) (count_over_time({job="nginx"}[1h]))'
{"metric":{"status":"200"},"value":[1792204112.184,"304"]}
{"metric":{"status":"404"},"value":[1792204112.184,"4"]}

(That is 304 requests answered 200 and 4 answered 404 in the last hour, counted from the log lines: the lab's own health checks and four test requests. Run instant-query for one value now, or range queries with "query" and a step for a graph.)

For the app logs, count the lines by level after parsing the JSON:

$ logcli instant-query 'sum by (level) (count_over_time({job="app"} | json [5m]))'
{"metric":{"level":"error"},"value":[...,"3"]}
{"metric":{"level":"info"},"value":[...,"4"]}
{"metric":{"level":"warn"},"value":[...,"1"]}

To compute with a number inside the line (a latency, a size) you unwrap it. avg_over_time, quantile_over_time, sum_over_time and the others work on the unwrapped values:

$ logcli instant-query 'avg_over_time({job="app"} | json | unwrap latency_ms [5m]) by (req)'
{"metric":{"req":"GET /api/items"},"value":[...,"260.25"]}
{"metric":{"req":"GET /health"},"value":[...,"3"]}
{"metric":{"req":"POST /api/pay"},"value":[...,"1633.3333333333333"]}
$ logcli instant-query 'quantile_over_time(0.95, {job="app"} | json | unwrap latency_ms [5m]) by (req)'
{"metric":{"req":"GET /api/items"},"value":[...,"781.1499999999996"]}
{"metric":{"req":"GET /health"},"value":[...,"3"]}
{"metric":{"req":"POST /api/pay"},"value":[...,"2040"]}

A trap that gave wrong numbers in the lab. The first attempt was avg by (req) (avg_over_time(... [5m])), which averages the averages. After | json every different combination of the parsed fields (level, status, req, latency) is its own series, so avg_over_time returns one average per combination and the outer avg takes the mean of those. The result for POST /api/pay was 1750 instead of the true 1633.33, and for GET /api/items it was 476.83 instead of 260.25. Put the grouping inside, with the "by (req)" after the range function as above, and the answer is computed over all the lines.

The same applies to any range query over parsed fields: if the numbers look slightly wrong, check what the series were before the outer sum or avg.

logcli is a thin client over the same API. Point it at the server with LOKI_ADDR or --addr, and give it credentials if there is a proxy:

$ export LOKI_ADDR=http://localhost:3100
$ logcli query '{job="app"} |= "error"' --since 30m --limit 50
$ logcli query '{job="nginx"}' --tail          (stream new lines as they arrive, like tail -f)
$ logcli labels job
app
bad
nginx
$ logcli --addr=http://localhost --username=alloy --password=PASSWORD labels job        (through the nginx proxy)
$ logcli series '{job="nginx"}' --since 24h
$ logcli instant-query 'sum(count_over_time({job="nginx"}[1h]))'

Without the password, the proxy answers 401 and logcli retries a few times and then gives up:
  Error doing request: run out of attempts while querying the server

Useful flags: --since 1h or --from/--to for the time range, --limit for the line count (the server caps it at max_entries_limit_per_query, 5000), --forward for oldest first, -o raw for the lines only, --quiet to hide the request URL. The lab's --tail printed the recent lines first (following new lines was not checked). Not run in the lab: logcli stats (it returned zeros for fresh data), and queries across several tenants.

Loki has its own rule evaluator, the ruler. You write rules in the same YAML format as Prometheus, with LogQL expressions, and the ruler sends firing alerts to the same Alertmanager that Prometheus uses (see the Prometheus guide). Routing, silences and notifications then work as they already do. The config's ruler block (above) says where the rules live and where Alertmanager is.

With auth_enabled false the tenant is called "fake", so the rule files go in a folder with that name:

# /var/lib/loki/rules/fake/app.yml
groups:
  - name: app
    rules:
      - alert: AppErrors
        expr: sum(count_over_time({job="app"} | json | level="error" [1m])) > 2
        for: 1m
        labels:
          severity: critical
          service: app
        annotations:
          summary: "The app logged {{ $value }} errors in the last minute"

$ sudo chown -R loki:nogroup /var/lib/loki/rules
$ curl -s localhost:3100/loki/api/v1/rules
app.yml:
    - name: app
      rules:
        - alert: AppErrors
          expr: sum(count_over_time({job="app"} | json | level="error" [1m])) > 2
          for: 1m
          ...

The ruler reads the folder periodically: a new file showed up in the rules API at once, but it was evaluated only after about a minute (state "unknown" for six checks, 10 seconds apart). The lab's timeline with a script pushing the error lines every 15 seconds:

$ curl -s localhost:3100/prometheus/api/v1/rules | jq -r '[.data.groups[].rules[]|.name+" "+.state+" "+.health] | join(",")'
AppErrors unknown unknown          (about a minute: the rule has not been loaded yet)
AppErrors pending ok               (the condition is true; the for: minute is running)
AppErrors firing ok

(Because of the for: 1m, the alert fires a minute after the condition first became true. Without errors arriving, the rule goes back to inactive on its own. A log-based alert needs lines to keep coming; "no lines at all" is a different condition, which you would write as absent_over_time({job="app"}[5m]) or with a metrics check on the agent.)

Alertmanager received it with the rule's labels, and its routes did the rest:

$ curl -s localhost:9093/api/v2/alerts | jq -c '.[] | select(.labels.alertname=="AppErrors") | {labels, status:.status.state}'
{"labels":{"alertname":"AppErrors","service":"app","severity":"critical"},"status":"active"}

The webhook got {"status":"firing", ... "alerts":[{"labels":{"alertname":"AppErrors","severity":"critical",...}, "annotations":{"summary":"The app logged 10 errors in the last minute"}, "generatorURL":"/explore?left=..."}]} and the on-call mailbox got the email with the subject [FIRING:1] AppErrors (app critical). The generatorURL is a link that opens the query in Grafana's Explore page when Alertmanager is configured to point at Grafana.

Use the same discipline as in the Prometheus guide: a for: time, a severity label that your routes match, and an annotation that tells the reader what happened. Be careful that a log alert measures a text pattern, not a fact: a changed log message silently turns the alert off, so also keep a metrics alert on the thing itself where you can.

Add Loki as a data source in Grafana with a file, the same way as Prometheus (see the Grafana guide). The lab's, for Loki behind the basic-auth proxy of the next section:

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki
    basicAuth: true
    basicAuthUser: alloy
    secureJsonData:
      basicAuthPassword: labpass-loki1
    jsonData:
      maxLines: 1000

$ curl -s -u admin:PASSWORD -X POST localhost:3000/grafana/api/admin/provisioning/datasources/reload
$ curl -s -u admin:PASSWORD localhost:3000/grafana/api/datasources/uid/loki/health | jq -c .
{"message":"Data source successfully connected.","status":"OK"}

Query it the same way as any data source (the /api/ds/query call, with the loki uid and a LogQL expression). A logs query returns a table with the labels, the time, the line and its nanosecond timestamp:

  fields: ["labels","Time","Line","tsNs","labelTypes","id"]
  Line:   {"level":"error","msg":"request done","req":"POST /api/pay","status":500,"latency_ms":1300}

A metric query ("queryType":"instant" with the expression sum by (level) (count_over_time({job="app"} | json [5m]))) returned one value per level: error 42, info 56, warn 14 (the lab's script had been pushing for a few minutes). In the web interface this is the Explore page and the Logs panel: pick Loki, type the LogQL, and click a line to see its labels and parsed fields. The lab did not open a browser, so what the page looks like was not checked.

Notes for dashboards: put a Logs panel next to the graph of the same service (a Prometheus panel and a Loki panel sharing the instance variable); use the "Logs volume" view to find the moment; link a metric panel to the logs by the same labels (job, host). The labels you chose for Prometheus and for Loki should match, because that is what lets you jump from a spike to its logs.

Loki has no login of its own. With auth_enabled false, anyone who can reach the port can push logs (filling your disk), read every log, and delete. Treat port 3100 like a database port. Three layers, in the order you should do them:

1. Listen on loopback only (http_listen_address: 127.0.0.1, as in the config above). The lab proved it: after the change, a request to port 3100 from another container got no answer (curl code 000).

2. Put nginx in front with basic auth and TLS. The lab's file, with a password file made by openssl (use htpasswd if you have it):

$ printf "alloy:%s\n" "$(openssl passwd -apr1 PASSWORD)" | sudo tee /etc/nginx/loki.htpasswd
$ sudo chgrp www-data /etc/nginx/loki.htpasswd && sudo chmod 640 /etc/nginx/loki.htpasswd

# /etc/nginx/conf.d/loki.conf
server {
    listen 80;
    server_name loki;
    client_max_body_size 10m;

    location = /ready {
        proxy_pass http://127.0.0.1:3100;
    }
    location / {
        auth_basic "Loki";
        auth_basic_user_file /etc/nginx/loki.htpasswd;
        proxy_pass http://127.0.0.1:3100;
        proxy_set_header Host $host;
    }
}

$ sudo nginx -t && sudo nginx -s reload

Results from another container:
  port 3100 direct                 no answer (loopback only)
  /ready through nginx             200 (open, so a monitor can check it; it was 503 for the first seconds after a Loki start)
  labels, no password              401
  labels, password alloy:...       200
  labels, wrong password           401

client_max_body_size matters: nginx's default is 1 MB, and an agent that sends a bigger batch gets "413 Request Entity Too Large" and drops it. The lab's file sets 10m (the 413 itself was not provoked). The lab ran plain HTTP between containers; on a real network add listen 443 ssl and a certificate, and change the URLs to https (not run). Give each agent its own user in the password file so you can remove one.

3. Multi-tenancy, if different teams must not see each other's logs. Set auth_enabled: true and every request has to name its tenant in an X-Scope-OrgID header. Loki does not check who you are, it only separates the data, so the proxy (or the agent's config) is what sets the header. The lab ran a second Loki on another port with auth_enabled true:

$ curl -s -w " [%{http_code}]\n" -H "Content-Type: application/json" -X POST localhost:3101/loki/api/v1/push -d '...'
no org id
 [401]
$ curl ... -H "X-Scope-OrgID: team-a" ... push      204
$ curl ... -H "X-Scope-OrgID: team-b" ... push      204
query as team-a:  ["hello from team-a"]
query as team-b:  ["hello from team-b"]
query as other:   []

Each tenant sees only its own lines, a tenant that never pushed sees nothing, and a request with no header is refused. In Alloy the tenant is the tenant_id setting of loki.write; in Grafana it is an X-Scope-OrgID header in the data source's HTTP headers (both not run in the lab). With multi-tenancy on, the ruler's rule folder is named after the tenant instead of "fake".

Run the web server and the agent as unprivileged users (the packages do), keep the password files at mode 640, and never put the push password in a world-readable config.

Loki keeps data until you tell it not to. Retention is done by the compactor, and it must be switched on in two places (both are in the config above):

  limits_config:   retention_period: 336h          how long to keep (14 days here); can be set per tenant or per stream selector
  compactor:       retention_enabled: true         without this the period does nothing
                   delete_request_store: filesystem

The lab's server log shows the compactor starting, and the surprise in it:

level=info ... caller=compactor.go:497 msg="this instance has been chosen to run the compactor, starting compactor"
level=info ... caller=tables_manager.go:70 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"

After every start the compactor waits ten minutes before it begins, so nothing is deleted for ten minutes, and a Loki that you restart often never gets there. The lab did not wait to see old data disappear (not run: retention actually removing chunks).

To delete particular lines (a log that held a password, say) use the delete API with a selector and a time range. The end time must not be in the future:

$ curl -s -w "[%{http_code}]\n" -X POST "localhost:3100/loki/api/v1/delete?query=%7Bjob%3D%22bad%22%7D&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)"
[204]
$ curl -s localhost:3100/loki/api/v1/delete | jq -c '.[] | {status,query}'
{"status":"received","query":"{job=\"bad\"}"}

With the end time one hour ahead the lab got "deletes in the future are not allowed" (400). The request is only "received": the delete_request_cancel_period (24h in the lab's server) gives you time to cancel it, and the lines are removed after that. The lab checked that the 25 streams were still there right after the request, as designed; the removal itself was not waited for.

Disk: the lab's data directory after a few hundred lines was about 400 KB (wal, tsdb, chunks). Plan from your own volume: bytes per day is the size of your compressed logs, times the retention days. Watch /var/lib/loki with an ordinary disk alert, and watch loki_ingester_memory_streams for the label problem. Loki exposes its own metrics at /metrics (through the proxy you need the basic-auth user to scrape it; not run in the lab).

Backups: the data directory is chunks plus index plus the write-ahead log. For a single server, stop Loki (or accept a crash-consistent copy) and copy /var/lib/loki and /etc/loki; for anything important use object storage (S3 or compatible) instead of the filesystem, and its own versioning (not run in the lab). Treat logs as a rolling window rather than an archive: if you must keep them for years, archive the raw files as well.

Upgrades: apt-get install --only-upgrade loki alloy logcli, after reading the release notes. Keep the schema_config entries you have, and add new schema versions with a later "from" date when the notes tell you to (not run in the lab, which installed the current version).

Install             add key + repo (apt.grafana.com); apt install loki logcli (server), apt install alloy (each machine)
Ready               curl -s localhost:3100/ready     (can take 40 s after a start)
Check the config    sudo -u loki loki -verify-config -config.file /etc/loki/config.yml
Alloy config        alloy validate FILE ; alloy fmt -w FILE ; status page on 127.0.0.1:12345
Push a line         curl -X POST localhost:3100/loki/api/v1/push -H "Content-Type: application/json" -d '{"streams":[{"stream":{"job":"x"},"values":[["NANOSECONDS","text"]]}]}'
Labels / values     logcli labels ; logcli labels job ; logcli series '{job="nginx"}'
Query lines         logcli query '{job="nginx"} |= "404"' --since 1h --limit 50 --quiet -o raw
Follow              logcli query '{job="nginx"}' --tail
Parse               | json   | logfmt   | pattern `<ip> - - [<_>] "<method> <path> <_>" <status> <_>`   | line_format "{{.path}}"
Filter lines        |= "text"   != "text"   |~ "regex"   !~ "regex"
Count               sum by (status) (count_over_time({job="nginx"}[5m]))      rate(...[1m]) for per second
Numbers in lines    avg_over_time({...} | json | unwrap latency_ms [5m]) by (req)      (group inside!)
Percentile          quantile_over_time(0.95, {...} | json | unwrap latency_ms [5m]) by (req)
Rules               /var/lib/loki/rules/TENANT/*.yml  (TENANT is "fake" when auth is off) ; curl localhost:3100/loki/api/v1/rules
Rule states         curl localhost:3100/prometheus/api/v1/rules
Delete lines        POST /loki/api/v1/delete?query=SELECTOR&start=S&end=E   (end not in the future; 24 h to cancel)
Limits in effect    curl -s localhost:3100/config | grep -E "retention|rate|streams|entries"
Tenants             X-Scope-OrgID header when auth_enabled is true
Streams count       curl -s localhost:3100/metrics | grep loki_ingester_memory_streams

Ready says "Ingester not ready: waiting for 15s"      normal for the first 40 seconds after a start; wait and ask again
Loki will not start, "field X not found"             a typo in config.yml; run loki -verify-config for the line number
Data disappears after a reboot                       path_prefix is /tmp/loki (the package default); use /var/lib/loki
Push returns "timestamp too old" (400)               the line is older than reject_old_samples_max_age; fix the timestamp stage, or raise the limit for a one-off import
Query returns "queries require at least one ... matcher"   {} or {job=~".*"}: use a selector that cannot match empty, such as {job=~".+"}
Old log lines show up as "now"                       Alloy stamps with the read time: add stage.timestamp for files that have history
Alloy: "dropping data ... status=401"                wrong or missing basic_auth in loki.write; those lines are lost
Alloy reads nothing from a file                      the alloy user cannot read it (group adm or add alloy to the file's group); check the positions file and Alloy's log
Push or query returns 429 or a limit error          a limit was hit (rate, streams, entries); find the cause (labels!) before raising it
Loki is slow, memory grows                           too many streams: a label with unbounded values. Check loki_ingester_memory_streams and logcli series
Wrong averages from avg_over_time                    the grouping is outside the range function: write avg_over_time(... [5m]) by (req)
Query returns far less than expected                 --limit (default 30 in logcli) or max_entries_limit_per_query (5000); and the time range
Alert rule is in the API but never fires             the ruler takes about a minute to load it; check /prometheus/api/v1/rules for the state, and that the tenant folder name is right
Alert fires but nothing is sent                      alertmanager_url in the ruler block is wrong or unreachable; check Alertmanager's /api/v2/alerts
Retention does not delete                            retention_enabled is false, retention_period is 0, or the compactor has not finished its 10 minute wait after a restart
Delete request returns "deletes in the future"       the end time is later than now
No org id (401)                                      auth_enabled is true: send the X-Scope-OrgID header
Proxy returns 413                                    client_max_body_size too small for the agent's batch
Grafana says the Loki data source failed            wrong URL or proxy password (the log of the Grafana server has the real error); Loki not ready yet

Keep the number of streams small. Label by job, host, env, service and a few more, never by anything with thousands of values. Put the rest in the line and parse it at query time.
Start every query with the narrowest selector and the shortest time range you can. Line filters (|=) are cheap; parsers (| json) and regular expressions cost more.
Log in a form that parses: JSON or key=value lines, with the same field names across services (level, msg, req, status, latency_ms).
Use one label naming scheme across Loki and Prometheus (job, host, env, service), so you can go from a metrics graph to the logs of the same thing.
Always set stage.timestamp for files with history, and look at the first queried lines after you add a new source.
Change the config, run -verify-config, then restart. Wait for ready, then look at the log for errors.
Put Loki on loopback behind a proxy with TLS and credentials. Give each agent its own login.
Set retention on purpose and remember the compactor's 10 minute wait. Delete requests are for the rare mistakes, not for housekeeping.
Alert on the thing itself (metrics) first and use log alerts for what only logs can tell you. A changed log message silences a log alert without any warning.
Watch Loki as you watch anything else: ready, disk use of /var/lib/loki, the stream count, and Alloy's own "dropping data" errors.
Treat logs as sensitive: they hold tokens, addresses and names. Limit who can query them, and do not keep them longer than you need.