A how-to manual: install and first login, configuration, data sources and dashboards as files, the API, users and permissions, alerting with real notifications, a reverse proxy, plugins, backup and restore, and the problems that actually came up.
Grafana: the dashboard and alerting front end for your metrics
curl -s localhost:3000/api/health
curl -s -u admin:PASSWORD localhost:3000/api/datasources
Grafana does not collect or store your metrics. It connects to data sources (Prometheus, Loki, a SQL database and many more), asks them questions, and draws the answers as panels on dashboards. It can also evaluate alert rules against those data sources and send notifications.
Everything in Grafana can be done by clicking in the web interface, and everything that matters can also be done with files and an HTTP API. This guide concentrates on the second way, because files and API calls can be checked into version control, repeated on a second server and run in a script. The web interface is the same Grafana, so what you build here is what you see there.
Every example below ran for real: Grafana 13.2.4 (the current release when this was written) installed from Grafana's own apt repository on Ubuntu 24.04, talking to the Prometheus lab from the Prometheus guide (Prometheus 3.15.0 with basic auth, node_exporter, Alertmanager). The lab is a container, so the program was started by hand instead of by systemd; the systemd commands are shown and marked as not run. The web interface itself was not driven (the lab had no browser): every page, panel and rule here was created and checked through files and the API, and where a result is something you would see on screen, the guide says so.
The pieces
Grafana is one program (a Go server with the web interface built in) plus the things around it:
grafana-server the server: web interface, HTTP API and alert evaluation, on port 3000
grafana cli the command-line tool for the admin password, plugins and data migrations
grafana.db a SQLite file with users, dashboards, folders, alert rules and settings (MySQL or PostgreSQL for bigger setups)
data sources where the numbers live: Prometheus, Loki, Elasticsearch, a database. Grafana only asks them questions
provisioning YAML files and JSON files that create data sources, dashboards and alert settings at start-up
plugins extra data sources, panels and apps; some are downloaded on the first start (see below)
The flow: you (or a file) define a data source; a dashboard holds panels; each panel has a query that Grafana sends to the data source when somebody looks at it. Alert rules run the same sort of query every minute or so on the server, with nobody looking, and send notifications when the answer is bad.
Grafana is not a metrics store. If the data source is down, the dashboard is empty. That makes a good first alert: Prometheus itself (see the Prometheus guide) should be monitored from outside Grafana as well.
Install from Grafana's apt repository
Ubuntu's own repositories have no Grafana package (the lab checked: apt-cache policy grafana prints nothing on a stock Ubuntu 24.04). Grafana Labs runs its own apt repository. Add the signing key and the repository, then install:
$ sudo apt-get install -y apt-transport-https gnupg wget ca-certificates
$ sudo mkdir -p /etc/apt/keyrings
$ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
$ echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main
$ sudo apt-get update 2>&1 | grep -i grafana
Get:2 https://apt.grafana.com stable InRelease [7661 B]
Get:6 https://apt.grafana.com stable/main amd64 Packages [423 kB]
Check that the key is the one you meant to trust before you install anything from the repository:
$ gpg --show-keys --with-fingerprint /etc/apt/keyrings/grafana.gpg | head -4
pub rsa3072 2023-08-24 [SC] [expires: 2027-08-22]
B53A E77B ADB6 30A6 8304 6005 963F A277 1045 8545
uid Grafana Labs <engineering@grafana.com>
sub rsa3072 2023-08-24 [E] [expires: 2027-08-22]
The signed-by option ties this key to this one repository only, which is the safer way than adding the key to the system-wide trust store. The key expires in 2027: when apt update starts complaining about it then, fetch the key again.
$ apt-cache policy grafana | head -3
grafana:
Installed: (none)
Candidate: 13.2.4
$ sudo apt-get install -y grafana
Setting up grafana (13.2.4) ...
info: Adding system user `grafana' (UID 101) ...
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-server
The package is large (about 1.3 GB unpacked: dpkg -s grafana reports Installed-Size: 1344296 KB), it creates the grafana user and does not start the service for you. The package named grafana is the open-source edition; grafana-enterprise is the same program with the enterprise features switched on by a licence (not run in the lab).
On a real server, start it with systemd (not run in the lab, which has no systemd; the lab ran the same command line the unit file runs):
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now grafana-server
$ systemctl status grafana-server
$ journalctl -u grafana-server -f
Check it is up. /api/health needs no login:
$ curl -s localhost:3000/api/health
{
"database": "ok",
"version": "13.2.4",
"commit": "0e79aef9d91abf3bdadae9af15763029072fb45b"
}
First login and the admin password
The first login is admin with the password admin, on http://SERVER:3000. Change it at once. In the web interface Grafana asks you to; from a script, use the API (the lab's first call proves the default works, then the change closes it):
$ curl -s -u admin:admin localhost:3000/api/user | jq -c '{login, isGrafanaAdmin, orgId}'
{"login":"admin","isGrafanaAdmin":true,"orgId":1}
$ curl -s -u admin:admin -X PUT -H "Content-Type: application/json" \
-d '{"oldPassword":"admin","newPassword":"NEWPASSWORD","confirmNew":"NEWPASSWORD"}' localhost:3000/api/user/password
{"message":"User password changed"}
$ curl -s -o /dev/null -w "old: %{http_code}\n" -u admin:admin localhost:3000/api/user
old: 401
If you forgot the password, reset it with the CLI on the server. It writes straight to the database, so it works even when you cannot log in:
$ cd /usr/share/grafana
$ sudo -u grafana grafana cli --config /etc/grafana/grafana.ini --homepath /usr/share/grafana admin reset-admin-password NEWPASSWORD
Admin password changed successfully ✔
Gotcha: run it from a directory the grafana user can read. From a home directory the lab got a Go panic ("error getting work directory: stat .: permission denied") and no password change. That is why the example starts with cd /usr/share/grafana.
The first start also installs a few plugins. The lab log shows the background installer fetching them from grafana.com:
$ ls /var/lib/grafana/plugins
grafana-advisor-app grafana-exploretraces-app grafana-lokiexplore-app grafana-metricsdrilldown-app grafana-pyroscope-app
On a server with no internet access the first start still works, but the installer logs failures. grafana.ini has a [plugins] preinstall setting for this (see the comments in the file; not run in the lab).
Where things live and how to configure Grafana
The deb package puts everything in the usual places:
/etc/grafana/grafana.ini the main configuration file (root:grafana, mode 640, so it is not world-readable)
/etc/grafana/provisioning/ data sources, dashboards, alerting, plugins and access-control files read at start-up
/etc/default/grafana-server environment file the systemd unit reads (paths, and any GF_ settings)
/var/lib/grafana/ data: grafana.db, plugins, and anything you keep there (the lab keeps dashboard JSON in /var/lib/grafana/dashboards)
/var/log/grafana/grafana.log the log
/usr/share/grafana/ the program and its web files (not configuration)
grafana.ini is about 1,860 lines long and every line is a commented-out default, so reading it tells you every setting. A semicolon at the start of a line means "default, not set". Remove the semicolon to set a value:
$ sudo grep -E "^;?(http_port|domain|root_url)" /etc/grafana/grafana.ini
;http_port = 3000
;domain = localhost
;root_url = %(protocol)s://%(domain)s:%(http_port)s/
The second way is environment variables. Any setting can be given as GF_SECTION_KEY, in capitals. With the deb package, put them in /etc/default/grafana-server, which the systemd unit reads as an EnvironmentFile. The lab used this for the proxy settings in a later section:
$ sudo tee -a /etc/default/grafana-server <<EOF
GF_SERVER_DOMAIN=graf.example.com
GF_SERVER_ROOT_URL=https://graf.example.com/grafana/
GF_SERVER_SERVE_FROM_SUB_PATH=true
GF_SERVER_HTTP_ADDR=127.0.0.1
GF_USERS_ALLOW_SIGN_UP=false
GF_ANALYTICS_REPORTING_ENABLED=false
EOF
An environment variable overrides the same setting in grafana.ini, so the file stays a clean copy of the defaults and your changes sit in one short file. Grafana reads its configuration only at start-up. After any change, restart it (sudo systemctl restart grafana-server) and look at the log line that shows what it bound to:
$ grep "HTTP Server Listen" /var/log/grafana/grafana.log | tail -1
logger=http.server ... msg="HTTP Server Listen" address=127.0.0.1:3000 protocol=http subUrl=/grafana socket=
Gotcha from the lab: if you start Grafana yourself (not through systemd) the variables in the file have to be exported. The lab's helper first read the file without exporting it and the settings silently did nothing, until the helper used set -a. Under systemd the EnvironmentFile already does the right thing.
Provisioning files (data sources, dashboards, alert rules) have a permission trap too. They must be readable by the grafana user. The lab made them root:grafana, mode 640:
$ sudo chgrp grafana /etc/grafana/provisioning/datasources/prometheus.yaml
$ sudo chmod 640 /etc/grafana/provisioning/datasources/prometheus.yaml
A root-only file (mode 600) made the reload fail with "open ...: permission denied" (shown in the data source section).
Data sources as a file
A data source is a named connection with a type, a URL and credentials. You can add one in the web interface (Connections, Data sources), but a file is repeatable and survives rebuilding the server. Grafana reads every YAML file in /etc/grafana/provisioning/datasources/. This is the lab's connection to the password-protected Prometheus from the Prometheus guide:
# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
- name: Prometheus
uid: prom
type: prometheus
access: proxy
url: http://prom:9090
isDefault: true
basicAuth: true
basicAuthUser: admin
secureJsonData:
basicAuthPassword: labpass-prom1
jsonData:
timeInterval: 15s
What matters:
uid a name you choose. Dashboards and alert rules refer to the data source by uid, so pick one and never change it
access: proxy the Grafana server makes the requests, not the viewer's browser. Prometheus then only needs to be reachable from Grafana
secureJsonData the password goes here. Grafana stores it encrypted and never returns it from the API
timeInterval the scrape interval of Prometheus, so that rate() windows are sensible
Load it without a restart. The reload call is asynchronous, so give it a few seconds, then ask the health check:
$ curl -s -u admin:PASSWORD -X POST localhost:3000/api/admin/provisioning/datasources/reload
{"message":"Datasources config reloaded"}
$ curl -s -u admin:PASSWORD localhost:3000/api/datasources | jq -c '.[] | {name,uid,type,url,isDefault,readOnly,basicAuth}'
{"name":"Prometheus","uid":"prom","type":"prometheus","url":"http://prom:9090","isDefault":true,"readOnly":true,"basicAuth":true}
$ curl -s -u admin:PASSWORD localhost:3000/api/datasources/uid/prom/health | jq -c .
{"details":{"application":"Prometheus","features":{"rulerApiEnabled":false}},"message":"Successfully queried the Prometheus API.","status":"OK"}
readOnly is true: a provisioned data source cannot be changed in the web interface or by the API. The lab tried and got "Cannot update read-only data source". Change the file and reload.
What the failures look like in the lab:
Wrong password (the file with a wrong basicAuthPassword, after a reload and a wait of a few seconds):
{"message":"401 Unauthorized - There was an error returned querying the Prometheus API.","status":"ERROR"}
Reading the health check too early gives the old answer ("OK") because the reload had not finished. If the check says OK when you expect an error, wait and ask again.
A file the grafana user cannot read (mode 600, owner root): the reload call returns {"message":"Internal Server Error"} and the log says
level=error msg="Failed to provision data sources" error="Datasource provisioning error: open /etc/grafana/provisioning/datasources/prometheus.yaml: permission denied"
A YAML mistake gives the same Internal Server Error and a log line such as
Datasource provisioning error: yaml: line 2: did not find expected '-' indicator
In both cases the old data source kept working (the list still showed Prometheus). The log is where the real message is, not the HTTP answer.
Asking a question: the query API
Under every panel is a query. You can send the same query yourself, which is the fastest way to test a data source and to debug a panel. POST to /api/ds/query with the data source uid and the data source's own query language. For Prometheus that is PromQL:
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/ds/query -d '{
"queries":[{"refId":"A","datasource":{"uid":"prom","type":"prometheus"},"expr":"up","instant":true}],
"from":"now-5m","to":"now"}' \
| jq -c '.results.A.frames[] | {labels:.schema.fields[1].labels, v:.data.values[1]}'
{"labels":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"v":[1]}
{"labels":{"__name__":"up","env":"lab","instance":"tgt:9100","job":"node","role":"web","team":"sre"},"v":[1]}
{"labels":{"__name__":"up","instance":"http://tgt/","job":"blackbox-http"},"v":[1]}
Three targets, all up, exactly what Prometheus itself reports. Use instant for one value, or "range":true with an intervalMs and maxDataPoints for a time series (the lab's CPU query over five minutes returned 8 points at the 15 second step).
Grafana can also pass on a call to the data source's own API, which is how its variable lists are filled. Asking Prometheus for the values of a label through Grafana:
$ curl -s -u admin:PASSWORD "localhost:3000/api/datasources/uid/prom/resources/api/v1/label/instance/values?match%5B%5D=node_uname_info"
{"status":"success","data":["tgt:9100"]}
In the web interface, the same thing is the Explore page: pick the data source, type the query, see the graph and the raw table. Treat Explore as your scratch pad and the dashboard as the thing you keep.
A dashboard as a file
A dashboard is one JSON document: a title, a time range, variables and a list of panels. Each panel has a type (stat, gauge, timeseries, table and so on), a position on a 24-column grid, and one or more queries. This is the lab's dashboard: a count of targets that are up, a memory gauge and a CPU graph, with a variable so that one dashboard serves every host:
# /var/lib/grafana/dashboards/node.json
{
"uid": "node-overview",
"title": "Node overview",
"tags": ["lab", "node"],
"timezone": "browser",
"schemaVersion": 41,
"version": 1,
"refresh": "30s",
"time": {"from": "now-1h", "to": "now"},
"templating": {"list": [{
"name": "instance", "type": "query", "label": "Instance",
"datasource": {"type": "prometheus", "uid": "prom"},
"query": {"query": "label_values(node_uname_info, instance)", "refId": "v1"},
"refresh": 1, "includeAll": false, "current": {}
}]},
"panels": [
{"id": 1, "type": "stat", "title": "Targets up",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
"datasource": {"type": "prometheus", "uid": "prom"},
"targets": [{"refId": "A", "expr": "count(up == 1)", "instant": true}],
"fieldConfig": {"defaults": {"unit": "none", "thresholds": {"mode": "absolute",
"steps": [{"color": "red", "value": null}, {"color": "green", "value": 3}]}}, "overrides": []}},
{"id": 2, "type": "gauge", "title": "Memory available",
"gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
"datasource": {"type": "prometheus", "uid": "prom"},
"targets": [{"refId": "A", "instant": true,
"expr": "node_memory_MemAvailable_bytes{instance=\"$instance\"} / node_memory_MemTotal_bytes{instance=\"$instance\"}"}],
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "thresholds": {"mode": "absolute",
"steps": [{"color": "red", "value": null}, {"color": "orange", "value": 0.1}, {"color": "green", "value": 0.25}]}}, "overrides": []}},
{"id": 3, "type": "timeseries", "title": "CPU busy",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
"datasource": {"type": "prometheus", "uid": "prom"},
"targets": [{"refId": "A", "legendFormat": "{{instance}}",
"expr": "1 - avg(rate(node_cpu_seconds_total{instance=\"$instance\",mode=\"idle\"}[$__rate_interval]))"}],
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1}, "overrides": []}}
]
}
Notes:
$instance the variable. Grafana replaces it in every query with the value picked at the top of the dashboard
$__rate_interval a built-in that picks a rate() window from the data source's timeInterval, so graphs stay correct when you zoom
the steps list colours by value; value null is "from minus infinity"
uid choose a stable uid, so links to the dashboard and the file keep working
Tell Grafana to load every JSON file in a folder with a dashboard provider:
# /etc/grafana/provisioning/dashboards/files.yaml
apiVersion: 1
providers:
- name: lab-files
folder: Lab
type: file
disableDeletion: true
allowUiUpdates: false
updateIntervalSeconds: 30
options:
path: /var/lib/grafana/dashboards
$ sudo cp node.json /var/lib/grafana/dashboards/ && sudo chown -R grafana:grafana /var/lib/grafana/dashboards
$ curl -s -u admin:PASSWORD -X POST localhost:3000/api/admin/provisioning/dashboards/reload
{"message":"Dashboards config reloaded"}
$ curl -s -u admin:PASSWORD "localhost:3000/api/search?type=dash-db" | jq -c '.[] | {title,uid,folderTitle,url}'
{"title":"Node overview","uid":"node-overview","folderTitle":"Lab","url":"/d/node-overview/node-overview"}
$ curl -s -u admin:PASSWORD localhost:3000/api/dashboards/uid/node-overview | jq -c '.meta | {provisioned,provisionedExternalId,folderTitle}'
{"provisioned":true,"provisionedExternalId":"node.json","folderTitle":"Lab"}
The folder Lab was created by the provider. The file is the source of truth: with allowUiUpdates false, a change made in the web interface cannot be saved over it (the editor offers to save the JSON so you can put it in the file instead). The provider re-reads the folder every updateIntervalSeconds, so editing the file is enough.
What the lab could and could not check. The variable query and all three panel queries were run through the API with $instance replaced by tgt:9100 (the variable list returned ["tgt:9100"]; the panels returned 3 targets up, memory available 0.797 and CPU busy 0.077). The lab had no browser, so how the page looks was not checked: open the dashboard once and look at it.
Folders and dashboards through the API
Everything the web interface does for folders and dashboards is an HTTP API call. Create a folder, then save a dashboard into it:
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/folders -d '{"title":"Team web"}' | jq -c '{uid,title}'
{"uid":"ag1gfcj79csg0c","title":"Team web"}
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/dashboards/db -d '{
"dashboard":{"title":"Web scratch","tags":["web"],"panels":[{"id":1,"type":"stat","title":"Probe success",
"gridPos":{"h":4,"w":8,"x":0,"y":0},"datasource":{"type":"prometheus","uid":"prom"},
"targets":[{"refId":"A","expr":"probe_success","instant":true}]}]},
"folderUid":"ag1gfcj79csg0c","overwrite":false,"message":"first version"}' | jq -c .
{"folderUid":"ag1gfcj79csg0c","id":3587052937670656,"slug":"web-scratch","status":"success","uid":"anrvkd","url":"/d/anrvkd/web-scratch","version":1}
Grafana made the uid (anrvkd) because the JSON had none. Things the lab found:
Two dashboards with the same title in the same folder were allowed. Posting the same title again created a second dashboard with its own uid. Older Grafana versions refused with "name already exists". Give your dashboards uids, and look for duplicates, not for an error.
Updating without folderUid moves the dashboard to the General folder. The lab fetched a dashboard, changed its title and posted it back with no folderUid: it silently left "Team web". The effect is a permissions surprise (see the permissions section): the people who had access to the folder lose it. Always send folderUid when you save.
Versions are kept: GET /api/dashboards/uid/UID/versions lists them with the message you gave (the lab showed "rename" at version 2 and the first save as "Initial save").
A stale edit is refused. Posting JSON with an old version number returned HTTP 409. Re-read the dashboard, apply your change, and post again, or send "overwrite":true if you really mean to replace it.
The put-it-back call, with the folder and overwrite:
$ J=$(curl -s -u admin:PASSWORD localhost:3000/api/dashboards/uid/anrvkd | jq -c .dashboard)
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/dashboards/db \
-d "{\"dashboard\":$J,\"folderUid\":\"ag1gfcj79csg0c\",\"overwrite\":true,\"message\":\"put back in Team web\"}" | jq -c '{status,version,folderUid}'
{"status":"success","version":3,"folderUid":"ag1gfcj79csg0c"}
Delete by uid:
$ curl -s -u admin:PASSWORD -X DELETE localhost:3000/api/dashboards/uid/avlg7w | jq -c .
{"message":"Dashboard Web scratch deleted","title":"Web scratch","uid":"avlg7w"}
Export is the same GET: the JSON you receive is the JSON you can put in a provisioning file or post to another server. (Grafana 13 also exposes dashboards on a Kubernetes-style API under /apis/dashboard.grafana.app/. The lab read the provisioned one there and it showed 3 panels and the file's path in its annotations. This guide sticks to the classic /api calls.)
A Grafana 13 detail: dashboards and folders are stored in a different table than before. The lab's sqlite3 query on the old dashboard table returned 0 rows while the dashboards worked; they live in the resource table. Back up the whole database file, not tables (see the backup section).
Users, teams and permissions
Grafana has three layers. A role in the organisation (Viewer, Editor, Admin) sets what a person can do in general. Teams group people. Folder and dashboard permissions then give a person, a team or a whole role access to one folder. Create a user with the admin API, change the role, make a team and add the user to it:
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/admin/users \
-d '{"name":"Ann Viewer","email":"ann@example.com","login":"ann","password":"labpass-ann1"}'
{"id":2,"uid":"eg1gfdblpsjr4a","message":"User created"}
$ curl -s -u admin:PASSWORD localhost:3000/api/org/users | jq -c '.[] | {login,role}'
{"login":"admin","role":"Admin"}
{"login":"ann","role":"Viewer"}
{"login":"ed","role":"Viewer"}
$ curl -s -u admin:PASSWORD -X PATCH -H "Content-Type: application/json" localhost:3000/api/org/users/3 -d '{"role":"Editor"}'
{"message":"Organization user updated"}
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/teams -d '{"name":"web-team","email":"web@example.com"}'
{"message":"Team created","teamId":1,"uid":"cg1gfdbz8qr5sa"}
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/teams/1/members -d '{"userId":2}'
{"message":"Member added to Team"}
New users are Viewers by default. Folders start with permissions for Admin, Editor (edit) and Viewer (view). Set a folder's permissions to: Admin role admin, team web-team edit, Editor role edit (this call replaces the whole list, so a role you leave out loses access):
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/folders/ag1gfcj79csg0c/permissions \
-d '{"items":[{"role":"Admin","permission":4},{"teamId":1,"permission":2},{"role":"Editor","permission":2}]}'
{"message":"Folder permissions updated"}
Permission numbers: 1 view, 2 edit, 4 admin. The lab then asked what Ann (a Viewer in the organisation, but in web-team) could do:
$ curl -s -u ann:labpass-ann1 "localhost:3000/api/search?type=dash-db" | jq -c '.[] | [.title,.folderTitle]'
["Node overview","Lab"]
["Web probes","Team web"]
$ curl -s -u ann:labpass-ann1 -X POST -H "Content-Type: application/json" localhost:3000/api/folders -d '{"title":"x"}'
{"message":"Access denied"}
Ann sees the Lab dashboard (the Viewer role can still view it) and, through her team, the dashboard in Team web. She cannot create folders, because that is an organisation-level right. As an Editor of the Team web folder through her team she can save dashboards in it, but a save with no folderUid was refused ("not allowed to create resource in the destination folder"), another reminder to send folderUid.
Gotchas that cost time in the lab:
The Viewer role was removed from the Team web folder by the permissions call above, so Viewers outside the team see nothing in it. That is the point; just do it on purpose.
When a dashboard that a team can see "disappears" for them, check which folder it is in. In the lab it had moved to General (the update-without-folderUid trap) and Ann got 403.
A team permission may take a moment to show (Grafana caches permissions for a short time). If a change does not show, wait half a minute before assuming it is wrong.
Service accounts are the right way to give a script access (do not share a person's password or use an old-style API key). Create the account with a role, then a token with an expiry:
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/serviceaccounts -d '{"name":"ci-deploy","role":"Editor"}' | jq -c '{id,name,login,role}'
{"id":4,"name":"ci-deploy","login":"sa-1-ci-deploy","role":"Editor"}
$ curl -s -u admin:PASSWORD -X POST -H "Content-Type: application/json" localhost:3000/api/serviceaccounts/4/tokens -d '{"name":"ci-token","secondsToLive":86400}' | jq -r .key
glsa_... (46 characters; shown once, store it now)
$ curl -s -H "Authorization: Bearer $TOKEN" "localhost:3000/api/search?type=dash-db" | jq length
2
$ curl -s -H "Authorization: Bearer $TOKEN" -X POST -H "Content-Type: application/json" localhost:3000/api/admin/users -d '{}' | jq -c .message
"You'll need additional permissions to perform this action. Permissions needed: users:create"
The token works with the Editor role of its service account and no more. Anonymous access is a setting (GF_AUTH_ANONYMOUS_ENABLED=true and GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer). The lab tried it: without a login, the search returned only the dashboards that Viewers may see ("Node overview", not the one in Team web), /api/user said Unauthorized, and creating a folder was refused. It is a quick way to put a wall display on a screen; keep it off otherwise. Removing the two lines and restarting brought back 401.
Alerting: rules, policies, contact points
Grafana's alerting has three parts, and it helps to keep them apart:
alert rule a query, a condition and a "for" time. It decides that something is wrong, and gives the alert labels (severity=critical)
notification policy a tree that looks at the labels and chooses who is told, how often, and how alerts are grouped
contact point where a message goes: email, webhook, Slack, PagerDuty and so on
The links between them are the labels. The rule puts severity=critical on the alert; a policy matches severity=critical and names a contact point. Like Prometheus with Alertmanager, but in one program. (You can use both. Grafana evaluates its own rules; Prometheus rules still go to Alertmanager.)
First, email needs a mail server. The lab used a local postfix and set the [smtp] section in grafana.ini (remove the semicolons, then restart):
[smtp]
enabled = true
host = localhost:25
from_address = grafana@graf.example.com
startTLS_policy = NoStartTLS
(For a real mail server use its host:port, a user and password, and keep TLS on. The lab's local server had none.)
Alert settings can be provisioned like everything else. One file holds contact points, the policy tree and rule groups. The lab's, with a webhook and an email contact point, a rule that fires when a Prometheus target is down, and the policy:
# /etc/grafana/provisioning/alerting/alerts.yaml
apiVersion: 1
contactPoints:
- orgId: 1
name: team-webhook
receivers:
- uid: cp-webhook
type: webhook
settings:
url: http://localhost:5001/alerts
httpMethod: POST
- orgId: 1
name: oncall-email
receivers:
- uid: cp-email
type: email
settings:
addresses: oncall@graf.example.com
singleEmail: true
policies:
- orgId: 1
receiver: team-webhook
group_by: ["alertname", "instance"]
group_wait: 10s
group_interval: 1m
repeat_interval: 4h
routes:
- receiver: oncall-email
object_matchers:
- ["severity", "=", "critical"]
continue: true
- receiver: team-webhook
groups:
- orgId: 1
name: node
folder: Lab
interval: 30s
rules:
- uid: instance-down
title: InstanceDown
condition: C
for: 1m
noDataState: NoData
execErrState: Error
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} is down"
data:
- refId: A
relativeTimeRange: { from: 300, to: 0 }
datasourceUid: prom
model:
refId: A
expr: up{job="node"}
instant: true
intervalMs: 15000
maxDataPoints: 43200
- refId: B
datasourceUid: __expr__
model: { refId: B, type: reduce, expression: A, reducer: last }
- refId: C
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator: { type: lt, params: [1] }
The rule is a small pipeline. A runs the query. B reduces its answer to one number per series (the last value). C compares that number with the threshold (below 1) and the rule's condition says "use C". Its "for: 1m" means the condition must hold for a full minute before the alert fires, which stops short blips from paging anyone.
$ sudo chgrp grafana /etc/grafana/provisioning/alerting/alerts.yaml && sudo chmod 640 /etc/grafana/provisioning/alerting/alerts.yaml
$ curl -s -u admin:PASSWORD -X POST localhost:3000/api/admin/provisioning/alerting/reload
{"message":"Alerting config reloaded"}
$ curl -s -u admin:PASSWORD localhost:3000/api/v1/provisioning/contact-points | jq -c '.[] | {name,type,provenance}'
{"name":"oncall-email","type":"email","provenance":"file"}
{"name":"team-webhook","type":"webhook","provenance":"file"}
$ curl -s -u admin:PASSWORD localhost:3000/api/v1/provisioning/alert-rules | jq -c '.[] | {title,ruleGroup,for,provenance}'
{"title":"InstanceDown","ruleGroup":"node","for":"1m","provenance":"file"}
provenance "file" means it came from a file: the web interface shows these as read-only. To change them, edit the file and reload. Policies and contact points apply the same way. The sample.yaml files in the provisioning folders are well commented: the lab removed the alerting one so only its own file was left.
An alert from start to finish. The lab broke the Prometheus target on purpose (changing its port in the Prometheus file_sd file, as in the Prometheus guide) and watched the rule through the rules API:
$ curl -s -u admin:PASSWORD localhost:3000/api/prometheus/grafana/api/v1/rules | jq -r '[.data.groups[].rules[] | .name+": "+.state+" "+([.alerts[]?|.state]|join(","))] | join(" | ")'
InstanceDown: pending Pending,Normal (20 s after the change: the target is down, the "for" minute has started)
InstanceDown: pending Pending,Normal (a minute later: still waiting)
InstanceDown: firing Alerting (about 80 s after the change)
Two instances appear at first: the new series (tgt:9199, Pending, then Alerting) and the old one (tgt:9100, Normal) which is still in the answer until it ages out. That is the label change problem from the Prometheus guide, seen from the other side. Once the alert fired, Grafana's Alertmanager grouped it for group_wait (10 s) and sent the notifications. The email:
From: "Grafana" <grafana@graf.example.com>
To: oncall@graf.example.com
Subject: [FIRING:1] InstanceDown tgt:9199 (lab Lab node web critical sre)
and the webhook body (15 top-level fields; the ones you will use are receiver, status, title, and alerts[] with labels, annotations and the start time):
{"receiver":"team-webhook","status":"firing","title":"[FIRING:1] InstanceDown tgt:9199 (lab Lab node web critical sre)",
"alerts":[{"status":"firing","labels":{"alertname":"InstanceDown","instance":"tgt:9199","severity":"critical",...},
"annotations":{"summary":"tgt:9199 is down"}, ...}], ...}
Putting the port back produced "[RESOLVED] InstanceDown tgt:9199 ..." by email. Resolved messages are on by default for Grafana's email and webhook contact points (unlike Prometheus's Alertmanager, where email needs send_resolved: true). The webhook got a resolved message too (status "resolved"); each contact point has a disableResolveMessage option if you do not want them.
Two things that went wrong in the lab, both worth knowing:
1. With the first version of the policy (a child route for severity=critical with no continue and nothing else), the critical alert went to the email contact point only. The root receiver (the webhook) did not get it: the webhook log stayed empty. A matching child route takes the alert away from its parent. To notify both, set continue: true on the critical route and add a sibling route that matches everything and names the webhook, as in the file above. With that, both the email and the webhook arrived.
2. The first email did not arrive and the log had nothing. The mail server had bounced it: "cannot update mailbox /var/mail/oncall for user". I had emptied the mailbox file as root, so postfix could not write to it (the bounce came back as a message to the grafana user). Grafana had done its part: a missing email is a mail-server problem until the mail log says otherwise. The same applies to webhooks: check that the receiving end answers 2xx.
Not run in the lab: mute timings and silences (they exist, under Alerting), Slack and PagerDuty contact points, notification templates, and recording rules.
Putting Grafana behind nginx
Do not expose port 3000 to the network. Bind Grafana to the loopback address and let a reverse proxy (nginx) take the public port and handle TLS. Two settings tell Grafana what its public address is:
GF_SERVER_ROOT_URL=https://graf.example.com/grafana/ the URL people type; used in links, emails and redirects
GF_SERVER_SERVE_FROM_SUB_PATH=true Grafana answers under /grafana/, so one host can serve more than one site
GF_SERVER_HTTP_ADDR=127.0.0.1 listen on loopback only
The lab's nginx file:
# /etc/nginx/conf.d/grafana.conf
map $http_upgrade $connection_upgrade { default upgrade; "" close; }
server {
listen 80;
server_name graf.example.com;
location /grafana/ {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:3000;
}
location /grafana/api/live/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:3000;
}
}
The second location is for Grafana Live, which uses WebSockets (live dashboards update). The first location passes the Host header on, so Grafana sees the name people used. TLS (listen 443 ssl and a certificate) was not run in the lab: it uses plain port 80 here. Add it for real use.
Test each step, from the server and from another machine:
$ sudo nginx -t && sudo nginx -s reload
$ curl -s http://127.0.0.1:3000/grafana/api/health | jq -c .database
"ok"
$ curl -s -H "Host: graf.example.com" http://127.0.0.1/grafana/api/health | jq -c .database
"ok"
$ curl -s -i -H "Host: graf.example.com" http://127.0.0.1/grafana/ | grep -iE "^(HTTP|location)"
HTTP/1.1 302 Found
Location: /grafana/login
$ curl -s -H "Host: graf.example.com" http://127.0.0.1/grafana/login | grep -o '<base href="[^"]*"'
<base href="/grafana/"
From another container: port 3000 gave no answer at all (it listens on 127.0.0.1 only) and port 80 through nginx returned 200. The bound addresses are in the Grafana log ("HTTP Server Listen address=127.0.0.1:3000 ... subUrl=/grafana").
Notes: with serve_from_sub_path on, the old path without the /grafana prefix still answered /api/health in the lab, so do not rely on the prefix as a security measure. If you see a redirect loop or a blank page behind a proxy, root_url is nearly always the cause: it must match exactly what the browser uses, including the trailing slash. Open the firewall for 80 and 443 only (see the UFW guide), never for 3000.
Plugins
Plugins add data sources, panel types and apps. List, install and remove them with the CLI as the grafana user, from a readable directory, then restart Grafana (it only looks at start-up):
$ cd /usr/share/grafana
$ sudo -u grafana grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini plugins ls
installed plugins:
grafana-exploretraces-app @ 3.0.0
grafana-lokiexplore-app @ 3.0.0
grafana-metricsdrilldown-app @ 3.0.0
grafana-pyroscope-app @ 3.0.0
grafana-advisor-app @ 1.1.0
$ sudo -u grafana grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini plugins install grafana-clock-panel
Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.
$ ls /var/lib/grafana/plugins
grafana-advisor-app grafana-clock-panel grafana-exploretraces-app ...
$ sudo -u grafana grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini plugins uninstall grafana-clock-panel
The lab installed and removed the clock panel and the directory changed as shown. Grafana only loads signed plugins by default (not tried in the lab); installing an unsigned one needs allow_loading_unsigned_plugins in grafana.ini, which is a decision to make carefully. Install plugins only from the catalogue you trust, as they run inside the server. Not run in the lab: provisioning plugin settings through /etc/grafana/provisioning/plugins.
Backup, restore and upgrade
What to back up: the database, the configuration and your provisioning files, and the plugin list. The database holds users, folders, dashboards made in the web interface, alert rules and settings; the files in /etc/grafana and your dashboard folder hold the rest. With SQLite, use sqlite3's .backup, which is safe while Grafana runs (a plain cp of a busy file can be inconsistent):
$ sudo sqlite3 /var/lib/grafana/grafana.db ".backup /var/backups/grafana-$(date +%F).db"
$ sudo sqlite3 /var/backups/grafana-2026-10-17.db "pragma integrity_check;"
ok
$ sudo tar czf /var/backups/grafana-etc-$(date +%F).tar.gz /etc/grafana /etc/default/grafana-server /var/lib/grafana/dashboards
Do not forget secret_key in grafana.ini (the lab used the default): it encrypts the data source passwords in the database. A backup of the database restored onto a server with a different secret_key should not be able to decrypt them (not run in the lab, but it is how the encryption works). Set your own secret_key before you have data sources, and keep it with the backup. (Changing it later is a separate procedure: not run in the lab.)
The lab did a full restore test. It backed up the database, deleted a dashboard through the API, stopped Grafana, copied the backup over grafana.db (owner grafana:grafana, mode 640, and removed the leftover -wal and -shm files), started Grafana, and asked for the dashboards:
$ curl -s -u admin:PASSWORD "localhost:3000/api/search?type=dash-db" | jq -c '[.[]|[.title,.folderTitle]]'
[["Node overview","Lab"],["Web probes","Team web"]]
The deleted dashboard was back. With MySQL or PostgreSQL use mysqldump or pg_dump instead (not run in the lab). Test your restore on a spare machine; a backup that was never restored is a hope.
Upgrade: back up first, then apt does the rest, because the repository is configured. Pin the version if you want upgrades to be a decision rather than a side effect of unattended-upgrades:
$ sudo apt-get update && sudo apt-get install --only-upgrade grafana
$ sudo apt-mark hold grafana
grafana set on hold.
$ apt-mark showhold
grafana
$ sudo apt-mark unhold grafana
On the first start after an upgrade Grafana migrates its database, so you cannot go back to the old version without the old backup. Read the release notes before a major version (the lab did not upgrade; it installed the current release).
Cheat sheet
Install add key + repo (apt.grafana.com), apt install grafana, systemctl enable --now grafana-server
Health curl -s localhost:3000/api/health (no login needed)
Logs /var/log/grafana/grafana.log or journalctl -u grafana-server
Config /etc/grafana/grafana.ini or GF_SECTION_KEY=value in /etc/default/grafana-server; restart after
Admin password cd /usr/share/grafana; sudo -u grafana grafana cli --config /etc/grafana/grafana.ini --homepath /usr/share/grafana admin reset-admin-password PW
Data source file /etc/grafana/provisioning/datasources/*.yaml reload: POST /api/admin/provisioning/datasources/reload
Dashboard files /etc/grafana/provisioning/dashboards/*.yaml (provider) + JSON in a folder reload: POST .../dashboards/reload
Alert settings /etc/grafana/provisioning/alerting/*.yaml reload: POST /api/admin/provisioning/alerting/reload
Check a data source GET /api/datasources/uid/UID/health
Run a query POST /api/ds/query {"queries":[{"refId":"A","datasource":{"uid":"UID"},"expr":"up","instant":true}],"from":"now-5m","to":"now"}
List dashboards GET /api/search?type=dash-db
Save a dashboard POST /api/dashboards/db {"dashboard":{...},"folderUid":"UID","overwrite":true}
Folder permissions POST /api/folders/UID/permissions {"items":[{"role":"Editor","permission":2},{"teamId":1,"permission":2}]} (1 view, 2 edit, 4 admin)
Service account POST /api/serviceaccounts, then POST /api/serviceaccounts/ID/tokens, use header "Authorization: Bearer TOKEN"
Rules and states GET /api/prometheus/grafana/api/v1/rules
Contact points GET /api/v1/provisioning/contact-points policies: .../policies rules: .../alert-rules
Plugins sudo -u grafana grafana cli ... plugins ls | install NAME | uninstall NAME (restart afterwards)
Backup sqlite3 /var/lib/grafana/grafana.db ".backup FILE" + tar of /etc/grafana and the dashboards folder
Hold a version sudo apt-mark hold grafana
Common problems
Reload returns Internal Server Error read /var/log/grafana/grafana.log: usually a YAML error or a provisioning file the grafana user cannot read (chgrp grafana, chmod 640)
Data source "OK" when it should fail the reload is asynchronous: wait a few seconds and run the health check again
Data source says 401 Unauthorized the password in secureJsonData is wrong, or the target needs basic_auth and the data source does not have it
Cannot edit a data source or dashboard it is provisioned (readOnly true / provenance "file"): change the file and reload
grafana cli panics "stat .: permission denied" run it from a directory the grafana user can read, for example cd /usr/share/grafana
GF_ settings have no effect Grafana was started by hand without exporting the variables; under systemd put them in /etc/default/grafana-server; check the "HTTP Server Listen" log line
Dashboard moved to General and the team lost it a save without folderUid; always send folderUid, then fix with the put-it-back call
Save returns 409 / "already exists" a stale version; re-read the dashboard, or send overwrite true
Two dashboards with the same title allowed in Grafana 13; use uids to tell them apart
A user cannot see a folder or dashboard check the org role, team membership and folder permissions; a permissions call replaces the whole list; wait a moment for the cache
Alert stays Pending the for: time keeps restarting because the condition flickers, or the rule interval is longer than you thought
Two alert instances after a target change the old series is still in the answer until it ages out; the new one is the real alert
Critical alert reaches only one contact point a matching child route takes the alert from its parent; set continue: true and add a sibling route for the other receiver
Email never arrives Grafana sent it; read the mail server's log and check that the mailbox is writable by the mail system (root-owned mailbox file bounced it)
No resolved message the contact point's disableResolveMessage is on, or the resolve arrives after group_interval; wait for the next group interval
Redirect loop or blank page behind a proxy root_url does not match the address in the browser (and the trailing slash); with a sub-path also serve_from_sub_path
Backup restored but data sources fail secret_key differs from the one the backup was made with
Plugin not visible after install restart Grafana; plugins are read only at start-up
First start is slow or logs plugin errors the preinstalled plugins are downloaded from the internet; allow it, or set the [plugins] preinstall option
Tips
Keep data sources, dashboards and alert rules in files in version control, and let the web interface be where you explore and prototype. Export what you built and commit it.
Give every data source and dashboard a uid you chose. Links, folders and alert rules refer to uids, and a generated uid is different on every server.
Run Grafana on the loopback address behind nginx with TLS, and never open port 3000 to the network.
Change the admin password on the first day, set secret_key yourself, disable sign-up, and keep anonymous access off unless a wall screen needs it.
Give people access through teams and folders, not one dashboard at a time. Use a service account with an expiry for each script.
Alert on what users feel and page only on that. Add the causes as warnings, and put a for: time on nearly every rule.
Test the whole chain whenever you change contact points or policies: break something on purpose in a test target and watch the email and the webhook arrive.
Look at the notification policy as a tree. Draw it on paper: root, children, which ones continue. Surprises come from the tree, not the rules.
Watch Grafana itself: its /api/health and a disk space check on /var/lib/grafana. A dashboard tool that is down tells nobody.
Back up grafana.db, /etc/grafana and your dashboard folder, keep the secret_key with them, and restore into a spare machine once to prove it works.
Pin the version with apt-mark hold and upgrade when you chose to, after reading the release notes and taking a backup.
