☁️ Ollama on a Cloud VM

Ollama on a Cloud VM (AWS, Azure and Google Cloud)

Run your own private AI on a rented server: no 8 GB laptop limit, and you can reach it from anywhere.
This guide builds it step by step, shows how to keep it secure, and how to keep the bill small.
The steps are the same on every cloud. Only the instance names differ.
Read "Costs" before you click Create. A GPU machine left running for a month can cost hundreds of dollars.
Prices, instance names and quotas change often. Confirm them on your cloud's pricing pages before you build.

First, the honest truth about "free tier"
The free VMs in the other guides (AWS t2/t3.micro, Azure B1s, GCP e2-micro) have about 1 GB of RAM.
That is too small to run any useful model. Even a 1B-parameter model needs about 1.5 GB to load.
To run Ollama in the cloud, plan on a paid machine: 16 GB of RAM for small models on the CPU, or a GPU for speed.
Use free trial credits for this, and set a budget alert first.

CPU only (cheap and slow: fine for learning and light use)
  4 vCPU / 16 GB RAM    3B to 8B models, a few words per second     AWS m7i.xlarge    Azure D4s_v5    GCP e2-standard-4
  4 vCPU / 32 GB RAM    Up to 14B models                            AWS r7i.xlarge    Azure E4s_v5    GCP n2-highmem-4
GPU (much faster, much more expensive)
  NVIDIA T4, 16 GB VRAM      7B to 8B models, quick                AWS g4dn.xlarge    Azure NC4as_T4_v3    GCP n1 with a T4
  NVIDIA L4 or A10G, 24 GB   Up to about 27B (4-bit)               AWS g6.xlarge      Azure NV A10 series  GCP g2-standard-4
  A100 / H100                70B-class models, very costly         AWS p4/p5          Azure ND series      GCP a2/a3
Rules of thumb:
- Model size (4-bit) plus 20 to 30 percent for context must fit in memory. On a GPU it must fit in VRAM to be fast
- Start with the smallest machine that fits your model. You can resize a VM later
- Use Ubuntu Server 24.04 LTS as the operating system

New cloud accounts usually have a GPU quota of ZERO, so creating a GPU VM fails until you ask for more:
AWS      Service Quotas > EC2 > "Running On-Demand G and VT instances" > request at least 4 vCPUs
Azure    Subscriptions > Usage + quotas > filter "NC" or "NV" > request an increase
GCP      IAM & Admin > Quotas > filter "GPUs (all regions)" and your GPU type > request 1
Requests can take minutes to days, and free-trial accounts may not be allowed GPUs at all. Ask early.

CPU 4 vCPU / 16 GB       about $0.15 to $0.20 per hour       roughly $110 to $150 a month if left on
GPU with a T4            about $0.50 to $0.75 per hour       roughly $400 to $550 a month if left on
GPU with an L4 or A10G   about $0.80 to $1.20 per hour
Spot / preemptible VMs   Often 60 to 90 percent cheaper, but the cloud can take the VM back at any time
Also billed: disks (a 100 GB disk costs a few dollars a month even when the VM is stopped), public IPs, and data leaving the cloud.
Cost habits that matter most:
1. Set a budget alert BEFORE you start (AWS Budgets, Azure Cost Management, GCP Billing budgets)
2. STOP the VM when you are done. Stopped VMs do not bill for compute, but disks and static IPs still bill
3. On Azure, stop it from the portal or CLI so it is "Deallocated". Shutting down inside the OS does not stop billing
4. Use an idle auto-shutdown (script below)
5. Delete everything when the project ends (see Cleanup)

Use the create-VM steps from the free VM guides for your cloud, with these settings:
Image        Ubuntu Server 24.04 LTS
Size         From the table above
Disk         100 GB or more of SSD (an 8B model is about 5 GB, a 27B model about 17 GB, and you will try several)
Login        SSH key only (no password login)
Firewall     Allow SSH (port 22) from YOUR IP only. Do NOT open port 11434 or 3000 to the internet
Public IP    Ephemeral is fine (it changes on restart). A static IP costs extra while idle
Region       Close to you, and one that has your GPU type in stock
Example CLI commands (names and flags change, so prefer the console the first time):
aws ec2 run-instances --image-id <ubuntu-24.04-ami> --instance-type g4dn.xlarge --key-name mykey --security-group-ids <sg-ssh-from-my-ip-only>
az vm create -g ollama-rg -n ollama-vm --image Ubuntu2404 --size Standard_NC4as_T4_v3 --os-disk-size-gb 100 --admin-username azureuser --generate-ssh-keys
gcloud compute instances create ollama-vm --machine-type=g2-standard-4 --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud --boot-disk-size=100GB
(GCP GPU machines need a few more flags. See the Google Cloud GPU docs for the exact command.)

ssh -i ~/.ssh/mykey.pem ubuntu@VM_PUBLIC_IP          (user: ubuntu on AWS and GCP, azureuser on Azure)
sudo apt update && sudo apt upgrade -y
sudo reboot                                          # if the kernel updated; reconnect after a minute
free -h        df -h        lscpu | head             # confirm RAM, disk and CPUs

nvidia-smi                                           # already installed? It lists your GPU
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers devices                          # shows the recommended driver
sudo ubuntu-drivers autoinstall
sudo reboot
nvidia-smi                                           # after the reboot it should show the GPU, driver and memory
Shortcuts: AWS Deep Learning AMIs ship with drivers, Azure has an NVIDIA GPU Driver extension, and GCP has a driver installer.
If nvidia-smi still fails: lspci | grep -i nvidia (is the GPU there?), then dmesg | grep -i nvidia.

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama                              # active (running)
ollama --version
The script detects an NVIDIA GPU and sets up support for it. AMD GPUs may need ROCm: see the Ollama docs.

For a separate data disk mounted at /data (check with lsblk first, and only format a NEW, EMPTY disk):
sudo mkfs.ext4 /dev/nvme1n1
sudo mkdir /data && sudo mount /dev/nvme1n1 /data
echo 'UUID=<uuid-from-blkid> /data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab
sudo mkdir -p /data/ollama && sudo chown -R ollama:ollama /data/ollama
sudo systemctl edit ollama
[Service]
Environment="OLLAMA_MODELS=/data/ollama"
Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl daemon-reload && sudo systemctl restart ollama
A separate disk survives a VM rebuild, and you can snapshot it so models never need downloading twice.

ollama pull llama3.2                                 # about 2 GB: a quick test
ollama run llama3.2 "Say hello in five words."
ollama pull llama3.1:8b                              # about 5 GB: a good general model on 16 GB RAM
ollama ps                                            # the PROCESSOR column should say 100% GPU on a GPU machine
watch -n 1 nvidia-smi                                # GPU memory and load while it answers (Ctrl+C to exit)
ollama run llama3.1:8b --verbose "Write a haiku about servers."      # prints tokens per second at the end
If ollama ps says "100% CPU" on a GPU machine, the driver is not working. See Common problems.

Ollama has NO password. Never expose port 11434 to the internet. Pick ONE of these methods.

Ollama stays on 127.0.0.1 on the VM. Your laptop forwards a local port through SSH:
ssh -N -L 11434:localhost:11434 -i ~/.ssh/mykey.pem ubuntu@VM_PUBLIC_IP
Leave it running. On your laptop, http://localhost:11434 now reaches the VM:
curl http://localhost:11434/api/tags
OLLAMA_HOST=127.0.0.1:11434 ollama run llama3.1:8b       # the ollama command on your laptop talks to the VM
Keep the tunnel alive with:  autossh -M 0 -N -L 11434:localhost:11434 ubuntu@VM_PUBLIC_IP

curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up       # on the VM and on each of your devices
tailscale ip -4                                                             # the VM's private address, for example 100.64.1.2
sudo systemctl edit ollama   ->   Environment="OLLAMA_HOST=100.64.1.2:11434"
sudo ufw allow in on tailscale0 to any port 11434 proto tcp
Use http://100.64.1.2:11434 from your devices. Nothing is exposed to the public internet, and the address does not change when the VM restarts.

sudo apt install -y caddy
caddy hash-password                                  # prints a hash for the password you choose
# /etc/caddy/Caddyfile
ai.example.com {
    basic_auth {
        alice PASTE_THE_HASH_HERE
    }
    reverse_proxy localhost:11434
}
sudo systemctl reload caddy
Open ports 80 and 443 (not 11434) in the cloud firewall. Clients send the login:
curl -u alice:PASSWORD https://ai.example.com/api/tags
Use a long random password. Public endpoints get scanned within hours.

sudo apt install -y docker.io && sudo usermod -aG docker $USER            # log out and back in
docker run -d --name open-webui --restart unless-stopped -p 127.0.0.1:3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main
ssh -N -L 3000:localhost:3000 ubuntu@VM_PUBLIC_IP                          # then open http://localhost:3000 on your laptop
Binding to 127.0.0.1:3000 keeps it off the internet. Create the first (admin) account right away.
The container must reach Ollama: use http://host.docker.internal:11434 in its settings, set OLLAMA_HOST=0.0.0.0:11434 on the VM,
and keep port 11434 blocked in the cloud firewall (or use --network=host).

Save this as /usr/local/bin/idle-shutdown.sh and chmod +x it. It shuts down after 60 minutes with no SSH sessions and no Ollama requests:
#!/usr/bin/env bash
set -euo pipefail
IDLE_MIN=60
# any SSH sessions (including tunnels)?
[ "$(ss -Htn state established '( sport = :22 )' | wc -l)" -gt 0 ] && exit 0
# any Ollama API requests recently?
[ "$(journalctl -u ollama --since "${IDLE_MIN} minutes ago" | grep -c '\[GIN\]')" -gt 0 ] && exit 0
# any model loaded right now?
ollama ps | tail -n +2 | grep -q . && exit 0
logger "idle-shutdown: no activity for ${IDLE_MIN} minutes, shutting down"
/sbin/shutdown -h now
Run it every 10 minutes:  echo '*/10 * * * * root /usr/local/bin/idle-shutdown.sh' | sudo tee /etc/cron.d/idle-shutdown
Test it with a short IDLE_MIN first. What "shutdown" does on each cloud:
AWS     Stops the instance (EBS-backed default): compute billing stops, disk billing continues
GCP     Stops the instance: compute billing stops
Azure   Shuts down but does NOT deallocate, so billing continues. Deallocate from outside: az vm deallocate -g ollama-rg -n ollama-vm
Start and stop from your laptop:
aws ec2 stop-instances --instance-ids i-0abc123          aws ec2 start-instances --instance-ids i-0abc123
az vm deallocate -g ollama-rg -n ollama-vm               az vm start -g ollama-rg -n ollama-vm
gcloud compute instances stop ollama-vm                  gcloud compute instances start ollama-vm
Public IPs change on restart. Look up the new address, or use Tailscale, whose address stays the same.

- Keep the VM and just stop it (you pay only for the disk)
- Snapshot the data disk or the VM (AWS EBS snapshot or AMI, Azure disk snapshot, GCP snapshot). Snapshots cost a little per GB per month
- Store models on a separate persistent disk and reattach it to new VMs
du -sh /data/ollama        ollama list        ollama rm MODEL

curl http://localhost:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"Explain RAID levels briefly.","stream":false}' | jq -r .response
Python and OpenAI-style tools: base URL http://localhost:11434/v1 and API key "ollama". Editor plugins, Open WebUI and shell helpers use the same address.

watch -n 1 nvidia-smi                       GPU memory and utilization
htop                                        CPU and RAM (CPU machines)
ollama ps                                   Loaded models, and GPU or CPU
journalctl -u ollama -f                     Server log: requests, errors, GPU detection
df -h /data                                 Disk space
Cloud budget alerts can also warn you when a VM has been running too long.

1. Terminate or delete the VM
2. Delete leftover disks (EBS volumes, managed disks, persistent disks)
3. Release static or elastic IP addresses
4. Delete snapshots and images you no longer need
5. Delete firewall rules, security groups and load balancers you created
6. On Azure, deleting the whole resource group is easiest
7. Check the billing page the next day to confirm charges have stopped

GPU quota error or "not enough capacity"           Request a quota increase, or try another region or zone. GPUs run out of stock
nvidia-smi not found or no devices                 Install the driver (Step 3) and reboot. Check lspci | grep -i nvidia
ollama ps shows 100% CPU on a GPU machine          Driver problem: check nvidia-smi, then sudo systemctl restart ollama and read journalctl -u ollama
"model requires more system memory"                Use a smaller model or quantization, or a bigger VM
Answers are very slow                              CPU machine, or a model too big for the GPU. Use a smaller model and check ollama ps
Out of disk space while pulling                    Grow the disk, or move OLLAMA_MODELS to a bigger one
Tunnel says "connection refused"                   On the VM run systemctl status ollama and curl localhost:11434
Tunnel keeps dropping                              Use autossh, or set ServerAliveInterval 30 in ~/.ssh/config
Cannot SSH after a restart                         The public IP changed. Look up the new one, or use a static IP or Tailscale
Bill is higher than expected                       A VM, disk, static IP or snapshot was left behind. Check the cost report by resource
Spot VM disappeared                                The cloud took it back. Keep models on a persistent disk and expect interruptions
Open WebUI cannot see Ollama                       Container-to-host address problem: use host.docker.internal or host networking
Port scanned or abused                             You exposed 11434. Close it, use a tunnel or private network, and review the logs

    Set a budget alert first, then do everything else. It is the cheapest safety feature there is.

    Start on a CPU machine with a 3B to 8B model to learn the workflow. Move to a GPU only if you need the speed.

    Never expose Ollama to the internet. Use an SSH tunnel, a private network, or a proxy with a password and HTTPS.

    Put an idle auto-shutdown on every GPU machine, and remember to deallocate on Azure.

    Keep models on a separate disk you can snapshot, so a new VM is ready in minutes.

    Lock SSH to your own IP, use keys only, and keep the operating system updated.

    Use spot or preemptible instances for experiments, and regular instances for anything other people rely on.

    Compare the monthly cloud cost with a used GPU or a mini PC at home. Heavy daily use is often cheaper at home.

    Treat the VM as disposable: script the build (this guide as a shell script or an Ansible playbook) so you can recreate it in one command.

    Cloud names, prices and quotas change. Confirm details in your provider's documentation before relying on any number here.