A timesaver: find out which language models fit your machine before you download them, and what a bigger GPU would change, with every command run for real.
▶Prefer to watch? This short video explains what llmfit is for, then runs it on a real Ubuntu 24.04 machine and simulates a bigger GPU, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
llmfit: check your hardware, then rank language models by how well they would run on it.
llmfit the terminal screen: every model, scored for this machine
llmfit system what it detected: CPU, RAM, GPU
llmfit fit --json -n 5 the same ranking as text for scripts
S (in the screen) simulate other hardware, such as a 24 GB GPU
llmfit is a program that reads your CPU, RAM and GPU, then goes through a built-in list of thousands of open language models and scores each one for fit, estimated speed and quality on that hardware. It tells you whether a model fits in memory, how it would run (on the GPU, split between GPU and RAM, or on the CPU alone), which quantization to choose, and a rough number of tokens per second. It is for deciding before you download a 20 GB file that would crawl, or before you buy a GPU.
This page is the written version of the Timesavers video. Every command below was run for real, in an Ubuntu 24.04 container with llmfit 1.1.17. The container ran on my laptop, and llmfit saw the laptop's real CPU and memory: an AMD Ryzen 5 3500U with 31 GB of RAM and an integrated GPU with almost no video memory of its own. No language model was downloaded or run, so every speed on this page is llmfit's estimate, not a measurement. The section "What was not run" lists what was not tried.
What llmfit is for
Use llmfit when the question is "which model should I even try on this machine?":
will this model fit? it compares the memory a model needs with your RAM and GPU memory
how will it run? GPU only, split between GPU and RAM, or CPU only, and the best quantization
how fast, roughly? an estimated number of tokens per second, with an error band
what if I had more? simulate another RAM or GPU size and the whole list is scored again
what hardware does this need? plan mode turns the question around
It does not run a model for you by default, and its speeds are estimates (the project's own output gives a band of about plus or minus 30 percent, and says the estimate is rough when your GPU is not in its bandwidth table). The list of models is large and made by the community, so many entries are fine-tunes of the well-known ones. It can also download model files and start them with llama.cpp (the help lists download, run, serve and bench), which was not tried here.
Install
The project's README gives several routes (Homebrew, Scoop, MacPorts, uv or pip, and an install script). The script is about 190 lines, so read it before you run it. What it does: it finds the latest release on GitHub, downloads the binary for your platform, downloads a checksum file and verifies it (if there is no checksum file it only prints a warning and continues), and installs the binary into /usr/local/bin, or into ~/.local/bin with --local. In the lab:
$ curl -fsSL https://llmfit.axjns.dev/install.sh -o install.sh
$ sh install.sh --local
> llmfit installer
> Downloading llmfit v1.1.17 for x86_64-unknown-linux-musl...
> Verifying checksum...
> Extracting...
> Installing to /home/ubuntu/.local/bin (--local mode)...
> Installed llmfit to /home/ubuntu/.local/bin/llmfit
> Done. Run 'llmfit' to get started.
$ llmfit --version
llmfit 1.1.17
On Ubuntu a new login shell finds ~/.local/bin by itself when the directory exists.
What it detected
llmfit system prints what llmfit sees. Everything it does is scored against this:
$ llmfit system
=== System Specifications ===
CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 cores)
Total RAM: 31.03 GB
Available RAM: 23.88 GB
RAM Bandwidth: ~26 GB/s (measured)
Backend: Vulkan
GPU: AMD GPU (0.25 GB VRAM, 0.03 GB free, Vulkan)
The CPU and RAM are the laptop's, not the container's, because the container had no limits. The GPU is the processor's built-in graphics, found through Vulkan, with a quarter of a gigabyte of video memory of its own. So nearly every model will run split between the GPU and system RAM, or on the CPU, and that shows in the screens below. The RAM bandwidth line is measured when it starts, so it changes a little from run to run (about 25 to 27 GB/s here).
The main screen
llmfit with no arguments opens the terminal screen. The top box is the hardware, with the runtimes it looked for (Ollama, MLX, llama.cpp, Docker, LM Studio, vLLM, none found here). Below it a row of search and filter boxes, then the table of models sorted by score. The columns are the model, its provider, the number of parameters, the score, the estimated tokens per second (tok/s*), the best quantization for this machine, the size on disk, the run mode, the share of memory it would use, the context length, the release date, the fit and the use case:
$ llmfit
┌ llmfit ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 cores) │ RAM: 23.7 GB avail / 31.0 GB total │ GPU: AMD GPU (0.2 GB, Vul│
│ Ollama: ✗ │ MLX: ✗ │ llama.cpp: ✗ (not in PATH, set LLAMA_CPP_PATH) │ Docker: ✗ │ LM Studio: ✗ │ vLLM: ✗ │ RamaLama: │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌ Search ────────────────────┐┌ Providers ┐┌ Use Case (┐┌ Caps (C) ┐┌ Sort [s] ─┐┌ Fit[f] Fil┐┌ Avail [a] ┐┌ TP [T] ──┐┌ Theme [t] ┐
│Press / to search... ││ All ││ All ││ All ││ Score ↓ ││All ││All ││All ││ Default │
└────────────────────────────┘└───────────┘└───────────┘└──────────┘└───────────┘└───────────┘└───────────┘└──────────┘└───────────┘
┌ Models (10421/10423) ────────────────────────────────────────────────────────────────────────────────────────────────────────────↑
│ In Model Provider Params Score▼ tok/s* Quant Disk Mode Mem % Ctx Date Fit Use Case █
│▶ ● — n3-8B-NVFP4-W4A16-P predoctor 4.7B 87 8.2 Q8_0 8.4G CPU+GP 42% 131k→1 2026-0 Good Reasoning ║
│ ● — mconcat/Qwen3.5-27… mconcat 22.1B 86 3.5 Q6_K 17.7G CPU+GP 79% 262k→9 2026-0 Good Reasoning ║
│ ● — mconcat/Qwen3.5-27… mconcat 22.1B 86 3.5 Q6_K 17.7G CPU+GP 79% 262k→9 2026-0 Good Reasoning ║
│ ● — second-state/DeepS… second-state 12.6B 86 5.2 Q8_0 13.2G CPU+GP 64% 131k→5 — Good Reasoning ║
│ ● — Alexzander85/Qwen3… alexzander85 7.5B 85 8.8 Q8_0 7.9G CPU+GP 37% 262k 2026-0 Good Reasoning ║
│ ● — deepseek-ai/DeepSe… DeepSeek 8.2B 85 8.1 Q8_0 8.6G CPU+GP 43% 131k→1 2025-0 Good Reasoning ║
│ ● — heseltiNe/barrier-… heseltine 8.2B 85 8.1 Q8_0 8.6G CPU+GP 43% 131k→1 2025-0 Good Reasoning ║
│ ● — mlx-community/Deep… mlx-communit 8.2B 85 8.1 Q8_0 8.6G CPU+GP 43% 131k→1 2025-0 Good Reasoning ║
│ ● — reasonwang/SkillGy… reasonwang 9.4B 85 7.0 Q8_0 9.9G CPU+GP 45% 262k 2026-0 Good Reasoning ║
│ ● — unsloth/DeepSeek-R… unsloth 8.4B 85 7.9 Q8_0 8.8G CPU+GP 44% 524k→1 — Good Reasoning ║
│ ● — deepseek-ai/DeepSe… DeepSeek 14.8B 85 4.5 Q8_0 15.5G CPU+GP 74% 131k→4 2025-0 Good Reasoning ║
│ ● — huihui-ai/DeepSeek… huihui-ai 14.8B 85 4.5 Q8_0 15.5G CPU+GP 74% 131k→4 2025-0 Good Reasoning ║
│ ● — saidutta69/DeepSee… saidutta69 14.8B 85 4.5 Q8_0 15.5G CPU+GP 74% 131k→4 — Good Reasoning ║
│ ● — unsloth/DeepSeek-R… unsloth 15.1B 85 4.4 Q8_0 15.8G CPU+GP 75% 131k→4 2025-0 Good Reasoning ║
│ ● — TeichAI/Qwen3-14B-… teichai 12.1B 84 5.5 Q8_0 12.7G CPU+GP 61% 40k 2025-1 Good Reasoning ║
The mark at the start of the table title, Models (10421/10423), is how many models match the filters out of how many are in the list. At the bottom, a status bar names the mode and the main keys. The speed column is marked with a star because it is an estimate. Here the top rows are small reasoning models that run split between CPU and GPU ("CPU+GP") at about 8 tokens per second, and the Fit column says Good for all of them.
Search, filter and sort
/ opens the search box. It matches part of the name, the provider, the size or the use case. Typing Qwen/Qwen3 and pressing Enter (which leaves the search box and keeps the filter) leaves 82 of the 10423 models, the official Qwen family from Alibaba:
keys: / Qwen/Qwen3 Enter
┌ llmfit ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 cores) │ RAM: 23.7 GB avail / 31.0 GB total │ GPU: AMD GPU (0.2 GB, Vul│
│ Ollama: ✗ │ MLX: ✗ │ llama.cpp: ✗ (not in PATH, set LLAMA_CPP_PATH) │ Docker: ✗ │ LM Studio: ✗ │ vLLM: ✗ │ RamaLama: │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌ Search ────────────────────┐┌ Providers ┐┌ Use Case (┐┌ Caps (C) ┐┌ Sort [s] ─┐┌ Fit[f] Fil┐┌ Avail [a] ┐┌ TP [T] ──┐┌ Theme [t] ┐
│Qwen/Qwen3 ││ All ││ All ││ All ││ Score ↓ ││All ││All ││All ││ Default │
└────────────────────────────┘└───────────┘└───────────┘└──────────┘└───────────┘└───────────┘└───────────┘└──────────┘└───────────┘
┌ Models (82/10423) ───────────────────────────────────────────────────────────────────────────────────────────────────────────────↑
│ In Model Provider Params Score▼ tok/s* Quant Disk Mode Mem % Ctx Date Fit Use Case █
│▶ ● — Qwen/Qwen3.5-9B Alibaba 9.7B 79 6.8 Q8_0 10.1G CPU+GP 46% 262k 2026-0 Good Multimodal█
│ ● — Qwen/Qwen3.6-35B-A… Alibaba 36.0B 79 31.6 Q4_K_M 20.9G CPU+GP 85% 262k→1 2026-0 Good Multimodal█
│ ● — Qwen/Qwen3-Coder-3… Alibaba 30.5B 79 24.7 Q5_K_M 20.8G CPU+GP 72% 262k→2 2025-0 Good Coding ║
│ ● — Qwen/Qwen3-Coder-3… Alibaba 30.5B 79 24.7 Q5_K_M 20.8G CPU+GP 72% 262k→2 2025-0 Good Coding ║
│ ● — Qwen/Qwen3-VL-8B-I… Alibaba 8.8B 78 7.5 Q8_0 9.2G CPU+GP 46% 262k→1 2025-1 Good Multimodal║
Other keys, from the project's docs: f cycles the fit filter (all, runnable, perfect, good, marginal), a cycles the availability filter, s cycles the sort column, and the capital letters P, U, C, L and R open popups for provider, use case, capability, license and runtime. Ctrl-U clears the search. h shows every key. llmfit remembers the search, the filters and the sort for next time in ~/.config/llmfit/filters.json, so a screen that opens already filtered is not a bug: delete the file or press Ctrl-U to start clean.
Details for one model
Enter on a row opens the detail view. It lists the model, its size, the best quantization for this hardware, the context length, the license, the use case, and on the right the notes and, when there are some, suggested GGUF files to download:
keys: Enter
│ ││ │
│ Model: Qwen/Qwen3.5-9B ││ ── GGUF Downloads ── │
│ Provider: Alibaba ││ │
│ Parameters: 9.7B ││ 📦 unsloth hf.co/unsloth/Qwen3.5-9B-GGUF │
│ Quantization: Q4_K_M ││ 📦 mradermacherhf.co/mradermacher/Qwen3.5-9B-GGUF │
│ Best Quant: Q8_0 (for this hardware) ││ │
│ Context: 262144 tokens ││ llmfit download \ │
│ Use Case: Multimodal, vision and text ││ unsloth/Qwen3.5-9B-GGUF \ │
│ Category: Multimodal ││ --quant Q8_0 │
│ Capabilities: Vision, Tool Use ││ │
│ Released: 2026-05-15 ││ ── Notes ── │
│ License: apache-2.0 ││ │
│ Runtime: llama.cpp (baseline est. ~6.8 tok/s) ││ Context capped at 8192 tokens for estimation (model │
│ Installed: - No runtime detected ││supports up to 262144; use --max-context to override) │
│ ││ GPU: insufficient VRAM, spilling to system RAM │
│ ── Score Breakdown ── ││ Performance will be significantly reduced │
│ ││ Best quantization for hardware: Q8_0 (model default: │
The same information is available without the screen:
$ llmfit info "Qwen/Qwen3.5-9B"
=== Qwen/Qwen3.5-9B ===
Provider: Alibaba
Parameters: 9.7B
Quantization: Q4_K_M
Best Quant: Q8_0
Context Length: 262144 tokens
Use Case: Multimodal, vision and text
Category: Multimodal
Released: 2026-05-15
License: apache-2.0
Runtime: llama.cpp (baseline est. ~6.8 tok/s)
Score Breakdown:
Overall Score: 79.4 / 100
Quality: 92 Speed: 17 Fit: 100 Context: 100
Baseline Est. Speed: 6.8 tok/s
Estimate Basis:
Confidence: estimated
Method: per-backend heuristic constant — GPU not in the bandwidth table,
so expect a wide error band. Please report your GPU model so we
can add real bandwidth data (github.com/AlexsJones/llmfit/issues).
Models single-request generation at ctx <= 8192 tokens.
Prompt processing (prefill/TTFT) is not estimated for this model/hardware.
Baseline error band is roughly +/-30%.
Verify on this machine:
llmfit bench "Qwen/Qwen3.5-9B" (against a running provider)
Resource Requirements:
Min VRAM: 4.9 GB
Min RAM: 5.4 GB (CPU inference)
Recommended RAM: 9.0 GB
Disk (est): 10.1 GB (at Q8_0)
Disk/quant: Q8_0: 10.1G Q6_K: 7.7G Q5_K_M: 6.6G Q4_K_M: 5.6G Q3_K_M: 4.6G Q2_K: 3.6G
Fit Analysis:
Status: 🟡 Good
Run Mode: CPU+GPU
Memory Utilization: 45.9% (10.9 / 23.7 GB)
GGUF Downloads:
unsloth → https://huggingface.co/unsloth/Qwen3.5-9B-GGUF
mradermacher → https://huggingface.co/mradermacher/Qwen3.5-9B-GGUF
Read the Estimate Basis part. "Method: per-backend heuristic constant" and "Baseline error band is roughly +/-30%" are the project saying that the speed is a guess for this GPU. It says llmfit bench can check it against a running provider, which was not run.
Compare two models
m marks the selected row as the baseline. Move to another row with j and k, then c opens the side-by-side view, with the differences from the baseline and an arrow for the direction. q closes it. Here the 9B model against a 30B coder model that mixes experts, with both from Alibaba's Qwen family:
keys: m j j c
│ Compare Qwen/Qwen3.5-9B vs Qwen/Qwen3-Coder-30B-A3B-Instruct │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌ Marked (baseline) ─────────────────────────────────────────────┐┌ Selected (delta vs baseline) ──────────────────────────────────┐
│ ││ │
│ Model: Qwen/Qwen3.5-9B ││ Model: Qwen/Qwen3-Coder-30B-A3B-Instruct │
│ Provider: Alibaba ││ Provider: Alibaba │
│ Use: Multimodal ││ Use: Coding │
│ Released: 2026-05-15 ││ Released: 2025-07-31 │
│ License: apache-2.0 ││ License: apache-2.0 │
│ Score: 79.4 ││ Score: 78.9 (-0.5) ↓ │
│ Fit: ● Good ││ Fit: ● Good │
│ tok/s: 6.8 ││ tok/s: 24.7 (+17.9) ↑ │
│ Mem%: 45.9% ││ Mem%: 72.1% (+26.2%) ↑ │
│ Disk: 10.1 GB ││ Disk: 20.8 GB │
│ Runtime: llama.cpp ││ Runtime: llama.cpp │
│ Mode: CPU+GPU ││ Mode: CPU+GPU │
│ Params: 9.7B ││ Params: 30.5B (+20.88B) ↑ │
└────────────────────────────────────────────────────────────────┘└────────────────────────────────────────────────────────────────┘
The bigger model has a similar score and a faster estimate (24.7 against 6.8 tok/s), (the A3B in its name means that only about 3 billion parameters are active at a time), but it takes 72% of memory against 46% and 20.8 GB of disk. The same comparison is available from the command line:
$ llmfit diff "Qwen/Qwen3.5-9B" "Qwen/Qwen3.5-4B"
=== Model Diff ===
Comparing 2 model(s) (sorted by Score)
Metric M1: Qwen/Qwen3.5-9B M2: Qwen/Qwen3.5-4B
-------------------- -------------------------------- --------------------------------
Score 79.4 75.6 (-3.8)
Baseline tok/s 6.8 14.2 (+7.3)
Fit 🟡 Good 🟡 Good
Run Mode CPU+GPU CPU+GPU
Runtime llama.cpp llama.cpp
Memory % 46.1% 23.9% (-22.2)
Params 9.7B 4.7B
Context 262144 tokens 262144 tokens
Best Quant Q8_0 Q8_0
Provider Alibaba Alibaba
Plan the hardware
The usual question is "what fits my machine?". Plan mode asks the other one: "what machine does this model need?". p opens it for the selected row, with a context length, a quantization and a target speed that you can edit with Tab and the keyboard, and Esc closes it:
keys: p (then Esc)
│ │
│ Model: Qwen/Qwen3-Coder-30B-A3B-Instruct │
│ Note: Estimate-based using current llmfit fit/speed heuristics. │
│ │
│ Inputs (editable) │
│ Context: 8192 tokens │
│ Quant: Q4_K_M │
│ KV Quant: <fp16> (fp16, fp8, q8_0, q4_0, tq) │
│ Target TPS: <none> tok/s │
│ │
│ Active KV: fp16 │
│ │
│ Minimum Hardware │
│ VRAM: 19.0 GB RAM: 8.0 GB CPU: 4 cores │
│ │
│ Recommended Hardware │
│ VRAM: 28.4 GB RAM: 12.0 GB CPU: 8 cores │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
On the command line, with the context length given:
$ llmfit plan "Qwen/Qwen3.5-9B" --context 8192
=== Hardware Planning Estimate ===
Model: Qwen/Qwen3.5-9B
Provider: Alibaba
Context: 8192
Quantization: Q4_K_M
Disk (est): 5.60 GB (weights only)
KV cache: fp16
Note: Estimate-based output using current llmfit fit/speed heuristics; not an exact benchmark.
Minimum Hardware:
VRAM: 6.3 GB
RAM: 8.0 GB
CPU Cores: 4
Recommended Hardware:
VRAM: 9.0 GB
RAM: 12.0 GB
CPU Cores: 8
Feasible Run Paths:
GPU: Yes
min: VRAM=6.3 GB RAM=8.0 GB cores=4
est speed: 17.9 tok/s
CPU offload: Yes
min: VRAM=2.0 GB RAM=6.3 GB cores=4
est speed: 8.9 tok/s
CPU-only: Yes
min: VRAM=n/a RAM=6.3 GB cores=4
est speed: 2.5 tok/s
Upgrade Deltas:
+6.1 GB VRAM -> Good
+8.8 GB VRAM -> Perfect
KV Cache Alternatives:
kv kv (GB) total savings notes
fp16 0.25 6.35 -
fp8 0.12 6.22 -50% vLLM and llama.cpp builds with fp8 KV support
q8_0 0.12 6.22 -50% llama.cpp --cache-type-k q8_0 --cache-type-v q8_0
q4_0 0.06 6.16 -75% llama.cpp --cache-type-k q4_0 --cache-type-v q4_0 (quality drop)
tq (n/a) 0.04 6.14 -83% Experimental: not in upstream vLLM, see 0xSero/turboquant; applies to 8 full-attention layers; 24 recurrent layers use fixed state covered by runtime overhead; requires vLLM + CUDA
The three run paths show the trade: the GPU path is the fastest and needs 6.3 GB of video memory, offloading to RAM needs far less video memory and is half as fast, and the CPU alone is the slowest of the three. These are estimates.
What if I had a bigger GPU?
S opens the hardware simulation. It shows the RAM, the video memory and the CPU cores, and you can type other values. The field keeps its old text, so press Ctrl-U to clear it before typing (in the lab, typing 24 straight after the 0.2 gave 0.224). Tab moves to the next field, Enter applies, Ctrl-R puts the real hardware back:
keys: S Tab Ctrl-U 24 Enter
┌ llmfit ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ SIM CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx (8 cores) │ RAM: 27.9 GB avail / 31.0 GB total │ GPU: AMD GPU (24.0 G│
│ Ollama: ✗ │ MLX: ✗ │ llama.cpp: ✗ (not in PATH, set LLAMA_CPP_PATH) │ Docker: ✗ │ LM Studio: ✗ │ vLLM: ✗ │ RamaLama: │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌ Search ────────────────────┐┌ Providers ┐┌ Use Case (┐┌ Caps (C) ┐┌ Sort [s] ─┐┌ Fit[f] Fil┐┌ Avail [a] ┐┌ TP [T] ──┐┌ Theme [t] ┐
│Qwen/Qwen3 ││ All ││ All ││ All ││ Score ↓ ││All ││All ││All ││ Default │
└────────────────────────────┘└───────────┘└───────────┘└──────────┘└───────────┘└───────────┘└───────────┘└──────────┘└───────────┘
┌ Models (82/10423) ───────────────────────────────────────────────────────────────────────────────────────────────────────────────↑
│ In Model Provider Params Score▼ tok/s* Quant Disk Mode Mem % Ctx Date Fit Use Case █
│▶ ● — truct-2507 Qwen/Q Alibaba 30.5B 87 49.4 Q5_K_M 20.8G GPU 65% 262k→2 2025-0 Good Chat █
│ ● — Qwen/Qwen3-30B-A3B… Alibaba 30.5B 87 49.4 Q5_K_M 20.8G GPU 65% 262k→2 2025-0 Good Chat █
│ ● — Qwen/Qwen3-Coder-3… Alibaba 30.5B 87 49.4 Q5_K_M 20.8G GPU 65% 262k→2 2025-0 Good Coding ║
│ ● — Qwen/Qwen3-Coder-3… Alibaba 30.5B 87 49.4 Q5_K_M 20.8G GPU 65% 262k→2 2025-0 Good Coding ║
│ ● — Qwen/Qwen3.6-35B-A… Alibaba 36.0B 86 63.2 Q4_K_M 20.9G GPU 77% 262k→1 2026-0 Good Multimodal║
│ ● — Qwen/Qwen3-VL-30B-… Alibaba 31.1B 86 48.6 Q5_K_M 21.1G GPU 66% 262k→2 2025-0 Good Multimodal║
│ ● — Qwen/Qwen3-VL-30B-… Alibaba 31.1B 86 48.5 Q5_K_M 21.1G GPU 66% 262k→2 2025-1 Good Multimodal║
The mark SIM at the top shows the numbers are not real. The same models now run on the GPU alone (Mode GPU) and the estimates jump, for example Qwen/Qwen3-30B-A3B-Instruct-2507 at 49.4 tokens per second, where the real machine was estimated at 24.7 for Qwen/Qwen3-Coder-30B-A3B-Instruct, a model of the same size. On the command line the same thing is the global option --memory, which overrides the video memory:
$ llmfit fit --json -n 3 --providers Alibaba | jq -r '.models[] | "\(.name) \(.run_mode) \(.estimated_tps) tok/s"'
Qwen/Qwen2.5-1.5B-Instruct CPU+GPU 42.8 tok/s
Qwen/Qwen2.5-Math-1.5B-Instruct CPU+GPU 42.8 tok/s
Qwen/Qwen1.5-MoE-A2.7B-Chat CPU+GPU 40.7 tok/s
$ llmfit --memory 24G fit --json -n 3 --providers Alibaba | jq -r '.models[] | "\(.name) \(.run_mode) \(.estimated_tps) tok/s"'
Qwen/Qwen3-30B-A3B-Instruct-2507 GPU 49.4 tok/s
Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 GPU 49.4 tok/s
Qwen/Qwen2.5-Coder-3B GPU 42.8 tok/s
--ram and --cpu-cores do the same for the other two numbers. --providers limits the list to one provider (Alibaba here), which keeps the community fine-tunes out of the way. This is the quickest way to see what you would gain from more video memory before you buy it. It only changes the arithmetic: the numbers are still estimates.
Hardware profiles
A profile is a stored description of a machine. llmfit ships three: an Apple M3 Max with 128 GB, an RTX 4090 PC with 64 GB and an AMD Ryzen AI Max with 128 GB. List them and look inside one:
$ llmfit hardware list
=== Hardware profiles ===
NAME RAM UNIFIED GPU BW FP16 TFLOP/S SOURCE
apple-m3-max-128gb 128 GB yes 400 GB/s - bundled
nvidia-rtx-4090 64 GB no 1008 GB/s 165.2 bundled
ryzen-ai-max-plus-395 128 GB yes 256 GB/s 29.7 bundled
Score models against one with: llmfit --profile <NAME> fit
$ llmfit hardware show nvidia-rtx-4090
=== Hardware profile: nvidia-rtx-4090 ===
Source: bundled
Schema version: 1
Describes GPU: RTX 4090 (provenance only — profiles are never auto-selected)
Hardware
Total RAM: 64.00 GB
Unified memory: no
GPU bandwidth: 1008 GB/s
DDR bandwidth: 90 GB/s
GPU fp16 throughput: 165.2 TFLOP/s
On this machine
RAM: 64.00 GB
VRAM pool: 0.25 GB
GPU bandwidth used: 1008 GB/s
Note: profiles carry no VRAM figure, so VRAM stays as detected here.
--profile scores the models against a profile instead of this machine. Read the last line of the output above: a profile carries RAM and speed figures but no video memory, so the video memory stays as detected, and in the lab the models still showed the CPU+GPU mode:
$ llmfit --profile nvidia-rtx-4090 fit --json -n 4 --providers Alibaba | jq -r '.models[] | "\(.name) \(.run_mode) \(.estimated_tps) tok/s"'
Qwen/Qwen2-57B-A14B-Instruct CPU+GPU 38.2 tok/s
Qwen/Qwen3.6-35B-A3B CPU+GPU 92.4 tok/s
Qwen/Qwen2-57B-A14B CPU+GPU 38.2 tok/s
Qwen/Qwen3.5-35B-A3B CPU+GPU 92.4 tok/s
--profile and --memory cannot be used together (the lab said so):
$ llmfit --profile nvidia-rtx-4090 --memory 24G fit -n 1
error: the argument '--profile <NAME|PATH>' cannot be used with '--memory <SIZE>'
Usage: llmfit --profile <NAME|PATH>
So use --memory with the real machine to try a video memory size, and --profile to try the memory bandwidth and RAM of another machine.
Use it in scripts
--json is a global flag and works on every command, and the text output uses the same data. The commands of this version, from llmfit --help:
Commands:
system Show system hardware specifications
hardware List, inspect, and validate hardware profiles
doctor Print a hardware diagnostic report for bug reports
claim Generate a Kubernetes DRA ResourceClaim encoding the model's fit
list List all available LLM models
fit Find models that fit your system (classic table output)
search Search for specific models
info Show detailed information about a specific model
diff Compare two models side-by-side, or auto-compare top N filtered models
concurrency Estimate concurrent-session capacity for a model (issue #140)
plan Plan hardware requirements for a specific model configuration
storage Estimate SSD capacity for a library of runnable models
recommend Recommend top models for your hardware (JSON-friendly)
download Download a GGUF model from HuggingFace for use with llama.cpp
hf-search Search HuggingFace for GGUF models compatible with llama.cpp
update Fetch the latest LLM models from HuggingFace and update the local cache
run Run a downloaded GGUF model with llama-cli or llama-server
serve Start llmfit REST API server for cluster/node scheduling workflows
bench Benchmark inference performance against running providers
help Print this message or the help of the given subcommand(s)
The flags before a command:
GLOBAL FLAGS:
--json Output structured JSON on every subcommand (for tool/agent
integration). Always exits 0 on success, 1 on error.
--memory <SIZE> Override GPU VRAM (e.g. "32G", "32000M", "1.5T").
--memory-percent <PERCENT>
Use this percentage of detected GPU VRAM (e.g. 90, 87.5).
--ram <SIZE> Override system RAM (e.g. "64G", "128000M").
--ram-percent <PERCENT>
Use this percentage of detected system RAM.
--cpu-cores <N> Override detected CPU core count.
--profile <NAME> Score against a whole hardware profile (name or path to a
profile JSON) instead of this machine. Sets capacity,
unified memory, and the bandwidth/compute figures the
throughput estimate needs. Conflicts with --memory,
--memory-percent, --ram, --ram-percent, and --cpu-cores.
See `llmfit hardware list`.
--llama-cpp-path <PATH>
Directory containing llama.cpp binaries. Overrides
LLAMA_CPP_PATH for this invocation when the directory exists.
--max-context N Cap context length for memory estimation (tokens).
Falls back to OLLAMA_CONTEXT_LENGTH env var if unset.
EXIT CODES:
recommend gives a short list for a use case (general, coding, reasoning, chat, multimodal or embedding). It was run for coding on this machine:
$ llmfit recommend --use-case coding -n 3 --json | jq -r '.models[] | "\(.name) \(.estimated_tps) tok/s \(.fit_level)"'
OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k 53.4 tok/s Good
NousResearch/NousCoder-14B 4.5 tok/s Good
ajayk007/Qwen2.5-Coder-1.5B-Shellsmith 42.8 tok/s Good
Notice the first result: a one-billion-parameter fine-tune that nobody has heard of, at the top because it is small and fast on this machine and scores well on fit. The score rewards fit and speed as well as quality, so a small model can beat a better big one. Look at the use case and the provider, not only the score, and use --providers or the search to look at models you know.
What was not run in the lab
No language model was downloaded, started or benchmarked, so llmfit download, run, serve and bench were not run, and no speed on this page is a measurement. These were not run either: update (it fetches a newer model list from HuggingFace), hf-search, concurrency, storage, claim (a Kubernetes resource claim), doctor, the web dashboard, the benchmark and leaderboard views (b and I in the screen), the download manager (D), the advanced configuration popup (A), themes (t), visual and select modes (v and V), the mouse, and the filters for provider, license, capability and runtime other than --providers. The hardware was one laptop with an integrated GPU, so what a discrete NVIDIA, AMD or Apple GPU looks like was only simulated, never detected. The measured RAM bandwidth changes a little on each start. The model list and the scores change as the project updates its data, so the numbers here will not match yours. Everything ran on Ubuntu 24.04 with llmfit 1.1.17.
Common problems and tips
The screen opens with a filter I did not set. llmfit saves the search, filters and sort in ~/.config/llmfit/filters.json. Ctrl-U clears the search, and deleting the file starts from nothing.
My GPU is not detected, or the memory is wrong. Check llmfit system first. The help says to use --memory to override the video memory when detection fails.
The speed is far from what I get. It is an estimate for your backend, with a band of about plus or minus 30 percent, and it can be wider for a GPU that is not in the bandwidth table. llmfit bench measures a running model (not run here).
I typed a number in the simulation and got 0.224. The field keeps its text. Press Ctrl-U before typing.
--profile and --memory together give an error. They cannot be combined. Use one or the other.
The best-scored model is one I do not know. The list is large and community made, and the score rewards fit and speed. Search for the family you want (Qwen/Qwen3, for example) or limit it with --providers.
Tips. Run llmfit system first so you know what it sees. Use S or --memory to see what more video memory would change before you buy anything. Treat the speed column as a way to compare models on the same machine, not as a promise. And remember that this tells you what should fit; only running the model tells you how it feels.
