Running a local LLM on a 2019 MacBook Pro GPU

I have a 2019 MacBook Pro with Radeon Pro Vega 20 with 4 GB of VRAM. For today’s standard it’s a small card — most local-LLM writing assumes 7B and up, and a 7B at 4-bit is already larger than this card. After the display compositor takes its reserve, about 3 GB is actually available to a model. I wanted to know what a machine like this can still do usefully, so I spent a day finding out what would run, what it costs, and whether any of it is fast enough for RAG agent work.

What failed

Ollama first, because it is one line to install. It runs, and it runs entirely on the CPU. Watching power while it generated made that plain:

sudo powermetrics --samplers gpu_power -n 3   # GPU sits at idle the whole time

This is not a misconfiguration. Ollama dispatches Metal acceleration on Apple Silicon only; on an Intel Mac it is CPU-only by design, whatever card is installed. There is no flag for it.

So I went one layer down and built llama.cpp myself. llama.cpp has a Vulkan backend, and MoltenVK translates Vulkan to Metal on macOS, so an AMD card ought to be reachable:

cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-cli -j6
./build/bin/llama-cli --list-devices

The Vega 20 shows up. Both the Vulkan backend and the Metal backend detect it correctly, memory allocates, and inference produces tokens at a reasonable rate.

The tokens are gibberish.

I worked through the obvious suspects and none of them mattered:

Tried	Result
Vulkan backend	gibberish
Metal backend (-dev MTL0)	gibberish
Flash attention off (-fa off)	gibberish
Models from 0.5B to 3B	gibberish

A configuration mistake does not survive that grid. This was a correctness bug, and the 1.4 GB build directory was dead weight from that point on.

ToshLLM, and the GPU finally doing something

ToshLLM is a GPL-3.0 macOS app that bundles a llama.cpp build patched specifically for AMD GPUs on Intel Macs. It prints the answer to the previous section in its own startup log:

ggml_metal: probed SIMD-group width = 64 (32 = Apple/AMD RDNA, 64 = AMD GCN/Vega)
ggml_metal: wave64 mode (SIMD width 64): GPU prefill matmul, CPU decode/reductions

Stock llama.cpp’s Metal kernels assume a SIMD-group width of 32, which is what Apple Silicon and AMD’s RDNA architecture use. The Vega 20 is GCN, which runs wave64 — a width of 64. Reductions across a SIMD group therefore read the wrong lanes and the numerical result is corrupted with no error raised anywhere. That is exactly the failure I had: allocation fine, tokens produced, content meaningless.

ToshLLM’s patch probes the width at runtime and routes the affected reductions to the CPU while keeping prefill matmul and quantized mat-vec on the GPU.

And the GPU is genuinely being used. Same model, same prompt, back to back:

	Generation	Prefill
Ollama (CPU only)	8.76 tok/s	49.2 tok/s
ToshLLM	14.69 tok/s	52.6 tok/s

1.7× on generation — real, but smaller than “GPU versus CPU” suggests, and prefill is nearly a tie. The reason is in that startup line: the wave64 workaround hands the decode-stage reductions back to the CPU, and decode is exactly what generation does. 14.69 tok/s is a hybrid number, not the card running alone.

The GUI is only a front-end. The bundled server can be driven directly, which is what anything scripted needs:

# GGML_METAL_VRAM_RESERVE_MB — keep 1 GB for the display compositor
# TOSH_FA_AMD                — the patched AMD attention kernel
# --no-mmap                  — 6.3x on this machine; see below
GGML_METAL_VRAM_RESERVE_MB=1024 \
TOSH_FA_AMD=1 \
/Applications/ToshLLM.app/Contents/Resources/bin/llama-server \
  -m ~/models/Qwen3-4B-Q4_K_M.gguf -ngl 99 -c 4096 -t 6 \
  --no-mmap --jinja --host 127.0.0.1 --port 8080

Note — Neither environment variable is optional. Without TOSH_FA_AMD=1 you lose the AMD attention patch; without the VRAM reserve, the display compositor and the model fight over the same 4 GB.

Then the laptop overheated

Correct output is not the same as a usable machine. I left a script benchmarking three models unattended overnight. Twenty-six minutes in, the chassis was at 87 °C, the fan sounded wrong, the laptop had stopped charging, and macOS had cut the CPU to a fifth of its clock:

pmset -g therm              # CPU_Speed_Limit = 20, CPU_Scheduler_Limit = 40
top -l 1 | grep "Load Avg"  # 119.77, on 12 logical cores
sysctl -n vm.swapusage      # climbing

The script ran the models in size order, and that order turns out to be the whole story:

Benchmark	Model	Outcome
1 of 3	Qwen3-4B, 2.3 GB	Finished in 422 s. Zero swaps.
2 of 3	GPT-OSS-20B, 11 GB, no -cmoe	Still running 17 minutes later. This is where it happened.
3 of 3	Qwen3-30B-A3B, 17 GB	Alive under a minute before I killed everything.

The model that cooked the laptop was the 20B. The model that ran fine was the one that fits in VRAM — it had finished cleanly eighteen minutes earlier.

That distinction is worth being pedantic about, because the obvious summary — “running a local LLM overheated my laptop” — is wrong in the one way that changes what you do next. Everything about it feels certain: the heat was real, the throttling was real, a local LLM was running. Only the subject of the sentence is wrong. Believing it would have meant deleting the 4B and keeping the 20B.

The give-away was in the same top output: 77.1% sys against 18.21% user. Almost all CPU time was going to the kernel moving pages, not to computing anything. Two causes, both avoidable.

llama-bench is a stress test, not a measurement tool. It drives CPU and GPU to 100% simultaneously and holds them there, repeating runs to average them. That is the correct design for a benchmark and the wrong thing to do to this chassis. The server already returns a timings field on every completion; that is where the numbers in this article come from.

Severe VRAM overcommit produces heat and nothing else. Forcing 11.2 GB into a 3 GB budget means the driver pages weights over PCIe continuously — CPU and GPU both burn power while producing almost no tokens. Mild overcommit is fine; Qwen3-8B runs 1.6 GB over the line without drama.

What I downloaded

Six GGUF files, chosen to bracket the budget from well under it to hopelessly over:

Model	Size	What it was for
Qwen3-4B-Q4_K_M	2.3 GB	Comfortably inside 3 GB — the optimistic case
gemma-3-4b-it + mmproj	3.1 GB	Same class, different family
Qwen3-8B	4.7 GB	One step over the line, to find out what that costs
gpt-oss-20b-mxfp4	11 GB	MoE — far over on paper, but most of it is experts
gpt-oss-20b-MXFP4.dflash	571 MB	Draft model for speculative decoding
Qwen3-30B-A3B-Instruct-2507	17 GB	The upper bound, to see where it breaks

Qwen3-30B answered the last question on a later, careful attempt of its own: it never finished loading in five minutes and drove swap to 2.9 GB trying. That is the shape of severe overcommit, and it is why the flags below matter more than the model list does.

The parameters that mattered

--no-mmap was the single biggest lever. With memory mapping on, weights stay as file-backed pages that are not guaranteed resident, and a short VRAM budget means re-reading them from the mapping over and over. --no-mmap reads the file once into anonymous memory, after which every upload to VRAM is clean.

Model	with mmap	with --no-mmap
Qwen3-4B	2.31 tok/s	14.54 tok/s	6.3×
Qwen3-8B	1.32 tok/s	3.27 tok/s	2.5×

The caveat is symmetrical: anonymous memory cannot be evicted cleanly, so if the model is larger than free RAM, --no-mmap forces swap and you have traded a slow problem for a worse one. Gate it on free memory.

-cmoe is mandatory for mixture-of-experts models. MoE splits each layer’s feed-forward network into many small experts and routes each token to a few. For GPT-OSS-20B that is 32 experts per layer with 4 used per token, which makes the asymmetry extreme — experts are roughly 91% of the weights but only 12% of the per-token work, while attention is 9% of the weights and runs for every token. -cmoe pins the expert weights to CPU RAM and lets only the rest onto the GPU.

GPT-OSS-20B	Generation	Stability
without -cmoe	3.95 tok/s	Metal e00002bd errors
with -cmoe	9.07 tok/s	none

VRAM demand drops from 11.2 GB to about 1 GB. This is the flag that turns the 20B from the thing that cooked the laptop into the thing that runs on it.

The rest, in descending order of how much they mattered:

• -c 4096. KV cache scales linearly with context length. ToshLLM’s GUI hardcodes -c 16384; chat and most RAG work needs nowhere near that, and dropping to 4096 cuts the cache to a quarter.
• -b 256 -ub 128. Smaller batches shrink the compute buffers, which is worth doing when the model already overcommits and negligible when it doesn’t.
• Thread count did not matter. -t 6 held CPU_Speed_Limit at 100 throughout and never throttled. My first theory was that threads were the heat source; that was simply wrong.
• Speculative decoding does not work here. GPT-OSS-20B ships a 571 MB draft model, but its dflash architecture isn’t fully implemented in the bundled engine — the loader expects 123 tensors and can construct 91. No flag reaches a failure that happens before parameters take effect.

Worth running alongside anything long:

while true; do
  SL=$(pmset -g therm | awk '/CPU_Speed_Limit/{print $3}')
  [ "${SL:-100}" -lt 80 ] && { pkill -f llama-server; break; }   # throttling: stop
  sleep 15
done

What I kept

One platform. Ollama came off the machine entirely — 17 GB of models, each with a ToshLLM equivalent that is 1.7× faster. It remains a valid CPU fallback in principle and costs one brew command to restore, but keeping strictly slower duplicates on disk was not worth the space.

Two models, with deliberately separated roles.

File	Size	Role	Generation
Qwen3-4B-Q4_K_M.gguf	2.3 GB	Daily driver. The only model that fits entirely inside the 3 GB budget.	14.7–16.3 tok/s
gpt-oss-20b-mxfp4.gguf	11 GB	Quality work with long outputs. Requires -cmoe.	9.07 tok/s

Everything between them turned out to be worthless. Qwen3-8B is both slower than the 20B and less capable than it, which is not a trade-off at all — it is just a worse option. gemma-3-4b-it was beaten in its own class by Qwen3-4B, and Qwen3-30B never got far enough to be compared with anything. Deleting all of it reclaimed 44 GB.

The two commands

Every finding above collapses into these. I ran both verbatim and checked with ps that the engine received what the line says, because that is exactly the check I had skipped when the numbers were wrong:

BIN=/Applications/ToshLLM.app/Contents/Resources/bin/llama-server

# Qwen3-4B — the daily driver. 14.98 tok/s on this run.
GGML_METAL_VRAM_RESERVE_MB=1024 TOSH_FA_AMD=1 "$BIN" \
  -m ~/models/Qwen3-4B-Q4_K_M.gguf \
  -ngl 99 -c 4096 -t 6 -b 256 -ub 128 \
  -fa auto --jinja --parallel 1 --no-mmap \
  --host 127.0.0.1 --port 8080

# GPT-OSS-20B — 8.59 tok/s. One flag differs, and it is not optional.
GGML_METAL_VRAM_RESERVE_MB=1024 TOSH_FA_AMD=1 "$BIN" \
  -m ~/models/gpt-oss-20b-mxfp4.gguf \
  -ngl 99 -cmoe -c 4096 -t 6 -b 256 -ub 128 \
  -fa auto --jinja --parallel 1 --no-mmap \
  --host 127.0.0.1 --port 8080

-cmoe is the only difference between them. An 11 GB model and a 2.3 GB model otherwise take identical flags, which is the clearest statement of what this whole exercise found: the model size is not what you tune for, the 3 GB line is.

One caveat carries over from --no-mmap — it is only correct while the model fits in free RAM. Below that, it forces swap and you have traded a slow problem for a worse one. My launcher gates on vm_stat rather than assuming.

The request side matters as much as the launch line, and differs per model:

# Qwen3 — thinking off. Left on, it will spend an entire 800-token budget
# reasoning and return empty content, with no error raised anywhere.
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","max_tokens":800,
       "chat_template_kwargs":{"enable_thinking":false},
       "messages":[{"role":"user","content":"..."}]}'

# GPT-OSS-20B — same call, but reasoning cannot be switched off, only
# shortened. Swap the one field and raise the budget to allow for it:
#   "reasoning_effort":"low"

Quality is a separate axis from speed, and it separates the two survivors less than the sizes suggest. On four scored tests covering what RAG work actually depends on — following the provided context instead of parametric memory, refusing to answer what isn’t in it, and emitting clean JSON — all three models passed, and Qwen3-4B and Qwen3-8B produced identical output. The 20B’s real cost doesn’t show in tok/s at all: it always produces a reasoning chain, and reasoning_effort: low only shortens it. Answering “1024” cost 127 tokens and 14 seconds. In an agent workflow made of many small calls, that fixed overhead dominates everything else. Qwen3 can switch it off outright with chat_template_kwargs: {enable_thinking: false}.

So: a 4 GB card from 2019 runs a 4B model at reading speed and a 20B MoE at two thirds of that. Fast enough to develop a RAG agent against, and the constraint that shaped every decision was never the parameter count. It was the 3 GB line.