CPU vs GPU Battle

A recurring experiment: take a real problem, solve it first the easy, CPU-friendly way, then bring in the GPU and deep learning — and keep an honest scorecard each round. The problem changes from post to post; the tension between the mature CPU toolchain and the raw power of the GPU stays the same.

Start Here

This is an open-ended series about the battle itself, not any single problem. Start with Part 1 to see the format in action, and expect more problems to enter the ring over time.

Questions This Series Answers

• When is the GPU actually worth it, and when does the mature CPU toolchain win?
• How have coding agents changed the economics of GPU optimization?
• What does an honest, on-device scorecard look like for a real problem?
• Where does classical, CPU-friendly engineering still beat deep learning?

Key Themes

• CPU versus GPU tradeoffs
• On-device machine learning
• Classical methods versus deep learning
• Coding agents and the new cost of optimization
• Honest benchmarking

Why this battle is worth watching

In graduate school I once spent hours — days, really — hand-optimizing a small GPU program. Tiling, memory coalescing, occupancy, the whole ritual, chasing a kernel that would finally saturate the hardware. It was painstaking, specialist work, and getting it right felt like an achievement in itself.

That kind of optimization is exactly what coding agents now absorb. The tuning that used to eat a week gets handled, and we get to harvest the benefit instead of grinding for it. Reaching for the GPU has never been cheaper.

But here is the twist that makes the fight interesting: CPUs have been around far longer, and the compilers, libraries, and habits of a whole generation of engineers have a much deeper grasp of the CPU than the GPU. So on any given problem it is genuinely unclear which corner wins — the mature, well-understood CPU stack that runs anywhere, or the raw, newly-accessible power of the GPU.

This series runs that battle one real problem at a time. Each round: take a problem, solve it first the easy CPU-friendly way, then bring in the GPU and deep learning — and keep an honest scorecard, wins and losses. The problem changes from post to post; the tension does not.

Articles in This Series

1. CPU vs GPU battle, Round 1: the agents brought the GPU

   How well does a small prompted LLM grade e-commerce search relevance on a 0-3 scale, entirely on a laptop? We start the CPU-vs-GPU battle with the two easy arms - BM25 and a prompted local model - and find the honest surprise is not accuracy but calibration and circularity.

2. CPU vs GPU battle, Round 1: train the judge, don't prompt it

   In Part 1 we prompted a general model to grade e-commerce search relevance and watched it miscalibrate. This round we stop prompting and train a specialist - a bge-reranker cross-encoder fine-tuned on Amazon ESCI, on the laptop’s own Apple-Metal GPU. It becomes the first arm to beat the shipped on-device judge on both test sets (ESCI QWK 0.360, WANDS 0.299) at ~28 ms/pair in ~1 GB of RAM.

3. CPU vs GPU battle, Round 1: adapt the generalist, read the logits

   Part 2 trained a specialist cross-encoder that won in-domain but leaned on its training data. This round we take the generative Llama-3.2-3B, LoRA-tune it on the laptop GPU with MLX, and grade by reading the label-token logits instead of prompting for JSON. In-domain it lands about even with the specialist (ESCI QWK 0.353), but it is the only arm that does not degrade out of domain - WANDS QWK 0.486, up from its own ESCI, while every other arm falls.

4. CPU vs GPU battle, Round 1: leave the laptop, the ceiling comes with you

   Three rounds of this experiment lived on a 16 GB laptop and nothing broke 0.5 QWK - a lexical floor at 0.222, and every learned or prompted arm crowded between 0.24 and 0.49. So we left the machine: the same prompt, the same parser, the same 3,000 held-out pairs, handed to a 7B judge on a datacenter GPU. It scored ESCI QWK 0.361 and WANDS 0.354 - better than the 3B it scaled up from, and still inside the same band. The ceiling is the task, not the model size. The result is now a poster at SCD 2026.