What Local AI Is Actually Good At in 2026, and What It Still Fumbles

Someone read my post about local models catching up and emailed me one sensible question: caught up at what?

Fair question. And it's the one nobody answers, because "the gap is four months" is a headline and "here is the task list" is work.

So here is the task list. Not vibes, not a leaderboard screenshot. A column of things I hand to a model running on my own hardware without thinking about it, a column of things that still go to a frontier API, and the evidence for why each item sits where it does. The useful version of local LLM use cases is not a list of ideas. It's a sorting rule.

The Column That Works Is Bigger Than You Think

Start with speech, because it's the cleanest case. On the Open ASR Leaderboard, NVIDIA's Parakeet TDT 0.6B v3 averages 6.34% word error rate across the eight benchmark sets. Whisper large-v3 averages roughly 7.4%. The better model is the smaller one, it's a 0.6 billion parameter file, and it runs on a laptop. There is no version of this task where paying an API is the technically superior choice. Transcription left the cloud and nobody sent a memo.

Embeddings are the same story. The Qwen3-Embedding family took the number one spot on the MTEB multilingual leaderboard at 70.58, and a January 2026 English snapshot still had it first at 70.6. Google's EmbeddingGemma is 308 million parameters and ranks 13th of 93, within 0.033 of the leader. Your semantic search, your RAG retrieval, your "find me the similar support ticket" feature: that is a solved local problem, and it has been for a while.

Then the bulk text work. Classification and routing. Structured extraction from invoices and contracts and emails. Summarisation. Tone rewriting. Translation for internal consumption where nobody's reputation rides on the phrasing. Code completion on patterns the model has seen a thousand times. These are the tasks where a 4B or 8B model does the job and the failure rate is low enough to design around.

What these have in common is worth naming, because it's the actual rule. Everything the model needs is in the prompt, and there is exactly one hop from input to output. No world knowledge it has to already possess. No chain of decisions where step four depends on step three going right. Give it the document, ask for the fields, get the fields.

That covers, honestly, most of the AI work I actually build for clients. Not the demo work. The volume work.

The Column That Still Fumbles

The other column starts where the chain gets long.

I described this in the previous post: thirty tool calls into a real codebase, the local model started confidently editing a file that wasn't in the project. That's not a one-off. It's the documented shape of the problem. The research paper title I keep coming back to is Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models, which found that renaming tool components to match what the model saw in pretraining cut schema misalignment errors by 80% and improved MetaTool and RoTBench scores by up to 17%. Read that backwards: a large share of small-model tool failure isn't reasoning failure, it's the model not recognising your naming convention. Which is both encouraging and slightly humiliating for everyone who blamed the model.

Long-horizon agentic work sits in this column. So does anything requiring current world knowledge, obviously, because the weights were frozen months ago and a local model has no idea what happened last week unless you tell it. So does complex multi-constraint reasoning. And so does large-context synthesis, which deserves its own paragraph below because the numbers there are genuinely alarming.

The honest caveat: the gap on agentic work is narrowing faster than I expected. GLM-5.2 now tops open-weight results on SWE-bench Pro and Terminal-Bench 2.1, and the pattern across BFCLv3, tau-2-bench and MCP-Universe is that open models generalise out of distribution better than they did a year ago. On open-ended long-horizon planning the closed models still edge ahead. But "edge ahead" is a different sentence than the one I would have written in 2025.

Where the Line Sits by Model Size

Most guides to local LLM use cases skip this part, which is odd, because the size question has a cleaner answer than the task question. Memory arithmetic doesn't have opinions.

At Q4, a 7B to 8B model needs 4 to 6 GB, which means 8 GB of VRAM or a base Mac runs it. That class does classification, extraction, rewriting, routing and short summarisation. It does not do sustained tool use, and I would not put one in front of a customer unsupervised.

The 20B class is the current sweet spot and it isn't close. gpt-oss-20b ships a 12.8 GiB checkpoint and runs in roughly 16 GB, which is an RTX 4080, a 4090, or a 24 GB Mac. It was also built with a chat schema designed around tool calls, and you can feel it. This is the tier where a local model stops being a text utility and starts being something you can wire into a pipeline.

Then 32B at Q4 wants around 20 GB, and 70B wants 40 GB and up, realistically 48 to 96 GB if you want context headroom. The 120B class is where it gets interesting and expensive at once: gpt-oss-120b activates only about 5B parameters per token so it runs fast, but you still hold all 117B weights in memory. Call it 60 to 65 GB on a single card. That's an H100 or a 96 GB workstation card, and at that point you are having a very different conversation about money, which I already had with myself in the break-even math on local hardware.

One thing that catches people out constantly: the weights are not the bill. At 128K context, KV cache can add tens of gigabytes on top, roughly 29% of total memory in one worked example. Every VRAM table you find online is quoting you the model at short context. Budget for the cache or your beautiful 20B setup falls over on the first long document.

Quantisation, Honestly

The community line is that Q4 is free and Q8 is for people with money to burn. That's roughly true, with one exception that matters enormously for the exact thing most people want to build.

The averages support the community: Q8 shows about 0.1% loss on MMLU, Q4 about 1.2%, and Q4_K_M lands at 95 to 97% of FP16 quality for roughly 35% of the VRAM. Fine. Take the Q4.

But averages hide the shape. The ACBench work published at ICML, Can Compressed LLMs Truly Act?, benchmarked compression against agentic capability specifically rather than perplexity, across 12 tasks and 15 models. Their finding: 4-bit quantisation preserves workflow generation and tool use with only a 1 to 3% drop, but degrades real-world application accuracy by 10 to 15%. Same quantisation. Same model. Ten times the damage, depending on what you asked it to do.

That is the whole point of this post in one result. "Is Q4 good enough" has no answer until you say good enough at what. Extraction and classification, take the Q4 and don't think about it. Anything where the model has to keep a plan straight across steps, go Q8 if you can hold it, and go in expecting to measure rather than assume. Separate signal-noise analysis found 4-bit largely preserves performance across task types and 3-bit is where knowledge-intensive and reasoning tasks fall apart, so 3-bit is a false economy no matter what the file size says.

The Failure Modes Nobody Puts on a Leaderboard

Four things break in production that no benchmark column warns you about.

Structured output is worse than you assume. A 2026 paper on small-model structured output tested Llama 3.1-8B, Gemma 2-9B and Qwen 2.5-7B and found that with no formatting guidance, no model produced valid JSON at all. Zero percent. With a minimal reference prompt, Llama hit 44.43% on GSM8K, Qwen hit 74.30% there and 0% on MATH, and Gemma scored 0% because it kept wrapping its JSON in markdown fences. Iterative prompt optimisation got them all to 84 to 87%. Constrained decoding enforced valid syntax but cost 3.6 to 8.2 times the latency without reliably improving results. The separate Structured Output Benchmark makes the same point differently: schema compliance is high while value accuracy tops out at 83% on text and 67.2% on images. The JSON parses. The fields are wrong.

Advertised context is not usable context. NVIDIA's RULER puts real usable context at 50 to 65% of advertised. Adobe's NoLiMa defines effective length as the point where a model still keeps 85% of its base score, and by that measure most models fall below half their short-context performance by 32K tokens. Chroma tested 18 models for context rot and every single one degraded as input grew. This is not a local model problem, it's everyone's problem, but it hits harder locally because you were counting on that context window to avoid buying more hardware.

Instruction drift over long inputs is the practical version of the same thing. The system prompt you wrote is competing with 40,000 tokens of document, and around the two-thirds mark it starts losing. I ran into this from the other direction when I looked at how much context tool definitions eat before you type a word.

Silent quality collapse. Nothing errors. The output still looks like output. This is the one that gets you, and the only defence is holding out a set of known-good examples and rerunning them every time you change a model, a quant or a prompt.

The Split I'd Actually Run

Local for volume, private and latency-sensitive. API for the hard ten percent. That's the whole pattern, and the decision rule fits in three questions.

Does the task need knowledge the model doesn't have in the prompt? API. Does it require more than about three dependent steps to go right in a row? API. Does it run thousands of times a day on your own data with a schema you control? Local, and stop paying per token for it.

Everything else is a measurement, not a debate. Take fifty real examples from your own workload, run them through both, and count. That's a morning's work and it settles more argument than any leaderboard.

I've left compliance out of this deliberately, because that's a whole separate reason to run local and it deserves its own treatment. If GDPR and data residency are what's driving you rather than capability, the EU-compliant local AI stack is the post you want. And if you're wondering what the setup actually looks like in practice, I wrote up pointing Claude Code at a local model start to finish.

The thing I'd want you to take away is smaller than a recommendation. Most people evaluating local LLM use cases test one model on one hard task, watch it fail, and conclude local isn't ready. The hard task was always going to fail. The interesting question is how much of your actual workload is boring, and in my experience building this stuff for clients, the answer is most of it.

If you're trying to work out which side of that line your own project falls on, get in touch. I'd rather tell you it's the API than sell you a machine you don't need.