Parameters, Quantization, VRAM: Why "It Fits" Is Not "It Runs" ​

We · October 2026

This is the third article in the "AI Tech Explainers" series, which takes the shape of one word, properly explained, per article: no product reviews, no rankings, no download or sign-up links. The previous article sorted out which layer each tool sits on; this one is about the first wall you hit right after installing: why some models open instantly, while others either throw an error or pump out two words a second. Every number here is an order of magnitude from community and vendor published figures (checked October 2026), not a guarantee—measure on your own machine.

AI moves fast. The tools, capabilities and numbers in this article describe where things stand right now, and are for reference only.

First, one formula to remember ​

There are many opinions online about "how big a model my machine can run," but the only starting point worth having is one line:

Model size ≈ parameter count × bytes per parameter

The parameter count is how many adjustable numbers the model holds, written as 7B, 14B or 32B—B means billion. Each parameter is stored as one number, and the higher the precision, the more bytes it takes: full precision (FP16, BF16) is 2 bytes, INT8 is 1 byte, 4-bit quantization is 0.5 bytes.

So an 8B model comes to roughly 16 GB at full precision, 8 GB at 8 bit and 4 GB at 4 bit. We want to stress that this formula is only for judging orders of magnitude, because a real quantized file never equals the theoretical number. Quantization does not simply shorten each number—it also stores an extra set of scaling information, so the actual file is always a little larger than theory: a 7B model at the 4-bit tier has a theoretical size of 3.5 GB, while files the community reports measuring sit between 4.2 and 4.6 GB. Sizing your machine off the theoretical value is digging your own hole.

Quantization: compression tiers of one model ​

This is the most misunderstood point. Q4, Q5 and Q8 are not three different models—they are several degrees of compression of the same model. The logic is the same as choosing original, high or standard quality when you save a photo.

Three practical conclusions:

First, the Q4 tier is the starting point for most people. These tier names (Q4_K_M, Q8_0 and the rest) come from the GGUF file format—the most common way open-source models are packaged for local tools. If a download page shows a file ending in .gguf, it was made for this kind of tool. Group quantization with K_M in the name is the most widely used form locally today; within one tier, Q4_K_M versus Q4_K_S is just a small tuning between size and quality, and an ordinary user can decide by "if it fits, take the higher tier."

Second, squeeze too hard and the VRAM you saved gets taken back somewhere else. The lower the bit width, the more the model loses on long reasoning chains and complex instruction following. Running a 32B model at Q3 until "nothing it answers is right" is worse than switching to a 14B model at Q4—the capability gap that parameter count makes is usually something quantization cannot patch back.

Third, in 2026 this column is still pushing downward. The 2-bit tier recently opened up, and the community has posted measurements of 8 GB-class models squeezed to about 4 GB. But mind its boundary: what works today is mainly model families designed for 2 bit, not any model hacked down to 2 bit. Low-bit quantization is still changing fast—when you read "model X holds up even at 2 bit," first check whether it is talking about the same thing.

"It fits" and "it runs" are two different things ​

The model file has to sit in VRAM—the graphics card's own memory—to compute. What happens if it does not fit? Most tools do not fail outright; they keep some layers in system memory and move them in when needed. That step is the entire reason so many people see "it started, but it is absurdly slow."

There is a set of measurements that circulates widely in the community: the same 32B model on the same 24 GB card. All of it in VRAM: about 40 tokens per second. Move 20% of the layers to system memory: down to 3.8 per second. All of it in memory: 1.6 per second. The difference is not tens of percent—it is an order of magnitude. The reason is blunt: generating every single token walks every layer, and if part of them sits on the slow pipe, the whole chain waits for the slowest segment.

So we want to correct one intuition: there is no such thing as "barely fits." Model plus context either goes into VRAM as a whole, or it grinds along at the speed of system memory—there is no middle setting. Offloading layers to memory to save a little VRAM is almost always a bad trade.

While we are here, this also explains why Apple machines often do surprisingly well at this: their VRAM and system memory are already one pool (unified memory), so there is no "moving up and down" at all. What remains is only whether the capacity is enough and the bandwidth keeps up. That is also why big-unified-memory machines get chosen for local large models—not because they beat a discrete GPU on speed, but because they can hold the thing.

The real hog is often the context ​

Many people budget VRAM for the weights only, then get confused at startup when usage runs well past the file size: the model file is 5 GB, so why is the run eating fifteen or more?

Because there is a second consumer—the KV cache. While generating, the model keeps a copy of the intermediate results computed from earlier text so it does not recompute everything for each new token. Give it how much context space, and it reserves that much room. This term grows linearly with context length, and it keeps climbing as the conversation gets longer.

Some orders of magnitude you can use directly:

• Short context (2–4K), one person using it: reserving 1–2 GB is enough.
• Same model, context stretched from 4K to 32K: this overhead goes up roughly 8-fold.
• Long-document Q&A, feeding in a whole PDF, or workflows where the AI rereads a pile of files (agents are the typical case): once context grows, the KV cache easily eats several GB.

An estimation formula that is good enough: VRAM needed ≈ weights × 1.1–1.2 + KV cache + 0.5 to 1 GB of system overhead.

This also answers another common misunderstanding directly: a page that says "supports 128K context" does not mean your machine can hold 128K of context at the same time. The vendor is stating the model's capability ceiling; what you need to compute is your machine's headroom. Context length can be adjusted in most local tools, and compressing it first to the length you actually need is the cheapest VRAM saving there is—far cheaper than scraping by on the quantization tier.

The parameter count can lie too: what MoE is ​

One more development of the past two years has made the parameter-count-to-size line less straight.

There is a class of models called MoE (Mixture of Experts): the total parameter count is large, but computing each token only uses a small share of it. The common shorthand from vendors and the community reads "120B total parameters, 12B active"—meaning it installs like a 120B giant but computes more like a 12B machine. Two conclusions:

• It wins on speed: fewer multiply-adds per token, so on the same hardware it is often faster than a plain model of the same total parameter count.
• It does not win on capacity: all the weights still have to be loaded, and that cannot be skipped. When you see "a huge model, and an ordinary computer can run it," first tell whether they mean active parameters or total parameters.

One table: which comfort zone your machine lands in ​

The table below is an order-of-magnitude reference from community and vendor published figures as of October 2026, estimated at the Q4 tier, for one person, with the context not stretched. Real results swing a lot with model architecture, GPU bandwidth and driver version—treat this as a starting point, not a promise.

Two notes on reading the table: subtract 1–2 GB from any "it fits" figure to get close to the headroom you actually have (the system and the KV cache take their share); and do not look up MoE models by total parameter count—first check how big their full weights are. When choosing a model, if you already have one that runs, use it to test the water: record its real usage and speed, then push up one tier in proportion. That beats any table.

Three rules for fewer pitfalls ​

1. Pick the parameter count by the capability you want, then pick the quantization tier by the headroom you have. Reverse the order and you get "squeezed into some tier nobody can use, just to fit inside 16 GB." If you want more capability, the right path is a bigger model, not a different quantization of the same one.
2. Do not max out the context. Most people's real conversations are only a few thousand characters long; setting the context to the length you need frees far more space than switching a quantization tier.
3. Spilling into system memory is not "a bit slower"—it is an order of magnitude slower. Better to step down one model size than to leave yourself a path that "runs, at two words per second."

One judgment we want to end on: the difficulty of running models locally has kept falling for the past three years—the quantization tiers have gotten finer, extreme compression like 2 bit has become usable, and big unified-memory machines route around the old "it does not fit" problem entirely. But one thing has not changed: what has never blocked ordinary people is compute—it is the environment. The same model opens in seconds on one well-configured machine, while on another you reinstall dependencies, change drivers and tune the context before it runs. There are already solutions on the market that gather the original, quantized and accelerated builds of common models under "one-click install, ready out of the box," and the real value of that kind of tool is exactly here: it finishes all the conversions above for you, and you only have to answer "which one do I want to try."

As for which model to pick for your graduation project or coursework, Open-Source AI Models for Undergraduate Graduation Projects: Organized by Direction is a list sorted by direction—worth reading alongside this one.