AirLLM: Running a 70B Model on a 4GB GPU, and When That Actually Helps

AirLLM is an open source Python library that runs inference for very large language models on GPUs that have no business running them. The headline claim from the project is a 70B parameter model on a single 4GB GPU, and Llama 3.1 405B on 8GB. It does this without distillation, without pruning, and without requiring quantization.

That claim is real, and the technique behind it is genuinely clever. It is also widely misread. AirLLM does not make large models cheap to serve. It makes them possible to run, which is a different thing, and the gap between those two words is where most people waste a weekend. This article covers how it works, the one piece of arithmetic that tells you whether it fits your workload, and the cases where a smaller model beats it outright.

Why VRAM is the wall in the first place

When people say a model does not fit on a GPU, they almost always mean the weights do not fit. A 70B parameter model in 16 bit precision needs roughly 140GB just to hold its parameters, before you allocate a single byte for activations or the KV cache. An 80GB A100 cannot hold it. Two of them can, barely, which is why serving large models normally means multi-GPU setups and the rental bills that come with them.

The standard responses are to shrink the model or rent more hardware. Quantization drops the weights to 8, 4, or fewer bits, trading some accuracy for a large memory saving. Distillation trains a smaller model to imitate a larger one. Both change the model. AirLLM takes a different route: it leaves the model exactly as it is and changes when the weights are in memory.

How layered inference works

A transformer is a stack of near identical layers executed in strict order. Layer 1 runs, produces an output, and hands it to layer 2. Layer 2 runs, hands off to layer 3, and so on to the top of the stack. Critically, once layer 1 has finished, its weights are not needed again for that forward pass.

That is the whole insight. There is no moment during a forward pass when you need all 80 layers of weights resident at once. You only ever need the weights of the layer you are currently executing. AirLLM exploits this by loading a single layer's weights onto the GPU, running that layer, freeing the memory, and loading the next.

1. Split the model checkpoint on disk into per layer shards, done once, ahead of time.
2. Load layer 1's weights from storage or system RAM into VRAM.
3. Run layer 1 on the current activations.
4. Free layer 1's weights and load layer 2 into the space just vacated.
5. Repeat to the top of the stack, then emit a token and start again.

Peak VRAM is now driven by the largest single layer plus your activations and KV cache, not by the full parameter count. For a 70B model, one layer is on the order of a couple of gigabytes, which is how a 4GB card gets into the game at all.

The arithmetic that decides whether it works for you

Here is the part the demos skip. You are no longer limited by memory capacity. You are limited by memory bandwidth, and the trade is worse than most people expect.

In a normal setup, weights sit in VRAM and the GPU reads them at something like 1 to 3 terabytes per second. Under layered inference, every weight has to travel from wherever it lives, system RAM or an SSD, across to the GPU, for every single forward pass. And a forward pass produces exactly one token. So generating a response means streaming the entire model, once per token.

The rough estimate is simple: seconds per token is approximately the model size in bytes divided by the bandwidth between where the weights live and the GPU. Work it through for a 70B model at 16 bit, roughly 140GB of weights:

• Weights on a SATA SSD at around 0.5 GB/s: on the order of several minutes per token. Unusable for anything interactive.
• Weights on a fast NVMe drive at around 5 GB/s: on the order of 30 seconds per token.
• Weights cached in system RAM, moving over PCIe 4.0 x16 at roughly 25 GB/s: on the order of 5 seconds per token.
• The same 70B quantized to 4 bit, roughly 35GB, from system RAM: on the order of 1.5 seconds per token.

Treat those as order of magnitude figures, not benchmarks. Real numbers depend on your drive, your PCIe generation, whether layers are prefetched while the previous one computes, and whether the operating system page cache is holding the shards. Prefetching in particular can overlap loading with compute and materially improve on the naive estimate. Benchmark your own setup before committing to anything.

What AirLLM is genuinely good for

Seconds per token sounds disqualifying until you notice how many real tasks do not care about latency. If nobody is sitting and waiting for the response, throughput per dollar matters and latency does not.

• Offline batch processing, such as classifying, extracting from, or summarising a document backlog overnight.
• Synthetic data generation for fine tuning a smaller model you will actually deploy.
• Evaluation runs, where you want a large model's judgement on a test set and can wait.
• One off analysis on sensitive data that is not permitted to leave your network.
• Checking whether a frontier sized model is meaningfully better at your task before you commit budget to serving one.

That last one is the most valuable and the most overlooked. Before you architect around a 70B model, you want evidence it beats a 7B or 13B model on your specific task. AirLLM lets you get that evidence on hardware you already own, in an afternoon, for the cost of electricity. If the large model turns out to be no better, and on narrow well defined tasks it frequently is not, you have saved yourself an entire infrastructure decision.

What it is not good for

Do not put this behind a user facing endpoint. Anything interactive, any chat interface, any API with a latency budget, any workload with concurrent users: layered inference is the wrong tool and no amount of tuning fixes it. The bottleneck is physics, not configuration.

It also scales badly with concurrency. Because each request streams the full model, two simultaneous requests do not share much work unless you batch them into the same forward pass. Serving frameworks built for throughput, such as vLLM or TensorRT-LLM, exist precisely because keeping weights resident and batching aggressively is what makes production serving economical.

And be honest about the KV cache. Layered inference solves the weights problem, not the context problem. The KV cache grows with sequence length and batch size and it stays resident. Long context work can still exhaust a small GPU even when the weights are being streamed neatly one layer at a time.

How it compares to the alternatives

AirLLM is one of four ways to deal with a model that will not fit, and it is rarely the first one to reach for.

• Use a smaller model. Usually the correct answer. A well chosen 7B to 14B model, fine tuned on your task, beats a general purpose 70B on narrow work more often than people expect, and it serves in real time.
• Quantize. A 4 bit 70B fits in around 35GB and runs at normal speed on a single large GPU. Some accuracy loss, enormous practical gain. Try this before layered inference.
• Rent a bigger GPU. Hourly cloud GPU time is cheap for intermittent work. If you need a 70B for six hours a month, renting is almost certainly cheaper than engineering around a 4GB card.
• Use an API. If the data is allowed to leave your network and volume is modest, a hosted frontier model costs less than the time you will spend on infrastructure.

AirLLM earns its place when several constraints stack up at once: the data cannot leave your network, you need the full precision behaviour of a large model, the work is batch shaped, and you are not willing to buy or rent the hardware to hold the whole thing. That combination is real, and when you hit it, nothing else does the job.

Getting started

Installation is a single pip command, and the API is deliberately close to the Hugging Face transformers interface so existing code needs little adjustment. The library supports several model families including Llama, Mistral, Qwen, ChatGLM, Baichuan, and InternLM, and it added macOS support for Apple silicon. It also offers optional block wise quantization, which reduces how many bytes cross the bus per pass and is the single most effective speed lever available to you.

Expect the first run to be slow beyond the inference itself. The library splits the checkpoint into per layer shards on first load, which is a substantial one time disk operation for a large model. Budget the disk space for both the original checkpoint and the sharded copy. Check the project documentation for current model support and configuration flags, since the library moves quickly.

The honest verdict

AirLLM is an elegant answer to a specific question: how do I run this model at all, on this hardware, when nothing else will. For offline batch work, for privacy constrained analysis, and above all for cheaply testing whether a large model is even worth the infrastructure, it is a genuinely useful tool that costs you nothing to try.

It is not a way to cut your inference bill for a production system. If you are serving users, the path to lower cost runs through choosing a smaller model, fine tuning it properly, quantizing it, and batching well. Those are less impressive than running a 405B model on a laptop GPU, and they are what actually reduces the number on the invoice.

The useful move is to use AirLLM as a measuring instrument rather than a serving strategy. Run the big model once, offline, on your real task. Compare it honestly against a small model you could actually deploy. Then make the infrastructure decision with evidence instead of an assumption about what size of model your problem requires.