AirLLM: Run 70B LLMs on a Single 4GB GPU — The Open-Source Tool With 25,000+ GitHub Stars

Running large language models locally has always been a dream for developers. The problem? Massive models like Llama 3.1 405B or DeepSeek-V3 require hundreds of gigabytes of VRAM — hardware that costs thousands of dollars and sits out of reach for most developers.

AirLLM changes that equation entirely.

This open-source tool, which has accumulated over 25,000 GitHub stars and continues to trend, lets you run 70B parameter models on a single 4GB GPU card. Yes, you read that right. No quantization. No model compression that degrades quality. No expensive A100s or H100s. Just your existing hardware, running models that were previously impossible to use locally.

What is AirLLM and Why Does It Matter?

AirLLM is a Python library that fundamentally rethinks how large language models are loaded and executed. Instead of loading the entire model into GPU memory at once (which requires massive VRAM), AirLLM loads and processes the model one layer at a time.

This approach, called layer-by-layer inference, means that at any given moment, only a small portion of the model resides in GPU memory. The rest stays on disk or in system RAM, loaded only when needed.

The result? Models that previously required 80GB+ of VRAM can now run on consumer-grade GPUs with 4-8GB of memory.

Supported Models (And They're Big)

AirLLM isn't just for small models. Here's what you can run on modest hardware:

• 70B models (Llama 3, Qwen2.5-72B) — runs on 4GB GPU
• 405B models (Llama 3.1 405B) — runs on 8GB GPU
• 671B models (DeepSeek-V3) — runs on ~12GB GPU
• 235B models (Qwen3-235B-A22B) — runs on ~3GB GPU (sparse MoE)
• 2.8T models (Kimi K3) — runs on under 4GB GPU (sparse MoE)

Yes, that last one is 2.8 trillion parameters running on a 4GB card. The magic? Sparse Mixture of Experts (MoE) models only activate a subset of parameters for each token, and AirLLM streams only the experts that are actually needed.

How AirLLM Works: The Technical Deep Dive

Traditional LLM inference loads the entire model into GPU memory. For a 70B parameter model in FP16, that's roughly 140GB of VRAM — impossible on consumer hardware.

AirLLM takes a different approach:

1. Model Sharding

When you first load a model, AirLLM splits it into individual layers and saves them separately. Each layer becomes its own file on disk.

2. Streaming Inference

During inference, AirLLM loads one layer at a time into GPU memory, processes the input through that layer, then moves to the next layer. At any given moment, only one layer (plus intermediate activations) occupies GPU memory.

3. Prefetching Optimization

AirLLM uses intelligent prefetching to overlap disk I/O with computation. While the GPU processes layer N, it simultaneously loads layer N+1 from disk. This reduces the performance penalty of layer-by-layer loading.

4. Optional Compression

For even faster inference, AirLLM supports 4-bit and 8-bit block-wise quantization. Unlike traditional quantization that affects model quality, AirLLM's approach only quantizes weights (not activations), preserving accuracy while achieving up to 3x speed improvement.

Real-World Example: Running Qwen3-32B on a Budget GPU

Let's walk through a complete example. We'll run Qwen3-32B on a GPU with just 4GB of VRAM.

Step 1: Installation

pip install airllm

That's it. No complex setup, no special dependencies.

Step 2: Load and Run

from airllm import AutoModel

MAX_LENGTH = 128

# Load a 32B model — it will run on 4GB GPU
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# Prepare input
input_text = ['What is the capital of France?']
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False
)

# Generate output
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True
)

# Decode and print
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

The first time you run this, AirLLM will download the model and split it into layers (make sure you have enough disk space). Subsequent runs will be faster since the sharded model is cached.

Step 3: Go Even Bigger

Want to run a 235B model? Just change one line:

model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")

That's it. Same code, same GPU, 7x more parameters. The sparse MoE architecture means only ~22B parameters are active per token, and AirLLM streams only those active experts.

Adding Compression for Speed

If you want faster inference and can tolerate minimal quality loss, enable 4-bit compression:

from airllm import AutoModel

model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",
    compression='4bit'  # or '8bit' for 8-bit quantization
)

This can provide up to 3x speed improvement with almost negligible accuracy loss, since AirLLM only quantizes weights (not activations).

Key Benefits of AirLLM

• Democratizes Large Models: Run 70B+ models on consumer GPUs that cost $200-400, not $10,000+ enterprise cards
• No Quality Loss: Unlike quantization or distillation, AirLLM preserves the full model quality — you're running the exact same model, just more efficiently
• Simple API: Drop-in replacement for Hugging Face Transformers. If you know AutoModel.from_pretrained(), you know AirLLM
• Cross-Platform: Works on Linux, Windows, and macOS (Apple Silicon). No CUDA required on Mac
• Wide Model Support: Llama 3.x/4, Qwen2.5/3, DeepSeek V2/V3, Phi-4, Gemma, ChatGLM, Mistral, and more
• Active Development: Regularly updated with support for the latest models (Kimi K3 2.8T added in July 2026)
• Open Source: MIT licensed, 25,000+ GitHub stars, active community

Performance Considerations

AirLLM trades speed for memory efficiency. Here's what to expect:

• First run: Slower, as the model is downloaded and sharded
• Subsequent runs: Faster, but still slower than full-memory inference
• With compression: Up to 3x faster than uncompressed AirLLM
• vs. traditional inference: Expect 2-10x slower token generation, depending on disk speed and model size

The tradeoff is clear: you sacrifice some speed to gain access to models that would otherwise be impossible to run. For many use cases (batch processing, research, prototyping), this is an excellent tradeoff.

When to Use AirLLM (And When Not To)

Perfect For:

• Developers who want to experiment with large models locally
• Batch processing jobs where speed isn't critical
• Research and prototyping with state-of-the-art models
• Running models on edge devices or budget hardware
• Privacy-sensitive applications where cloud APIs aren't acceptable

Not Ideal For:

• Real-time chatbots requiring sub-second responses
• High-throughput production systems serving thousands of requests
• Situations where you have access to enterprise GPUs with sufficient VRAM

Frequently Asked Questions

1. Does AirLLM really work with just 4GB of VRAM?

Yes. AirLLM has been tested with 70B models on GPUs with as little as 4GB of VRAM. The key is layer-by-layer loading — only one layer occupies GPU memory at a time. Larger models (405B, 671B) require proportionally more VRAM but still far less than traditional inference.

2. How much slower is AirLLM compared to traditional inference?

Expect 2-10x slower token generation, depending on your disk speed and model size. With 4-bit compression enabled, you can achieve up to 3x speedup. For batch processing or non-real-time applications, this is often acceptable.

3. Does AirLLM work on macOS?

Yes, AirLLM supports macOS with Apple Silicon (M1/M2/M3/M4). You'll need to install MLX (Apple's machine learning framework) and torch. The API remains identical to the Linux/Windows version.

4. Can I use AirLLM with models I've already downloaded?

Absolutely. Just pass the local path instead of a Hugging Face repo ID:

model = AutoModel.from_pretrained("/path/to/your/local/model")

AirLLM will shard the local model and cache the layers for future use.

5. Does compression affect model quality?

AirLLM's compression only quantizes weights (not activations), which preserves accuracy much better than traditional quantization. In practice, the quality loss is negligible for most tasks. The compression provides up to 3x speed improvement.

6. What about sparse MoE models like DeepSeek-V3 or Qwen3-235B?

AirLLM has special handling for Mixture of Experts models. Instead of loading entire layers, it streams only the experts that are actually activated for each token. This is why a 235B MoE model can run on just 3GB — only ~22B parameters are active per token.

7. Is AirLLM suitable for production use?

AirLLM is excellent for prototyping, research, and batch processing. For high-throughput production systems requiring low-latency responses, you'll want traditional inference with sufficient VRAM or cloud-based solutions. However, for many production workloads (batch summarization, document processing, etc.), AirLLM is perfectly suitable.

Getting Started

Ready to run large models on your existing hardware? Here's your action plan:

1. Install AirLLM: pip install airllm
2. Pick a model: Start with a 32B or 70B model to test
3. Run the example: Copy the code from the Real-World Example section above
4. Experiment: Try larger models, enable compression, test on your specific use case

For developers looking to deepen their understanding of large language models and AI development, check out CoddyKit's comprehensive AI and machine learning courses. Whether you're just starting with Python or diving deep into transformer architectures, structured learning accelerates your progress far beyond trial-and-error.

The Bottom Line

AirLLM represents a paradigm shift in how we think about running large language models. By rethinking the inference process — loading models layer by layer instead of all at once — it makes previously impossible tasks achievable on consumer hardware.

With 25,000+ GitHub stars and support for the latest models (including the massive 2.8T Kimi K3), AirLLM has proven itself as a reliable, actively maintained tool in the AI developer's toolkit.

Whether you're a researcher experimenting with cutting-edge models, a developer building privacy-focused applications, or simply someone who wants to run Llama 3.1 405B without buying a $30,000 GPU, AirLLM delivers. The future of large model inference isn't just in the cloud — it's on your desk, running on hardware you already own.

Ready to try it? Install AirLLM today and experience what was once thought impossible: running 70B parameter models on a 4GB GPU.