MiniMind: The Open-Source Project With 56,800+ GitHub Stars That Lets You Train an LLM From Scratch in Just 2 Hours

Why Training an LLM From Scratch Matters in 2026

Large language models have transformed how we build software. ChatGPT, Qwen, DeepSeek — these systems feel almost magical. But for most developers, the inner workings of these models remain a black box. You can fine-tune them with LoRA, wrap them with high-level libraries like transformers or trl, and get impressive results with a dozen lines of code.

But here's the uncomfortable truth: using a model and understanding a model are fundamentally different skills. And in a field moving this fast, the developers who truly understand the mechanics — tokenization, attention mechanisms, training loops, alignment — are the ones who build the next breakthrough, not just consume the last one.

That's exactly the gap MiniMind was built to close.

What Is MiniMind?

MiniMind is an open-source project created by developer Jingyao Gong that provides a complete, from-scratch implementation of a small but fully functional language model. The project's philosophy is beautifully simple:

"Building a plane from LEGO bricks yourself is far more exciting than flying first class."

Unlike most LLM tutorials that stop at fine-tuning or rely heavily on pre-built frameworks, MiniMind implements every core algorithm in raw PyTorch — no high-level abstractions, no magic wrappers. You see and write every line of the transformer architecture, the tokenizer, the training loop, and the alignment pipeline.

The result? A 64M-parameter model (MiniMind-3) that's approximately 1/2700th the size of GPT-3, yet demonstrates real conversational ability, tool calling, and even adaptive reasoning — all trainable on consumer hardware.

Key Project Stats

• 56,800+ GitHub stars and 7,400+ forks
• Apache 2.0 license — completely free and open
• 2-hour training time on a single RTX 3090
• ~$0.40 cloud GPU cost for full training run
• Compatible with transformers, vllm, llama.cpp, and ollama
• Active development since July 2024, with major v3 release in April 2026

The Complete Training Pipeline: From Pretrain to Agentic RL

What makes MiniMind exceptional isn't just its small size — it's the completeness of its training pipeline. Most educational LLM projects show you pretraining or maybe fine-tuning. MiniMind covers the entire journey:

1. Pretraining

The foundation. MiniMind trains on curated text data using next-token prediction, building the model's base language understanding from scratch. The project provides both a lightweight dataset (pretrain_t2t_mini.jsonl, 1.2GB) for quick experiments and a full dataset (pretrain_t2t.jsonl, 10GB) for thorough training.

# Start pretraining from scratch
cd trainer && python train_pretrain.py

# Resume from checkpoint if interrupted
python train_pretrain.py --from_resume 1

2. Supervised Fine-Tuning (SFT)

This is where the model learns to follow instructions and have conversations. MiniMind's SFT data includes multi-turn dialogues, tool-calling examples, and reasoning templates — all in a unified format:

{
  "conversations": [
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."}
  ]
}

3. LoRA Fine-Tuning

MiniMind implements LoRA from scratch — no peft wrapper. You see exactly how low-rank adapters modify attention weights, making it an invaluable learning resource for understanding parameter-efficient fine-tuning.

4. RLHF and RLAIF

The project includes native PyTorch implementations of:

• DPO (Direct Preference Optimization) — RLHF without a reward model
• PPO (Proximal Policy Optimization) — the classic RLHF algorithm
• GRPO (Group Relative Policy Optimization) — DeepSeek's approach
• CISPO — a newer alignment algorithm

5. Agentic RL & Tool Use

The latest v3 release introduced train_agent.py, a dedicated script for training the model to use tools across multi-turn interactions. The SFT data already includes ~100,000 tool-calling examples synthesized from Qwen3-4B, giving the base model foundational tool-use capabilities out of the box.

6. Adaptive Thinking & Model Distillation

MiniMind supports adaptive reasoning — the model can dynamically decide whether to "think" before answering, controlled via a simple open_thinking toggle. It also includes white-box distillation code, letting you transfer knowledge from larger models into MiniMind's compact architecture.

Architecture: Aligned With Qwen3

MiniMind-3's architecture aligns with the Qwen3 / Qwen3-MoE ecosystem, making it a practical learning platform for understanding production-grade model designs. The Dense model runs at 64M parameters, while the MoE (Mixture of Experts) variant reaches 198M total parameters with 64M active per forward pass.

The tokenizer uses a custom BPE + ByteLevel implementation with a compact 6,400-token vocabulary — deliberately small to minimize embedding layer overhead on such a tiny model. Special tokens include <tool_call>, <tool_response>, and <think> for structured generation.

# Quick inference with MiniMind-3
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("./minimind-3")
model = AutoModelForCausalLM.from_pretrained("./minimind-3")

inputs = tokenizer("Explain quantum entanglement simply:", return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Real-World Example: Training Your Own Domain-Specific Model

Let's say you're building a medical Q&A assistant and want to understand every step of the process. Here's how MiniMind makes it possible on a budget:

# 1. Clone and set up (~5 minutes)
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt

# 2. Download datasets (~10 minutes)
# Place pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl in ./dataset/

# 3. Pretrain the base model (~1.5 hours on RTX 3090)
cd trainer && python train_pretrain.py

# 4. Fine-tune on your medical Q&A data (~30 minutes)
# Replace sft_t2t_mini.jsonl with your curated medical dataset
python train_full_sft.py

# 5. Apply DPO for safer medical responses (~20 minutes)
python train_dpo.py

# 6. Test your model
cd .. && python eval_llm.py --weight full_sft

# Total time: ~2.5 hours | Total cost: ~$0.60 on cloud GPU

The project also includes a Streamlit-based chat UI for interactive testing and an OpenAI-compatible API server (serve_openai_api.py) that you can connect to any chat frontend like Open-WebUI or FastGPT.

Key Benefits of MiniMind for Developers

• True understanding — Every algorithm implemented from scratch in PyTorch, no black-box abstractions
• Affordable experimentation — Full training pipeline runs for under $1 on consumer GPUs
• Production-compatible — Works with transformers, vllm, llama.cpp, ollama out of the box
• Complete pipeline coverage — Pretrain → SFT → LoRA → RLHF → Agentic RL → Distillation
• Multi-GPU ready — Supports DDP and DeepSpeed for scaling beyond a single GPU
• Educational datasets included — Curated, cleaned, and ready-to-use training data
• Active community — 56,800+ stars, 7,400+ forks, regular updates since 2024
• Extensible — Vision model (MiniMind-V), Omni model (MiniMind-O), and experimental architectures

Getting Started in Under 10 Minutes

If you just want to try MiniMind without training:

# Option 1: Ollama (easiest)
ollama run jingyaogong/minimind-3

# Option 2: vLLM for production serving
vllm serve ./minimind-3 --served-model-name "minimind"

# Option 3: Download from Hugging Face
git clone https://huggingface.co/jingyaogong/minimind-3

For the full training experience, grab the mini datasets and follow the step-by-step trainer scripts. The checkpoint system means even if your training gets interrupted, you can resume exactly where you left off with --from_resume 1.

MiniMind vs. Other LLM Learning Resources

There's no shortage of "build your own LLM" tutorials online. So what sets MiniMind apart?

Most educational projects either (a) give you a pre-trained model and teach fine-tuning, or (b) provide a toy implementation that doesn't scale beyond a few thousand tokens. MiniMind is the rare project that does both — it's genuinely educational and genuinely functional.

The fact that it covers the entire modern LLM pipeline — including cutting-edge techniques like GRPO, agentic reinforcement learning, and adaptive thinking — makes it unmatched as a comprehensive learning resource.

Frequently Asked Questions

Q: Do I need an NVIDIA GPU to use MiniMind?

No. While an NVIDIA GPU (like an RTX 3090) provides the best training speed, MiniMind also supports CPU and Apple MPS (Metal Performance Shaders) for training. Training will be significantly slower on CPU, but inference works well everywhere.

Q: Can MiniMind models be used in production?

MiniMind models are educational and experimental — they're not designed to compete with GPT-4 or Claude in capability. However, the 64M model can serve as a lightweight conversational agent for specific narrow tasks, and the training pipeline is production-grade code you can adapt for larger models.

Q: How does MiniMind compare to nanoGPT?

Andrej Karpathy's nanoGPT is an excellent introduction to transformer pretraining, but it stops at the pretraining stage. MiniMind goes much further — covering SFT, LoRA, RLHF, DPO, PPO, GRPO, tool use, and agentic RL — making it a more complete educational resource for the modern LLM stack.

Q: Is the training data included?

Yes. MiniMind provides curated datasets on both HuggingFace and ModelScope, including pretraining data (1.2GB–10GB), SFT data with tool-calling examples (1.6GB–14GB), DPO preference data, and agentic RL datasets. All data is cleaned, deduplicated, and formatted for immediate use.

Q: What languages does MiniMind support?

MiniMind-3 has bilingual capabilities in English and Chinese. The custom 6,400-token vocabulary is optimized for both languages, though English proficiency is stronger in the latest release. You can adapt the tokenizer and retrain for other languages.

Q: Can I use MiniMind with existing tools like LangChain or LlamaIndex?

Yes. Since MiniMind-3 is compatible with the Hugging Face transformers library and includes an OpenAI-compatible API server, it integrates with any tool that supports those interfaces — including LangChain, LlamaIndex, Open-WebUI, and FastGPT.

Q: What's the MoE variant and should I use it?

The MoE (Mixture of Experts) variant, minimind-3-moe, has 198M total parameters but only activates 64M per forward pass — giving you MoE architecture understanding without massive compute requirements. It's excellent for learning how production models like Mixtral and Qwen3-MoE work internally.

Final Thoughts

In an era where AI education is increasingly dominated by paid courses and surface-level tutorials, MiniMind stands out as a genuinely generous, technically rigorous, and refreshingly honest open-source project. It doesn't promise to make you an "AI expert" in 30 minutes — it gives you the tools, code, and data to actually understand how language models work, one line of PyTorch at a time.

Whether you're a developer looking to move beyond API calls, a student studying NLP, or an engineer who believes the best way to learn is by building — MiniMind offers the most complete, affordable, and transparent path from zero to a trained language model.

With 56,800+ stars and counting, the community has spoken: this is how LLM education should be done.