LoRA Fine-Tuning with Hugging Face PEFT
Configure LoRA rank, alpha, and target modules, run supervised fine-tuning with the TRL SFTTrainer, monitor training loss, and save merged and adapter-only checkpoints.
LoRA Fine-Tuning with Hugging Face PEFT is a free AI Engineering Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why LoRA Instead of Full Fine-Tuning?
Full fine-tuning updates every parameter in a model. For a 7-billion parameter model at float32 precision, that requires ~28GB of GPU memory just for the weights, plus optimizer states, gradients, and activations — easily 80-120GB total. LoRA (Low-Rank Adaptation) instead adds a tiny number of trainable parameters (typically 0.1-1% of total) as low-rank matrix pairs that are applied to selected layers, reducing the GPU requirement by 10-50x while achieving comparable results.
# LoRA math intuition:
# Full fine-tuning: update W (large matrix, e.g., 4096 x 4096 = 16.7M parameters)
# LoRA: instead train W = W_0 + A @ B where:
# A has shape (4096, r) - only r*4096 params
# B has shape (r, 4096) - only r*4096 params
# r (rank) is typically 4, 8, or 16 - much smaller than 4096
# Memory comparison for 7B model:
# Full fine-tuning: ~80GB GPU RAM
# LoRA (rank=8): ~12GB GPU RAM - fits on a single A100 or 3090
print('LoRA makes fine-tuning accessible without massive GPU clusters')Installing the Required Libraries
LoRA fine-tuning with Hugging Face requires three libraries: transformers (model loading and tokenization), peft (Parameter-Efficient Fine-Tuning, which implements LoRA), and trl (Transformer Reinforcement Learning, which provides the SFTTrainer for supervised fine-tuning). Together these provide a high-level, production-ready fine-tuning workflow.
# pip install transformers peft trl accelerate bitsandbytes datasets
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer, SFTConfig
from datasets import Dataset
import torch
print('Libraries imported successfully')
print(f'GPU available: {torch.cuda.is_available()}')
if torch.cuda.is_available():
print(f'GPU: {torch.cuda.get_device_name(0)}')
print(f'GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')Loading the Base Model with Quantization
For most LoRA fine-tuning, load the base model in 4-bit quantization using bitsandbytes. This reduces the base model's memory footprint by 75% (a 7B model goes from ~14GB to ~4GB) while retaining most of the model's capability. Only the LoRA adapter layers are trained in full precision. The combination of 4-bit quantization + LoRA is called QLoRA and makes fine-tuning 7B models accessible on consumer GPUs.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
MODEL_NAME = 'mistralai/Mistral-7B-Instruct-v0.2' # or 'meta-llama/Llama-3.2-3B-Instruct'
# 4-bit quantization config (QLoRA)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # quantize base model to 4-bit
bnb_4bit_quant_type='nf4', # NF4 quantization type
bnb_4bit_compute_dtype=torch.float16, # compute in float16
bnb_4bit_use_double_quant=True # double quantization for extra savings
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
quantization_config=bnb_config,
device_map='auto', # automatically distribute across GPUs
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token # required for training batches
print(f'Model loaded. Parameters: {model.num_parameters():,}')Configuring LoRA Hyperparameters
The three most important LoRA hyperparameters are: r (rank) — the dimension of the low-rank matrices; higher rank means more trainable parameters and more expressive adaptation but more memory and risk of overfitting. lora_alpha — a scaling factor usually set to 2x the rank. target_modules — which weight matrices to apply LoRA to; attention layers (q_proj, v_proj) are the most common choice. Start with r=8 and tune from there.
from peft import LoraConfig, TaskType
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM, # causal language modeling
r=8, # rank: 4, 8, 16, 32 — higher = more params
lora_alpha=16, # scaling: usually 2*r
lora_dropout=0.1, # dropout on LoRA layers for regularization
target_modules=['q_proj', 'v_proj', # which weight matrices to adapt
'k_proj', 'o_proj', # common to target all attention projections
'gate_proj', 'up_proj', 'down_proj'], # and FFN layers
bias='none', # do not adapt bias parameters
)
# Wrap the model with LoRA adapters
from peft import get_peft_model, prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model) # prepares quantized model for training
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 3,752,071,168 || trainable%: 0.11%Preparing the Dataset
The SFTTrainer expects datasets in a specific format. The simplest format is a dataset with a single text column containing the fully formatted prompt + response string. Use the tokenizer's chat template to format conversation examples consistently. The SFTTrainer handles tokenization, batching, and loss masking (computing loss only on the assistant's responses, not on the input prompt).
from datasets import Dataset
import json
def load_and_format_dataset(jsonl_path: str, tokenizer) -> Dataset:
examples = []
with open(jsonl_path) as f:
for line in f:
ex = json.loads(line.strip())
# Apply chat template to format as expected by the model
formatted = tokenizer.apply_chat_template(
ex['messages'],
tokenize=False,
add_generation_prompt=False
)
examples.append({'text': formatted})
return Dataset.from_list(examples)
# Load training and validation datasets
train_dataset = load_and_format_dataset('train.jsonl', tokenizer)
val_dataset = load_and_format_dataset('validation.jsonl', tokenizer)
print(f'Train examples: {len(train_dataset)}')
print(f'Validation examples: {len(val_dataset)}')
print('Sample formatted text:')
print(train_dataset[0]['text'][:300])Training with SFTTrainer
The SFTTrainer from TRL wraps the Hugging Face Trainer with supervised fine-tuning defaults. Configure it with training hyperparameters: number of epochs (1-3 is usually sufficient for instruction fine-tuning), batch size, gradient accumulation steps (to simulate larger batches with limited memory), learning rate (2e-4 is a common starting point), and the output directory for checkpoints.
from trl import SFTTrainer, SFTConfig
training_args = SFTConfig(
output_dir='./fine-tuned-model',
num_train_epochs=2, # 2-3 epochs for instruction fine-tuning
per_device_train_batch_size=4, # increase if GPU memory allows
gradient_accumulation_steps=4, # effective batch size = 4*4 = 16
learning_rate=2e-4, # typical LoRA learning rate
warmup_ratio=0.05, # warmup for 5% of steps
lr_scheduler_type='cosine', # cosine decay learning rate schedule
logging_steps=10,
eval_strategy='steps',
eval_steps=50, # evaluate on validation set every 50 steps
save_steps=100,
max_seq_length=2048, # max token length per example
fp16=True, # mixed precision training
report_to='none' # or 'wandb' for experiment tracking
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=val_dataset,
)
print('Starting LoRA fine-tuning...')
trainer.train()Monitoring Training Progress
During training, watch two key metrics: training loss should decrease steadily. validation loss should decrease at first, then plateau or slightly increase (overfitting). If validation loss increases significantly while training loss continues decreasing, stop training early — the model is memorizing training examples rather than generalizing. The optimal stopping point is just before validation loss starts rising.
# Reading training logs
# Training step logs look like:
# {'loss': 1.4523, 'grad_norm': 0.85, 'learning_rate': 0.0002, 'epoch': 0.2, 'step': 20}
# {'loss': 1.2341, 'grad_norm': 0.72, 'learning_rate': 0.00018, 'epoch': 0.4, 'step': 40}
# Validation results look like:
# {'eval_loss': 1.1823, 'eval_runtime': 12.3, 'eval_samples_per_second': 8.1, 'step': 50}
# {'eval_loss': 1.0923, 'eval_runtime': 12.1, 'eval_samples_per_second': 8.3, 'step': 100}
# {'eval_loss': 1.1234, 'eval_runtime': 12.4, 'eval_samples_per_second': 8.2, 'step': 150}
# ^ validation loss went UP at step 150 - overfitting starting
# Load the best checkpoint (lowest validation loss)
from transformers import TrainerCallback
print('Best model checkpoint is saved automatically by SFTTrainer (load_best_model_at_end=True)')Saving LoRA Adapters and Merging
After training, save the LoRA adapter weights separately from the base model. The adapter is tiny (a few MB to a few hundred MB) and can be applied to the base model at inference time. For production deployment, you can also merge the LoRA weights into the base model, creating a single model file that does not require the PEFT library at inference time — this reduces inference overhead.
# Save only the LoRA adapter (tiny - typically 10-100MB)
model.save_pretrained('./lora-adapter-only')
tokenizer.save_pretrained('./lora-adapter-only')
# Load the adapter for inference (requires base model + peft)
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.float16, device_map='auto')
model_with_adapter = PeftModel.from_pretrained(base_model, './lora-adapter-only')
# Alternative: Merge adapter into base model (no PEFT needed at inference)
print('Merging LoRA weights into base model...')
merged_model = model_with_adapter.merge_and_unload() # creates a regular model
merged_model.save_pretrained('./merged-model')
print('Merged model saved - can be used with standard transformers, no PEFT needed')Running Inference on the Fine-Tuned Model
Test your fine-tuned model on a set of held-out prompts before declaring success. Compare outputs side-by-side with the base model on the same prompts to verify that the fine-tuning achieved its goals. Check both the cases it should handle better (the target task) and cases it should still handle well (general tasks that you do not want to have degraded).
def generate(model, tokenizer, prompt: str, max_new_tokens=512) -> str:
inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.1,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
# Decode only the new tokens (not the input prompt)
new_tokens = outputs[0][inputs['input_ids'].shape[1]:]
return tokenizer.decode(new_tokens, skip_special_tokens=True)
# Compare base vs fine-tuned
test_prompt = 'Extract JSON from: "Alice Johnson, 28, software engineer in NYC since 2021"'
print('=== BASE MODEL ===')
print(generate(base_model, tokenizer, test_prompt))
print('\n=== FINE-TUNED MODEL ===')
print(generate(merged_model, tokenizer, test_prompt))LoRA Key Hyperparameters Summary
To tune LoRA for your use case, start with these defaults and adjust one at a time. r=8 is a safe starting point — increase to 16 or 32 if the model needs more expressive capacity. lora_alpha = 2*r is a stable choice. Learning rate 2e-4 works for most instruction fine-tuning; decrease to 1e-4 if you see unstable training loss. Epochs 1-3: stop when validation loss stops decreasing.
When to Use OpenAI Fine-Tuning Instead
Hugging Face PEFT LoRA requires a GPU. If you do not have GPU infrastructure, OpenAI's fine-tuning API is a managed alternative that handles the training infrastructure for you. Upload your JSONL file, call the API to start a training run, and receive a fine-tuned model ID you can use in API calls. OpenAI fine-tuning supports GPT-4o-mini and GPT-3.5-turbo. It is more expensive per token but eliminates infrastructure management entirely.
from openai import OpenAI
client = OpenAI()
# Upload training file
train_file = client.files.create(
file=open('train.jsonl', 'rb'),
purpose='fine-tune'
)
val_file = client.files.create(
file=open('validation.jsonl', 'rb'),
purpose='fine-tune'
)
# Create fine-tuning job
job = client.fine_tuning.jobs.create(
training_file=train_file.id,
validation_file=val_file.id,
model='gpt-4o-mini', # base model to fine-tune
hyperparameters={
'n_epochs': 3,
'batch_size': 'auto',
'learning_rate_multiplier': 'auto'
}
)
print(f'Fine-tuning job created: {job.id}')
# Monitor: client.fine_tuning.jobs.retrieve(job.id)
# Use: client.chat.completions.create(model=job.fine_tuned_model, ...)Quick Check
Test your understanding of LoRA fine-tuning from this lesson.
Lesson Recap
In this lesson you learned: LoRA reduces fine-tuning GPU requirements by 10-50x by training only small low-rank adapter matrices instead of all model parameters, QLoRA (4-bit quantization + LoRA) makes 7B model fine-tuning accessible on consumer GPUs with ~12GB VRAM, and the SFTTrainer from TRL provides a high-level API that handles tokenization, batching, loss masking, and checkpoint saving. Next up we evaluate and deploy the fine-tuned model.
Frequently asked questions
Is the “LoRA Fine-Tuning with Hugging Face PEFT” lesson free?
Yes — the full text of “LoRA Fine-Tuning with Hugging Face PEFT” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “LoRA Fine-Tuning with Hugging Face PEFT”?
Configure LoRA rank, alpha, and target modules, run supervised fine-tuning with the TRL SFTTrainer, monitor training loss, and save merged and adapter-only checkpoints. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “LoRA Fine-Tuning with Hugging Face PEFT” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- When Fine-Tuning Beats Prompting
- Preparing a High-Quality Training Dataset
- LoRA Fine-Tuning with Hugging Face PEFT
- Evaluating and Deploying Your Fine-Tuned Model