Overview
Fine-tune large language models efficiently using Parameter-Efficient Fine-Tuning (PEFT) methods. Train 7B to 70B parameter models on consumer GPUs (16-48 GB VRAM) using LoRA, QLoRA, and 25+ adapter methods from the Hugging Face PEFT library. Avoid the cost and hardware requirements of full fine-tuning while achieving comparable results. Written for transformers 5.x, TRL 1.x and PEFT 0.21: several arguments older tutorials use (torch_dtype, warmup_ratio, max_seq_length on the trainer) were renamed or moved.
Instructions
When a user asks to fine-tune a model, determine the approach:
Task A: Set up the environment
pip install torch transformers datasets peft accelerate trl
pip install bitsandbytes # only for QLoRA; needs an NVIDIA GPU (also AMD/Intel builds exist)
hf auth login # gated models such as Llama need accepted license + token
Verify GPU availability:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
Flash Attention is optional: the default sdpa attention is fast enough on recent PyTorch, and flash-attn needs a long compile.
Task B: Fine-tune with LoRA (16+ GB VRAM)
TRL's SFTTrainer accepts a model name and a peft_config, so you do not wrap the model yourself:
import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
model_name = "meta-llama/Llama-3.1-8B" # gated: accept the license on Hugging Face first
# 1. Dataset in prompt-completion form (loss is computed on the completion only)
raw = load_dataset("json", data_files="support_tickets.jsonl", split="train")
dataset = raw.map(
lambda ex: {
"prompt": [{"role": "user", "content": ex["question"]}],
"completion": [{"role": "assistant", "content": ex["answer"]}],
},
remove_columns=raw.column_names,
).train_test_split(test_size=0.05)
# 2. LoRA configuration
lora_config = LoraConfig(
r=16, # rank (8-64; higher = more capacity)
lora_alpha=32, # scaling factor (start at 2x rank)
target_modules="all-linear", # every linear layer except the output head
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
# 3. Training arguments
args = SFTConfig(
output_dir="./lora-output",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
bf16=True, # Ampere or newer; use fp16=True on older GPUs
logging_steps=10,
eval_strategy="epoch",
save_strategy="epoch",
warmup_steps=0.03, # a float below 1 is a ratio (warmup_ratio was removed)
lr_scheduler_type="cosine",
max_length=2048, # was max_seq_length
gradient_checkpointing=True,
model_init_kwargs={"dtype": torch.bfloat16}, # was torch_dtype
)
trainer = SFTTrainer(
model=model_name,
args=args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
peft_config=lora_config,
)
trainer.model.print_trainable_parameters() # about 0.5-1% of the weights
trainer.train()
trainer.save_model("./lora-adapter") # saves only the adapter (tens of MB)
SFTTrainer also accepts plain {"text": ...} rows and conversational {"messages": [...]} rows. Add assistant_only_loss=True to SFTConfig to train only on assistant turns of a conversational dataset. Pass packing=True to pack short examples together for speed.
Task C: Fine-tune with QLoRA (8+ GB VRAM)
QLoRA loads the frozen base model in 4-bit and trains LoRA adapters on top. Reuse Task B and add a quantization config through model_init_kwargs:
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
args = SFTConfig(
output_dir="./qlora-output",
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=2e-4,
bf16=True,
gradient_checkpointing=True,
model_init_kwargs={"quantization_config": bnb_config, "dtype": torch.bfloat16, "device_map": "auto"},
)
# SFTTrainer(model=..., args=args, peft_config=lora_config, ...) exactly as in Task B
If you build the model yourself instead, call peft.prepare_model_for_kbit_training(model) before get_peft_model. Rough VRAM for weights plus training state at a short context: a 7-8B model fits in about 6-10 GB, 13B in about 12-16 GB, 70B in about 40-48 GB. Longer sequences and larger batches need more.
Task D: Merge and export the fine-tuned model
Merge into a model loaded in 16-bit, not the 4-bit one:
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "./lora-adapter")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B").save_pretrained("./merged-model")
Convert to GGUF for Ollama or llama.cpp with the script in a llama.cpp checkout (the old llama_cpp.convert module does not exist):
python convert_hf_to_gguf.py ./merged-model --outfile model-q8_0.gguf --outtype q8_0
convert_lora_to_gguf.py converts the adapter alone if you prefer to keep it separate.
Task E: Prepare a custom dataset
from datasets import Dataset, load_dataset
tickets = [
{"question": "Where can I download my March invoice?", "answer": "Open Billing > Invoices and choose March 2026."},
{"question": "How do I reset two-factor authentication?", "answer": "Use the recovery codes, or ask an admin to reset 2FA for you."},
]
dataset = Dataset.from_list(tickets).train_test_split(test_size=0.1)
# Or load from a JSONL file with one object per line
dataset = load_dataset("json", data_files="training_data.jsonl", split="train")
Convert to prompt/completion (or messages) columns as in Task B. The tokenizer needs a chat template for conversational data; base models without one need SFTConfig(chat_template_path=...).
Examples
Example 1: Fine-tune Llama 3.1 8B for customer support
User request: "Fine-tune Llama 8B on our support ticket data, I have a 12 GB GPU"
Use Task C (QLoRA) with support_tickets.jsonl mapped to prompt/completion. Settings: 3 epochs, learning_rate=2e-4, r=16, max_length=1024. Watch eval_loss after each epoch and keep the best checkpoint. Result: an adapter of tens of MB in ./lora-adapter, loadable with PeftModel.from_pretrained. Training time depends on dataset size and GPU: measure on 200 steps before committing to a full run.
Example 2: Domain-adapt a model for medical text
User request: "Adapt Mistral 7B to understand medical terminology"
Run continued pre-training with LoRA on a plain-text medical corpus ({"text": ...} rows, r=32), then instruction-tune on medical QA pairs with the Task B setup. Evaluate on held-out questions and compare with the base model before trusting answers; a fine-tune does not make clinical output reliable.
Example 3: Fine-tune a 70B model with QLoRA on 2x A100 80 GB
User request: "Fine-tune Llama 70B on our internal documents"
Use QLoRA with device_map="auto" to shard the 4-bit weights across both GPUs. Set per_device_train_batch_size=1, gradient_accumulation_steps=16, gradient_checkpointing=True. Run one short job first to measure tokens per second and extrapolate the total time.
Guidelines
- Start with QLoRA if VRAM is limited; published results show quality close to 16-bit LoRA, with slower steps from dequantization.
- Use rank
r=16as a default. Increase tor=32-64for complex domain adaptation; decrease tor=8for simple style tuning. - Start with
lora_alpha = 2 * r; usetarget_modules="all-linear"unless you have a reason to limit it. - LoRA tolerates a higher learning rate than full fine-tuning (1e-4 to 2e-4); lower it if loss spikes.
- Monitor training loss: it should decrease steadily. Evaluate on a held-out split every epoch to catch overfitting.
- On CPU-only machines TRL 1.x training failed in a test (its fused loss kernel is GPU-only); use a CUDA GPU or a hosted notebook.
- Pin library versions in a lockfile: transformers, TRL and PEFT change argument names between majors.
- Clean, well-formatted training data matters more than quantity. 1,000 high-quality examples often beat 10,000 noisy ones.
- Check the base model license before redistributing a merged model; adapters alone are usually easier to share.