---

title: LoRA Fine-Tuning for Production LLMs — Ops Guide description: LoRA fine-tuning complete guide: adapter training, GPU resource planning, model merging, deployment strategies, and 5 production pitfalls. Original ops tutorial. tags: [ai, lora, finetuning, llm, ops] lang: en date: 2026-10-23 original: true faq: - q: "How much GPU memory do I need for LoRA fine-tuning?" a: "For 7B models: 1× A100 80GB or 2× RTX 4090. For 70B models: 4× A100 80GB minimum. Use qlora (4-bit quantization) to reduce to 1× A100 40GB for 7B models." - q: "How to choose LoRA rank (r) and alpha?" a: "Start with r=16, alpha=32 (alpha=2*r is a good rule). Higher rank = more capacity but slower training. Lower rank = faster but may underfit. Typical range: r=8-64." - q: "When to merge adapters vs keeping them separate?" a: "Merge when deploying to production for simplicity and latency. Keep separate when you need to switch between tasks dynamically or A/B test different adapters." - q: "How to prevent catastrophic forgetting during fine-tuning?" a: "Use a mixed dataset (80% task-specific, 20% general), set lower learning rate (1e-5 to 5e-5), enable prompt tuning alongside LoRA, and monitor validation loss." - q: "What's the difference between LoRA and full fine-tuning?" a: "LoRA trains only low-rank adaptation matrices while freezing base model weights. Full fine-tuning updates all parameters. LoRA is 10-100x faster and uses 10-100x less memory, with comparable quality for most tasks."

---

LoRA Fine-Tuning for Production LLMs — Ops Guide

Use Case: Fine-tune open-source LLMs (Llama, Qwen, DeepSeek) for specific tasks using LoRA (Low-Rank Adaptation) — cost-effective, resource-efficient, production-ready.

---

1. What is LoRA and Why Use It

LoRA (Low-Rank Adaptation) freezes pre-trained model weights and injects trainable rank decomposition matrices into each layer. This dramatically reduces trainable parameters while preserving model capability.

Why LoRA over full fine-tuning:

Aspect	Full Fine-Tuning	LoRA
GPU Memory	4× model size	~2GB for 7B model
Training Time	Days	Hours
Storage	Full model weights	Small adapters (100MB)
Multi-task	One model per task	Multiple adapters per model
Quality	Slightly better	Near-parity for most tasks

---

2. Environment Setup

2.1 Docker Environment

FROM nvidia/cuda:12.1-devel-ubuntu22.04

RUN apt-get update && apt-get install -y \
    python3.10 python3.10-dev python3-pip git curl \
    && rm -rf /var/lib/apt/lists/*

RUN pip3 install --upgrade pip \
    && pip3 install \
    transformers==4.40.0 \
    peft==0.11.1 \
    accelerate==0.28.0 \
    datasets==2.17.0 \
    trl==0.8.1 \
    bitsandbytes==0.43.1 \
    flash-attn==2.5.8 \
    torch==2.2.0

WORKDIR /workspace

2.2 Build and Run

docker build -t lora-finetune .
docker run --gpus all -it -v /data/models:/models -v /data/output:/output lora-finetune

---

3. LoRA Configuration

3.1 Key Parameters

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                    # Rank of decomposition
    lora_alpha=32,           # Scaling factor (typically 2*r)
    lora_dropout=0.1,        # Dropout for regularization
    bias="none",             # Train only LoRA biases
    task_type="CAUSAL_LM",   # Task type
    target_modules=[         # Layers to apply LoRA
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    inference_mode=False     # Set True for inference-only
)

3.2 Quantization Options

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit quantization (Q LoRA)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",      # Normalized float 4
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True  # Double quantization
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=quantization_config,
    device_map="auto"
)

---

4. Training Pipeline

4.1 Dataset Preparation

from datasets import Dataset
import json

# Load and preprocess data
def format_example(example):
    return {
        "text": f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example.get('input', '')}\n\n### Response:\n{example['output']}"
    }

# Example: JSONL format
# {"instruction": "Explain Kubernetes deployments", "input": "", "output": "A Kubernetes deployment..."}
dataset = Dataset.from_json("/data/train.jsonl")
dataset = dataset.map(format_example)

4.2 Training Configuration

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="/output/model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    warmup_steps=100,
    logging_steps=10,
    save_strategy="epoch",
    eval_strategy="epoch",
    fp16=True,                  # Mixed precision
    report_to="tensorboard",
    overwrite_output_dir=True,
)

4.3 Execute Training

from transformers import Trainer
from peft import get_peft_model

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Show trainable parameter count

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
)

trainer.train()
trainer.save_model("/output/model")

---

5. Model Merging and Deployment

5.1 Merge Adapter with Base Model

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model and adapter
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
model = PeftModel.from_pretrained(base_model, "/output/model")
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("/output/merged-model")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokenizer.save_pretrained("/output/merged-model")

5.2 Deployment with vLLM

# Serve merged model
vllm serve /output/merged-model \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 4096

5.3 Keep Adapter Separate (Multi-Task)

# Load base model once, swap adapters dynamically
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")

def get_task_model(task: str):
    adapter_paths = {
        "support": "/output/adapters/support",
        "code": "/output/adapters/code",
        "analysis": "/output/adapters/analysis"
    }
    return PeftModel.from_pretrained(base_model, adapter_paths[task])

---

6. Production Pitfalls (5 Common Issues)

Issue 1: Out of Memory During Training

Symptom: torch.cuda.OutOfMemoryError

Fix:

# Enable gradient checkpointing
training_args.gradient_checkpointing = True

# Reduce batch size
training_args.per_device_train_batch_size = 2
training_args.gradient_accumulation_steps = 16

# Use 4-bit quantization (QLoRA)
bnb_4bit_quant_type="nf4"

---

Issue 2: Model Collapses (Degraded Output)

Symptom: Model generates gibberish or repeats tokens

Causes: - Learning rate too high - Training on too few examples - Catastrophic forgetting

Fix:

# Lower learning rate
learning_rate=1e-5  # Instead of 2e-4

# Add more training data (minimum 500 examples)
# Add general domain data to prevent forgetting

---

Issue 3: Adapter Too Large

Symptom: Adapter file > 1GB for a 7B model

Fix:

# Reduce rank
lora_config = LoraConfig(r=8, lora_alpha=16)  # Instead of r=16, alpha=32

# Target fewer modules
target_modules=["q_proj", "v_proj"]  # Only attention layers

---

Issue 4: Slow Inference After Merging

Symptom: Merged model is slower than base model

Fix: - Ensure float16 precision is maintained - Use vLLM for serving (not HuggingFace Transformers directly) - Quantize to INT8/FP8 if latency is critical

---

Issue 5: Evaluation Score Drops After Deployment

Symptom: Training accuracy is high but production performance is poor

Fix: - Use held-out validation set for evaluation - Test with real production prompts, not synthetic ones - Implement A/B testing between base and fine-tuned model

---

7. Monitoring Fine-Tuning Quality

7.1 Training Metrics

import matplotlib.pyplot as plt

# Plot training loss
losses = trainer.state.log_history
plt.plot([l["loss"] for l in losses if "loss" in l])
plt.title("Training Loss")
plt.xlabel("Step")
plt.ylabel("Loss")
plt.savefig("/output/loss_curve.png")

7.2 Evaluation Checklist

• Perplexity on validation set < 10
• No repetition in generated text
• Task-specific accuracy > 85%
• General knowledge preserved (test with random prompts)
• Inference latency < 2x base model

---

8. Further Reading

• LoRA Paper
• QLoRA Paper
• Hugging Face PEFT Documentation

---

FAQ

How much GPU memory do I need for LoRA fine-tuning? For 7B models: 1× A100 80GB or 2× RTX 4090. For 70B models: 4× A100 80GB minimum. Use qlora (4-bit quantization) to reduce to 1× A100 40GB for 7B models.

How to choose LoRA rank (r) and alpha? Start with r=16, alpha=32 (alpha=2*r is a good rule). Higher rank = more capacity but slower training. Lower rank = faster but may underfit. Typical range: r=8-64.

When to merge adapters vs keeping them separate? Merge when deploying to production for simplicity and latency. Keep separate when you need to switch between tasks dynamically or A/B test different adapters.

How to prevent catastrophic forgetting during fine-tuning? Use a mixed dataset (80% task-specific, 20% general), set lower learning rate (1e-5 to 5e-5), enable prompt tuning alongside LoRA, and monitor validation loss.

What's the difference between LoRA and full fine-tuning? LoRA trains only low-rank adaptation matrices while freezing base model weights. Full fine-tuning updates all parameters. LoRA is 10-100x faster and uses 10-100x less memory, with comparable quality for most tasks.

---