title: Deploying Local LLMs with Ollama — Production Guide description: Complete guide to deploying LLMs locally with Ollama: Docker Compose setup, NVIDIA GPU acceleration, model management, API debugging, and 8 common troubleshooting cases. Original ops guide. tags: [ai, ollama, llm, ops] lang: en date: 2026-10-23 original: true faq: - q: "What to do when Ollama reports 'no visible GPU'?" a: "Check if nvidia-container-toolkit is installed and whether the Docker runtime is set to nvidia. Run docker run --rm --gpus all nvidia/cuda:12.6-base nvidia-smi to verify GPU visibility." - q: "Model download is extremely slow or times out?" a: "Configure a domestic mirror by adding \"proxy\": \"https://ollama.myhloli.com\" to /etc/ollama/config.json. Alternatively, manually download .gguf files into the ~/.ollama/models/ directory." - q: "Ollama API returns 401 / 403?" a: "Ollama has no authentication by default. If you need auth, use a reverse proxy (Nginx/Caddy) with Basic Auth or JWT, or configure the OLLAMA_ORIGINS environment variable in Docker Compose to restrict sources." - q: "How to switch the default Ollama model?" a: "Run ollama list to view downloaded models, ollama run <model> to start interactive mode, ollama pull <model> to download new models. Specify the model field in API calls to switch." - q: "Ollama memory usage causes OOM?" a: "Limit concurrent requests, use smaller models (e.g., llama3.2:1b), enable GPU offload, or increase swap. Check dmesg | grep -i oom to locate the trigger point."

Deploying Local LLMs with Ollama — Production Guide

Use Case: Privately deploy LLMs on your own servers or edge devices to prevent data leakage, reduce API costs, and enable offline availability.

1. Quick Deployment (Docker Compose)

# docker-compose.yml
version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama_data:

Start the service:

docker compose up -d
ollama list  # Confirm service is ready

2. NVIDIA GPU Acceleration Setup

2.1 Prerequisites Check

# Verify driver
nvidia-smi

# Verify Docker GPU support
docker run --rm --gpus all nvidia/cuda:12.6-base-ubuntu22.04 nvidia-smi

If the second command returns no output or errors, install nvidia-container-toolkit:

# Ubuntu/Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

2.2 GPU Declaration in Docker Compose

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1          # Number of available GPUs
          capabilities: [gpu]

Note: If using Podman, you need to additionally set --device /dev/dri:/dev/dri --device /dev/nvidia*:/dev/nvidia*.

3. Model Download and Management

3.1 Common Models Quick Reference

Model	Parameters	Min VRAM	Use Case
`llama3.2:1b`	1B	2 GB	Edge devices, low latency
`llama3.2:3b`	3B	4 GB	General conversation
`qwen2.5:7b`	7B	8 GB	Strong Chinese understanding
`deepseek-r1:8b`	8B	10 GB	Reasoning enhancement
`llama3.1:70b`	70B	40 GB	High-quality output

3.2 Download Acceleration (Domestic Mirrors)

Method 1: Environment Variables

export OLLAMA_MODELS=/data/ollama/models
ollama pull llama3.2:3b

Method 2: Mirror Configuration (Recommended)

Edit /etc/ollama/config.json:

{
  "proxy": "https://ollama.myhloli.com"
}

Method 3: Manual GGUF Import

# Download .gguf file to ~/.ollama/models/blobs/
# Directory structure: ~/.ollama/models/blobs/sha256-<hash>
ollama create my-model -f Modelfile

4. API Debugging

4.1 Basic Calls

# Generate text
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Explain the lifecycle of Docker volumes",
  "stream": false
}'

# Chat endpoint
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    {"role": "user", "content": "What are the reasons a Kubernetes Pod stays in Pending state?"}
  ]
}'

4.2 OpenAI-Compatible Mode

Ollama has a built-in OpenAI API-compatible endpoint:

curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "llama3.2:3b",
  "messages": [{"role": "user", "content": "hello"}]
}'

This feature allows frameworks like LangChain and LlamaIndex to connect directly.

5. Common Troubleshooting (8 Pitfalls)

Issue 1: `no CUDA-capable device is detected`

# Check nvidia runtime
docker info | grep -A5 "Runtime"

# Confirm nvidia-container-toolkit is installed
nvidia-ctk runtime inspect --runtime docker | grep nvidia

Fix: Reconfigure Docker runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Issue 2: Extremely slow inference after model load

Cause: Model is running on CPU instead of GPU.

# Check GPU utilization
nvidia-smi

# Check Ollama logs
docker logs ollama --tail 50 | grep -i gpu

Fix: Ensure the devices configuration in Docker Compose is correct and the host has drivers installed.

Issue 3: Out of memory (OOM during load)

# View current VRAM usage
nvidia-smi

# Check model requirements
ollama show llama3.1:70b --modelfile

Fix: - Switch to a smaller model (7B → 3B → 1B) - Enable quantization: ollama pull llama3.2:3b:q4_K_M - Reduce concurrent request count

Issue 4: API response timeout

Fix: Adjust timeout parameters:

# Set on startup
OLLAMA_NUM_PARALLEL=2 OLLAMA_MAX_LOADED_MODELS=1 ollama serve

Or add to Docker Compose:

environment:
  - OLLAMA_NUM_PARALLEL=2
  - OLLAMA_MAX_LOADED_MODELS=1
  - OLLAMA_HOST=0.0.0.0:11434

Issue 5: Multiple model conflicts

Symptom: Loading multiple large models simultaneously exhausts VRAM.

Fix: Limit concurrent model count:

environment:
  - OLLAMA_MAX_LOADED_MODELS=1
  - OLLAMA_NUM_PARALLEL=1

Issue 6: GPU unavailable in Windows WSL2

Fix: Ensure WSL2 version is ≥ 5.15 and NVIDIA drivers support WSL2. Run nvidia-smi on the Windows side to confirm drivers are working, then run the same command inside WSL2 to verify.

Issue 7: Docker container time desynchronization

Symptom: Some model inference results are abnormal.

Fix:

services:
  ollama:
    # ...
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

Issue 8: Corrupted model files

Symptom: Model cannot be loaded after an interrupted pull.

Fix:

# Remove corrupted model
ollama rm llama3.2:3b

# Re-download
ollama pull llama3.2:3b

6. Production Hardening

6.1 Network Isolation

services:
  ollama:
    networks:
      - ai-internal
    # Do not expose ports to the host
    # Remove ports configuration

networks:
  ai-internal:
    driver: bridge
    internal: true

Provide external access through a reverse proxy (Nginx/Caddy):

location /ai/ {
    proxy_pass http://ollama:11434/;
    proxy_set_header Host $host;
    limit_req zone=ai burst=5;
}

6.2 Resource Limits

services:
  ollama:
    deploy:
      resources:
        limits:
          memory: 16G
          cpus: '8'
        reservations:
          memory: 8G
          cpus: '4'
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

6.3 Health Check

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
  interval: 30s
  timeout: 10s
  retries: 3

7. Toolchain

Tool	Purpose
`ollama` CLI	Model management, API calls
`ollama show <model>`	View model metadata
`ollama list`	List installed models
`ollama cp`	Copy models (cross-instance migration)
LangChain / LlamaIndex	Application development frameworks
Open WebUI	Visual web interface

8. Further Reading

FAQ

What to do when Ollama reports 'no visible GPU'? Check if nvidia-container-toolkit is installed and whether the Docker runtime is set to nvidia. Run docker run --rm --gpus all nvidia/cuda:12.6-base nvidia-smi to verify GPU visibility.

Model download is extremely slow or times out? Configure a domestic mirror by adding "proxy": "https://ollama.myhloli.com" to /etc/ollama/config.json. Alternatively, manually download .gguf files into the ~/.ollama/models/ directory.

Ollama API returns 401 / 403? Ollama has no authentication by default. If you need auth, use a reverse proxy (Nginx/Caddy) with Basic Auth or JWT, or configure the OLLAMA_ORIGINS environment variable in Docker Compose to restrict sources.

How to switch the default Ollama model? Run ollama list to view downloaded models, ollama run <model> to start interactive mode, ollama pull <model> to download new models. Specify the model field in API calls to switch.

Ollama memory usage causes OOM? Limit concurrent requests, use smaller models (e.g., llama3.2:1b), enable GPU offload, or increase swap. Check dmesg | grep -i oom to locate the trigger point.