---

title: Ollama for Enterprise — Production Guide description: Enterprise Ollama deployment: multi-instance clustering, GPU sharing, RBAC, CI/CD integration, monitoring, and cost optimization. Original ops tutorial. tags: [ai, ollama, enterprise, llm, ops] lang: en date: 2026-10-23 original: true faq: - q: "How to deploy Ollama in a multi-team enterprise?" a: "Use Kubernetes with GPU node pools, implement RBAC with Ollama's native auth or OAuth proxy, set resource quotas per team, and use a shared model registry. Consider Ollama Gateway for centralized access control." - q: "How to share GPU resources across multiple Ollama instances?" a: "Use NVIDIA MPS (Multi-Process Service) for GPU time-slicing, or deploy Ollama with --num-parallel=N to handle concurrent requests on shared GPUs. For strict isolation, use GPU dedlication with MIG (Multi-Instance GPU) on A100/H100." - q: "How to integrate Ollama into CI/CD pipelines?" a: "Pull required models in Docker build stage, use ollama list to verify, run inference tests in CI, and cache model layers in container registry. Add a 'model health check' step that validates response quality before deployment." - q: "How to monitor Ollama at enterprise scale?" a: "Export Ollama metrics to Prometheus (enable /api/metrics), track: active models, GPU utilization, request latency, token throughput, error rates. Set up Grafana dashboards and PagerDuty alerts for SLA breaches." - q: "How to optimize Ollama costs in production?" a: "Right-size models per use case (1B for simple tasks, 7B for complex), implement request caching, use quantized models, set concurrency limits, and schedule off-peak batch processing. Target: < $0.001 per inference."

---

Ollama for Enterprise — Production Guide

Use Case: Deploy and operate Ollama at enterprise scale — multi-instance clustering, GPU sharing, access control, monitoring, and cost optimization for production AI workloads.

---

1. Enterprise Architecture

1.1 Reference Architecture

                        ┌──────────────────┐
                        │   Load Balancer   │
                        │   (Nginx/Traefik) │
                        └────────┬─────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              │                  │                  │
              ▼                  ▼                  ▼
    ┌───────────────┐  ┌───────────────┐  ┌───────────────┐
    │  Ollama Node  │  │  Ollama Node  │  │  Ollama Node  │
    │  (GPU: A100)  │  │  (GPU: A100)  │  │  (GPU: A100)  │
    │               │  │               │  │               │
    │  Model Cache  │  │  Model Cache  │  │  Model Cache  │
    └───────┬───────┘  └───────┬───────┘  └───────┬───────┘
            │                  │                  │
            └──────────────────┼──────────────────┘
                               │
                    ┌──────────┴──────────┐
                    │   Shared Storage     │
                    │   (NFS/EBS/GCS)     │
                    │   /models, /data     │
                    └─────────────────────┘

1.2 Kubernetes Deployment

# ollama-cluster.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai-platform
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: nvidia.com/gpu
                  operator: Exists
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          env:
            - name: OLLAMA_HOST
              value: "0.0.0.0:11434"
            - name: OLLAMA_ORIGINS
              value: "https://app.example.com"
            - name: OLLAMA_NUM_PARALLEL
              value: "4"
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "32Gi"
              cpu: "16"
            requests:
              memory: "16Gi"
              cpu: "8"
          volumeMounts:
            - name: model-cache
              mountPath: /root/.ollama
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: ollama-models-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ai-platform
spec:
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434
  type: ClusterIP

---

2. GPU Sharing Strategies

2.1 Time-Slicing with NVIDIA MPS

# Enable MPS on host
nvidia-ctk mps enable
nvidia-ctk mps start

# Each Ollama container gets a slice
OLLAMA_NUM_PARALLEL=8 ollama serve

2.2 GPU Dedication with MIG (A100/H100)

# Create MIG partitions
nvidia-ml-smi --set-gpu-identity-string MIG-GPU-uuid
nvidia-ml-smi --mig-config 1g.5gb  # 1 GPU, 5GB VRAM per partition

# Each Ollama instance gets isolated GPU
docker run --gpus '"device=0"' ollama/ollama:latest

2.3 Concurrency Control

environment:
  - OLLAMA_NUM_PARALLEL=4      # Max parallel requests per GPU
  - OLLAMA_MAX_LOADED_MODELS=2 # Max models loaded simultaneously
  - OLLAMA_MAX_QUEUE=100       # Request queue size

---

3. Access Control and RBAC

3.1 JWT Authentication

from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import jwt, JWTError

app = FastAPI()
security = HTTPBearer()
SECRET_KEY = os.environ["OLLAMA_JWT_SECRET"]
ALGORITHM = "HS256"

def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
    try:
        payload = jwt.decode(credentials.credentials, SECRET_KEY, algorithms=[ALGORITHM])
        return payload
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid token")

@app.get("/api/tags")
def list_models(token: dict = Depends(verify_token)):
    return {"models": ["llama3.2:3b", "qwen2.5:7b"]}

@app.post("/api/generate")
def generate(prompt: str, token: dict = Depends(verify_token)):
    # Check permissions
    if token.get("team") not in ALLOWED_TEAMS:
        raise HTTPException(status_code=403, detail="Team not allowed")
    return {"response": llm.generate(prompt, model=token.get("default_model"))}

3.2 Team-Based Quotas

TEAM_QUOTAS = {
    "data-science": {"max_tokens_per_day": 1_000_000, "max_concurrent": 10},
    "engineering": {"max_tokens_per_day": 500_000, "max_concurrent": 5},
    "research": {"max_tokens_per_day": 10_000_000, "max_concurrent": 20},
}

def check_quota(team: str, tokens_used: int) -> bool:
    quota = TEAM_QUOTAS.get(team)
    if not quota:
        return False
    daily_usage = get_daily_usage(team)
    return daily_usage + tokens_used <= quota["max_tokens_per_day"]

---

4. CI/CD Integration

4.1 Dockerfile with Model Pre-loading

FROM ollama/ollama:latest

# Pre-pull required models during build
RUN ollama pull llama3.2:3b && \
    ollama pull qwen2.5:7b && \
    ollama pull bge-m3

# Copy application
COPY . /app
WORKDIR /app

# Health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://localhost:11434/api/tags || exit 1

CMD ["ollama", "serve"]

4.2 CI Pipeline

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - docker build -t ollama-app:${CI_COMMIT_SHA} .
    - docker push ollama-app:${CI_COMMIT_SHA}

test:
  stage: test
  services:
    - name: ollama:latest
      alias: ollama
  script:
    - curl http://ollama:11434/api/tags
    - python tests/test_llm_quality.py

deploy:
  stage: deploy
  script:
    - kubectl set image deployment/ollama ollama=ollama-app:${CI_COMMIT_SHA}

---

5. Monitoring and Observability

5.1 Metrics Export

# Enable Ollama metrics
# Add to ollama service configuration
OLLAMA_KEEP_ALIVE=-1  # Keep models loaded indefinitely

# Export to Prometheus via exporter
docker run -d \
  -p 9103:9103 \
  -e OLLAMA_URL=http://ollama:11434 \
  promcommunity/ollama-exporter

5.2 Key Metrics to Track

Metric	Description	Alert Threshold
ollama_gpu_utilization	GPU compute utilization	< 50% or > 95%
ollama_request_latency_seconds	Request latency P99	> 5s
ollama_tokens_per_second	Token throughput	< 10 tok/s
ollama_active_models	Number of loaded models	> max_loaded_models
ollama_queue_depth	Pending requests	> 50
ollama_error_rate	4xx/5xx rate	> 5%

5.3 Grafana Dashboard

{
  "dashboard": {
    "title": "Ollama Enterprise Dashboard",
    "panels": [
      {
        "title": "GPU Utilization",
        "type": "graph",
        "targets": [{"expr": "ollama_gpu_utilization"}]
      },
      {
        "title": "Request Latency P99",
        "type": "graph",
        "targets": [{"expr": "histogram_quantile(0.99, rate(ollama_request_latency_seconds_bucket[5m]))"}]
      },
      {
        "title": "Token Throughput",
        "type": "graph",
        "targets": [{"expr": "rate(ollama_tokens_generated_total[5m])"}]
      }
    ]
  }
}

---

6. Cost Optimization

6.1 Model Right-Sizing

MODEL_RECOMMENDATIONS = {
    "simple_qa": "llama3.2:1b",      # 1B params, < 2GB VRAM
    "code_generation": "qwen2.5:7b", # 7B params, ~8GB VRAM
    "complex_reasoning": "deepseek-r1:8b", # 8B params, ~10GB VRAM
    "high_quality": "llama3.1:70b",  # 70B params, ~40GB VRAM
}

def select_model(task_type: str) -> str:
    return MODEL_RECOMMENDATIONS.get(task_type, "llama3.2:3b")

6.2 Caching Strategy

import hashlib
import redis

r = redis.Redis(host="redis", port=6379)

def cached_generate(model: str, prompt: str, **kwargs) -> str:
    """Cache generation results to avoid redundant computation"""
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()[:16]
    cache_key = f"ollama:{model}:{prompt_hash}"

    cached = r.get(cache_key)
    if cached:
        return cached.decode()

    # Generate and cache
    response = llm.generate(model=model, prompt=prompt, **kwargs)
    r.setex(cache_key, 3600, response)  # 1 hour TTL

    return response

6.3 Cost Tracking

COST_PER_TOKEN = {
    "llama3.2:1b": 0.0,      # Local: free
    "llama3.2:3b": 0.0,
    "qwen2.5:7b": 0.0,
}

def track_cost(model: str, prompt_tokens: int, completion_tokens: int) -> float:
    rate = COST_PER_TOKEN.get(model, 0.0)
    return (prompt_tokens + completion_tokens) * rate

---

7. Further Reading

• Ollama Enterprise Guide
• Kubernetes GPU Scheduling
• NVIDIA MPS Documentation

---

FAQ

How to deploy Ollama in a multi-team enterprise? Use Kubernetes with GPU node pools, implement RBAC with Ollama's native auth or OAuth proxy, set resource quotas per team, and use a shared model registry. Consider Ollama Gateway for centralized access control.

How to share GPU resources across multiple Ollama instances? Use NVIDIA MPS (Multi-Process Service) for GPU time-slicing, or deploy Ollama with --num-parallel=N to handle concurrent requests on shared GPUs. For strict isolation, use GPU dedlication with MIG (Multi-Instance GPU) on A100/H100.

How to integrate Ollama into CI/CD pipelines? Pull required models in Docker build stage, use ollama list to verify, run inference tests in CI, and cache model layers in container registry. Add a 'model health check' step that validates response quality before deployment.

How to monitor Ollama at enterprise scale? Export Ollama metrics to Prometheus (enable /api/metrics), track: active models, GPU utilization, request latency, token throughput, error rates. Set up Grafana dashboards and PagerDuty alerts for SLA breaches.

How to optimize Ollama costs in production? Right-size models per use case (1B for simple tasks, 7B for complex), implement request caching, use quantized models, set concurrency limits, and schedule off-peak batch processing. Target: < $0.001 per inference.

---