Skip to content

Migrating Conda to Slim Docker Images

Objective: Transform bloated conda environments into production-ready, slim Docker images. Eliminate conda bloat while maintaining package compatibility and build reproducibility.

When your conda environment is dragging gigabytes of unnecessary packages, you're paying for storage and transfer costs with your soul. This guide shows how to migrate from conda to Docker while keeping images lean and production-ready.

0) Principles (Read Once, Live by Them)

The Five Commandments

  1. Choose the right base image
  2. Prefer python:3.12-slim over continuumio/miniconda3
  3. Use mamba for faster conda operations when needed

  4. Avoid conda in production unless absolutely necessary

  5. Multi-stage always

  6. Build conda environment in heavy stage, copy only artifacts
  7. Use conda-pack for environment portability

  8. Strip conda metadata and unnecessary files

  9. Exploit BuildKit

  10. Use RUN --mount=type=cache for conda/pip operations
  11. Use --mount=type=secret for private channels

  12. Use targeted COPY --link for environment files

  13. Kill the conda bloat

  14. Remove conda metadata, cache, and unused packages
  15. Use --no-deps when possible, install only what you need

  16. Strip conda environment to essentials

  17. Freeze the world

  18. Pin conda packages, Python version, and base image
  19. Use conda-lock for reproducible builds
  20. Lock file for exact reproducibility

Why These Principles: Conda environments are bloated by default. Docker images need to be lean. The key is understanding what your application actually needs and eliminating everything else.

1) Base Image Selection: The Foundation

# Production runtime (slim)
FROM python:3.12-slim

# Development build (heavy)
FROM python:3.12-slim AS builder

Why Python Slim: Minimal overhead, no conda bloat, faster builds, smaller images.

Conda Base (When Necessary)

# Only if you absolutely need conda
FROM continuumio/miniconda3:latest AS conda-builder

# Or use mamba for faster operations
FROM condaforge/mambaforge:latest AS mamba-builder

When to Use Conda: When you have complex scientific dependencies that are difficult to install with pip, or when you need specific conda packages.

Hybrid Approach (Best of Both)

# Use conda for scientific packages, pip for everything else
FROM python:3.12-slim AS builder

# Install conda only for specific packages
RUN apt-get update && apt-get install -y wget && \
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
    bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda && \
    rm Miniconda3-latest-Linux-x86_64.sh

ENV PATH="/opt/conda/bin:$PATH"

Why Hybrid: Get conda's scientific package ecosystem without the full conda environment bloat.

2) Environment Analysis: What You Actually Need

Analyze Current Environment

# Export current conda environment
conda env export > environment.yml

# Analyze package sizes
conda list --show-channel-urls | grep -v "pypi" | sort -k2 -nr

# Find largest packages
conda list --show-channel-urls | grep -v "pypi" | awk '{print $2, $1}' | sort -nr | head -20

Why Analysis: Understanding your dependencies prevents over-engineering and helps identify optimization opportunities.

Identify Essential Packages

# Find packages your code actually imports
python -c "
import sys
import pkg_resources
import importlib

# Get all installed packages
installed = [pkg.project_name for pkg in pkg_resources.working_set]

# Find packages your code imports
imported = set()
for module in sys.modules:
    if '.' in module:
        imported.add(module.split('.')[0])

print('Installed:', len(installed))
print('Imported:', len(imported))
print('Unused:', len(installed) - len(imported))
"

Why This Matters: Many conda environments include packages that are never used. Identifying unused packages saves significant space.

3) Multi-Stage Build Pattern: The Optimization

Complete Dockerfile (Conda → Slim)

# syntax=docker/dockerfile:1.9

############################
# 1) Conda Builder Stage
############################
FROM continuumio/miniconda3:latest AS conda-builder

# Set working directory
WORKDIR /app

# Copy environment file
COPY environment.yml .

# Create conda environment
RUN conda env create -f environment.yml && \
    conda clean -afy

# Activate environment and install additional packages
RUN conda activate myenv && \
    pip install --no-cache-dir additional-packages

# Use conda-pack to create portable environment
RUN conda install -c conda-forge conda-pack && \
    conda-pack -n myenv -o /tmp/env.tar.gz && \
    mkdir /tmp/env && \
    tar -xzf /tmp/env.tar.gz -C /tmp/env && \
    rm /tmp/env.tar.gz

############################
# 2) Python Slim Runtime
############################
FROM python:3.12-slim AS runtime

# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgomp1 \
    libgfortran5 \
    libopenblas0 \
    liblapack3 \
    && rm -rf /var/lib/apt/lists/*

# Copy conda environment
COPY --from=conda-builder /tmp/env /opt/conda

# Set environment variables
ENV PATH="/opt/conda/bin:$PATH" \
    PYTHONPATH="/opt/conda/lib/python3.12/site-packages" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /opt/conda

USER appuser

# Copy application code
WORKDIR /app
COPY --chown=appuser:appuser app/ /app/

# Health check
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
    CMD python -c "import sys; sys.exit(0)"

# Run application
CMD ["python", "app.py"]

Why This Pattern: Conda environment is built in heavy stage, only the essential environment is copied to slim runtime.

Alternative: Mamba for Speed

# syntax=docker/dockerfile:1.9

############################
# 1) Mamba Builder (Faster)
############################
FROM condaforge/mambaforge:latest AS mamba-builder

WORKDIR /app
COPY environment.yml .

# Use mamba for faster package resolution
RUN mamba env create -f environment.yml && \
    mamba clean -afy

# Pack environment
RUN mamba install -c conda-forge conda-pack && \
    conda-pack -n myenv -o /tmp/env.tar.gz && \
    mkdir /tmp/env && \
    tar -xzf /tmp/env.tar.gz -C /tmp/env && \
    rm /tmp/env.tar.gz

############################
# 2) Runtime
############################
FROM python:3.12-slim AS runtime

# Install minimal system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgomp1 \
    && rm -rf /var/lib/apt/lists/*

# Copy environment
COPY --from=mamba-builder /tmp/env /opt/conda
ENV PATH="/opt/conda/bin:$PATH"

# App code
WORKDIR /app
COPY app/ /app/
CMD ["python", "app.py"]

Why Mamba: Faster package resolution, better dependency handling, reduced build times.

4) Environment Optimization: The Slimming

Remove Conda Bloat

# In builder stage, after conda-pack
RUN find /tmp/env -name "*.pyc" -delete && \
    find /tmp/env -name "__pycache__" -type d -exec rm -rf {} + && \
    find /tmp/env -name "*.dist-info" -type d -exec rm -rf {} + && \
    find /tmp/env -name "tests" -type d -exec rm -rf {} + && \
    find /tmp/env -name "*.a" -delete && \
    find /tmp/env -name "*.la" -delete

Why This Cleanup: Removes Python bytecode, cache files, test directories, and static libraries that aren't needed at runtime.

Strip Unnecessary Packages

# Remove packages not needed in production
RUN conda remove --force-remove \
    jupyter \
    notebook \
    ipython \
    spyder \
    matplotlib \
    seaborn \
    && conda clean -afy

Why Package Removal: Development tools and visualization libraries aren't needed in production. Remove them to save space.

Optimize Environment File

# environment.yml (optimized)
name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.12
  - numpy=1.24.0
  - pandas=2.0.0
  - scikit-learn=1.3.0
  - pip
  - pip:
    - fastapi==0.104.1
    - uvicorn==0.24.0

Why Optimization: Pin versions, use conda-forge for better packages, minimize dependencies.

5) Alternative: Pure Python Approach

Convert Conda to Pip

# Export conda environment to requirements.txt
conda list --export > requirements-conda.txt

# Convert to pip-compatible format
pip freeze > requirements-pip.txt

# Or use conda-to-pip converter
pip install conda-to-pip
conda-to-pip environment.yml > requirements.txt

Why Conversion: Pip is faster, more compatible with Docker, and produces smaller images.

Pure Python Dockerfile

# syntax=docker/dockerfile:1.9

############################
# 1) Builder Stage
############################
FROM python:3.12-slim AS builder

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    gfortran \
    libopenblas-dev \
    liblapack-dev \
    && rm -rf /var/lib/apt/lists/*

# Create virtual environment
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install packages
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

############################
# 2) Runtime Stage
############################
FROM python:3.12-slim AS runtime

# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgomp1 \
    libgfortran5 \
    libopenblas0 \
    liblapack3 \
    && rm -rf /var/lib/apt/lists/*

# Copy virtual environment
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# App code
WORKDIR /app
COPY app/ /app/
CMD ["python", "app.py"]

Why Pure Python: Smaller images, faster builds, better compatibility, easier maintenance.

6) Advanced Optimization: The Production Pattern

Conda-Lock for Reproducibility

# Install conda-lock
pip install conda-lock

# Generate lock file
conda-lock -f environment.yml -p linux-64

# Use lock file in Dockerfile
COPY conda-lock.yml .
RUN conda-lock install --prefix /opt/conda conda-lock.yml

Why Conda-Lock: Ensures exact reproducibility across environments and builds.

BuildKit Optimization

# syntax=docker/dockerfile:1.9

FROM continuumio/miniconda3:latest AS builder

# Use BuildKit cache for conda operations
RUN --mount=type=cache,target=/opt/conda/pkgs \
    conda env create -f environment.yml && \
    conda clean -afy

# Use BuildKit cache for pip operations
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir additional-packages

Why BuildKit: Faster rebuilds, better caching, improved build performance.

Multi-Architecture Support

# syntax=docker/dockerfile:1.9

FROM --platform=$BUILDPLATFORM continuumio/miniconda3:latest AS builder

# Platform-specific optimizations
RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
        conda config --set subdir linux-aarch64; \
    fi

# Build for target platform
RUN conda env create -f environment.yml && \
    conda clean -afy

Why Multi-Arch: Support for ARM64, Apple Silicon, and other architectures.

7) Monitoring and Debugging: The Operations

Image Size Analysis

# Analyze image layers
docker history myapp:latest

# Compare image sizes
docker images | grep myapp

# Use dive for detailed analysis
dive myapp:latest

Why Analysis: Understanding image composition helps identify optimization opportunities.

Runtime Verification

# Test conda environment
docker run --rm myapp:latest conda list

# Test Python imports
docker run --rm myapp:latest python -c "import numpy, pandas, sklearn"

# Test application
docker run --rm myapp:latest python app.py

Why Verification: Ensure the optimized environment still works correctly.

Performance Monitoring

# Monitor container resource usage
docker stats myapp:latest

# Check memory usage
docker exec myapp:latest ps aux

# Monitor disk usage
docker exec myapp:latest du -sh /opt/conda

Why Monitoring: Ensure optimization doesn't impact performance.

8) Complete Example: Production Setup

Project Structure

conda-docker-app/
├── Dockerfile
├── environment.yml
├── requirements.txt
├── docker-compose.yml
├── .dockerignore
└── app/
    ├── main.py
    ├── models/
    └── utils/

environment.yml

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.12
  - numpy=1.24.0
  - pandas=2.0.0
  - scikit-learn=1.3.0
  - pip
  - pip:
    - fastapi==0.104.1
    - uvicorn==0.24.0

requirements.txt

# Core dependencies
numpy==1.24.0
pandas==2.0.0
scikit-learn==1.3.0

# Web framework
fastapi==0.104.1
uvicorn==0.24.0

# Optional: additional packages
pydantic==2.5.0

Dockerfile

# syntax=docker/dockerfile:1.9

############################
# 1) Builder Stage
############################
FROM continuumio/miniconda3:latest AS builder

WORKDIR /app
COPY environment.yml .

# Create conda environment
RUN conda env create -f environment.yml && \
    conda clean -afy

# Pack environment
RUN conda install -c conda-forge conda-pack && \
    conda-pack -n myenv -o /tmp/env.tar.gz && \
    mkdir /tmp/env && \
    tar -xzf /tmp/env.tar.gz -C /tmp/env && \
    rm /tmp/env.tar.gz

# Clean up conda bloat
RUN find /tmp/env -name "*.pyc" -delete && \
    find /tmp/env -name "__pycache__" -type d -exec rm -rf {} + && \
    find /tmp/env -name "tests" -type d -exec rm -rf {} + && \
    find /tmp/env -name "*.a" -delete

############################
# 2) Runtime Stage
############################
FROM python:3.12-slim AS runtime

# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgomp1 \
    libgfortran5 \
    libopenblas0 \
    liblapack3 \
    && rm -rf /var/lib/apt/lists/*

# Copy conda environment
COPY --from=builder /tmp/env /opt/conda
ENV PATH="/opt/conda/bin:$PATH" \
    PYTHONPATH="/opt/conda/lib/python3.12/site-packages" \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

# Create non-root user
RUN useradd -m -u 1000 appuser && \
    chown -R appuser:appuser /opt/conda

USER appuser

# Copy application code
WORKDIR /app
COPY --chown=appuser:appuser app/ /app/

# Health check
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
    CMD python -c "import sys; sys.exit(0)"

# Run application
CMD ["python", "app.py"]

docker-compose.yml

version: '3.8'

services:
  app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - PYTHONPATH=/opt/conda/lib/python3.12/site-packages
    volumes:
      - ./app:/app
    healthcheck:
      test: ["CMD", "python", "-c", "import sys; sys.exit(0)"]
      interval: 30s
      timeout: 3s
      retries: 3

.dockerignore

# Python
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
env/
venv/
.venv/

# Conda
.conda/
conda-meta/

# Git
.git/
.gitignore

# IDE
.vscode/
.idea/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db

# Documentation
*.md
docs/

9) TL;DR Quickstart

Migration Checklist

  • Analyze current conda environment
  • Identify essential packages
  • Choose base image (python:3.12-slim recommended)
  • Use multi-stage build
  • Remove conda bloat
  • Optimize environment file
  • Test and verify
  • Monitor performance

Build Commands

# Build with BuildKit
DOCKER_BUILDKIT=1 docker build -t myapp:slim .

# Build with cache
docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --cache-to type=local,dest=/tmp/.buildx-cache .

# Run and test
docker run --rm myapp:slim python -c "import numpy, pandas, sklearn"

Size Comparison

# Before optimization
docker images | grep myapp-before
# myapp-before    latest    2.5GB

# After optimization
docker images | grep myapp-after
# myapp-after     latest    800MB

10) The Machine's Summary

Migrating from conda to slim Docker images requires understanding your dependencies and eliminating bloat. The key is using multi-stage builds, removing unnecessary packages, and optimizing for production.

The Dark Truth: Conda environments are bloated by default. Docker images need to be lean. The migration process reveals what you actually need versus what you think you need.

The Machine's Mantra: "In minimalism we trust, in multi-stage we build, and in the slim container we find the path to efficient deployment."

Why This Migration Matters: Slim images start faster, use less memory, and cost less to store and transfer. Production systems need efficiency, not bloat.

This tutorial provides the complete machinery for migrating conda environments to slim Docker images. The patterns scale from development to production, from megabytes to terabytes.