12 Containerisation

12.1 Containerisation with Docker

As your data science projects grow more complex, you may encounter the “it works on my machine” problem — where code runs differently in different environments. Containerisation solves this by packaging your code and its dependencies into a standardised unit called a container. Building on the Reproducible Environments chapter (conda and uv for Python, renv for R), containers extend that isolation to the whole runtime — the operating system libraries, the system tools, and the language version itself, not just the packages.

12.1.1 Why Containerisation for Data Science?

Containerisation offers several advantages for data science:

Reproducibility: Ensures your analysis runs the same way everywhere
Portability: Move your environment between computers or cloud platforms
Dependency Management: Isolates project dependencies to avoid conflicts
Collaboration: Easier sharing of complex environments with colleagues
Deployment: Simplifies deploying models to production environments

Think of containers as lightweight, portable units that package everything your code needs to run. Unlike virtual machines, containers don’t boot their own operating system; they share the host’s kernel — the core of the operating system that manages memory, files, and processes — and isolate themselves using features that kernel already provides. That makes them much faster to start and much cheaper in memory than a VM, while still giving you a reproducible, isolated environment.

12.1.2 Installing Docker

Docker is the most widely used containerisation platform, and its command-line tool (docker) is effectively the lingua franca of containers. Even the alternative tools below aim to be drop-in replacements for it. We’ll use Docker Desktop in this chapter, but note the following licensing caveat before installing:

Docker Desktop licensing

Since 2021, Docker Desktop requires a paid subscription for commercial use at companies with more than 250 employees or over $10 million in annual revenue. It remains free for personal use, education, small businesses, and open-source projects. If your workplace falls above that threshold, the free alternatives below all provide the same docker command:

Rancher Desktop: open source, works on Windows/macOS/Linux, bundles Kubernetes
Podman Desktop: open source, Red Hat–maintained, and the closest drop-in replacement for the Docker CLI. It runs without a background service (a daemon) holding root privileges, which some IT departments prefer
OrbStack (macOS only): very fast, polished; free for personal use, paid for commercial

All the Dockerfiles and docker commands in this chapter work identically on each of these tools.

12.1.2.1 On Windows

Download Docker Desktop for Windows (or one of the alternatives above)
Run the installer and follow the prompts
Windows 10 Home users should ensure WSL 2 is installed first

12.1.2.2 On macOS

Download Docker Desktop for Mac (or OrbStack / Rancher Desktop)
Run the installer and follow the prompts

12.1.2.3 On Linux

# For Ubuntu/Debian
sudo apt update
sudo apt install docker.io
sudo systemctl enable --now docker

# Add your user to the docker group to run Docker without sudo
sudo usermod -aG docker $USER
# Log out and back in for this to take effect

12.1.2.4 Verifying Installation

Open a terminal and run:

docker --version
docker run hello-world

If both commands complete successfully, Docker is installed correctly.

12.1.3 Docker Fundamentals

Before creating our first data science container, let’s understand some Docker basics:

Images: Read-only templates that contain the application code, libraries, dependencies, and tools
Containers: Running instances of images
Dockerfile: A text file with instructions to build an image
Docker Hub: A registry of pre-built Docker images
Volumes: Persistent storage for containers

The relationship between these components works like this: you create a Dockerfile that defines how to build an image, the image is used to run containers, and volumes allow data to persist beyond the container lifecycle.

12.1.4 Creating Your First Data Science Container

Let’s create a basic data science container using a Dockerfile:

Create a new directory for your project:

mkdir docker-data-science
cd docker-data-science

Create a file named Dockerfile with the following content:

# Use a base image with Python installed
FROM python:3.13-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy requirements file
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Create a non-root user (before copying files so we can set ownership)
RUN useradd --create-home --shell /bin/bash jovyan

# Copy the rest of the code, owned by the non-root user
COPY --chown=jovyan:jovyan . .

USER jovyan

# Command to run when the container starts
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", "--no-browser"]

Create a requirements.txt file with your Python dependencies:

numpy
pandas
matplotlib
scipy
scikit-learn
jupyter
jupyterlab

Build the Docker image:

docker build -t data-science-env .

This command tells Docker to build an image based on the instructions in the Dockerfile and tag it with the name “data-science-env”. The . at the end specifies that the build context is the current directory.

Run a container from the image:

docker run -p 8888:8888 -v $(pwd):/app data-science-env

This command does two important things:

Maps port 8888 in the container to port 8888 on your host machine, allowing you to access Jupyter Lab in your browser
Mounts your current directory to /app in the container, so changes to files are saved on your computer

$(pwd) on Windows

$(pwd) is shell syntax for “the current directory”. It works in Git Bash, WSL, and PowerShell. It does not work in the old cmd.exe prompt, where the equivalent is %cd%:

docker run -p 8888:8888 -v %cd%:/app data-science-env

If the volume mount appears to do nothing — the container starts but your files aren’t in /app — an unexpanded $(pwd) is the usual reason.

Open the Jupyter Lab URL shown in the terminal output

You now have a containerised data science environment that can be shared with others and deployed to different systems.

12.1.5 Understanding the Dockerfile

Let’s break down the Dockerfile we just created:

# Use a base image with Python installed
FROM python:3.13-slim

The FROM statement specifies the base image to use. We’re starting with a lightweight Python 3.13 image. Always pick a currently-supported Python version; consult the official release schedule before pinning in production, since each version’s support window eventually ends.

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    && rm -rf /var/lib/apt/lists/*

The RUN statement executes commands during the build process. Here, we’re updating the package list and installing gcc, which is required for building some Python packages.

# Set working directory
WORKDIR /app

The WORKDIR statement sets the working directory within the container.

# Copy requirements file
COPY requirements.txt .

The COPY statement copies files from the host to the container. We copy the requirements file separately to take advantage of Docker’s caching mechanism.

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

Another RUN statement to install the Python dependencies listed in requirements.txt.

# Create a non-root user (before copying files so we can set ownership)
RUN useradd --create-home --shell /bin/bash jovyan

COPY --chown=jovyan:jovyan . .

USER jovyan

Three things happen here. COPY . . copies everything from the current directory on the host into the working directory in the container. The --chown flag sets the owner as it copies, and USER jovyan means every command after this line — including the CMD below — runs as that user rather than as root.

That last part is the one worth internalising. By default everything in a container runs as root, which is fine until the container is exposed to the network or mounts a host directory, at which point a process that escapes its intended boundary is doing so with root privileges. Creating an unprivileged user costs two lines. (jovyan is not a special name — it’s the convention the official Jupyter images use, and following it keeps file ownership consistent if you later switch to one of those base images.)

Note what --chown does not cover. It sets ownership on the files baked into the image at build time. The -v $(pwd):/app mount in the run command replaces /app with a directory from your host, so anything the image had at that path is hidden underneath it, and the ownership the container sees is the host’s, not what the Dockerfile set.

If Jupyter can’t save notebooks into a mounted directory

On macOS and Windows, Docker Desktop translates ownership for you and this rarely comes up. On Linux the host’s numeric user ID passes through unchanged, so if your account isn’t UID 1000 the mounted files belong to someone jovyan isn’t, and saving fails with a permission error.

Run the container as yourself instead of rebuilding the image:

docker run -p 8888:8888 -u $(id -u):$(id -g) -v $(pwd):/app data-science-env

id -u and id -g are your own user and group numbers, so files the container writes come back owned by you.

# Command to run when the container starts
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--port=8888", "--no-browser"]

The CMD statement specifies the command to run when the container starts. In this case, we’re starting Jupyter Lab. We bind to 0.0.0.0 so the server accepts connections from outside the container (otherwise the port mapping from the host wouldn’t reach it).

12.1.6 Using Pre-built Data Science Images

Instead of building your own Docker image, you can use popular pre-built images:

12.1.6.1 Jupyter Docker Stacks

The Jupyter team maintains several ready-to-use Docker images:

# Basic Jupyter Notebook. Pin a specific dated tag for reproducibility
docker run -p 8888:8888 quay.io/jupyter/minimal-notebook:2025-10-13

# Data science-focused image with pandas, matplotlib, etc.
docker run -p 8888:8888 quay.io/jupyter/datascience-notebook:2025-10-13

# All the above plus TensorFlow
docker run -p 8888:8888 quay.io/jupyter/tensorflow-notebook:2025-10-13

These pre-built images offer a convenient way to get started without creating your own Dockerfile. The Jupyter Docker Stacks project provides a range of images for different needs, from a bare Python and JupyterLab up to images with the scientific stack, R, and deep learning frameworks already installed.

Two small but important habits:

Pull from quay.io/jupyter/*: the project moved off Docker Hub, and quay.io is now the canonical location.
Pin a specific dated tag (e.g. 2025-10-13) rather than latest. Using latest means the image can silently change under you between runs, which defeats the whole point of containers for reproducibility. Check the jupyter/docker-stacks releases page for current tags.

12.1.6.2 RStudio

For R users, the Rocker project maintains RStudio Server images. Keep the password out of your shell history by reading it from an environment variable or .env file rather than typing it on the command line:

# Bash, Git Bash, WSL, macOS. Set the password once for this shell session:
export RSTUDIO_PASSWORD='a-strong-password-you-chose'

docker run -p 8787:8787 \
  -e PASSWORD="$RSTUDIO_PASSWORD" \
  rocker/rstudio:4.5

PowerShell has no export, and uses a backtick rather than a backslash to continue a line:

$env:RSTUDIO_PASSWORD = 'a-strong-password-you-chose'

docker run -p 8787:8787 `
  -e PASSWORD="$env:RSTUDIO_PASSWORD" `
  rocker/rstudio:4.5

Access RStudio at http://localhost:8787 with username rstudio and the password you set. Pin to a specific Rocker tag (e.g. rocker/rstudio:4.5) rather than using latest, for the same reproducibility reasons as above. Rocker publishes tags on Docker Hub: browse there to pick a specific patch version if you need one.

12.1.7 Docker Compose for Multiple Containers

For more complex setups with multiple services (e.g., Python, R, and a database), Docker Compose allows you to define and run multi-container applications:

Create a .env file alongside your compose file with the secrets, and add .env to your .gitignore so it never reaches a repository:

RSTUDIO_PASSWORD=a-strong-password-you-chose
POSTGRES_PASSWORD=another-strong-password

Create a file named compose.yaml (the newer Compose V2 name; docker-compose.yml still works):

services:
  jupyter:
    image: quay.io/jupyter/datascience-notebook:2025-10-13
    ports:
      - "8888:8888"
    volumes:
      - ./jupyter_data:/home/jovyan/work

  rstudio:
    image: rocker/rstudio:4.5
    ports:
      - "8787:8787"
    environment:
      - PASSWORD=${RSTUDIO_PASSWORD}
    volumes:
      - ./r_data:/home/rstudio

  postgres:
    image: postgres:17
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - ./postgres_data:/var/lib/postgresql/data

The ${RSTUDIO_PASSWORD} and ${POSTGRES_PASSWORD} placeholders are filled in from your .env file at runtime, so the secrets never live in the file you commit.

Start all services (Compose V2 uses docker compose, a space, not a hyphen):

docker compose up

Access Jupyter at http://localhost:8888 and RStudio at http://localhost:8787

Docker Compose creates a separate container for each service in your configuration while allowing them to communicate with each other. This approach makes it easy to run complex data science environments with multiple tools.

12.1.8 Docker for Machine Learning Projects

For machine learning projects, containers are particularly valuable for ensuring model reproducibility and simplifying deployment.

This example needs a trained model to copy

Docker’s COPY fails the build if the source path doesn’t exist, so the Dockerfile below needs this layout in place before it will build:

ml-api/
├── Dockerfile
├── requirements.txt
├── app.py
└── models/
    └── model.pkl        <- your trained model

If you have no trained model to hand, write a throwaway one so the example builds. Run this once from the ml-api directory, in an environment that has scikit-learn:

# make_model.py
import pickle, pathlib
import numpy as np
from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(np.array([[0.0], [1.0]]), np.array([0.0, 1.0]))
pathlib.Path("models").mkdir(exist_ok=True)
with open("models/model.pkl", "wb") as f:
    pickle.dump(model, f)

Create a project-specific Dockerfile:

FROM python:3.13-slim

WORKDIR /app

# Copy and install requirements first for better caching
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the model artefact and the application that serves it
COPY models/ models/
COPY app.py .

# Expose port for API
EXPOSE 5000

# Run the API service under a production web server, not Flask's
# built-in development server (see the Deployment chapter)
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "app:app"]

Create a requirements.txt for the API. This is a different, much smaller set than the notebook environment earlier in the chapter — a serving container needs the web server and whatever library loads your model, and has no use for JupyterLab:

flask
gunicorn
scikit-learn
numpy

A pickled model needs the same library version that created it

pickle stores a reference to the classes that built the object, not a self-contained description of it. Loading a scikit-learn model in a container with a different scikit-learn version produces either a warning about version mismatch or a failure to unpickle at all. Pin the version here to whatever you trained with — scikit-learn==1.9.0 rather than bare scikit-learn — which is one more reason the environment chapter’s advice about recording versions matters.

Create a simple model serving API (app.py):

from flask import Flask, request, jsonify
import pickle
import numpy as np

app = Flask(__name__)

# Load pre-trained model
with open('models/model.pkl', 'rb') as f:
    model = pickle.load(f)

@app.route('/health')
def health():
    # A cheap endpoint for the container runtime to poll - see HEALTHCHECK below
    return jsonify({'status': 'ok'})

@app.route('/predict', methods=['POST'])
def predict():
    data = request.json
    features = np.array(data['features']).reshape(1, -1)
    prediction = model.predict(features)[0]
    return jsonify({'prediction': prediction.tolist()})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Build and run the container:

docker build -t ml-model-api .
docker run -p 5000:5000 ml-model-api

This creates a containerised API service for your machine learning model that can be deployed to any environment that supports Docker.

12.1.9 Best Practices for Docker in Data Science

To get the most out of Docker for data science, follow these best practices:

Keep images lean: Smaller images pull faster, deploy faster, and ship less software that could turn out to have a vulnerability in it. Use -slim (or -alpine for pure-Python workloads) base images when you can.
```
FROM python:3.13-slim  # ~150 MB; the full python:3.13 image is ~1 GB
```
Note that Alpine-based images use musl instead of glibc, which occasionally breaks scientific Python packages that ship precompiled wheels for glibc only. When in doubt, start with -slim.
Use multi-stage builds for production: Separate building dependencies from runtime
```
# Build stage
FROM python:3.13 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Runtime stage
FROM python:3.13-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY . .
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "app:app"]
```
The benefit: your final image doesn’t carry compilers, header files, or build tools, only the installed Python packages and your application code. That usually takes a substantial bite out of the image size, and every tool you leave out is one that can’t be used against you if someone gets into the container.

The second COPY --from=builder line is easy to leave out and the resulting failure is confusing. site-packages holds the importable library code, but packages that install a command — gunicorn, jupyter, streamlit, dvc — put an executable script in /usr/local/bin, which is a different directory. Copy only site-packages and import gunicorn works fine while gunicorn at the command line reports not found.

Layer your Dockerfile logically: Order commands from least to most likely to change

# System dependencies change rarely
RUN apt-get update && apt-get install -y gcc

# Requirements change occasionally
COPY requirements.txt .
RUN pip install -r requirements.txt

# Application code changes frequently
COPY . .

Use volume mounts for data: Keep data outside the container

docker run -v /path/to/local/data:/app/data my-data-science-image

Implement proper versioning: Tag images meaningfully
```
docker build -t mymodel:1.0.0 .
```
Create a .dockerignore file: Exclude unnecessary files
```
# .dockerignore
.git
__pycache__/
*.pyc
venv/
data/
```
Use environment variables for configuration (not secrets):
```
ENV MODEL_PATH=/app/models/model.pkl
```
ENV values are baked into the image and visible to anyone who can pull it, so they are fine for non-sensitive defaults like file paths or log levels. Pass actual secrets at runtime via docker run -e, --env-file, or your platform’s secret store. Never hard-code them in a Dockerfile.
Add a HEALTHCHECK so the runtime can detect and restart broken containers:
```
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD python -c "import urllib.request; \
    urllib.request.urlopen('http://localhost:5000/health')"
```
Two things have to line up here, and both are easy to get wrong. The port and path must match something your application actually serves — 5000 and /health above correspond to the Flask API earlier in this chapter; point a healthcheck at a route that doesn’t exist and Docker will report a perfectly healthy container as unhealthy forever. And the command must exist inside the image: healthcheck examples online almost always use curl, which the -slim images this chapter recommends don’t include, so the check errors every time it runs. Python is guaranteed to be there, so use Python.
Run as a non-root user in production images. Most official base images provide one (e.g. the Jupyter stacks use jovyan, Rocker uses rstudio). For your own images, add a user near the end of the Dockerfile:
```
RUN useradd --create-home appuser
USER appuser
```
Running as root inside the container isn’t catastrophic on its own, but combined with volume mounts or a container escape it makes things much worse than they need to be.

12.1.10 Common Docker Commands for Data Scientists

Here are some useful Docker commands for day-to-day work:

# List running containers
docker ps

# List all containers (including stopped ones)
docker ps -a

# List images
docker images

# Stop a container
docker stop container_id

# Remove a container
docker rm container_id

# Remove an image
docker rmi image_id

# View container logs
docker logs container_id

# Execute a command in a running container
docker exec -it container_id bash

# Clean up unused resources
docker system prune

docker system prune deserves a particular mention: images and stopped containers accumulate quickly, and reclaiming tens of gigabytes is a common first result.

12.2 Conclusion

Packaging your code, dependencies, and configuration into a single unit gets you consistent behaviour across machines, and turns “it works on mine” into a claim anyone can test.

By the end of this chapter you should have:

Docker installed, with docker run hello-world completing successfully
An image you built yourself from a Dockerfile, serving Jupyter Lab on localhost:8888
A compose.yaml bringing up more than one service at once, with its passwords read from a .gitignored .env file rather than written into the file
A rough sense of when a pre-built image (the Jupyter stacks, Rocker) is the better starting point than writing your own

Docker is the default but not the only option: Podman runs the same images without a background daemon and without Docker Desktop’s licensing terms, and its CLI is close enough that most commands in this chapter work unchanged. Kubernetes, the tool most often mentioned in the same breath as Docker, solves a problem — running many containers across many machines — that most individual data scientists never have.