14 Optimising Workflows and Next Steps
14.1 Optimising Your Data Science Workflow
The tools are installed and your work deploys. What remains is the difference between a project that runs and a project that keeps running: a structure someone else can navigate, a single command that rebuilds the whole analysis, and checks that catch a mistake before a stakeholder does.
14.1.1 Project Organisation Best Practices
A well-organised project makes collaboration easier and helps maintain reproducibility:
14.1.1.2 Making Your Code Importable
The project_name/ directory is a Python package, which is what lets you write from project_name.features import add_growth_rate in a notebook or a test rather than juggling relative paths. That only works once you have told Python the package exists. From the project root, with your project environment activated:
pip install -e .The -e stands for editable: Python records where your source lives instead of copying it, so your next edit takes effect without reinstalling. It reads the package name from pyproject.toml, which is why the tree above includes one. Do this once per environment, and again if you rename the package.
Skip it and the tests later in this chapter fail with ModuleNotFoundError: No module named 'project_name', which is the most common first stumble with this layout.
14.1.1.3 Generating the Structure
You can create this structure automatically. The template above is Cookiecutter Data Science, which since version 2 ships its own command, ccds.
It is distributed as a command-line application rather than a library, so install it with pipx, which puts each such application in its own environment where it cannot conflict with your projects:
# Install pipx itself, once per machine
pip install pipx
pipx ensurepath # adds pipx's install directory to your PATH
# Close and reopen your terminal, then install the template tool
pipx install cookiecutter-data-science
# Create a new project - run this in the directory that should contain it
ccdsccds asks a short series of questions (Python version, environment manager, testing framework, whether you want documentation) and generates the tree accordingly, so your project won’t match the listing above in every detail.
Plenty of tutorials still say pip install cookiecutter followed by cookiecutter https://github.com/drivendata/cookiecutter-data-science. Both halves are now wrong: the project moved to the drivendataorg organisation, and v2 of the template requires the ccds command rather than plain cookiecutter. If you meet a version of this instruction that generates a src/ directory and a setup.py, you are looking at v1.
14.1.2 Data Version Control
While Git works well for code (covered in Editors and Version Control), it’s not designed for large data files. Data Version Control (DVC) extends Git to handle data:
# Install DVC. The bracketed extra pulls in the libraries for your chosen
# storage backend - use dvc[gdrive], dvc[azure] or dvc[gs] instead of
# dvc[s3] if you're not on AWS. Plain `pip install dvc` installs no
# backends at all and `dvc push` will fail.
pip install "dvc[s3]"
# Initialise DVC in your Git repository
dvc init
# Hand a large file over to DVC
dvc add data/raw/large_dataset.csv
# That created two things to commit: a small pointer file, and a
# .gitignore entry so Git stops tracking the data itself
git add data/raw/large_dataset.csv.dvc data/raw/.gitignore
git commit -m "Track raw dataset with DVC"
# Configure remote storage and upload
dvc remote add -d storage s3://mybucket/dvcstore
dvc push.gitignore template
The template in Editors and Version Control ignores data/raw/ and data/processed/ wholesale, which is the right default when you have no other way to keep large files out of Git. DVC writes its own, narrower rules — it ignores the data file itself and leaves the .dvc pointer trackable — so the blanket rule now works against you, and the git add above fails with “The following paths are ignored by one of your .gitignore files”.
Remove the data/ lines from your .gitignore when you adopt DVC and let DVC manage those directories instead.
The two-file result is the whole mechanism. large_dataset.csv.dvc is a few lines of YAML containing a hash of the data, and it lives in Git like any other source file; the multi-gigabyte CSV lives in S3. A colleague clones the repository, gets the pointer, runs dvc pull, and receives exactly the version of the data that matches the code they checked out. Checking out an old commit and running dvc pull reconstructs the data as it was then.
The benefits of using DVC include:
- Tracking changes to data alongside code
- Reproducing exact data states for past experiments
- Sharing large datasets efficiently with teammates
- Creating pipelines that track dependencies between data processing stages
14.1.3 Automating Workflows with Make
Make is a build tool that can automate repetitive tasks in your data science workflow:
- Create a file named
Makefile:
.PHONY: data features model report clean
# Download raw data
data:
python project_name/dataset.py
# Process data and create features
features: data
python project_name/features.py
# Train model
model: features
python project_name/modeling/train.py
# Generate report
report: model
quarto render reports/final_report.qmd --to html
# Remove generated files. This deletes outputs only - the report source,
# and everything under data/raw/, are deliberately left alone.
clean:
rm -rf data/processed/*
rm -rf models/*
rm -f reports/*.html reports/*.pdf
rm -rf reports/*_files reports/figures/*The paths match the project_name/ package from the structure above. If you generated your project with ccds and answered its questions differently, adjust them to whatever it created.
Make is notoriously picky: the commands under each target (the lines like python project_name/dataset.py) must start with a real tab character, not spaces. If your editor auto-converts tabs to spaces, you’ll get a cryptic missing separator error when you run make. Most editors (VS Code, Vim, RStudio) can be told to leave tabs alone in Makefile specifically, and VS Code does this automatically. If in doubt, run cat -A Makefile and make sure each recipe line starts with ^I (a tab).
- Run tasks with simple commands:
# Run all steps
make report
# Run just the data processing step
make features
# Clean up generated files
make cleanMake tracks dependencies between tasks and only runs the necessary steps. For example, if you’ve already downloaded the data but need to rebuild features, make features will skip the download step.
Automation tools like Make help ensure consistency and save time by eliminating repetitive manual steps. They also serve as documentation of your workflow, making it easier for others (or your future self) to understand and reproduce your analysis.
14.1.4 Continuous Integration for Data Science
Continuous Integration (CI) automatically tests your code whenever changes are pushed to your repository:
- Create a GitHub Actions workflow file at
.github/workflows/python-tests.yml:
name: Python Tests
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.13'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest pytest-cov
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Install the project itself, so tests can import it
run: pip install -e .
- name: Test with pytest
run: |
pytest --cov=project_name tests/- Write tests for your code in the
tests/directory
Push a commit and GitHub runs this on a fresh Linux machine. A green tick next to the commit means the tests passed; a red cross links to the log, with the failing test and its traceback. The useful side effect is that CI runs in an environment that has nothing but your requirements.txt — so a package you installed locally six months ago and forgot to record shows up as a failure here rather than in front of a stakeholder.
14.1.5 Writing Tests for Analysis Code
Testing is the piece of software practice that data science teams most often skip, usually on the grounds that “the output is a number, not a program”. The counter-argument is that a number informing a decision is exactly the thing you want to be sure about.
Here is the function we’re going to test. Put it in project_name/features.py:
import numpy as np
import pandas as pd
def add_growth_rate(df: pd.DataFrame, column: str) -> pd.DataFrame:
"""Add a period-over-period growth rate for `column`.
Where the previous period was zero, the growth rate is undefined and we
return NA rather than inf.
"""
df = df.copy()
previous = df[column].shift(1)
df["growth_rate"] = (df[column] - previous) / previous.replace(0, np.nan)
return dfA test is an ordinary function whose name starts with test_, containing an assert. Put this in tests/test_features.py:
import pandas as pd
import pytest
from project_name.features import add_growth_rate
def test_growth_rate_computes_period_over_period_change():
df = pd.DataFrame({"revenue": [100.0, 110.0, 121.0]})
result = add_growth_rate(df, column="revenue")
assert result["growth_rate"].tolist() == pytest.approx(
[float("nan"), 0.10, 0.10], nan_ok=True
)
def test_growth_rate_handles_zero_denominator():
df = pd.DataFrame({"revenue": [0.0, 50.0]})
result = add_growth_rate(df, column="revenue")
assert result["growth_rate"].isna().iloc[1] # not inf, not a crashRun them with pytest from the project root.
Two details in the first test are worth pausing on, because both are easy to get wrong in a way that fails a correct implementation. The expected first value is float("nan") and not None: the first row has no prior period, and a float column represents that gap as nan, so None would never match. And nan_ok=True is what tells pytest.approx to count two NaNs as equal — in ordinary Python, nan == nan is False, so without it the test fails no matter what your function returns.
The second test is the more instructive one, and it is the reason features.py bothers with previous.replace(0, np.nan). The obvious one-line implementation — df[column].pct_change() — passes the first test and fails the second, because dividing 50 by 0 gives inf rather than NA. That is exactly the edge case that produces a nonsense number in a spreadsheet, propagates quietly through three more calculations, and surfaces as an absurd figure in a board pack. The test is what forces the guard to exist.
Four kinds of test earn their keep in analysis work:
- Unit tests for individual transformations — the example above
- Data validation tests asserting your assumptions about incoming data: no negative prices, no duplicate IDs, dates within an expected range. Great Expectations and Pandera are built for this; a handful of
assertstatements works too - Model performance tests that fail the build if accuracy on a held-out set — data the model never saw while training — drops below a threshold you set
- Integration tests that run the whole pipeline end to end on a small sample and check it produces output at all
R users have the same facilities through testthat: test_that("growth rate handles zero denominators", { expect_true(is.na(...)) }), run with devtools::test().
14.1.6 Catching Problems Before They Reach CI
CI tells you something broke only after you have pushed it. Pre-commit hooks catch it while the change is still on your machine. The pre-commit framework runs a set of checks against your staged files every time you commit:
# .pre-commit-config.yaml
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.1
hooks:
- id: ruff # lint
- id: ruff-format # format
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: check-added-large-files # stop a 2GB CSV entering Git history
- id: detect-private-key # stop a credential entering Git history
- id: end-of-file-fixerInstall with pip install pre-commit then pre-commit install. Run pre-commit autoupdate occasionally to bump those rev: pins — they’re exact tags, so they don’t move on their own. The last two hooks are the ones to care about: a large file or a leaked key committed to Git is genuinely painful to remove afterwards, and both are trivial to prevent.
14.1.7 When Make Isn’t Enough: Workflow Orchestrators
Make handles “run these steps in this order, skip what’s already done” on one machine, on demand. When a pipeline needs to run on a schedule, retry failed steps, alert someone when it breaks, or coordinate work across machines, you’ve reached the limits of a Makefile.
The tools for that job are workflow orchestrators. The three you’ll encounter:
- Airflow — the incumbent, widely deployed, heavier to operate. If your organisation already runs a data platform, it probably runs Airflow.
- Prefect — Python-native and considerably lighter to start with; you decorate ordinary functions with
@taskand@flow. - Dagster — organised around data assets rather than tasks, which maps well onto analytics work where the thing you care about is “is this table up to date”.
Don’t reach for any of them early. A Makefile plus a scheduled GitHub Actions workflow — on: schedule: with a cron expression, the five-field 0 6 * * 1 notation meaning “06:00 every Monday” — covers a surprising amount of ground, and none of these tools is free to operate. Adopt one when you have several interdependent pipelines and a real need to know when one of them fails overnight.
14.2 Advanced Topics and Next Steps
As you grow more comfortable with the data science infrastructure we’ve covered, here are some advanced topics to explore:
14.2.1 MLOps (Machine Learning Operations)
MLOps combines DevOps practices with machine learning to streamline model deployment and maintenance:
- Model Serving: Tools like TensorFlow Serving, TorchServe, or MLflow for deploying models
- Model Monitoring: Tracking performance and detecting drift — the gradual decay in accuracy as live data stops resembling the data the model was trained on
- Feature Stores: Centralised repositories that compute a feature once and serve the same definition to both training and production
- Experiment Tracking: Recording parameters, metrics, and artefacts from experiments
14.2.2 Distributed Computing
For processing very large datasets or training complex models:
- Spark: Distributed data processing
- Dask: Parallel computing in Python
- Ray: Distributed machine learning
- Kubernetes: Container orchestration for scaling
14.2.3 AutoML and Model Development Tools
These tools help automate parts of the model development process:
- AutoML: Automated model selection and hyperparameter tuning
- Feature Engineering Tools: Automated feature discovery and selection
- Model Interpretation: Understanding model decisions
- Neural Architecture Search: Automatically discovering optimal neural network architectures
14.2.4 Staying Current
Tooling churns, and keeping up with all of it is neither possible nor useful. Two habits are worth more than a subscription list:
- Read the release notes for the handful of tools you actually depend on. Pandas, Quarto, and your cloud provider each publish them, and they are where breaking changes are announced before they break something of yours.
- Teach it. The Carpentries run workshops on exactly this material — the shell, Git, R, and Python for researchers — and their instructor training is free and well designed. Explaining
git rebaseto a room of ecologists is an efficient way to discover what you don’t understand about it.
For everything else, PyData conference talks (all on YouTube, free) and the Stack Overflow question you find while debugging will cover more ground than any curated feed.
14.3 Conclusion
The tools in this chapter share a purpose: making the second run of an analysis cheaper than the first.
By the end of this chapter you should have:
- A project directory whose structure someone else could navigate without asking you
- A
Makefilethat rebuilds your analysis end to end with one command - Tests in a
tests/directory and a CI workflow that runs them on every push - Pre-commit hooks installed, so a large file or a stray API key can’t reach your Git history by accident
- Data too large for Git under DVC, with the pointer files committed alongside the code
Skip all of it on a one-off script you’ll delete next week. Do it on anything a colleague will inherit, anything that runs more than once, and anything whose output someone will act on.