Introduction

The measure of success most often is speed. Doing things better is synonymous with doing things faster so that we can do even more things efficiently and effectively. In buying into this premise, we enter a spiral of acceleration that we can never hope to master.

—Diana Scharf Hunt and Pam Hait

Welcome, dear reader.

Oh, tests.

Tests are great. Tests show us where bugs are… at least some of them. Tests help us to not add new bugs.

But… tests. Tests are also a frequent source of frustration. When you first started writing tests, you probably found them an extra nuisance to write. You probably still think the same, at least sometimes.

Even if you love writing tests, running them can be boring. Slow tests can make you lose concentration, juggle tasks, and end up getting less stuff done. They can suck all the fun out of programming. And programming is meant to be fun!

There’s a phase shift when you speed up your tests, restoring that joy. If your tests take 1000 seconds and you reduce them to 600 seconds, you’ll feel it. Your whole team will feel it. The same happens if you can reduce a test run from 30 seconds down to 10 seconds.

And there’s no limit to “fast enough”. Reducing test run time is the easiest and safest way to increase your speed of delivery. And your organization wants to deliver as fast as possible, to get valuable features and fixes in front of users.

Fast tests also tend to be more accurate tests. They’re well targeted, so they only fail when a feature is broken. And with less overhead, you can exercise features more completely.

The content is based on my eight years of experience with tests on Django projects. I’ve sped up many projects’ test suites, improved Django’s own testing framework, and created several pytest plugins.

In your hands, or at least on your screen, is my best guide for speeding up tests on your project. I’ve tried to cover most situations, so you should be able to find something relevant to your project.

May it reduce your test speed and your test pain. Enjoy!

—Adam

P.S. Please send me any and all feedback through the contact details on my site.

Who Is This Book For?

If you are a Django developer working with tests, this is for you. No matter your level.

My aim is to make the book readable by a junior developer who has been given the vague task of “improving the tests”, but also to include material that even long time Django users may not have come across. I’ve tried to be opinionated enough to be actionable, but provide advice that’s applicable to all projects.

A Brief Tour of This Book

The first two chapters, Toolbox and Measure!, explain the various tools you have at hand for modifying and measuring your tests respectively.

The next six chapters cover ways of speeding up your tests without the bother of rewriting your test code (much). Easy Wins covers a number of smaller, non-invasive changes that can speed up your tests. Upgrades then re-iterates the importance of keeping things upgraded, especially with respect to test performance. Parallelize covers using parallelized testing, and the challenges you might face adding it. Migrations covers tactics for improving your workflow with database migrations. Finally, Database Configuration and CI Configuration cover ways of speeding up your tests in configuration for those environments.

The three chapters that follow cover how to write fast tests. Test Structure and Test Data go into depth on writing faster tests that use less data. And Targeted Mocking shows some focussed techniques for replacing components during testing that can help you write better, faster tests.

Finally, the Outroduction gives some closing remarks and links to further reading.

In a rush?

To get the 80% of improvements that take 20% of the effort:

Record your current test runtime with a shell timing command as in Measure!.

Skim Easy Wins for any changes that are applicable to your project and make them. When necessary, follow the references back to Toolbox to understand the pieces you’re changing.

If you don’t yet run your tests in parallel, read Parallelize.

Learn how to reuse your database and squash your migrations in Migrations.

Do as many of the upgrades described in Upgrades as possible.

Compare your test runtime!

If after this, your tests are still too slow, profile them with one of the tools described in the second half of Measure!. Then when you know where to focus, read the later chapters to determine ways of speeding up the slowest tests.

Examples

This book has a lot of code examples. I hope that is not a surprise!

Here’s some information on their layout.

Commands

Commands to run in your shell look like this:

$ python --version

The $ represents the prompt - don’t type that. I’m using macOS but hope to have removed any macOS or *nix bias. Commands should work cross-platform, but where that’s not possible I’ve added a separate Windows example with a PowerShell prompt:

> python --version

If I’ve written something incompatible with your platform (most likely Windows), please let me know.

Versions

Examples have been prepared with Python 3.9 and Django 3.2. I’m only using the official Python distribution, CPython, since Django or its dependencies often don’t work on other Python interpreters.

Python code is formatted with Black. (Like Django itself will be one day.) Import statements are sorted and grouped with isort. And all code is also linted with Flake8.

Throughout, I refer to installation commands with:

$ python -m pip install Django

I’m using python -m pip as per pip developers’ recommendations - see my blog post on the subject.

Please read these commands as “install using your dependency management solution”. There are several tools for this in the Python ecosystem, and I can’t cover them all.

If you aren’t using any dependency management tool (beyond pip), pip-compile is a great start as it’s a small layer on top of pip. It’s also my preference.

Django Projects

The example projects I’ve used are based on my “simple-core” startproject template. This has directory structure like this:

$ tree
.
├── example
│   ├── __init__.py
│   ├── core
│   │   ├── __init__.py
│   │   ├── models.py
│   │   └── tests
│   │       ├── __init__.py
│   │       └── test_models.py
│   ├── settings.py
│   └── urls.py
└── manage.py

3 directories, 8 files

This is slightly different to Django’s normal startproject template. The whole project lives inside a single Python package, example. example.core is an app, inside which there are models and tests. This is counter to the Django default of having apps in modules that are peers to the “project app”.

I prefer this layout because it makes it clear when you’re importing something belonging to your project. Owned names always start with example (or whatever your project is called).

resources.zip

This PDF also came with a resources.zip. This includes all the example projects. They’re structured with one directory per chapter, within which there are several individual projects.

When a file would be particularly useful to open, I mention the file name to open. Otherwise, the code examples don’t refer to their file names, since they should stand alone. If you do find yourself wanting to refer to some code, the files should be named well enough to figure out which one you want.

Snipped Output

Quite often in examples, I “snip” the code or output to keep it short and discard irrelevant details. You can spot this whenever there’s a .... For example, here I’ve snipped the list of subcommands:

$ python manage.py --help

Type 'manage.py help <subcommand>' for help on a specific subcommand.

Available subcommands:

[django]
    check
    ...
    testserver

In Python, ... is also syntactically valid - it’s the rarely needed Ellipsis object. This never has any meaning in my examples.

Acknowledgements

Thank you to everyone who helped me put this book together.

Thanks to my partner Mafalda for her love, encouraging me to carry on when I needed it, and encouraging me to stop at the end of the day. Thanks to my parents - my mum for sharing her experience as an author, and my dad for proofreading this book with the same care he applied to every homework essay and CV. And thanks to my brother and sister-in-law for their support.

I was very happy to have three excellent friends offer their help as reviewers. You should check out their blogs! Thanks to David Seddon for helping organize the London Django Meetup, giving many great testing and design talks there, and balancing my Aristotelian tendencies with his Platonist ones. Thanks to Jeroen Janssens for letting me deploy his recommendation system, letting me crash on the way to African Python conferences, and having an eye for both design and wording. And thanks to Julius Šėporaitis for pushing for a culture of excellence every time I’ve worked with him, bringing real magic everywhere he goes, and suggesting a broad range of enhancements.

I’d also like to thank these people for their support and feedback: Ahter Sönmez, Carlton Gibson, Dan Palmer, Gordon Wrigley, James Cooke, Jeff Triplett, Kristian Glass, Marco Alabruzzo, Mariusz Felisiak, Peter Inglesby, Rupert Baker, Tom Grainger, Will Vincent, Çağıl Uluşahin, Matt Layman. And Kevin the puppy.

And finally, thank you reader for purchasing my book. I hope you find it useful.

Changelog

2021-07-08

Added ePub file to the release. Thanks to Mike Lissner and Paolo Melchiorre for requesting and testing it.

Upgraded to Python 3.9 and Django 3.2 and updated content to reflect changes.

Added Mock Time section to the “Targeted Mocking” chapter, describing my library time-machine.

Added a section to “Parallelize” on implementing shared resource locking with a pytest fixture.

Added note in Chapter 9 about Circle CI test splitting.

Updated section on_commit() Callbacks to cover Django 3.2’s captureOnCommitCallbacks() method, and its backport in django-capture-on-commit-callbacks.

Updated section Use setUpTestData to cover Django 3.2’s merging of django-testdata, and link to my “How to convert” post.

Updated section Prevent Output to cover Django 4.0’s --buffer --parallel support, and fix error reporting in example code.

Updated section Activate Parallelization to better describe the multiprocessing start methods, and reflect the current status of the spawn method ticket.

Updated the parallel testing macOS workaround in a safer way, without setting the environment variable globally.

Moved the Mock Settings section from “Toolbox” to “Targeted Mocking”.

Renamed chapter 12 to “Targeted Mocking” as “targetted” is a misspelling.

Fixed some errors reported by Matt Layman, Nat Dunn, and Paolo Melchiorre.

Fixed links to pytest documentation after it was reorganized.

2020-05-18

First release!

Toolbox

“If all you have is a hammer, everything looks like a nail.”

—The Law of the Instrument

Let’s look at the tools you can use to improve test speed. The following chapters make use of these extensively.

Test Framework

I’m going to cover two test frameworks in this book:

Django’s built-in test framework - the one behind python manage.py test.

pytest: helps you write better programs - a popular, more Pythonic test runner, that you invoke with pytest.

(I’ll also briefly touch on the now-disused framework nose at the end of this section.)

Normally Django developers first learn testing through its built-in framework. Therefore, while popular, pytest sees a bit less use, at least in the Django community.

In each section I’ll start with Django’s test framework, then cover how things differ in pytest (if they do).

My personal preference is to use pytest. It has a lot of features built-in, it has an extensive plugin ecosystem, and it’s under active development. As we’ll see, many of the things that take extra work with Django’s test framework are included in pytest.

However, this book is not intended to convince you to switch. It’s not necessarily feasible to switch framework, especially only to increase your test speed. Although pytest tries to provide drop-in compatibility, it doesn’t always work out - there are always edge cases. Maybe in reading though, you will see more of what pytest is about and try it out.

Django’s Test Framework

When we use Django’s test framework, we’re actually using two things:

The standard library’s unittest module. It’s based on the jUnit framework from Java, which is why names are in camelCase rather than snake_case.

Django’s test framework. This is comprises extensions to unittest, such as django.test.TestCase and test database setup.

As we’ll see, both provide us with a lot of hooks for extending the testing process. However, there’s no plugin system, so if we want to use others’ extensions, we have to copy-paste snippets or use multiple inheritance.

pytest

When we use pytest for Django test suites, we need to use:

pytest itself

The pytest-django plugin.

pytest also has many more built-ins and hooks, and there’s a whole ecosystem of plugins that work with it. We’ll touch on many of these throughout the book, but there are so many, it will still be just a preview. You can find out more in the excellent documentation for pytest, pytest-django, and the many blog posts and plugins they link to.

pytest has built-in support for unittest test cases and pytest-django reorganizes parts of Django’s test framework to work under pytest. With pytest-django set up, pytest can run existing Django test cases with no modification, so converting to pytest can be fairly straightforward. See pytest-django’s Getting Started Guide.

There are four styles we can use for Django tests running under pytest:

Django’s TestCase with its self.assert* functions:

from django.test import SimpleTestCase

class AdditionTests(SimpleTestCase):
    def test_one(self):
        self.assertEqual(2 + 2, 4)

This style is the same as using Django’s test framework.

Django’s TestCase with plain assert statements:

from django.test import SimpleTestCase

class AdditionTests(SimpleTestCase):
    def test_one(self):
        assert 2 + 2 == 4

This style takes advantage of pytest’s assertion rewriting, which decomposes assert statements on failure to show the components.

Plain test functions:

def test_one():
    assert 2 + 2 == 4

pytest collects module level functions with names starting “test_” as tests. This style makes it easy to start writing tests.

Plain test classes:

class TestAddition:
    def test_one(self):
        assert 2 + 2 == 4

pytest finds module-level classes with names starting “Test”, and collects their methods with names starting “test_” as tests. There’s no base class to inherit from. This style also makes it easy to start writing tests, with some grouping.

The first style, using vanilla TestCases does not take advantage of any pytest features. This style is mostly useful as a first step when converting to pytest. Using the TestCase.assert* functions requires extra learning and recall for developers, and reduces readability, compared to the assert statement. So it’s normally better to convert any such such function calls to the assert statement, as in the second style. There’s even a pytest tool called unittest2pytest that can help you do the conversion.

The third style, using functions, is “the pytest default” as it appears most in the pytest documentation, although the fourth class-based style is just as well supported. Either style allows us to use pytest markers and fixtures seamlessly, including those in pytest-django.. Using classes, as in the fourth style, is a bit more maintainable in the long run, tests for the same thing, such as a view, can be grouped together.

That said there’s a downside to using plain test functions or classes. Not every feature from Django’s test framework is available via pytest-django, and some would be especially hard for it to copy, such as Django’s class-level transactions. Going in the other direction is possible: whilst a little clunky, we can use pytest fixtures in TestCase classes.

Therefore, my preference is to use the second style, using the best of both Django’s test framework with pytest’s features.

During this book we’ll be using the first style. This allows the examples to work on both frameworks, and allows us to use all the features in Django’s testing framework.

Historical: nose

nose was the previous “better python test runner”, popular before pytest took over. At time of writing it is long abandoned, with the last commit four years ago in 2016.

It was popular with Django projects too, via the django-nose project. One of its best features was parallel test execution, which it supported before Django did.

The nose2 project was started as a rewrite, but it hasn’t been as successful as its predecessor. It’s still going but I don’t know of any maintained Django runner.

If you have an old project using nose, it’s probably better at this time to move to either Django’s test runner or pytest. pytest has some nose compatibility features, see its documentation page How to run tests written for nose.

Custom test Management Command

Note

This section only applies to the Django test framework. pytest does not use a management command to run the tests.

This is the outermost layer of the test process. When you run python manage.py test, Django loads and runs the “test” management command.

Normally it finds and uses the built-in command. But if you add a custom management command called “test” in one of your project’s apps, it will use that instead. If you subclass and override parts of the built-in command, you can extend its functionality.

For example, you could create example/core/management/commands/test.py with this content:

import platform

from django.core.management.commands.test import Command as BaseCommand

class Command(BaseCommand):
    def handle(self, *test_labels, **options):
        print(f"Python version: {platform.python_version()}")
        return super().handle(*test_labels, **options)

Then when you run your tests, you’ll get a message with the current Python version before the tests start:

$ python manage.py test
Python version: 3.8.1
System check identified no issues (0 silenced).

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

You won’t need a custom “test” command often. If you read the source, you’ll see Django doesn’t even do much there.

The test runner is the most common extension point, and can even add extra options to the management command. Let’s look at that next.

Test Runner

The test runner is the part of the test framework that handles finding and running your tests. You can customize various parts of this process.

With Django’s Test Framework

Django’s test process uses a “test runner” class called DiscoverRunner. This co-ordinates the whole test process, including discovery, setup, and execution.

You can extend it by creating a custom subclass and pointing the TEST_RUNNER setting at your subclass. If you’re using Django’s test framework, this is the main place to add changes that affect the whole test run.

For example, imagine you wanted to ensure all warnings are shown, without needing to pass -W to python each time you run the tests. You could add the below content to a file example/tests/runner.py:

import warnings
from django.test.runner import DiscoverRunner

class ExampleTestRunner(DiscoverRunner):
    def run_tests(self, *args, **kwargs):
        # Show all warnings once, especially to show DeprecationWarning
        # messages which Python ignores by default
        warnings.simplefilter("default")

        return super().run_tests(*args, **kwargs)

This overrides the run_tests() method, the core method for actually executing test classes and methods. You also need to configure your runner to be used in your settings:

TEST_RUNNER = "example.tests.runner.ExampleTestRunner"

You can extend many different points of the testing process using similar method overrides.

DiscoverRunner also has some attributes that refer to unittest classes that the test runner uses. You can replace these attributes with pointers to your own subclasses to customize some deeper details of the test process. You won’t need these so often, so we’ll cover these when we need to.

Django’s methods and attributes are documented in the Django test runner documentation.

With pytest-django

If you are using pytest-django, the TEST_RUNNER setting is ignored. But pytest offers many granular hooks which you can use to achieve all the same things.

You’ll normally make global changes in a conftest.py file in the root of your project.

For example to add a warning filter we’d need to just edit example/conftest.py to contain:

import warnings

# Show all warnings once, especially to show DeprecationWarning
# messages which Python ignores by default
warnings.simplefilter("default")

(This example is actually unnecessary, as pytest already has built-in warning capture and control.)

pytest finds and executes this file at the start of tests. You can run code at import time, as in this example, or hook into test lifecycle through either fixtures or hook functions. I’ll show use of these as they’re required.

Test Settings

It’s common to want to change some Django settings for the whole test run. We’ll discuss many built-in and third party settings and how they can affect test speed throughout the book.

You might also want to change your own custom settings during tests to help with test speed. For example, you might want to set a pagination count to a low value in tests, so you need less data to test paged views using it.

The Django community seems to have a hundred ways of organizing settings files. I can’t hope to cover them all here. But they can be categorized on at least one useful dimension: using multiple files (normally one per environment), or a single file (controlled through environment variables or similar).

My personal preference is to use a single settings file, driven by environment variables. This is following the “Config” section of The Twelve-Factor App guide, by Heroku. I normally use a package like django-environ to help with parsing the environment variables and optionally allowing a local .env file to set them.

But you probably aren’t looking to change the layout of your settings files right now. Let’s look at how you can configure settings for tests with both approaches, then how to override settings for individual tests and some things to avoid in your test settings.

Multiple Settings Files: Use a Separate File for Tests

This approach is easiest if you’re already using a multiple settings file approach.

Say you have a base settings file example/settings/base.py with default settings:

...

PAGINATION_COUNT = 100

You could override this value during tests with a test settings file example/settings/test.py:

from example.settings.base import *  # noqa

PAGINATION_COUNT = 10

In general from ... import * is a bit of an anti-pattern, since it prevents you from knowing all the names being imported. But it would be annoying to type out all the names to import here, and it would be a bug to not import a setting. The # noqa comment disables Flake8’s warnings about using import *.

You’d also have similar settings files for development, production, and any other environments:

$ tree example/settings
example/settings
├── __init__.py
├── base.py
├── development.py
├── production.py
└── test.py

0 directories, 5 files

Note

Extending dictionary-based settings

Many of Django’s settings are nested dictionaries, for example DATABASES. Often when extending them you’ll want to just change a few keys. The cleanest way to do this is to copy.deepcopy() the base setting, then mutate appropriately. For example to configure persistent database connections in one environment only:

from copy import deepcopy

from example.settings.base import *  # noqa
from example.settings.base import DATABASES

DATABASES = deepcopy(DATABASES)
DATABASES["default"]["CONN_MAX_AGE"] = 120

(The # noqa comment and second explicit import of DATABASES are there to help Flake8’s analysis.)

With a test setting file in place, you can run your tests with python manage.py test --settings example.settings.test, or pytest --ds example.settings.test. This works, but quickly gets tedious. You can easily forget the extra option, leading you to accidentally run the tests with the base settings which could cause errors or failures.

Instead, you can make your test runner automatically use your test settings.

With Django’s Test Framework

You’ll want to customize your manage.py:

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys

def main():
    argv = sys.argv
    try:
        command = argv[1]
    except IndexError:
        command = "help"

    if command == "test":
        default = "example.settings.test"
    else:
        default = "example.settings.production"
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", default)

    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(argv)

if __name__ == "__main__":
    main()

This code checks the running command by parsing sys.argv the same way that execute_from_command_line() does internally. Checking sys.argv is rarely needed - normally it’s better to use a custom “test” management command or runner. But in this case it’s warranted since there’s no other point in the Django startup process to modify settings before they are loaded. I covered this technique in a little more depth in my blog post How to Check the Running Django Command.

With pytest-django

If you’re using pytest-django, you can configure the Django settings file to use in your pytest.ini:

[pytest]
DJANGO_SETTINGS_MODULE = example.settings.test

Single Settings File: Override Settings in the Test Runner

This approach uses a custom test runner to change settings values as the tests start. If your settings are driven by environment variables or similar, it means you don’t have to set those variables to run the tests correctly. It also works when using multiple settings files, but it could be surprising as settings values would be split between the settings files and the runner.

Let’s use the same example as above - a PAGINATION_COUNT setting that defaults to 100 but we want to set to 10 in tests. We’d set it to the default in our settings file:

PAGINATION_COUNT = 100

Then after the settings file is loaded for the test run, we’d replace it using Django’s @override_settings tool. Where to use @override_settings depends on your test runner.

With Django’s Test Framework

You can change settings for the duration of the tests in a custom runner like so:

from django.test.runner import DiscoverRunner
from django.test.utils import override_settings

class ExampleTestRunner(DiscoverRunner):
    def run_tests(self, *args, **kwargs):
        with override_settings(**TEST_SETTINGS):
            return super().run_tests(*args, **kwargs)

TEST_SETTINGS = {
    "PAGINATION_COUNT": 10,
}

You can change as many settings as you like by adding them to the TEST_SETTINGS dictionary.

With pytest-django

You can override settings for the duration of tests from within a fixture:

import pytest
from django.test.utils import override_settings

@pytest.fixture(scope="session", autouse=True)
def test_settings():
    with override_settings(**TEST_SETTINGS):
        yield

TEST_SETTINGS = {
    "PAGINATION_COUNT": 10,
}

This fixture is marked as session-scoped, which means it will only be created once per test run. And as it’s marked for automatic use, pytest will start it as the tests begin, rather than wait for a test that requests it. You can change as many settings as you like by adding them to the TEST_SETTINGS dictionary.

Don’t Run Tests in Debug Mode

You shouldn’t run tests with DEBUG set to True. It decreases test accuracy since the test environment is less similar to production, and imposes extra overheads on the CPU and memory. Django does a few things extra in debug mode, such as keeping a log of every database query and creating debugging responses for 404 errors. Some third party packages also impose extra overheads in debug mode, such as django-debug-toolbar’s profiling.

There are a couple of options in the TEMPLATES setting that default to copying their value from DEBUG. You should check these aren’t overridden to True during tests as well:

DjangoTemplates - the debug option

Jinja2- the auto_reload and undefined options

Ensuring that DEBUG and these related settings default to False during tests depends on your settings file layout. Check the relevant settings file(s) and environment variable(s). If you’re not sure of the value, you can run your tests with a temporary assert statement in a test or AppConfig.ready():

assert not settings.DEBUG, "DEBUG mode should not be on!"

It is occasionally useful to run tests in debug mode, for example to check the executed database queries. If you’re using Django’s test framework, you can use its --debug-mode option (documentation) to do this. Or if you’re using an environment variable, you can run the tests with it set on.

Avoid a TESTING setting

I’ve come across many projects and pieces of community advice that recommend adding a TESTING setting that your code can use to determine if it’s running in tests:

TESTING = True

I’d advise against this for several reasons.

Firstly, during tests your project should be configured as similarly as possible to production so your tests are accurate. Having a TESTING setting encourages its use, such as skipping particular code paths during tests. Over the long term this will cause drift between the two environments.

But yes, we’re pragmatists with deadlines here. Sometimes it is necessary to change behaviour in tests.

Thus, my second objection is that such a setting is quite a blunt instrument. If you see that TESTING is set, it doesn’t actually provide information about how the code’s behaviour changes. You need to search the code for its use, in statements like if settings.TESTING. As your project grows in size, it can become quite hard to find out all the changes it makes.

It’s much better to use targeted settings, that you can change individually for tests. For example, the PAGINATION_COUNT example we used above is much more focussed and comprehensible. We can also reconfigure it for development, staging, and production if required, without refactoring the code.

Custom TestCase Classes

Django provides four test case classes out of the box. They add extra functionality to unittest.TestCase.

It’s quite common that you want to add extra project-specific functionality. This can be done by subclassing Django’s classes and using your versions throughout your project.

For example, imagine your tests often assert on the value of the Clear-Site-Data header (documentation). To reduce verbosity, you can add a custom assertion function for all your tests. You can do this by creating a mixin and using it in test case subclasses, in a global file such as example.test:

from django import test

class ExampleTestMixin:
    def assertClearSiteData(self, response, value=None):
        if value is None:
            value = ["*"]

        self.assertEqual(
            response.get("Clear-Site-Data", ""),
            ", ".join(f'"{v}"' for v in value),
        )

class SimpleTestCase(ExampleTestMixin, test.SimpleTestCase):
    pass

class TestCase(ExampleTestMixin, test.TestCase):
    pass

class TransactionTestCase(ExampleTestMixin, test.TransactionTestCase):
    pass

class LiveServerTestCase(ExampleTestMixin, test.LiveServerTestCase):
    pass

If you’re not using some classes, such as LiveServerTestCase, you could skip creating your own subclasses of them. I showed them all here to be exhaustive.

You could then use them in your tests like so:

from http import HTTPStatus

from example.test import TestCase

class LogoutTests(TestCase):
    def test_log_out_clears_all_site_data(self):
        response = self.client.post("/log-out/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertClearSiteData(response)

There are many use cases for custom test case classes, and we’ll only cover some in this book. Here are some examples:

Adding “shortcut” data factory functions (covered later in Use Factories).

Customizing the test client with a subclass by setting client_class.

Adding extra logic that runs before or after each test method or class, for example resetting an internal cache.

Attaching shortcuts to commonly used functions to reduce the amount of import statements needed to write tests.

Be careful of any execution overhead you add in custom test case classes, such as creating test data in setUp(), since it will apply to your entire test suite. If you do use something with noticeable overhead, you can always create your own extra test case class for those tests that need it. I’ve seen test suites run twice as fast after eliminating a few calls to create default data in setUp(), when that data was only used by a fraction of the tests.

Ensure Only Your Custom TestCase Classes Are Used

If you have some custom TestCase classes, they may end up containing vital logic for your environment. In this case, you probably want to make sure you use them throughout your project, and not Django’s classes, the plain unittest.TestCase, or others.

With Django’s Test Framework

You can do this in a custom runner that uses a custom unittest.TestLoader subclass. This is another piece of the unittest framework that can be overridden to control how tests get loaded. It would look like this:

import sys
from unittest import TestLoader

from django.test.runner import DiscoverRunner

from example.test.cases import (
    LiveServerTestCase,
    SimpleTestCase,
    TestCase,
    TransactionTestCase,
)

class ExampleTestLoader(TestLoader):
    allowed_test_case_classes = (
        LiveServerTestCase,
        SimpleTestCase,
        TestCase,
        TransactionTestCase,
    )

    def loadTestsFromTestCase(self, testCaseClass):
        if not issubclass(testCaseClass, self.allowed_test_case_classes):
            print(
                f"{testCaseClass.__module__}.{testCaseClass.__name__} does not"
                + " inherit from an example.test TestCase class",
                file=sys.stderr,
            )
            sys.exit(1)
        return super().loadTestsFromTestCase(testCaseClass)

class ExampleTestRunner(DiscoverRunner):
    test_loader = ExampleTestLoader()

Then when you added a bad class, you’d see this message:

$ python manage.py test
example.core.tests.test_bad.BadTest does not inherit from an example.test TestCase class

With pytest-django

Install the pytest-restrict plugin that I maintain, and configure it in your pytest.ini. For example:

[pytest]
addopts = --restrict-types example.test.cases.LiveServerTestCase,example.test.cases.SimpleTestCase,example.test.cases.TestCase,example.test.cases.TransactionTestCase

Third-Party Packages

There are many third-party packages available for extending Django’s built-in test framework.

There are some general purpose Python testing extensions, such as parameterized. Unfortunately I don’t know of any good index for these.

For Django specific packages, you can see many of them in the testing section of djangopackages.org. These can plug in at any of the layers covered in this chapter - the framework, the command, the runner, test case classes - or elsewhere. For example, django-test-plus provides custom test case classes and a runner.

If you’re using pytest, you also have its huge plugin ecosystem to employ. See How to install and use plugins for a start.

I’ll mention any relevant packages throughout the book.

Measure!

To understand God’s thoughts we must study statistics, for these are the measure of his purpose.

—Florence Nightingale

Before you improve your test runtime, you should measure it. This will inform you of the best places to look for improvements.

Let’s look at the various profiling tools available.

Throughout this chapter I’ll be using a project with some tests that are deliberately slow due to use of time.sleep():

import time

from django.test import TestCase

class VariableTimingTests(TestCase):
    def test_fast_one(self):
        time.sleep(0.1)

    def test_fast_two(self):
        time.sleep(0.1)

    def test_slow_one(self):
        time.sleep(1.0)

    def test_slow_two(self):
        time.sleep(2.0)

Built-in Output

Both test frameworks output a message at the end about the time taken to run tests.

With Django’s Test Framework this looks like:

$ python manage.py test
...
Ran 4 tests in 3.214s

OK

With pytest, this looks like:

$ pytest
...
========================== 4 passed in 3.55s ==========================

pytest is slower for two reasons. First, it is a larger framework, so it needs more time to start up and provide all its features. Second, it includes the time for database setup, whilst Django’s Test Framework does not.

From Django 3.2, Django’s Test Framework can time database setup separately with the --timing flag:

$ python manage.py test --timing
...
Ran 4 tests in 3.215s

OK
Destroying test database for alias 'default'...
Total database setup took 0.156s
  Creating 'default' took 0.156s
Total database teardown took 0.000s
Total run took 3.383s

All the above timing numbers are measured from within the test framework. This means they don’t include any overhead incurred getting to the point the test framework starts timing, such as slow-to-import modules.

So, if we can’t quite trust these numbers for “total time taken”, what can we trust?

Shell Timing Commands

Rather than measuring “from the inside” of your Python test process, you can measure “from the outside” using your shell’s timing command.

On Unix-like systems, you can use the time command:

$ time python manage.py test
...
Ran 4 tests in 3.208s

OK
real         3.56
user         0.29
sys          0.05

The number to pay attention to here is the “real” time. The other two measure parts of this - the CPU time spent in the program and the operating system - see more on Wikipedia.

On Windows, you can use PowerShell’s Measure-Command:

> Measure-Command {python manage.py test}
...

Ran 3 tests in 0.011s

OK

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 563
...

These commands will always produce a larger number than “from the inside”. This is because it includes things that can never be measured in the test runner, such as the time both Python and Django take to initialize. Here it’s an extra 0.3 seconds, which isn’t much, but that can be significant if you have some slow-to-import libraries.

At the end of the day, this total is the actual duration you care about reducing. Therefore you might want to update your CI (Continuous Integration) system to always run your tests with your shell’s timing command, so you always have your test run time measured in your logs.

Measure Individual Tests

Although you want to reduce the total test time, it’s still mostly made up of the time spent running individual tests. If you measure the timing of these individually, you can find the slowest and prioritize speeding them up.

Your tests’ runtime will tend to follow the law of nature known as the Pareto principle. That is, 20%-ish of your tests will take 80%-ish of the total run time. Therefore, you want to find that 20% so you can speed it up!

Let’s look at how you can time your tests and get a report of the slowest ones.

With Django’s Test Framework

Unfortunately, neither Django nor unittest have a built-in mechanism for test timing. Thankfully though there are some community resources.

There’s a package called django-slowtests that provides a custom test runner with test timing functionality. At time of writing it appears to only have official support up until Django 2.1, but it’s likely it works with later versions too.

When installed it gives output like:

$ python manage.py test
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 3.208s

OK
Destroying test database for alias 'default'...

2 slowest tests:
2.0001s test_slow_two (example.core.tests.VariableTimingTests)
1.0000s test_slow_one (example.core.tests.VariableTimingTests)

You may want to use it alongside other test runner customizations suggested in this book. If so, you should only need to change the examples to inherit from its DiscoverSlowestTestsRunner class instead of Django’s DiscoverRunner.

If django-slowtests doesn’t work for you, there’s also a great blog post from Haki Benita titled Timing Tests in Python For Fun and Profit. It covers the steps to making a custom Django test runner that implements similar functionality.

With pytest-django

pytest has a built-in ability to time test durations, including the split between setup, call, and teardown. It displays this in a report at the end of its normal output. You do this by running your tests with the --durations argument and the number of top slowest tests to capture:

$ pytest --durations 10
========================= test session starts =========================
...
collected 4 items

example/core/tests.py ....                                      [100%]

====================== slowest 10 test durations ======================
2.00s call     example/core/tests.py::VariableTimingTests::test_slow_two
1.00s call     example/core/tests.py::VariableTimingTests::test_slow_one
0.15s setup    example/core/tests.py::VariableTimingTests::test_fast_one
0.10s call     example/core/tests.py::VariableTimingTests::test_fast_one
0.10s call     example/core/tests.py::VariableTimingTests::test_fast_two

(0.00 durations hidden.  Use -vv to show these durations.)
========================== 4 passed in 3.42s ==========================

Note that setup and teardown are measured independently to the tests. This is a small project so “10” captured all 5 durations, but on larger projects using 10 or 20 is most useful.

You probably don’t want to run tests in development with --durations every time. But you could add it to your CI system’s configuration to keep the data readily available in its logs.

Profile

Looking at individual tests’ timing provides a great start for investigations. However it’s not very granular. We still don’t know what is slow inside the individual tests, nor are we timing what happens outside the test functions.

We can solve this by using a profiling tool. These time individual function calls, or even lines of code. With this increased level of information, we can find slow points quickly.

There are many Python profilers out there. Here we’re going to look at two:

py-spy - a separate tool.

cProfile - a library built-in to Python.

They use two different techniques.

py-spy is a sampling profiler. This means that it pauses the program repeatedly (by default, 100 times per second) to record which stack of functions is running.

cProfile is an event-based profiler. It records every function execution so it sees everything that happens.

py-spy’s approach has a much lower overhead, which means you can even use it to profile your application in production. But it can miss some things, like a fast function that is called many times. cProfile will capture this, but it’s too much overhead to use in production. You can read more about the difference between the two techniques in the cProfile documentation.

I’d recommend trying py-spy first on your tests, to get a picture of what’s happening. Then if you need to dig deeper, turn to the higher resolution of cProfile. We’ll cover them in this order in the following sections, along with relevant visualization tools.

If you’ve never used profiling tools, you might find this quite daunting. Just remember, you don’t need to understand everything at once. If you can navigate the visualizations enough to spot the slowest points in your tests, that’s a win.

py-spy

py-spy’s documentation lives in its GitHub readme. It’s a standalone tool that inspects the memory of your Python process to find the running functions.

You can install it from pip:

$ python -m pip install py-spy

There are also packages and binaries for some platforms; see the installation instructions.

py-spy has a few sub-commands. Here we’ll only use record, since that’s most useful for profiling a whole test run.

Record a Test Run

You can record a whole test run by running it under py-spy:

$ sudo py-spy record -o profile.svg --idle --subprocesses -- python manage.py test
py-spy> Sampling process 100 times a second. Press Control-C to exit.

System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 3.209s

OK

py-spy> Stopped sampling because process exitted
py-spy> Wrote flamegraph data to 'profile.svg'. Samples: 329 Errors: 0

Let’s take apart that command.

First, it uses sudo to run as root. This is necessary on macOS, but not always on Linux - see the FAQ. You shouldn’t need extra privileges on Windows.

Second, it uses -o profile.svg to save the default flame graph output as an SVG image. We’ll look at this image shortly.

Third, the --idle option ensures that our slow tests which use time.sleep() are recognized as “active” tests. You shouldn’t normally need this - it’s only to make our demo slow tests appear in the output. It’s unlikely a real project’s tests are slow because of calls to time.sleep().

Fourth, the --subprocesses argument means py-spy will also profile launched subprocesses. This is useful if you are using parallel testing. Our command doesn’t here, but I hope after Parallelize, you will be.

Finally, the “--” separates py-spy’s arguments from the program it should run and record. After this we list our test command. If you’re using pytest-django, swap python manage.py test for pytest. You can also pass any arguments on to the test command. For example, you can run an individual test module, class, or function by passing its path.

View the Flame Graph

The output graph looks like this:

Note

Sorry the text is small here. py-spy intends you to use its SVG’s in a web browser, where you can click to re-slice the data, zoom in, and search. If you have the included resources.zip, you can open 03_measure/time-tests/profile.svg for the interactive experience.

This is called a flame graph, because its colouring makes it look like flames of function call stacks. In this example the “flames” are pointing downwards, but they’re often also shown the other way up to really look like flames.

At the top is the root “all” node, which contains all measured processes and threads. The node below that is the python manage.py test process, and that contains the <module> execution for manage.py. Each node from there down is a Python function call. Nodes are coloured randomly from the palette of flamey colours, for visual separation only.

On the fifth line, we can see the separation between two execute() calls. The first is Django setting itself up, and the second is the actual execution of the “test” command.

The first execute() column is Django starting up. It’s not particularly interesting on this project, and it’s quite a lot of calls so I’ve cropped it.

The second execute() is the test command running. Underneath it, the various test runner and suite calls are stacked, until we reach our test methods at the bottom. There we can see two short boxes with the names cropped (the fast tests), then the two slow ones.

This is a great first visualization to look at, but there’s a more powerful tool which py-spy can work with: speedscope.

speedscope

speedscope is a cross-language profiling visualization tool. It loads data from a JSON format and presents it with multiple interactive views. It’s a browser-based tool using local storage only, so it won’t upload your data anywhere. However if you are concerned about the security implications of uploading your program’s stack traces, you can run it yourself - see its documentation.

py-spy can write out data in the speedscope format with --format speedscope:

$ sudo py-spy record --format speedscope -o profile.speedscope --idle --subprocesses -- python manage.py test

To open this, you can load the speedscope page at https://www.speedscope.app and drag-and-drop the file onto it.

Note

If you have the included resources.zip, you can follow along in your browser. Open speedscope and drop 03_measure/time-tests/profile.speedscope onto it.

Once loaded, you have three views of the data, based on the tabs at the top left.

The first is the “time order” flame graph:

This is much the same as py-spy’s default output, although prettier and more interactive. The graph shows the same stack of nodes from python manage.py test down. The X axis is measured in samples, for which py-spy defaulted to 100 per second.

The colouring now carries some information, grouping functions from the same module with similar colours. There are five groups in the graph:

pink/purple for Python internals

green for Django’s command execution

blue for for Django’s test framework

darker olive for unittest

light olive for our tests

Above the the flame graph is a mini-map, which is useful for navigating when zooming in. You can focus on a single function call by clicking, and zoom in to its sub-tree by double-clicking.

The second view is “left heavy”:

This rearranges nodes left-to-right by execution time, so the slowest, or “heaviest” ones appear first. This allows you to find the slowest tests and then inspect why they are slow.

Here I’ve clicked on the test_slow_two node to expand it in the bottom panel, which has two sections.

The left section shows a summary of time spent within this function versus its callees. py-spy doesn’t pick up on time.time() so it shows all the time (48% of the total runtime) being spent within the test. The “This instance” and “All instances” sections are equivalent here because test_slow_two was only run once.

The right section shows the inverted stack trace leading to this function call. This is useful for navigating back into the actual code.

The third view is “sandwich”:

This allows you to sort by total runtime. This is useful for functions that are called many times in different places, adding up to a significant portion of the runtime. For example, if most of your tests used a particular helper function, you could find it here by sorting by “total”.

You can then check which functions call it (“callers”) and which functions it calls (“callees”), with the mini flame graphs on the right.

For more information, including the shortcuts, see the speedscope documentation.

cProfile

cProfile is one of two profilers built into Python. The other is called “profile”. They have the same API - the only difference is that, while profile is built in Python, cProfile is built in C, so it’s much faster. I’m referring only to cProfile here since it’s normally the one you want to use, and its name is more distinctive. They’re documented together in The Python Profilers.

Record a Test Run

cProfile has both a module entry point and a Python API. We’ll use the entry point here.

To use it to wrap your Django tests, run:

$ python -m cProfile manage.py test
System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 3.211s

OK
         297743 function calls (287955 primitive calls) in 3.898 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    33/23    0.000    0.000    0.193    0.008 <frozen importlib._bootstrap>:1002(_gcd_import)
  673/317    0.001    0.000    0.381    0.001 <frozen importlib._bootstrap>:1017(_handle_fromlist)
      753    0.002    0.000    0.002    0.000 <frozen importlib._bootstrap>:103(release)
      557    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:143(__init__)
      557    0.001    0.000    0.007    0.000 <frozen importlib._bootstrap>:147(__enter__)
      557    0.000    0.000    0.002    0.000 <frozen importlib._bootstrap>:151(__exit__)
...

To run pytest, swap manage.py test for -m pytest:

$ python -m cProfile -m pytest

Note

The -m option was added in Python 3.7. On older versions, you’ll need to add a wrapper script that calls pytest.main().

At the end of the test runner output, you can see the profiling report. This is generated by another built-in module called pstats, meaning “profiler stats”. It includes of the number of calls, the total time the program took, and then a table of the called functions. It’s similar to the “sandwich” view on speedscope.

I have had to snip off a lot of output here. I’ve only shown the first six function calls. But the total, on this example Django project with no dependencies, is 3821 functions. And in the real world you could have 10 or 100 times more!

Weirdly, the default sort order is by function name. This has made the functions inside Python’s import machinery rise to the top, with its odd module name <frozen importlib._bootstrap>.

To find our tests’ slow points, we want to sort by one of the time columns, set with the -s flag. The best starting point is cumtime, standing for cumulative time. This is the total time spent in the function across all calls, including time spent in callees.

The next most useful is tottime, which is the total time spent in the function across all calls, excluding time spent in callees. For definitions of the other columns, see the documentation.

You can also can tame the massive amount of output by piping it into a pager program.

Let’s add those two options.

On Unix-like systems, run:

$ python -m cProfile -s cumtime manage.py test 2>&1 | less

The 2>&1 combines the stdout and stderr. Then | less pipes the output into the less command.

On Windows, swap less for more:

> python -m cProfile -s cumtime manage.py test 2>&1 | more

(It’s also possible to install less on Windows through various packages. It may be useful as it has more features than more.)

When you run this, you’ll see output like:

....
----------------------------------------------------------------------
Ran 4 tests in 3.208s

OK
System check identified no issues (0 silenced).
         297535 function calls (287740 primitive calls) in 3.545 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    523/1    0.003    0.000    3.545    3.545 {built-in method builtins.exec}
        1    0.000    0.000    3.545    3.545 manage.py:2(<module>)
        1    0.000    0.000    3.545    3.545 manage.py:7(main)
        1    0.000    0.000    3.470    3.470 __init__.py:398(execute_from_command_line)
        1    0.000    0.000    3.470    3.470 __init__.py:321(execute)
        1    0.000    0.000    3.237    3.237 test.py:16(run_from_argv)
        1    0.000    0.000    3.236    3.236 base.py:311(run_from_argv)
      2/1    0.000    0.000    3.226    3.226 base.py:347(execute)
        1    0.000    0.000    3.226    3.226 test.py:49(handle)
        1    0.000    0.000    3.226    3.226 runner.py:669(run_tests)
        1    0.000    0.000    3.208    3.208 runner.py:627(run_suite)
        1    0.000    0.000    3.208    3.208 runner.py:151(run)
        1    0.000    0.000    3.208    3.208 suite.py:83(__call__)
        1    0.000    0.000    3.208    3.208 suite.py:102(run)
        4    0.000    0.000    3.207    0.802 testcases.py:253(__call__)
        4    0.000    0.000    3.207    0.802 case.py:735(__call__)
        4    0.000    0.000    3.207    0.802 case.py:641(run)
        4    0.000    0.000    3.207    0.802 case.py:632(_callTestMethod)
        4    3.207    0.802    3.207    0.802 {built-in method time.sleep}
        1    0.000    0.000    2.000    2.000 tests.py:16(test_slow_two)
        1    0.000    0.000    1.001    1.001 tests.py:13(test_slow_one)
...

Note

With both less and more, you can scroll the output with the arrow keys, and quit by pressing the “q” key.

In the bottom two lines, just before I snipped the output, you can see the two slow tests. And above that, there’s time.sleep, showing the cumulative time spent inside it - 3.207 seconds. Our tests’ four calls sleep a total of 3.2 seconds - the extra 0.007 comes from operating system rounding, and profiling overhead.

(Also note that py-spy didn’t capture time.sleep() because it’s written in C. It might be able to in a future version.)

This view gives you a great clue of what to investigate to speed things up. For any function with a high cumulative time, you can either try write it in a faster method, or reduce the number of calls.

But how can we find what’s calling it? cProfile records that information, but provides no built-in visualization tool. Unfortunately there’s no speedscope integration, but we can use another tool: kcachegrind.

kcachegrind

kcachegrind is a popular, cross-language profile visualization tool. It has been developed as an open source project since 2002. While its GUI looks a bit dated, it has a lot of useful features.

Installing can be a bit confusing. It’s available as either kcachegrind or qcachegrind.

On Linux, you should be able to find one of these through your system’s package manager.

On macOS, you can brew install qcachegrind.

On Windows, you should be able to install it via the conda package.

We also need one other tool, pyprof2calltree. This can convert the Python profile format to the “calltree” format used by kcachegrind. Run:

python -m pip install pyprof2calltree

Then you can record a profile of your tests using cProfile’s -o flag:

$ python -m cProfile -o profile.prof manage.py test
System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 3.212s

OK

With the -o flag, cProfile doesn’t display a report. It only generates the output file:

$ du -sh profile.prof
544K    profile.prof

You can then use pyprof2calltree to convert this profile file and open it directly in kcachegrind with:

$ pyprof2calltree -i profile.prof -k

kcachegrind looks quite dense:

The left panel lists all functions. The “Incl.” column is inclusive runtime, the same as “cumtime” in the cProfile table report, and this is how I’ve sorted the list (by clicking on the column heading). I’ve selected the _callTestMethod() function, the unittest wrapper around each test.

The right hand side includes two visualization panels with many tabs.

In the top right panel, I’ve selected “Callers”. This lists the functions that called the selected one. In this case, it’s another unittest internal function run().

In the bottom right panel, I’ve selected the call graph. This is a very useful view for tracing up and down called functions to find the reason for the slowness. This shows us that _callTestMethod() took 83.3% of the total runtime, and how that divides amongst the functions it called. At the bottom we can trace and find that time.sleep() is the cause of the slowness, as it took 77.96% of the total runtime.

I’ve tried to select one of the most useful views, but there’s a lot you can discover through the other options. It’s worth playing around for a bit. The next most useful thing to check is sorting functions by their “Self” runtime, which is equivalent to cProfile’s “tottime” column.

pytest Integration

As with many things, “there’s a pytest plugin for that”. pytest-profiling can profile your tests with cProfile. It can then output the profiling report at the end of the run and store the profile data in a prof/ directory.

You can use that profiling data with kcachegrind as above, or through its direct integration with another visualization tool, gprof2dot.

PyCharm Integration

If you’re using PyCharm as your editor, it has a built-in cProfile integration. This provides some of the same visualization capabilities as kcachegrind. Check out its documentation.

Easy Wins

“No matter how smart you are, you’re smarter if you take the easy ways when they are available.”

—Daniel C Dennett, Intuition Pumps and Other Tools for Thinking

Here are some “easy wins” - smaller, less invasive changes that you can try on your project without editing the tests themselves. They won’t magically speed up your tests by an order of magnitude, but saving a few seconds here and there can add up. Most are quite dependent on the nature of your project, so read to see if they’re relevant, try, and find out!

Use a Faster Password Hasher

This is a recommendation straight from Django’s documentation section Speeding up the tests. It might one day be made into a default - see Ticket #20917.

If your project is like most Django projects and uses django.contrib.auth, it probably creates users during tests. Django’s default password hashing algorithm is slow by design - it’s intended to prevent attackers from testing logins with many different passwords quickly. But this means every user created in tests has this overhead imposed. If you switch to the much less secure MD5 based password hasher, you can speed things up. Set PASSWORD_HASHERS in your test settings to just use that:

PASSWORD_HASHERS = [
    "django.contrib.auth.hashers.MD5PasswordHasher",
]

Be careful that this has no way of making its way into your production settings.

The Django docs also warn:

Don’t forget to also include in PASSWORD_HASHERS any hashing algorithm used in fixtures, if any.

If you do use more than one hasher, make sure MD5PasswordHasher is first, so that it applies to passwords stored in tests. The other hashers in the list are used for decoding passwords only.

Technically you could make this a tiny bit faster by implementing a password hasher that doesn’t do any hashing. But don’t, since if that accidentally made its way into production, you’d be storing unhashed passwords, breaking your users’ security entirely. (Not that MD5 hashing is very secure these days, as it’s very cheap to break even with brute-forcing.)

Always Rebuild the Test Database if It Exists

Note

This section only applies to the Django test framework. pytest-django automatically rebuilds the test database if it exists.

This tip won’t make your actual test run any shorter, but it will save you developer time.

(I previously posted a version of this on my blog and it has been quite popular.)

When using Django’s test runner, you can encounter this prompt when not passing --keepdb:

$ python manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: database "example_test" already exists

Type 'yes' if you would like to try deleting the test database 'example_test', or 'no' to cancel:

The test database is normally left around when a previous test run crashed and couldn’t clean it up.

The prompt to rebuild it can be useful on some projects which need consideration around their test database, but for most it is only an annoyance. You start a test run, go for a cup of tea, and come back to find this prompt still waiting, rather than your results.

You can stop this prompt by passing --noinput, in which case Django doesn’t ask and always deletes the existing database. You could make --noinput default to on in a custom test management command that subclasses Django’s built-in test command. Indeed this is the approach I used on my blog post.

However always running the tests in “non-interactive” mode might unintentionally disable some other features. I can’t find any such features in Django 3.0, but future versions might add something.

Instead of overriding the flag for the whole test run, you can replace it for only the database setup phase in a custom test runner:

from django.test.runner import DiscoverRunner

class ExampleTestRunner(DiscoverRunner):
    def setup_databases(self, **kwargs):
        # Force to always delete the database if it exists
        interactive = self.interactive
        self.interactive = False
        try:
            return super().setup_databases(**kwargs)
        finally:
            self.interactive = interactive

This function is the point at which Django creates the database (source). The --noinput flag maps internally to set interactive = False. Here we’re temporarily forcing it to False.

With this in place, you’ll never see that prompt again!

Disable Database Serialization

Django serializes your whole database into a SQL string at the start of a test run, which can take a few seconds. This SQL string is then used for rollback on TestCase classes where the serialized_rollback flag is set to True. The flag is documented at Rollback emulation.

Most projects do not use this rollback flag. Therefore, you can probably disable the serialization. You can do this by setting the SERIALIZE option to False in your database’s TEST options:

 DATABASES = {
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
         "NAME": os.path.join(BASE_DIR, "db.sqlite3"),
         "TEST": {"SERIALIZE": False},
     }
 }

The SERIALIZE option will be deprecated in Django 4.0. Instead, Django will detect when no test cases use serialized_rollback and automatically skip the serialization step. This change is thanks to Simon Charette in Ticket #32446.

If you have any test cases that required serialized_rollback, there’s normally a way to rewrite them to be faster. More on such techniques later in Test Structure and Test Data.

Disable Instrumentation Packages during Tests

You may be using some packages that add instrumentation, for example:

A development tool, such as django-debug-toolbar.

An error handling tool, such as Rollbar or Sentry.

An Application performance Monitoring (APM) tool, such as New Relic or Scout APM.

Such packages often work by tracking every database query, cache operation, etc. The overhead for all this instrumentation is normally significant. For example, on a couple of project’s test suites, I’ve seen Sentry add about 8% to the total test run time.

Since you often don’t need their functionality during tests, you can disable these packages in your test settings for a speed boost. Some packages provide an option to disable their instrumentation features. If that’s not available, you can change your settings to never load them during tests.

Here are some examples on how to disable such packages. If you are using a different instrumentation package, hopefully you can find similar configuration. (And please let me know the package and configuration so I can add it in a future edition.)

django-debug-toolbar

If you’ve followed the normal install instructions and haven’t customized SHOW_TOOLBAR_CALLBACK, the toolbar is only active when DEBUG = True. Therefore you should only need to check that you Don’t Run Tests in Debug Mode.

Rollbar

Rollbar’s Django setup documentation lists its various settings, configured in Django through the ROLLBAR setting dict. If you set enabled to False, the Rollbar middleware will disable itself and never install instrumentation:

ROLLBAR['enabled'] = False

Sentry

If you use a single settings file powered by environment variables, you can do this by installing Sentry only when the environment variable is there:

if "SENTRY_DSN" in os.environ:
    import sentry_sdk

    sentry_sdk.init(...)

If you use multiple settings files, only include the calls to sentry_sdk.init() in the relevant files, such as staging and production.

New Relic

If you’re using the default installation method, New Relic only adds its instrumentation when you run your application via newrelic-admin run-program. This means there’s no overhead during tests.

But if you’re using the manual integration, you’ll have a call to newrelic.agent.initialize() in your wsgi.py or similar. You should make sure that this isn’t called during tests, perhaps by using an environment variable to prevent it.

Scout APM

Full disclosure: I’m the maintainer for the Scout APM Python agent.

Scout only installs its instrumentation if its monitor setting is True. This can be set via your Django settings or environment variables as SCOUT_MONITOR. It defaults to False, so ensure it’s either not set, or set explicitly to False, during tests.

Use an In-Memory File Storage Backend

Your project probably uses Django’s file storage API somewhere, for example through FileFields. If your tests exercise this functionality, you are probably using the built-in FileSystemStorage and are writing files to disk.

Writing to disk is about an order of magnitude slower than writing to memory. There’s a third party package, dj-inmemorystorage that you can use to keep files in memory instead. Install it and configure your test settings to use it:

DEFAULT_FILE_STORAGE = "inmemorystorage.InMemoryStorage"

Nice! Another advantage over FileSystemStorage is that files won’t be left on disk at the end of tests.

You may have multiple storage backend instances in your project. For example, FileField and ImageField both take an optional storage argument to use a non-default backend. In this case, you’ll want to swap all those instances too.

You can do this by adding a custom setting similar to DEFAULT_FILE_STORAGE that contains the path to the backend class. Then in module that you create your custom storage backend, you can use Django’s import_string() to import the named class. For example:

from django.conf import settings
from django.db import models
from django.utils.module_loading import import_string

image_storage = import_string(settings.EXAMPLE_IMAGE_STORAGE)()

class Book(models.Model):
    cover = models.ImageField(storage=image_storage)

This technique will work on some projects. But reading a setting at import time is not recommended, since it can change at run time through @override_settings (see Mock Settings). In particular, if you use the technique mentioned previously for a single settings file, its overrides won’t have been applied at import time. Django gets around this by providing some proxy objects that read settings at run time only, one of which is django.core.files.storage.default_storage.

You can create a similar proxy for your custom storage instance like so:

from django.conf import settings
from django.core.signals import setting_changed
from django.dispatch import receiver
from django.db import models
from django.utils.functional import SimpleLazyObject
from django.utils.module_loading import import_string

@SimpleLazyObject
def image_storage():
    return import_string(settings.EXAMPLE_IMAGE_STORAGE)()

@receiver(setting_changed)
def reset_image_storage(*, setting, **kwargs):
    if setting == "EXAMPLE_IMAGE_STORAGE":
        image_storage._setup()

class Book(models.Model):
    cover = models.ImageField(storage=image_storage)

This uses the undocumented but stable Django class SimpleLazyObject to create a proxy around the decorated function. The function will be run “lazily”, when the proxy is first interacted with. The second function is a receiver for the setting_changed signal, which resets the proxy whenever the setting changes. This will work with usage of @override_settings and any of its alternatives.

Nice!

Use an In-Memory Cache Backend

You’re very likely using Django’s cache framework, either directly or through third party packages. By default, caching uses the fast local-memory backend, so if you haven’t configured CACHES, you don’t need to do anything here. But if are using a different backend, you can normally speed up your tests by using the local-memory backend in your test settings:

CACHES = {
    "default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}
}

You will need to add the "LOCATION" key per backend if you’re using more than one - see the documentation.

Whichever backend you’re using, Django does not clear caches between tests. This is bad for test isolation - a test can set a cache key that is read by a later test, changing its results. More on this later in Check Your Tests Are Isolated.

Test cache clearing is an unfortunately old open issue in Django: Ticket #11505. It’s a tricky issue because there are many different cache backends to consider. Hopefully we’ll see a resolution for it in a future version of Django.

You can implement it for your project with custom TestCase classes that clear all caches after each test:

from django.conf import settings
from django.core.cache import caches
from django.test import SimpleTestCase, TestCase

class ExampleTestCaseMixin:
    def _post_teardown(self):
        super()._post_teardown()
        for name in settings.CACHES:
            caches[name].clear()

class SimpleTestCase(ExampleTestCaseMixin, SimpleTestCase):
    pass

class TestCase(ExampleTestCaseMixin, TestCase):
    pass

This will slow down your tests slightly, but with the local-memory backend it shouldn’t be significant.

Another option is to use Django’s dummy cache backend in your test settings:

CACHES = {"default": {"BACKEND": "django.core.cache.backends.dummy.DummyCache"}}

The dummy cache backend does not do any caching - it never stores anything, and it always returns misses for lookups. This means state cannot leak between tests, and it’s slightly faster than using the local-memory cache backend. The downside is that your tests won’t cover any of the cached pathways in your code. You can test those pathways by setting CACHES to use the local-memory backend temporarily, using @override_settings (see Mock Settings). For example, to override for a single test method:

from django.test.utils import override_settings

from example.core.views import get_book_aggregates
from example.test import TestCase

local_memory_caching = override_settings(
    CACHES={
        "default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}
    }
)

class GetBookAggregatesTests(TestCase):
    def test_with_dummy_cache(self):
        with self.assertNumQueries(1):
            value1 = get_book_aggregates()
        self.assertEqual(value1, {"total_books": 9999})

        with self.assertNumQueries(1):
            value2 = get_book_aggregates()
        self.assertEqual(value1, value2)

    @local_memory_caching
    def test_with_local_memory_caching(self):
        with self.assertNumQueries(1):
            value1 = get_book_aggregates()
        self.assertEqual(value1, {"total_books": 9999})

        with self.assertNumQueries(0):
            value2 = get_book_aggregates()
        self.assertEqual(value1, value2)

The first test test_with_dummy_cache() checks that by default, with the dummy cache backend in place, the get_book_aggregates() function makes a database query on every access. Then test_with_local_memory_caching(), with its @override_settings decorator, checks that its second access does not make a query.

Great!

django-lrucache-backend

If you’re using the local-memory backend, there’s a faster version available in the package django-lrucache-backend. It will only make a difference for your tests if you cover a lot of caching code. But if you’re also using local-memory caching in production, it could be a big win. Long time Django contributor Josh Smeaton created it and covered it on on his company’s blog.

Fake Redis Caching

You might be using Redis as your cache backend via the popular django-redis package. In this case, you might not be able to use the local-memory backend for your tests, because it doesn’t have Redis-specific features.

You can still speed it up though, by using fakeredis. This is an in-memory Python only replacement for the Redis library. You can configure django-redis to use it by reconfiguring django-redis’ CONNECTION_POOL_KWARGS and REDIS_CLIENT_CLASS options in your test settings:

from fakeredis import FakeConnection, FakeServer
from functools import partial

create_fake_connection = partial(FakeConnection, server=FakeServer())

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://localhost:6379/0",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "CONNECTION_POOL_KWARGS": {
                "connection_class": create_fake_connection,
            },
            "REDIS_CLIENT_CLASS": "fakeredis.FakeStrictRedis",
        },
    }
}

Great!

Use In-Memory Backends for your Task Queues

Most Django projects use a third party task queue package for background processing. These normally have two configuration options to speed up the tests: an “eager” mode, and in-memory storage.

Eager Mode

Eager mode means that wherever a task would be enqueued, it is instead executed immediately. This alone is normally not useful during tests, since it can mean a lot of unnecessary processing. It can also be bad for testing how your tasks deal with data consistency, since they run earlier than they would in a production environment.

But if you are correctly using Django’s transaction.on_commit() for your enqueues, and TestCase for its transactional behaviour, it will be less of a problem. TestCase always rolls back its transactions, so the on_commit() callbacks don’t run, and the tasks won’t be eagerly executed. The callbacks and tasks can still be tested though, as covered later in on_commit() Callbacks.

Eager mode won’t necessarily speed up your tests, but it will make them faster to write. It’s not necessarily an easy switch though.

In-Memory Storage

Using in-memory storage can be a big win when it comes to task queues. Not only does it have the same speed boosts of memory versus disk, but it can also reduce the time needed to maintain your development and test environments.

Configuring Packages

I’ve covered how to configure the most popular queue packages in your test settings below. If yours isn’t listed, check its documentation, as it probably has similar options.

(While you’re looking at your task queue setup, you might want to run through my list of common issues using them.)

Celery

Celery supports both an eager mode and in-memory storage out of the box.

To use its eager mode, set these settings:

CELERY_ALWAYS_EAGER = True
CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

Activating exception propagation with CELERY_EAGER_PROPAGATES_EXCEPTIONS is optional, but it’s useful in tests as it prevents failures from being silenced. See the documentation.

Celery also supports an in-memory backend out of the box, but its documentation doesn’t mention it (at least that I can find). This is because it uses a second library called Kombu for its transport layer implementations. Kombu documents its in-memory transport without any restrictions, so it should be drop-in.

To use it, set the broker setting:

CELERY_BROKER = "memory://"

That should be all that’s needed.

Django-Q

Django-Q’s “eager” mode is called “sync”, as the task will be executed synchronously rather than asynchronously. You can add it by setting:

Q_CLUSTER = {
    ...
    "sync": True,
    ...
}

See the documentation on asynchronous testing.

Django-Q doesn’t support in-memory storage by default.

If you’re using Redis, you can use the django-redis integration and configure django-redis to use fakeredis during tests - see Fake Redis Caching for an example.

Otherwise, you could try using the orm storage. This should mean at least some speed increase due to Django’s TestCase using transactions to rollback the database changes.

Dramatiq

Dramatiq doesn’t provide an “eager” mode but it does provide in-memory storage.

You’re probably using django_dramatiq to use Dramatiq under Django. In this case you can configure Dramatiq to use its in-memory broker and result backends in your test settings:

DRAMATIQ_BROKER = {
    "BROKER": "dramatiq.brokers.stub.StubBroker",
}

DRAMATIQ_RESULT_BACKEND = {
    "BACKEND": "dramatiq.results.backends.stub.StubBackend",
}

If you’re not using django_dramatiq, you’ll want to switch to these stub classes directly where you set up Dramatiq in tests.

See the dramatiq documentation on unit testing and also the django-dramatiq documentation, which has some examples including with pytest.

Huey

Huey’s eager mode is called “immediate mode”. Setting this on both activates eager execution and moves storage to in-memory. You can use it with the built-in Django integration by setting:

HUEY = {
    # ...
    "immediate": True,
    # ...
}

If you aren’t moving to immediate mode, you can move to the in-memory backend only. You can do this by using MemoryHuey:

HUEY = {
    "huey_class": "huey.MemoryHuey",
    # ...
}

RQ

You’re probably using RQ via django-rq. If not, you’ll need to adapt these instructions.

RQ’s eager mode is configured by the ASYNC option. To activate it set the option to False on each queue you have configured:

RQ_QUEUES = {
    "default": {
        # ...
        "ASYNC": False,
        # ...
    },
}

Unfortunately there doesn’t seem to be a built-in way to use in-memory storage with django-rq. But see this GitHub comment for a monkey-patch you could use to switch to fakeredis, similar to Fake Redis Caching.

Skip Slow Tests Locally

If you’ve measured your tests, you’ll likely have found the slowest tests at the top end of your Pareto distribution. These might be integration tests that run several views, make full requests with LiveServerTestCase, or even run a browser through Selenium.

A typical workflow has developers run tests during development, as well as running on a Continuous Integration (CI) system. In this case, it can make sense to move your slowest tests to run only on CI. You trade off full coverage in development for a faster workflow, which is normally worthwhile.

Moving slow tests to CI only doesn’t replace rewriting them in a faster way, as we’ll cover later in Test Structure, but it can be done without much work.

To detect we’re on CI, we can check for an environment variable. All CI systems that I’ve seen set some variables that you can use to detect when your tests are running there. For example, several CI systems set CI=true. To check for your system, see the list I made on my blog for the most popular CI systems.

We can use this detection along with the test frameworks’ tagging features to disable slow tests locally.

With Django’s Test Framework

Django’s test framework provides a basic test tags feature. These act as extra “labels” on your test methods or classes that can be used to selectively run them.

For example, here are two tests, one tagged as slow:

from django.test import TestCase, tag

class MyTests(TestCase):
    def test_one(self):
        ...

    @tag("slow")
    def test_two(self):
        ...

You can also tag whole test cases and tags are inherited. See the documentation for more details.

You can run these tests with the --exclude-tag option to exclude the slow one:

$ python manage.py test -v 2 --exclude-tag slow
Excluding test tag(s): slow.
Skipping setup of unused database(s): default.
System check identified no issues (0 silenced).
test_one (example.core.tests.test_django_runner.MyTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

To avoid needing to remember to pass the tag on your development environment, you can set it automatically. You’d do this in your custom test runner:

import os
from django.test.runner import DiscoverRunner

class ExampleTestRunner(DiscoverRunner):
    def __init__(self, *args, exclude_tags=None, **kwargs):
        on_ci = os.environ.get("CI", "") == "true"
        if exclude_tags is None and not on_ci:
            exclude_tags = ["slow"]
        super().__init__(*args, exclude_tags=exclude_tags, **kwargs)

Make sure to adapt for the correct environment variable for your CI system - see my blog post covering most popular CI systems.

Once you’ve done this, running the tests locally automatically excludes the slow tag:

$ python manage.py test -v 2
Excluding test tag(s): slow.
Skipping setup of unused database(s): default.
System check identified no issues (0 silenced).
test_one (example.core.tests.test_django_runner.MyTests) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

But if the CI environment variable is set, or if you pass --exclude-tag, they run:

$ CI=true python manage.py test -v 2
Skipping setup of unused database(s): default.
System check identified no issues (0 silenced).
test_one (example.core.tests.test_django_runner.MyTests) ... ok
test_two (example.core.tests.test_django_runner.MyTests) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
$ python manage.py test -v 2 --exclude-tag another
Excluding test tag(s): another.
Skipping setup of unused database(s): default.
System check identified no issues (0 silenced).
test_one (example.core.tests.test_django_runner.MyTests) ... ok
test_two (example.core.tests.test_django_runner.MyTests) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Great!

With pytest-django

Similar to Django’s tags, pytest features the concept of test marks. These are richer since you can attach parameters to the marks too.

To add a slow mark, add it with a description to your pytest.ini configuration file:

[pytest]
DJANGO_SETTINGS_MODULE = example.settings
addopts = --strict-markers
markers =
    slow: marks tests as slow

This also adds the --strict-markers option, which means Pytest will error for unknown marks. This guards against typos like “solw” being added to tests.

You can then mark your tests with @pytest.mark.slow:

from django.test import SimpleTestCase

import pytest

class MyTests(SimpleTestCase):
    def test_one(self):
        ...

    @pytest.mark.slow
    def test_two(self):
        ...

You can then exclude the slow tests with the -m flag:

$ pytest -m 'not slow'
========================= test session starts =========================
...
collected 2 items / 1 deselected / 1 selected

example/core/tests/test_example.py .                            [100%]

=================== 1 passed, 1 deselected in 0.07s ===================

The -m flag also allows further boolean logic, such as -m 'core and not slow'.

To implement skipping outside of CI, we can adapt the pattern from the Pytest documentation Control skipping of tests according to command line option. This means adding a collection-time hook in our conftest.py like so:

import os

import pytest

def pytest_collection_modifyitems(config, items):
    on_ci = os.environ.get("CI", "") == "true"
    if on_ci or config.getoption("markexpr") != "":
        # On CI, or user passed -m, do not skip
        return

    skip_slow = pytest.mark.skip(reason="Skipping slow tests")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

Make sure to adapt for the correct environment variable for your CI system. (See my blog post.)

Once you’ve done this, running the tests locally automatically skips the slow tests:

$ pytest -v
========================= test session starts =========================
...
collected 2 items

example/core/tests/test_example.py::MyTests::test_one PASSED    [ 50%]
example/core/tests/test_example.py::MyTests::test_two SKIPPED   [100%]

==================== 1 passed, 1 skipped in 0.07s =====================

But if the CI environment variable is set, or if you pass -m, they run:

$ CI=true pytest -v
========================= test session starts =========================
...
collected 2 items

example/core/tests/test_example.py::MyTests::test_one PASSED    [ 50%]
example/core/tests/test_example.py::MyTests::test_two PASSED    [100%]

========================== 2 passed in 0.11s ==========================

Nice!

Prevent Output

Note

This section only applies to the Django test framework. Pytest automatically captures output for passing tests by default.

If your application creates a lot of output on stdout or stderr, using print(), logging, or similar, it can slow down your tests noticeably. Your terminal program takes time to store, render, and scroll all that output. In extreme cases, when you output several screens of text per test, your tests might run three times slower. Also, lots of output from passing tests makes it hard to debug failures!

Ideally, Django’s test framework would follow pytest here, which automatically captures output in an in-memory buffer, and outputs it on test failure.

Python’s unittest actually provides this behaviour as a command-line option --buffer. Django supports this since version 3.1, thanks to Baptiste Mispelon on Ticket #27430, but not when using parallel testing. And as we’ll see in Parallelize, parallel testing is one of the biggest speed boosts available.

Ticket #31370, worked on by your author, added support for --buffer with parallel testing. This change will be released in Django 4.0.

Until then, you can add your own output buffering with Custom TestCase Classes. You can override the unittest internal method _callTestMethod to add capturing, and output it on failure:

import sys
from io import StringIO
from unittest import mock

from django import test

# Make unittest treat this module as part of itself. This means it will skip
# frames in this module in tracebacks, allowing us to override
# _callTestMethod() without breaking where errors are reported.
__unittest = True

class ExampleTestMixin:
    def _callTestMethod(self, method):
        try:
            out = StringIO()
            err = StringIO()
            with mock.patch.object(sys, "stdout", new=out), mock.patch.object(
                sys, "stderr", new=err
            ):
                super()._callTestMethod(method)
        except Exception:
            print(out.getvalue(), end="")
            print(err.getvalue(), end="", file=sys.stderr)
            raise

class SimpleTestCase(ExampleTestMixin, test.SimpleTestCase):
    pass

class TestCase(ExampleTestMixin, test.TestCase):
    pass

class TransactionTestCase(ExampleTestMixin, test.TransactionTestCase):
    pass

class LiveServerTestCase(ExampleTestMixin, test.LiveServerTestCase):
    pass

Then when running tests, the output will only be displayed when an exception is raised, due to a failing assertion or an error. For example, imagine you had these tests:

from example.test import SimpleTestCase

class OutputTests(SimpleTestCase):
    def test_pass(self):
        print("Hello from test_pass()")
        self.assertTrue(True)

    def test_fail(self):
        print("Hello from test_fail()")
        self.assertTrue(False)

Only the output from the failing test will be displayed:

 $ python manage.py test
 System check identified no issues (0 silenced).
 Hello from test_fail()
 F.
 ======================================================================
 FAIL: test_fail (example.core.tests.test_captured.OutputTests)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File "/.../example/core/tests/test_captured.py", line 11, in test_fail
     self.assertTrue(False)
 AssertionError: False is not true

 ----------------------------------------------------------------------
 Ran 2 tests in 0.001s

 FAILED (failures=1)

Note that this is an imperfect copy of the --buffer feature. Output from code run outside of tests, such as during imports or setUp(), will still be displayed. And the output is interleaved with test execution, rather than displayed alongside the assertion.

We’ll look at output mocking in more depth later, in Mock Output.

Prevent Logging

Note

This section only applies to the Django test framework. Pytest automatically captures all logging by default.

This follows on from the above. Just like other kinds of output, if your logging is configured to appear on standard output, it slows down your tests.

Python’s logging module configuration is quite layered, but you can turn it off in one command with logging.disable(). You can call this in a custom test runner before tests start:

import logging
from django.test.runner import DiscoverRunner

class ExampleTestRunner(DiscoverRunner):
    @classmethod
    def add_arguments(self, parser):
        parser.add_argument(
            "--enable-logging",
            action="store_true",
            default=False,
            help="Actually edit the database",
        )

    def __init__(self, *args, enable_logging, **kwargs):
        super().__init__(*args, **kwargs)
        self.enable_logging = enable_logging

    def run_tests(self, *args, **kwargs):
        if not self.enable_logging:
            logging.disable(level=logging.CRITICAL)

        return super().run_tests(*args, **kwargs)

(Passing level=logging.CRITICAL is not necessary on Python 3.7+ since it’s the new default.)

This extension allows you to re-enable logging if you need to see it for a failing test by passing --enable-logging to python manage.py test.

Reduce pytest Test Collection

pytest tries harder than unittest to find test files, recursing through all directories, even if they aren’t directly importable in Python. It can thus check many unnecessary directories, such as installed packages or generated files. If we see pytest spending a noticeable amount of time on its collecting... step, it may be searching too many directories.

We can prevent pytest from searching inside certain directories by providing patterns in its norecursedirs option, in our pytest.ini (or other configuration file). Any directory matching any of these fnmatch-style patterns will be ignored. It’s a good idea to include any large directories in your project that definitely do not contain tests.

For example, to avoid any directory called templates we would list it as a pattern:

[pytest]
DJANGO_SETTINGS_MODULE = example.settings
norecursedirs = .* build dist *.egg node_modules templates venv

Note that defining the value overrides the default, so we need to copy in applicable entries from pytest’s default, as listed in its documentation. For example, it’s normally useful to keep .*, which avoids recursing through hidden directories such as Git’s .git.

Thanks to this book, I made a PR to pytest to add node_modules to the default value. This has been released in pytest 6.2.0. node_modules is a common culprit on Django projects since it can be very large whilst not containing any tests.

Prevent WhiteNoise From Scanning All Static Files

The WhiteNoise package is popular for serving static files. It serves static files from Python, so it’s easy to set up compared to configuring another web server. But by default, it imposes a test startup overhead, as it inspects all your static files.

The problem is described under the heading “WhiteNoise makes my tests run slow!” in its Django guide. The solution is to disable the startup scan by using “auto-refresh” mode in your test settings:

WHITENOISE_AUTOREFRESH = True

Upgrades

The future belongs to those who believe in the beauty of their dreams.

—Eleanor Roosevelt

You probably already know this, but keeping everything up-to-date is a generally good idea. Security improvements, bug fixes, and new features are the main reasons to upgrade. But most things also tend to get faster over time, so if you are looking for more justifications to upgrade, your test speed can be one.

It can be hard to schedule upgrades alongside other work. There isn’t an operating system dialogue or paperclip jumping up and down on your screen to pressure you. But I’d like to indulge in an analogy.

Upgrades are like washing dishes.

Letting them all “for later” can leave you with a threatening task that drains you when you approach it.

On the other hand, washing them regularly keeps your mental load low. You have familiarity with the task, and experience the pleasant reward of a clean kitchen.

I hope the hints in this chapter show you some of the performance reasons to upgrade and make the process easier.

Upgrade Django

Speed Improvements

Both core Django features and tests tend to get a bit faster with every version of Django. Here are some examples of test runner improvements in recent versions:

Version 2.2 made the test runner avoid setting up unused databases, and changed the order that TransactionTestCase loads its serialized data.

Version 3.0 added some easier test selection flags: -k, --start-at, and --start-after.

Version 3.1 will bring output buffering (see Prevent Output), and faster table flushing in TransactionTestCase (Ticket #31276).

In what can now be considered the deep past of 2009, Version 1.1 introduced transactions around tests in TestCase. This meant an 8-12x speed increase for most projects!

Additionally, other parts of Django tend so see some performance improvements in each version, often unmentioned in the release notes.

Django has a benchmarking project called djangobench. Although it’s currently somewhat disused, it has been used in the past to verify performance improvements.

Upgrade Process

There are two kinds of upgrade to do on Django: feature version and minor version. The feature versions are 2.0, 2.1, 2.2, 3.0, etc. The minor versions are the final numbers at the end: 3.0.0, 3.0.1, 3.0.2, etc.

You should always use the latest minor version of the feature version you’re on. These include vital bug fixes, sometimes fixing critical security issues. Follow the Django blog to keep up to date on these. Also consider using a service like Dependabot (now part of GitHub), pyup, or Snyk.

For feature versions, you should track when they are released, and how long they are supported for. See the timeline diagram on the download page. You can choose to stick on LTS (Long Term Support) versions and upgrade in between, but I’d recommend only doing that if your organization imposes it. The non-LTS feature versions tend to be just as stable, only with shorter support windows. Yes, you’ll need to upgrade more often, but this avoids “the dishes piling up”.

The documentation has a pretty thorough how-to guide to follow. I don’t think I can add anything to it here.

Upgrade Python

Python has had some performance improvements in recent releases. For example:

Python 3.6 changed the representation of dictionaries, making them faster and use less memory.

Python 3.8 added the new “vectorcall” convention in the C code, speeding up many built-ins.

The Python team has official benchmarks running on speed.python.org. There’s one that’s particularly relevant to us Django developers - the time it takes to render a Django template. Here are the results for the currently measured versions of Python (source):

While Python 3.6 is a little slower than 2.7, the following versions are faster and faster. There’s a ~25% improvement from Python 3.6 to 3.8 - nice!

Upgrade Process

Upgrading can be a short process, if your dependencies are ready. You should be able to follow mostly the same process you use to set up your project, just swapping for the newer version.

Here’s a quick checklist:

Check your Django version is compatible with the new Python version.

This is noted at the top of the version release notes, for example see Django 3.0. If it’s not - you need to upgrade Django first.

Check your other dependencies are compatible with the new Python version.

Most packages list this somewhere in their documentation or changelog.

Other, less maintained packages don’t track this information, and may not even be tested against the new Python version. For these, you should be cautious and make a note to check their functionality after upgrading.

Read the Python “What’s New” version guide, for example What’s New In Python 3.8.

You don’t need to read the whole thing - the “Release Highlights” section covers the main things to know.

Install the new version of Python.

This depends on how you installed it to begin with.

For local development, I prefer using pyenv. It allows multiple versions to be installed easily, and is updated quickly with new versions. This way, you can test on the new version without uninstalling the old version.

On Debian/Ubuntu, try the deadsnakes PPA for up-to-date packages. See an example installation at installpython3.com.

If you’re using Docker, upgrading is normally a matter of changing your base Python image tag.

If you can’t find a package for your specific operating system, you can download and compile Python from the source at python.org/downloads.

Build a new virtual environment with the new version of Python.

If you’re using Docker or similar, this is probably done through rebuilding the image.

Rebuild your requirements lock file with the new version.

For example if you’re using pip-tools, you’ll want to rerun pip-compile. This is because some packages require certain dependencies depending on your version.

Run your tests.

Hopefully they all pass, and they’re a little faster!

Release.

If you do manual testing on a staging environment, it’s worth doing a little extra testing there.

If your project is larger, you might find some of your dependencies are blockers, or you have a lot of code to upgrade. In this case, try running both versions of Python in parallel on your CI system until you’re ready to upgrade.

When it comes to release time, be prepared with a rollback plan if you have problems. This mostly means ensuring you don’t write data with the new version of Python that can’t be read with the previous version.

The main culprit here is pickle, which occasionally introduces new, incompatible versions of its protocol. The easiest solution is to separate pickled data by Python version.

Since pickle is used by Django’s caching framework, you’d need to do this there by specifying the Python version in your KEY_PREFIX:

import platform

CACHES = {
    "default": {
        ...
        "KEY_PREFIX": platform.python_version(),
    }
}

You could also subclass Django’s cache backends to specify a pickle protocol version available in both Python versions.

You’ll need to do similar anywhere else you are using pickle. Check what your third party packages do - especially those related to caching and task queues.

Help, I’m on Python 2.7!

You probably know this already, but this is a big upgrade! A lot has changed, but all for the better.

As we saw above, there are many performance improvements. And some of the breaking changes also mean you’ll write more performant code.

If you’re still using Python 2.7, that means you’re also using Django 1.11, which ended support in April 2020. That’s another good reason to upgrade.

How to do this upgrade is a big topic and out of scope for this book. There are lots of resources online though.

Here are some resources to help convince your organization to invest in the upgrade:

The Python 2.7 countdown clock, which has reached zero

Python.org sunset statement

Powerful Python’s guide on the main difference

And here are some practical guides:

A free book: Supporting Python 3: An in-depth guide

The official Python documentation porting guide

My blog post on upgrading with zero downtime

I hope they help.

Upgrade Your Database Server

Okay, one last thing to look at upgrading. Just like Django and Python, database servers tend to improve their query performance in nearly every new version.

Unlike the application layer, upgrading your database server normally requires some downtime unless you do extra work with replicas. There’s also a risk of data loss so you should ensure your backups are working and you can roll back if anything goes wrong.

Minor Versions

The first thing to check is that you’re on the latest minor version available for your feature release. Minor versions are normally focussed on bug fixes, but they can often bring some performance improvements, especially earlier in a feature release cycle. They’re also important for stability and security, so you should aim to always be on the latest minor version available, even if you can’t upgrade feature releases often.

For example, if you use PostgreSQL 12, open up “PostgreSQL 12” on the release notes page, and check you’re using the latest 12.x release available.

Major Versions

Major versions are normally where performance advantages lie. But they are also riskier and more work to schedule.

You should check that Django is compatible with the version you’re aiming for. This is documented in the Databases reference page.

You should also read your Database server’s release notes. For example the PostgreSQL 12.0 release notes starts with this juicy detail:

General performance improvements, including:

Optimizations to space utilization and read/write performance for B-tree indexes

B-tree indexes are the default type of indexes, so this sounds like a real performance win. That said, there are no details on relative performance or links to benchmarks, so we don’t know how much of an improvement we can really expect. Although upgrades tend to be inevitable, it’s a good idea to track your application’s performance so you can check how much it helped you, or if it caused a slow down.

MySQL versus MariaDB

About 90% of Django users use PostgreSQL. So if you’re using MySQL or MariaDB, you’re in a minority. But don’t worry, I’m here with you as a MariaDB fan!

If you’re using MySQL (from Oracle), you should consider moving to MariaDB. The two have played catch up with each other over the years, but I still believe MariaDB is ahead in both features and performance.

My anecdote: we saw all query times drop by 30% when we moved YPlan from MySQL 5.6 to MariaDB 10.0. Naturally such a performance improvement translated to our tests being faster as well.

Although MariaDB has strived to maintain MySQL compatibility, Django didn’t officially support it until version 3.0. But now it does, it’s a good time to look at moving.

Migration can normally done by uninstalling MySQL, installing MariaDB, and running the mysql_upgrade command line tool. See MariaDB versus MySQL: Compatibility in the documentation.

Note

MySQL = MariaDB on most Linux distributions

Most Linux distributions have moved their mysql-server package to install MariaDB, so you may be using it without realizing. If you aren’t sure which version you’re using, connect with the CLI and you’ll see the server version reported in the opening headers:

 $ python manage.py dbshell  # or 'mysql'
 Welcome to the MariaDB monitor.  Commands end with ; or \g.
 Your MariaDB connection id is 9
 Server version: 10.4.11-MariaDB Homebrew
 ...

Parallelize

A problem shared is a problem halved. A joy shared is a joy doubled.

—Anonymous

Django’s documentation section on speeding up your tests recommends using parallel testing. It can speed up your tests dramatically, but unfortunately it’s not always easy to activate. Let’s look at how you can move your project to parallel testing.

What Is Parallel Testing?

Parallel testing is the splitting of your tests across multiple processes. Each process can run on separate processor cores, at the same time, so the tests can finish faster. For example, if you use four parallel processes to each run 25% of the test suite, your tests could run up to four times faster.

That is, you could take this serial (non-parallel) execution:

And turn it into this:

In practice there are some overheads, such as database setup, that prevent you from achieving that perfect N times speed-up. This is known as Amdahl’s law. But the larger your project, the smaller the overheads are, relative to the whole runtime, so the closer you can get to a full speed-up.

How to Move To Parallel Testing

Parallel testing is built-in to the Django test framework, and available in pytest via a well-maintained plugin. Activating these features is relatively straightforward, but you can encounter problems when moving a test suite from serial to parallel. These are more likely the larger your project is. So if you’re working on a new project, you should add parallelism as soon as possible.

Parallelism problems are all caused by the same thing: parallel tests accessing shared resources. These resources could be your database, cache, Python variables, or anything else that stores data.

When your tests run serially, they can assume two things:

They run in a particular order

They’re the only thing working with these resources.

Once you start running them in parallel, both of these assumptions are broken.

If two tests assume they run in a particular order, the later one can end up depending on the first one. This can only happen if the first test is not fully isolated. That is, it doesn’t cleanly reset the shared resources it uses. Django provides some test isolation by default, such as TestCase’s database transactions, but it can’t do everything.

If two tests assume they have sole access to a shared resource, they can change the data the other is using. This can lead to failure because the data is not in the expected state. It can be random too since tests run at slightly different speeds each time. Django protects the test database against this by cloning it. But other data stores, such cache servers, are subject to this.

A Checklist

Here’s a rough checklist for converting your project to parallel testing while dealing with these problems:

Check tests are isolated.

Activate parallel testing and see if it works.

Chase any errors, which could be from non-isolated tests or shared resource access.

We’ll follow this order in the next sections.

If you’re in a rush, you can try to activate parallel testing before checking for isolation. But if you see failures, go back to checking for isolation.

After this, we’ll see how you can get a bit more speed from splitting test cases, and how to parallelize across multiple CI servers if necessary.

Check Your Tests Are Isolated

Let’s look at an example of fixing non-isolated tests, then how you can detect them.

Fix Non-Isolated Tests

Take this model:

from django.db import models

class Book(models.Model):
    title = models.CharField(max_length=120)

    SHORT_TITLE_LIMIT = 30

    @property
    def short_title(self):
        if len(self.title) > self.SHORT_TITLE_LIMIT:
            return self.title[: (self.SHORT_TITLE_LIMIT - 3)] + "..."
        return self.title

    def to_api_data(self):
        return {"title": self.title, "short_title": self.short_title}

It has two functions to test. You can test them like so, but beware, these tests are not isolated:

from django.test import SimpleTestCase

from example.core.models import Book

class BookTests(SimpleTestCase):
    def test_short_title(self):
        Book.SHORT_TITLE_LIMIT = 10
        book = Book(title="A Christmas Carol")

        self.assertEqual(book.short_title, "A Chris...")

    def test_to_api_data(self):
        book = Book(title="A Song of Ice and Fire")

        self.assertEqual(
            book.to_api_data(),
            {"title": "A Song of Ice and Fire", "short_title": "A Song ..."},
        )

If you run the tests together, they pass:

$ python manage.py test
System check identified no issues (0 silenced).
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

But if you run the second one alone, it fails:

$ python manage.py test -k test_to_api_data
System check identified no issues (0 silenced).
F
======================================================================
FAIL: test_to_api_data (example.core.tests.test_models_not_isolated.BookTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../test_models_not_isolated.py", line 16, in test_to_api_data
    self.assertEqual(
AssertionError: {'title': 'A Song of Ice and Fire', 'short_title': 'A Song of Ice and Fire'} != {'title': 'A Song of Ice and Fire', 'short_title': 'A Song ...'}
- {'short_title': 'A Song of Ice and Fire', 'title': 'A Song of Ice and Fire'}
?                         ^^^^^^^^^^^^^^^

+ {'short_title': 'A Song ...', 'title': 'A Song of Ice and Fire'}
?                         ^^^

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

This is an indication that it depends on something changed by the other test.

If you look again, you’ll see the first test modifies Book.SHORT_TITLE_LIMIT, but doesn’t reset it. This change then affects the second test via the short_title returned in to_api_data(). The change to Book.SHORT_TITLE_LIMIT remains in place for the rest of the test suite, so it could potentially affect many tests across different files.

The solution is to make sure that SHORT_TITLE_LIMIT is reset correctly. You can do this with mock.patch.object, which will reset the attribute to its original value, even if the test fails with an exception. You can use it like so:

from unittest import mock

from django.test import SimpleTestCase

from example.core.models import Book

class BookTests(SimpleTestCase):
    @mock.patch.object(Book, "SHORT_TITLE_LIMIT", new=10)
    def test_short_title(self):
        book = Book(title="A Christmas Carol")

        self.assertEqual(book.short_title, "A Chris...")

    def test_to_api_data(self):
        book = Book(title="A Song of Ice and Fire")

        self.assertEqual(
            book.to_api_data(),
            {
                "title": "A Song of Ice and Fire",
                "short_title": "A Song of Ice and Fire",
            },
        )

(More on using unittest.mock later in Targeted Mocking.)

These tests can then be run alone, or together in any order. Solved!

Detecting Non-Isolated Tests

So non-isolated tests can be a pain to track down, and they block you from using parallel testing. How can you detect them?

The key is in the ordering. By changing the order in which your tests run, you can discover non-isolated tests. With the right workflow, you can detect them early and avoid merging them in your code base.

There are two easy to use re-ordering techniques: reversal, and randomization. Reversal is simply running the tests in reverse order to normal. Randomization is shuffling the test order like a pack of cards, normally with restrictions to keep tests from the same test case together.

Other re-ordering techniques exist, such as exhaustively trying all possible ordering spaces, but they’re not practical for every day use.

In a paper titled Empirically Revisiting the Test Independence Assumption, computer scientists from the University of Washington evaluated several techniques. They checked them with known non-isolated tests in several open source Apache projects. They found that reversal finds most non-isolated tests, but randomization finds more of them - eventually.

Reversal works well because most test dependencies are simple, with one non-isolated test affecting others. Randomization finds trickier cases, where two or more non-isolated tests work together to affect others. The downside of randomization is that it it’s a slow search across all possible orderings. You might be unlucky and miss simple dependencies when adding them, only to find them weeks or months later.

Workflows

Using reversal is relatively easy. A good workflow is to use it on your CI system only. Locally, developers work with the “forward” test order, and CI works with the “backward” order. This means that each change will be tested in both directions, by default, without any extra thought from developers. If a test fails on CI but not locally, you know you likely have an isolation issue.

Using randomization takes a bit more work. You want to use it in every environment, but also be able to reproduce a given order. The simplest solution here is to use pseudo random ordering, like Python’s random module, and record its seed. If you find an isolation failure, you can re-run the same set of tests with the same seed to reproduce the order.

Let’s look at how you can implement these techniques.

With Django’s Test Framework

For reversal, Django’s test runner provides the --reverse flag (documentation) out of the box. You can add --reverse to your CI system’s test command to use it there.

For randomization, Django will support it from version 4.0, with a --shuffle option. This is thanks to work by Chris Jerdonek in Ticket #24522.

With pytest-django

For both these techniques, I maintain pytest plugins.

For reversal, you can use the pytest-reverse plugin. This provides a --reverse flag that works similarly to Django’s. You can add this flag on your CI system’s test command.

(This feature should be merged into pytest core at some point: see pytest issue #7042.)

For randomization, you can use the pytest-randomly plugin. This randomly shuffles tests within test cases, then shuffles test cases within files, and finally shuffles files.

The order uses random and prints the random seed at the start of tests. You can then reuse it with the --randomly-seed flag.

The plugin also controls the generation of random data through random and other libraries. This is useful if your tests do any random (“fuzzy”) data generation, and can give you a “double benefit” for randomization.

Activate Parallelization

Let’s now activate the parallelization feature in your test runner. If any tests fail, they’re either due to non-isolation as above, or shared resources, which we’ll tackle below.

With Django’s Test Framework

First, install tblib:

$ python -m pip install tblib

This allows Django’s parallel test runner to display tracebacks for failed tests.

Then you can pass the --parallel option (documentation) when running tests:

$ python manage.py test --parallel
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
....
----------------------------------------------------------------------
Ran 4 tests in 0.053s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

We can see that the tests above ran in parallel created the test database and cloned it for each test process.

Passing --parallel without any argument automatically uses as many test processes as your computer has CPU cores. We can also pass a number of processes to run. Django parallelizes by test case, so if you run fewer test cases than you have cores, it won’t use all your cores. In the above example project I have two test cases, so only two processes are used.

Unfortunately, --parallel doesn’t work at all on Windows. And it doesn’t work on macOS since Python 3.8+, without a tweak which we’ll cover. In both cases this is due to unavailability of the “fork” method of launching new processes.

Python’s multiprocessing module supports several methods of starting new processes. Django relies on the “fork” method, which uses the Unix system call fork(). Because Windows is not a Unix-based operating system, the “fork” method is unavailable there, so Django’s parallel testing is unavailable.

The situation on macOS is a little more complicated. As a Unix-based operating system, it does support fork(), but recent macOS versions have blocked the system call. This is due to fork() breaking some system libraries. Because of this block, Python 3.8 changed multiprocessing to default to the “spawn” method on macOS. To carry on parallel testing on macOS, we can disable the block - see the “macOS workaround” sidebar below.

In order to support parallel testing on all operating systems, Django is moving to use multiprocessing’s “spawn” method. This was worked on by Ahmad A. Hussein as part of Google Summer of Code 2020 (Ticket #31169). At time of writing, the change is awaiting merge and will hopefully be released in Django 4.0.

Note

macOS workaround

As covered above, to use parallel testing on Python 3.8+ on macOS, we need to disable the operating system block on fork(). This has a small risk of introducing system library crashes to your test suite, but in practice I’ve never seen any problems.

The workaround requires two steps. First, we disable the block by setting the environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY to YES. Second, we change the default multiprocessing start method to “fork”, undoing the change to “spawn” from Python 3.8.

We can do both of these in a Custom test Management Command, by adding a management/commands/test.py file to one of our apps containing:

import multiprocessing
import os
import sys

from django.core.management.commands import test

class Command(test.Command):
    def __init__(self, *args, **kwargs):
        if sys.platform == "darwin":
            # Workaround for https://code.djangoproject.com/ticket/31169
            os.environ["OBJC_DISABLE_INITIALIZE_FORK_SAFETY"] = "YES"
            if multiprocessing.get_start_method(allow_none=True) != "fork":
                multiprocessing.set_start_method("fork")
        super().__init__(*args, **kwargs)

We do the change in __init__() to be before the main test command’s argument parsing, which reads the multiprocessing start method. The check before changing the start method is required for cases when the method has already been changed - notably PyCharm does this for you.

With this in place you should find parallel testing works, and you see multiple test databases being created.

Make --parallel the Default

It’s annoying to remember to pass --parallel all the time. Also, since it optionally takes an argument you can’t easily use it before a named test, as the test name becomes the flag value. This raises an error since it’s not an integer:

$ python manage.py test --parallel example.core.tests.test_a
usage: manage.py test [-h] [--noinput] [--failfast] ...
manage.py test: error: argument --parallel: invalid int value: 'example.core.tests.test_a'

You can make it default to “on” with all your computer’s cores in your custom test runner like so:

from django.test.runner import DiscoverRunner

class ExampleTestRunner(DiscoverRunner):
    @classmethod
    def add_arguments(self, parser):
        super().add_arguments(parser)

        # Modify parallel option to default to number of CPU cores
        # Find the action as already created in super(), and change its
        # 'default' (1) to its 'const' (the number of CPU cores)
        parallel_action = next(
            a for a in parser._optionals._actions if a.dest == "parallel"
        )
        parallel_action.default = parallel_action.const

This uses a little manipulation of argparse internals to change the argument’s default after Django adds it in the super() method. The argument optionally takes the number of cores to run on, but if you don’t provide a value, it uses the const value, which Django sets to the number of CPU cores. Here we’re copying that into the default, so the flag doesn’t need setting.

Then you can run in parallel by default with:

$ python manage.py test

When necessary, you can run in serial by specifying a single process:

$ python manage.py test --parallel 1

Great, you’ve set up parallel testing and made it the default!

With pytest-django

Install the distributed testing plugin pytest-xdist:

python -m pip install pytest-xdist

pytest-xdist does more than multi-core parallel testing. It can also distribute tests across multiple machines, or multiple Python versions. We’ll stick with multi-core parallel testing here. If you have a use case for the other modes, check out the documentation.

(That said, if you do need to test with multiple Python versions, it’s probably better to use tox.)

pytest picks up plugins and their options automatically. To activate pytest-xdist’s parallel testing capabilities, pass -n auto to pytest:

$ pytest -n auto
========================= test session starts =========================
...
plugins: xdist-1.31.0, forked-1.1.3, django-3.8.0
gw0 [4] / gw1 [4] / gw2 [4] / gw3 [4] / gw4 [4] / gw5 [4] / gw6 [4] / gw7 [4] / gw8 [4] / gw9 [4] / gw10 [4] / gw11 [4]
....                                                            [100%]
========================== 4 passed in 1.73s ==========================

-n is short for -numprocesses. Using auto uses as many test processes as your computer has CPU cores. These are shown starting up and collecting the tests in the line before tests begin, labelled gw0, gw1, etc. “gw” stands for “gateway”, and my computer has 12 cores, so I have gw0 through gw11.

Because pytest-xdist doesn’t collect the tests until it has started all the processes, it can spin up more processes than necessary. For this project, I have 12 workers despite only having four tests. This is not normally a problem, as the processes shouldn’t do much initialization. Thanks to pytest’s fixture system, idle processes won’t do any test setup by default, such as creating test databases.

By default, pytest-xdist splits tests individually and sends them to ready processes. This means a test case class might be split across multiple processes, so each process has to run its class-level setup. The same applies to module-level fixtures. This duplicate work unnecessarily slows down your tests.

You can change this by setting a distribution strategy set with the --dist option. This can be set to loadscope (group by file or test case) or loadfile (group by file).

If you you’re using Django test cases, loadscope should work best, since it allows parallelization inside files with many test cases. This is the same behaviour as Django’s test framework. If you’re using module level test functions, there are no test cases, so the two options are equivalent.

So you can run:

$ pytest -n auto --dist loadscope
========================= test session starts =========================
...
plugins: xdist-1.31.0, forked-1.1.3, django-3.8.0
gw0 [4] / gw1 [4] / gw2 [4] / gw3 [4] / gw4 [4] / gw5 [4] / gw6 [4] / gw7 [4] / gw8 [4] / gw9 [4] / gw10 [4] / gw11 [4]
....                                                            [100%]
========================== 4 passed in 1.84s ==========================

You can make this the default by adding it to the addopts setting in your pytest.ini (or other configuration file):

[pytest]
addopts = --numprocesses auto --dist loadscope

Great, you’ve set up parallel testing and made it the default!

Deal with Shared Resources

Okay, you’ve made sure your tests are isolated, and added parallel testing. If you’re lucky and don’t see any errors or failures, great! But if you do see some failures, don’t worry. This is pretty normal for larger projects that have been using serial testing for a while.

These failures are likely related to some use of shared resources. If you dig into the failures, you should be able to figure out which resources are the problem. You can then tackle them with one these techniques:

Sharding resources per test process.

Locking shared resources so only one test process can access them at a time.

Sharding resources is faster since the test processes can act independently. Locking forces some tests to pause until they can acquire the lock and thus slows them down, so should be done with caution.

Let’s look at these techniques now.

Sharding Resources by Test Process

Take these tests:

from django.core.cache import cache
from django.test import SimpleTestCase

class CacheOneTests(SimpleTestCase):
    def test_it(self):
        cache.set("example-key", 1)
        self.assertEqual(cache.get("example-key"), 1)

class CacheTwoTests(SimpleTestCase):
    def test_it(self):
        cache.set("example-key", 2)
        self.assertEqual(cache.get("example-key"), 2)

They’re using the memcached cache backend:

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyLibMCCache",
        "LOCATION": "127.0.0.1:11211",
    }
}

(We previously looked at how an in-process in-memory cache backend can be an easy win for test speed. Let’s assume that’s not feasible here.)

These tests look reasonable, and they pass when run serially. But if you run them in parallel, they fail:

$ python manage.py test --parallel
System check identified no issues (0 silenced).
.F
======================================================================
FAIL: test_it (example.core.tests.CacheTwoTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/.../python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/.../python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/.../example/core/tests.py", line 14, in test_it
    self.assertEqual(cache.get("example-key"), 2)
  File "/.../python3.8/unittest/case.py", line 912, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/.../python3.8/unittest/case.py", line 905, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 2 tests in 0.165s

FAILED (failures=1)

The failures are also random - sometimes they manage to pass, sometimes CacheOneTests fails with 2 != 1 instead. The problem is they’re both concurrently using a shared resource - the memcached key "example-key".

For example, to reach the failure above, the tests’s memcached requests are interleaved like this:

CacheTwoTests sets "example-key" to 2.

CacheOneTests sets "example-key" to 1.

CacheTwoTests reads "example-key", and asserts it should be 2.

To solve this problem, you need to make it so each test process accesses different resources. In general, for any shared resource, you need to do one of the following:

Use different resource processes per test process.

This is what Django does for test databases. For caches, that would mean running one cache server per test process.

Use a different resource “container” per test process.

For caches, this would mean a different “database” per test process. memcached doesn’t support databases, but other cache servers do, like Redis.

Format resource names differently per test process.

For caches, this means changing the keys that are used.

The last option is the least invasive, and least wasteful. It lets you use the same cache processes and storage space you already have.

You could change the individual tests to access separate keys. But this is not always feasible, is fragile to changes in your project, and doesn’t stop the problem from recurring in other tests.

Instead, you can change the key function that Django uses to calculate the full key. If you make this return different keys per test process, the two processes’ cache usage can never overlap. You can do this in your test settings like so:

def make_cache_key(key, key_prefix, version):
    return f"{key_prefix}:{os.getpid()}:{version}:{key}"

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyLibMCCache",
        "LOCATION": "127.0.0.1:11211",
        "KEY_FUNCTION": make_cache_key,
    }
}

Django runs this function for each cache operation to calculate the full key that it sends to the cache server. The new key function make_cache_key() includes the current process ID from os.getpid() in the full cache key. These are guaranteed to be unique by the operating system. So each parallel test process gets a different “key space” and they can never use the same keys.

Make sure this key function isn’t used in your production settings files, otherwise server processes won’t share the same key space and caching won’t work properly.

A similar technique can be used for shared filenames, queue names, etc. For a more complicated example, see this blog post by David Kwan which covers separating Redis databases per test process for the RQ package.

Locking Shared Resources

If you can’t modify the shared resource to shard, you can instead add locking. You can do this with the SerializeMixin mixin for TestCase classes that Django provides. SerializeMixin uses the operating system’s ability to lock a file to coordinate between test processes. It’s documented in Enforce running test classes sequentially.

Take the above example again. If you couldn’t shard the cache keys with KEY_FUNCTION, you could instead use SerializeMixin like so:

from django.core.cache import cache
from django.test import SimpleTestCase
from django.test.testcases import SerializeMixin

class CacheOneTests(SerializeMixin, SimpleTestCase):
    lockfile = __file__

    def test_it(self):
        cache.set("example-key", 1)
        self.assertEqual(cache.get("example-key"), 1)

class CacheTwoTests(SerializeMixin, SimpleTestCase):
    lockfile = __file__

    def test_it(self):
        cache.set("example-key", 2)
        self.assertEqual(cache.get("example-key"), 2)

CacheOneTests and CacheTwoTests both declare lockfile = __file__. __file__ is the Python magic variable that contains the path of current Python file, such as ".../example/core/tests.py". lockfile can be any file, but using the current Python file is convenient.

Because these two test cases use the same lockfile, they each exclude the other one from running at the same time. If the two test cases run at the same in different test processes, the locking will force them to run serially. Whichever test case happens to start first will obtain the lock on the named file, and hold that lock until it all its tests have finished. The second test case will pause until it can acquire the lock itself, before running its tests then releasing the lock.

If we run just these two test cases, there are two orders. If CacheTwoTests happens to start first, the order will be:

CacheTwoTests acquires the file lock.

CacheTwoTests.test_it() runs.

CacheTwoTests releases the file lock.

CacheOneTests acquires the file lock.

CacheOneTests.test_it() runs.

CacheOneTests releases the file lock.

Done!

Here we used lockfile = __file__. If you have test cases in multiple test files that need to be run in serial, you should use a different reference file. For example, you could use the path to a shared models file.

Be careful though. If you overuse the same lockfile between many test cases, you could undo much of the advantage of parallel testing. Your tests would run mostly in serial, with the overhead of multiple processes.

With pytest-django

SerializeMixin is only usable with Django TestCases. If you’re writing test functions, you can create a fixture that does the same thing.

You can create and use such a fixture like so:

import pytest
from django.core.files import locks
from django.core.cache import cache

@pytest.fixture
def serialize():
    with open(__file__) as lockfile:
        locks.lock(lockfile, locks.LOCK_EX)
        yield

def test_cache_one(serialize):
    cache.set("example-key", 1)
    assert cache.get("example-key") == 1

def test_cache_two(serialize):
    cache.set("example-key", 2)
    assert cache.get("example-key") == 2

The serialize fixture opens the current Python file as referred to by __file__. It then locks it with the undocumented Django file-locking functions, which are wrappers around the standard library’s functions. The lock is held until the file is closed, which happens when the fixture is torn down after yield.

Note

pytest-xdist’s --dist=loadscope will already serialize tests in the same file. Using that test distribution, you’ll find such file-locking fixtures are only necessary for tests in different files.

Split Up Large Test Groups

Both Django’s test framework and pytest-xdist with --dist loadscope group tests together in parallel processes. They are grouped by class in the case of TestCases, and by module in the case of pytest test functions. Let’s call this a “test group”.

The grouping is a good thing as it allows tests to share setup, through TestCase.fixtures, setUpTestData(), and pytest fixtures. This shared setup reduces the overhead of multiple test processes.

However, as covered in Measure Individual Tests, you should expect the runtimes of your tests to follow a Pareto distribution. This will also apply to your test groups, whether they’re test cases or modules.

The Pareto distribution is a law of nature - you can basically never stop your tests from following it. But you can “flatten the curve” to make the distribution less severe by dividing the groups. This way, no particular test case or module can dominate the overall test runtime.

Essentially, you want to turn situations like this:

…into this:

Dividing is only worth it if you have a large test group extending the overall runtime, as in the first diagram. Unnecessary division increases the amount of work to do by repeating any group-level setup in each process, slowing down the overall runtime. To find your longest test group, you can measure all your test processes with py-spy --subprocesses and speedscope, as covered previously.

If you do see a large test group, you should divide it by splitting it into several smaller groups, whether that’s test cases or modules. For example, take this slow test case:

from django.test import TestCase

class BookViewTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.user = ...
        cls.books = ...
        ...

    def test_logged_in_one(self):
        ...

    def test_logged_in_two(self):
        ...

    ...

    def test_logged_out_one(self):
        ...

    def test_logged_out_two(self):
        ...

    ...

This class has a lot of tests, so they dominate the overall runtime. Setup is minimal and shared because it’s done with setUpTestData(), which is covered later.

You can divide this class by splitting it in two like so:

from django.test import TestCase

class BaseBookViewTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.user = ...
        cls.books = ...
        ...

class LoggedInBookViewTests(BaseBookViewTests):
    def test_logged_in_one(self):
        ...

    def test_logged_in_two(self):
        ...

    ...

class LoggedOutBookViewTests(BaseBookViewTests):
    def test_logged_out_one(self):
        ...

    def test_logged_out_two(self):
        ...

    ...

The shared setup is now extracted into the shared BaseBookViewTests class. It doesn’t have any test_* methods so it’s not run alone like an “abstract” shared base.

Here there is a logical split to make, between logged in and logged out tests. While you might be splitting for performance, using a logical boundary helps keep the tests organized. It can also mean that parts of the setup aren’t necessary for one of the split cases. For example, here it might be possible to remove the creation of the user for the logged out tests.

Migrations

Humans are allergic to change. They love to say, “We’ve always done it this way.” I try to fight that. That’s why I have a clock on my wall that runs counter-clockwise.

—Grace Hopper, The Wit and Wisdom of Grace Hopper (1987)

Your database is probably the centre of your project, and therefore used in most of your tests. Setting up the test database with migrations can be the most noticeable test startup cost. And the longer your project goes on, the longer your migrations take to run.

Let’s look at some strategies for countering this overhead.

Reuse the Test Database Between Runs

This is noted in Django’s documentation section Speeding up the tests.

Both Django’s test framework and pytest-django provide command line options for reusing your test database between runs, to avoid re-running migrations. There’s no fully automated way of doing this without caveats, so they’re something for you to learn and master.

Let’s look at how this works via Django’s Test Framework, then how pytest extends it.

With Django’s Test Framework

Django’s test framework provides the --keepdb flag:

$ python manage.py test --keepdb

This tells Django to reuse any existing test databases from a past run, rather than destroying and recreating them with migrations. (There can be multiple test databases from either multiple entries in DATABASES or parallel testing.)

Mostly you’ll want to add the flag whenever you need to run the same set of tests repeatedly. But if your migration history changes to edit, rename, or remove a migration, you’ll need to run the tests without the flag, once. This can happen when you:

…rename a migration file

…edit a migration file

…check out another source control branch

…merge in other developers’ changes

…rename an app

…install or upgrade a third party package

(This is not an exhaustive list.)

If you don’t run the tests without --keepdb in these situations, you’ll potentially be testing against the wrong version of your database. Sometimes tests pass, sometimes you get a descriptive error, but sometimes you get unhelpful database errors about missing tables or columns.

The only migration-changing situation that it’s safe to use --keepdb with is when you add new migrations. In this case, Django will run the new migrations on the reused test databases, updating them to the expected state.

Let’s look at an example workflow. Imagine you might start a new Git branch to work on a feature:

$ git checkout add-comments

You make a new model and edit some existing ones. These changes require you to make a new migration:

$ python manage.py makemigrations

You then write some tests for the new feature. When you run them, you can use --keepdb to prevent the databases from being destroyed at the end of the run:

$ python manage.py test --keepdb example.core.tests.views.test_comments

Then as you work on the feature (without changing the migration file), you can continue running your tests with --keepdb. Once you’re done working on the feature, you might commit and then check out your default branch to work on something else:

$ git commit -m "Add comments feature"
$ git checkout main

At this point, you’ve switched to a migration history that doesn’t include your new migration. But your test databases still include the changes from that migration. So when you run the tests, you should do so without the --keepdb flag:

$ python manage.py test

On this test run, Django rebuilds the database from scratch, without your in-progress migration. You can then continue using --keepdb until the next change to migrations.

You then fetch changes from your other team members:

$ git pull

If you inspect all the changed files and third party packages, you might be able to predict if the migration history has changed. But if your team has changed a lot of files, it can be faster to assume the history has changed and avoid using --keepdb for the next test run.

With pytest-django

pytest-django has a slightly different approach here. It comes with two command line flags to help your workflow.

The flag to keep the database is --reuse-db, which works identically to Django’s --keepdb. This is overridden by the --create-db flag, which forces the database to be recreated.

By placing --reuse-db in your pytest.ini file, you can have it always apply. Then when you do something that changes your migration history, the first time you run the tests, you add the --create-db flag on the command line. This can be a little easier to work with since it requires adding a flag when things change, rather than when they stay the same.

Setting up the work flow is well documented on the pytest-django database page under “Example work flow”, so I won’t repeat it here.

Let’s look at with the same example as above. First, you start the new branch:

$ git checkout add-comments

You make a new model and edit some existing ones. These changes require you to make a new migration:

$ python manage.py makemigrations

You then write some tests for the new feature, and want to run them. Because the migration is new, you don’t need to add --create-db yet. You can simply run your tests:

$ pytest example/core/tests/views/test_comments.py

The --reuse-db flag is automatically added from pytest.ini. Django reuses the database, but detects and applies the new, unapplied migration.

As you continue working on the feature (without changing the migrations), you can run the same command.

Once you’re done working on this feature, you might commit and then check out your default branch to work on something else:

$ git commit -m "Add comments feature"
$ git checkout main

Again, since the migration history has changed to remove a migration you need to recreate the test database on the first run. You do this by adding the --create-db flag, which negates the --reuse-db flag in your pytest.ini:

$ pytest --create-db

You then fetch changes from your other team members:

$ git pull

Rather than inspect all the changed files and third party packages, you assume the history has changed, and again use --create-db for the next test run.

Squash Your Migrations

If you use Django’s test framework, you can see your individual migrations running at the start of tests by using the increased verbosity of -v 2:

 $ python manage.py test -v 2
 Creating test database for alias 'default' ('test_db')...
 Operations to perform:
   Synchronize unmigrated apps: core
   Apply all migrations: (none)
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
 Running migrations:
   Applying core.0001_initial... OK
   Applying core.0002_add_book... OK
   Applying core.0003_add_author... OK
   Applying core.0004_add_bookauthor... OK
   Applying core.0005_change_book_title... OK
   Applying core.0006_change_author_name... OK
   ...

Thankfully Django has a built-in solution for large migration histories, called squashing. This can “squash” a run of migrations back into a single one, and remove redundant operations in the process. This greatly reduces the time it takes to build your database.

The Django documentation on squashing is very extensive, and covers the workflow well. On most projects it runs without a hitch in a few minutes, but it’s not always smooth.

If you use any RunSQL or RunPython operations, you should re-inspect them to see if you can add elidable = True first. This allows Django to drop them in the squashed migration, and do more optimization on the remaining operations. You’d normally add this for one-off data migrations that don’t need to be applied in the squashed history.

You might also encounter some unoptimized edge cases or bugs. I know I have. The optimization phase of migration squashing is complex and it’s hard for Django to account for every case. Be prepared to search through Django’s ticket tracker for others having similar issues, and workarounds in comments. And maybe contribute a fix :)

Once you have squashed your migrations, it’s best to set up a regular schedule to do it - perhaps monthly. This will keep your test runtime down and your migrations more navigable.

Don’t Disable Migrations in Tests

Warning

As explained below, I recommend you do not use this technique. I’ve described it because it’s fairly common and your project might already be using it.

An alternative to squashing your migrations is to bypass them entirely in tests. You can do this by disabling your apps’ migration histories during tests.

When migrations run on an app without a migration history, Django “synchronizes” the app’s models by generating and running an in-memory migration for them. This in-memory migration is similar to a squashed migration, as it directly creates the models in their current state. The big difference is that any custom operations, such as RunPython, do not appear in this in-memory migration, even though they’d be preserved in a squashed migration.

You can disable an app’s migration history by mapping it to None in MIGRATION_MODULES in your test settings:

MIGRATION_MODULES = {
    "core": None,
}

If you then run the tests with Django’s test framework and increased verbosity (-v 2), you’ll see that app’s tables created under the header “Synchronizing apps without migrations”:

 $ python manage.py test -v 2
 Creating test database for alias 'default' ('test_db.sqlite3')...
 Destroying old test database for alias 'default' ('test_db.sqlite3')...
 Operations to perform:
   Synchronize unmigrated apps: core
   Apply all migrations: (none)
 Synchronizing apps without migrations:
   Creating tables...
     Creating table core_author
     Creating table core_book
     Running deferred SQL...
 Running migrations:
   No migrations to apply.
 System check identified no issues (0 silenced).
 test_create (example.core.tests.test_models.BookTests) ... ok
 ...

To make this apply to all apps, you have two options. From Django 3.1, you can put set the per-database test option TEST_MIGRATE to False:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": "db.sqlite3",
        "TEST": {
            "NAME": "test_db.sqlite3",
            "MIGRATE": False,
        },
    }
}

On older Django versions, you can make MIGRATION_MODULES into a dictionary-like object that responds with None for every key:

class DisableMigrations:
    def __contains__(self, item):
        return True

    def __getitem__(self, item):
        return None

MIGRATION_MODULES = DisableMigrations()

(DisableMigrations implements the two Python “magic” methods __contains__(), for the in operator, and __getitem__(), for indexing with []. These are the only two dictionary operations that Django performs with MIGRATION_MODULES.)

Additionally pytest-django has the --nomigrations flag (documentation).

This technique can give you a good speed boost. And there are a lot of community resources recommending it.

However, it was more useful historically than with modern versions of Django. Migrations used to take longer to run, especially before the big internal rewrite in Django 1.9.

It’s also dangerous. Your database migrations permanently change your project’s most important asset: its data.

By ignoring this vital code in tests, you expose yourself to a lot of risk. You also have much less guarantee that your test database matches what’s used in production, and therefore that your code will actually work. This becomes more important when you write custom migrations including the RunSQL, RunPython, and SeparateDatabaseAndState operations.

These mismatches could cause bugs and even data loss. This is why I recommend you don’t use this technique. It’s much better to run your migrations in tests, keeping the overhead minimal by squashing them regularly and reusing your database between test runs.

If you are using this technique, the least you can do is make it apply only in local development. Your CI system can then run your full migration history to verify tests against it.

With Django’s Test Framework

You can use the “are we on CI?” check from Skip Slow Tests Locally to conditionally change your Test Settings. For example:

import os

from django.test.runner import DiscoverRunner
from django.test.utils import override_settings

class ExampleTestRunner(DiscoverRunner):
    def run_tests(self, *args, **kwargs):
        with override_settings(**TEST_SETTINGS):
            return super().run_tests(*args, **kwargs)

TEST_SETTINGS = {
    "PAGINATION_COUNT": 10,
}

class DisableMigrations:
    def __contains__(self, item):
        return True

    def __getitem__(self, item):
        return None

on_ci = os.environ.get("CI", "") == "true"
if not on_ci:
    TEST_SETTINGS["MIGRATION_MODULES"] = DisableMigrations()

With pytest-django

You can make tests default to bypassing migrations by adding the flag to the addopts setting in your pytest.ini (or other configuration file):

[pytest]
addopts = --nomigrations

Then configure your CI system to run tests with --migrations, which forces them to run again:

$ pytest --migrations

For more information refer to the documentation.

Database Configuration

Past data has a lot of good in it, but it is the bad side that is bad.

—Yogi Berra

Your tests probably use your database a lot. Later we’ll look at rewriting some tests to avoid using your database in some cases, but for the most part your database is an integral, unavoidable part of your Django project. Let’s see what we can do to speed it up during tests.

Use In-Memory Storage

Memory is much faster than disk storage. According to the interactive latency calculator set to the year of writing, 2020, a 1MB read from memory takes 3μs, while the same from SSD takes 49μs. That’s about 16 times faster.

By moving your database storage from disk to memory, you can take advantage of this speed-up.

There are two ways to do this: use your database’s native in-memory storage, or use an in-memory filesystem. Let’s look at them in turn.

Native In-Memory Storage

This can be an easier option, but it’s really only feasible on SQLite.

SQLite

SQLite supports in-memory databases. This is done by declaring their location as ":memory:".

If you’re using SQLite for your database, Django will automatically use :memory: as database location during tests, unless you set TEST_NAME. So this configuration is enough to use in-memory databases during tests:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": "/path/to/non-test-database.sqlite3",
    }
}

If you’re not using SQLite in production, it’s not a good idea to switch to it just for tests though. This is covered later in Don’t Swap Your Database to SQLite in Tests.

MariaDB/MySQL

MariaDB and MySQL come with an in-memory storage engine called MEMORY (MariaDB documentation / MySQL documentation). In theory this would be useful for tests, but it has too many limitations:

BLOB and TEXT aren’t supported, preventing you from using Django’s BinaryField and TextField.

Foreign keys are not supported.

It’s non-transactional, preventing Django’s TestCase from rolling back with transactions.

Because of these reasons, it’s best to stick to using the default InnoDB storage engine with an in-memory filesystem.

In-Memory Filesystems

A filesystem is the software layer between the OS file operations and the disk itself. It chooses how to organize files as bytes on disk, and index them for access.

In-memory filesystems avoid actually storing data on disk, and instead keep it in-memory. By using one for your database, you can gain the speed advantage of memory over disk with no changes to the database itself. When I moved the testing for my library Django-MySQL to use an in-memory filesystem, I saw the tests go from ~9 minutes down to ~7 minutes - a nearly 25% saving.

Linux includes an in-memory filesystem called tmpfs. I’ll be covering that below since it’s the most common operating system for developing Django projects, be it directly or via virtual machines or containers. You should be able to adapt the instructions for other operating systems.

macOS includes the ability to make an in-memory filesystem, by creating an in-memory disk device and mounting a normal filesystem on that. Check out this Stack Overflow question on RAM disks for instructions.

Windows does not include an in-memory filesystem. You’ll need to take your pick of extra software - check out this list on Wikipedia, which starts with open source options.

With Docker

Docker comes with the ability to create tmpfs volumes. If you pass type=tmpfs for a --mount option, it will be created via tmpfs, and live only as long as its container. By placing this mount at your database’s default data directory, your database will automatically use it.

For example, for PostgreSQL:

docker run \
    --mount type=tmpfs,destination=/var/lib/postgresql/data \
    --detach \
    --name postgres \
    --env POSTGRES_PASSWORD=password \
    --publish 5432:5432 \
    postgres:12

Or for MariaDB:

docker run \
    --mount type=tmpfs,destination=/var/lib/mysql \
    --detach \
    --name mariadb \
    --env MYSQL_ALLOW_EMPTY_PASSWORD=true \
    --publish 3306:3306 \
    mariadb:10.5

With docker-compose

If you are using docker-compose to control Docker containers, you can specify the tmpfs option in the volumes section of your docker-compose.yml. For example, with PostgreSQL 12:

 version: '3.7'
 services:
   db:
     image: postgres:12
     environment:
       POSTGRES_PASSWORD: postgres
     ports:
       - 5432:5432
     volumes:
       - type=tmpfs,destination=/var/lib/postgresql/data/

On CI Systems Supporting Docker

If you’re using your CI system to launch your test database in a Docker container, you can normally configure the volume in a similar way to docker-compose. For example, with MariaDB on GitHub Actions, you’d do this:

 jobs:
   tests:
     name: tests

     ...

     services:
       database:
         image: mariadb:10.5
         env:
           MYSQL_ALLOW_EMPTY_PASSWORD: true
         ports:
         - 3306:3306
         volumes:
         - type=tmpfs,destination=/var/lib/mysql

     ...

Some CI systems do not support passing volume options to Docker. In this case, you can instead use the database images built by Circle CI with memory support, tagged ending "-ram". Because these are public on Docker Hub, you can use them with any CI system. They have a documentation section covering these images: Optimizing Postgres Images. While this describes PostgreSQL only, their DockerHub account has many more database server images with "-ram" tags.

Manually Managed

If you’re managing the installation of your database on your computer or a virtual machine, you’ll want to create a tmpfs before your database starts. It’s most convenient if you create the tmpfs in your database’s default data directory, as this avoids extra configuration. But you can also place it elsewhere and configure your database to use that.

For example, for PostgreSQL you could run this before starting it:

$ mkdir -p /var/lib/postgresql/data
$ mount -t tmpfs none /var/lib/postgresql/data

This normally requires root permissions, and may require you to set the directory’s ownership and permissions for your database server’s user.

To run this before your database server starts, you can define filesystem in your /etc/fstab file.

For a full tutorial with PostgreSQL on Ubuntu, see the blog post Faster Testing with RAM Drives by Mark Litwintschik. The tutorial also covers using a separate data directory.

Don’t Swap Your Database to SQLite in Tests

Warning

For reasons explained below, I recommend you do not use this technique. I’ve described it because some other resources recommend it. It’s only appropriate to use SQLite for tests if your production database is SQLite, or if you’re building a third party package that’s database-agnostic.

One practice sometimes recommended in Django resources is to use a client-server database such as PostgreSQL in production, but to test with SQLite. This is normally a double-edged sword. It speeds up your tests, but at a great cost to test accuracy.

Why Is SQLite Faster?

SQLite’s test speed boost comes from three main factors.

First, Django automatically uses its in-memory mode during tests. But as we’ve seen above, any database can be used with an in-memory filesystem.

Second, it’s weakly typed. SQLite allows you to store integers in columns declared as strings (!), and there are no length limits on columns. This allows it to work a bit faster, but it also invites bugs! And testing is meant to catch bugs. The ORM does not shield you from its weak typing.

Third, it’s not a client-server database, so there’s less work communicating queries and data to/from a server. This is only a small advantage though, since client-server databases are highly optimized for mitigating this overhead.

So for these three factors: the first we can replicate on any database, the second invites bugs, and the third is only a small advantage.

The Disadvantages of Using Two Databases

Although SQL is a standard, there are many irreconcilable differences between databases. Django’s ORM tries to make them more equivalent, but it’s impossible to make them identical. By using SQLite alongside another database, you’ll be affected by these differences sooner or later.

SQLite is not only weakly typed, but it also has different behaviour in casting and querying. For example, see my blog post where I investigated different databases’ implicit type conversion.

SQL syntax also varies. Although you might try and stick to only the ORM, you may eventually want to write some raw SQL, particularly in migrations.

Features vary between databases too. You may want to take advantage of database-specific features at some point on your project, using Django extensions like django.contrib.postgres or Django-MySQL. If your tests don’t use the same database, this won’t be possible.

And finally, different databases have different bugs. Although they’re rare and normally quickly fixed, database bugs are regularly found at all layers: the ORM, the Python drivers, and the databases themselves. Using two databases exposes your project to twice as many such bugs. You don’t want to add workarounds for database bugs solely for test speed.

More on the Topic

If you want to read more about this, see the section “Use the Same Database Engine Everywhere” in Two Scoops of Django.

CI Configuration

The maxim “Nothing avails but perfection” may be spelt shorter: “Paralysis”.

—Winston Churchill

CI (continuous integration) systems are normally treated as “set and forget”, once an initial working setup is found. But there are normally some easy changes you can make to reduce your CI test run time.

Caching

Many modern CI systems use completely fresh environments between builds and build steps. This is a good thing! Isolating your build environments helps prevent bugs, just like individual test isolation.

Unfortunately, installing everything from scratch is slow.

The solution is to add some caching. This tells your CI system to keep and restore certain directories between test environments.

Let’s look at an example of doing this with the most common package manager for Django projects, pip, and other tools this might work for.

pip Caching

When you install a set of packages from scratch with pip, it downloads and unpackages each one individually. The downloading can take a noticeable amount of time, and it can consume significant bandwidth. Globally, this adds up. As of 5 March 2020, PyPI was transferring 300TB a day.

During installation, pip stores downloaded packages in its own cache directory. The location of the cache directory varies by platform – the default on Linux is ~/.cache/pip. Future installs of the same packages use the local copies, avoiding downloading them.

To keep builds fast, you should tell your CI system to cache this cache directory, restoring it in new build environments. pip will then avoid downloading anything, except in the case of new packages.

Let’s look at how to do this with Git Actions, Jenkins, and Travis. If you’re using another CI system, check its documentation - it should work similarly to one of these.

GitHub Actions

Use the official GitHub cache action. Its pip documentation shows how to use it (as of v1):

 name: CI

 on:
   ...

 jobs:
   tests:
     ...

     steps:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v1
       ...
     - uses: actions/cache@v1
       with:
         path: ~/.cache/pip
         key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
         restore-keys: |
           ${{ runner.os }}-pip-
     ...

Jenkins

On Jenkins, your tests run with the same OS user each time (on each server). Normally this means the pip cache can work already. Check it exists in $JENKINS_HOME/.caches/pip (on Linux).

Travis

Pip caching is built-in and only needs activating in your .travis.yml:

 language: python

 cache: pip

Mirror PyPI

If your CI system doesn’t have the ability to preserve a cache between runs, or you have many independent CI servers, you can’t restore a single pip cache directory. But you can still set up a kind of caching by running the PyPI mirror server devpi. This caches and serves the packages as served by PyPI.

By running devpi on your network, packages won’t need to be fetched from the public internet but from your much closer server. You’ll also have the ability to keep working if PyPI goes down.

While relatively easy to set up, it has a higher maintenance cost. It’s normally only worth it on larger teams.

To set it up, start with the quickstart guide, including the index-url configuration for pip.

Other Caching

pip is not unique here - most other package managers have a cache directory that your CI system should preserve between environments. For example, if you’re using Node.js, both its package managers npm and yarn have their own cache directories.

The configuration to preserve these should be similar to that for pip. Refer to your CI system’s documentation for details.

Scale Up

Scaling up your CI system’s hardware to a faster configuration is normally a fairly cheap way of buying more test speed. It’s not unreasonable to even scale above the size of your production web servers.

Only some CI systems offer the ability to do this. Of those mentioned previously, GitHub Actions and Travis CI don’t, while Jenkins does since it’s self-hosted. Check your CI system’s documentation, and maybe pricing plans.

The easy ways to scale up are:

Move from hard drive (magnetic disk) to SSD

Add more CPU cores

Add more memory

Parallelize Across Multiple CI Servers

In Parallelize, we looked at parallelizing your tests across different CPU cores on the same computer. You can make your tests run even faster in CI by parallelizing them across different servers, allowing them to run on even more CPU cores. This ability is built-in to many CI systems.

This has some draw backs though. Amdahl’s law still applies, and the “fixed work” on CI is bigger, due to installing dependencies etc. It can also be hard to recombine test results. But if your tests still take many minutes, it can be worth it.

Split by Tags

This is an extension of the technique in Skip Slow Tests Locally. It’s the easiest way to split your tests across multiple servers. It can work well for splitting into two, but doesn’t scale so well for more pieces.

The idea is to split your tests into two sets by a tag. It could be any tag, but I’ll stick with the “slow” tag as referenced previously.

With this split, you can run two test processes on two CI servers, one to run those matching the tag, and one to run those without.

It’s quite hard to achieve, or maintain a perfect 50/50 split. So one of the test processes will still dominate the runtime - similar to the diagram in Split Up Large Test Groups.

With Django’s Test Framework

Run these two commands on your separate servers:

$ python manage.py test --exclude-tag slow
$ python manage.py test --tag slow

With pytest-django

Run these two commands on your separate servers:

$ pytest -m 'not slow'
$ pytest -m 'slow'

Split With Workers

Instead of splitting manually with tags, it would be better to treat your CI servers similarly to CPU cores. This should scale to any number of workers, and allow a more even split of work between them. However it’s more advanced and might prevent other tools from working, such as Coverage.py.

With Django’s Test Framework

I’m not aware of any tooling readily available for this. You could build a custom test runner to allow selection though, inspired by the below pytest features.

With pytest-django

If you’re using pytest, and your “main” CI server can connect to its workers over SSH or TCP, you can use pytest-xdist’s remote execution features. See its documentation.

Otherwise, you can use a custom plugin to collect the tests on each worker, and reduce which get run to only a subset based on the server. For an example of doing this on Heroku CI, see Pierre de Wulf’s blog post Python Tests Run 10x Faster By Using Continuous Integration on Heroku.

With Circle CI

Circle CI has test splitting feature, which is unique among the CI systems that I know of. This feature takes over test discovery and passes a list of file names to your test runner. It records test run times per file and uses that to split your tests evenly among workers.

Long time Django core contributor Josh Smeaton has written a blog post on using it with Django and pytest: Django Test Splitting on Circle CI.

Test Structure

Order doesn’t come by itself.

—Benoit Mandelbrot

Right, it’s time to roll up your sleeves. We’ve gone past the easy wins of reconfiguring parts that run your test suite. It’s time to dive into the tests themselves and rewrite them in faster forms.

In this chapter, we’ll cover restructuring your tests to be fast. A well structured test suite is a fast test suite.

AAA: Arrange-Act-Assert

Well structured tests follow the AAA pattern. (Or, at least, are guided by it.)

The pattern has three steps:

Arrange - set up the object(s) to be tested.

Act - run the tested function.

Assert - make assertions about what the tested function should have done.

This keeps tests clear on what they’re trying to check, and helps reduce redundant steps.

For example, here’s a test structured strictly according to the pattern:

from django.test import SimpleTestCase

from example.core.models import Book

class BookSimpleTests(SimpleTestCase):
    def test_short_title_truncated(self):
        # Arrange
        book = Book(
            title=(
                "An Inquiry into the Nature and Causes of the Wealth of"
                + " Nations"
            )
        )

        # Act
        result = book.get_short_title()

        # Assert
        self.assertEqual(result, "An Inquiry into the Nature ...")

Sometimes it’s more practical to condense the steps together. For example, you can combine the above “Act” and “Assert” steps by not using the temporary variable result:

from django.test import SimpleTestCase

from example.core.models import Book

class BookSimpleTests(SimpleTestCase):
    def test_short_title_truncated(self):
        # Arrange
        book = Book(
            title=(
                "An Inquiry into the Nature and Causes of the Wealth of"
                + " Nations"
            )
        )

        # Act + Assert
        self.assertEqual(
            book.get_short_title(), "An Inquiry into the Nature ..."
        )

Sometimes you can even fit all three steps in one line. For example:

from django.test import SimpleTestCase

from example.core.models import Book

class BookSimpleTests(SimpleTestCase):
    def test_short_title_not_truncated(self):
        # Arrange + Act + Assert
        self.assertEqual(
            Book(title="A Song of Ice and Fire").get_short_title(),
            "A Song of Ice and Fire",
        )

Steps can also appear out of order. For example, assertRaises() may be written before the “Act” step, although it actually makes its assertion when the context manager exits:

from django.test import SimpleTestCase

from example.core.models import Book

class BookSimpleTests(SimpleTestCase):
    def test_short_title_not_truncated(self):
        # Arrange
        book = Book(title="")

        # Assert
        msg = "Can't get short_title when there is no title!"
        with self.assertRaisesMessage(ValueError, msg):
            # Act
            book.get_short_title()

There are other caveats to the AAA pattern. Often your “Arrange” is split between your test case class, setUp(), setUpTestData(), fixture files, etc. And sometimes your “Act” takes more than one function call. Overall though, you should use it as guidance for telling how well structured your tests are.

For more on writing tests in the AAA style, see James Cooke’s blog post, which introduced me to the concept. And Bill Wake’s post, which first described the pattern.

Avoid “One Assertion per Test”

A commonly recommended extension of the AAA pattern is to write tests with only a single assertion. This is a good idea to increase the signal-to-noise ratio from test failures. If several tests fail for the same component with different assertions, you have more information about the problem.

Unfortunately the single assertion idea can be taken too far and slow down tests beyond its benefit. Tests that make only one assertion can become a performance drain as they unnecessarily repeat the same “Arrange” and “Act” steps. They also take longer to write and maintain.

For example, take these tests:

from django.test import SimpleTestCase

class IndexTests(SimpleTestCase):
    def test_get_200(self):
        response = self.client.get("/")

        self.assertEqual(response.status_code, 200)

    def test_get_content_type(self):
        response = self.client.get("/")

        self.assertEqual(response["Content-Type"], "text/plain")

    def test_get_body(self):
        response = self.client.get("/")

        self.assertEqual(response.content.decode(), "Welcome to example.com")

Each test function makes a single, granular assertion on part of the response.

In theory, test failures will give you the maximum information to track down the error. If only one part of the response changes, you see an individual test failure pointing to the problem. If several parts change, you see several failures.

In practice, the tests are correlated, because the response is a single object. If one test fails, it’s likely they will all fail. In this case, the several test failures can be a hindrance to debugging - there is more output and it’s less clear the assertions are all related to the same request.

You could combine the above tests like so:

from django.test import SimpleTestCase

class IndexTests(SimpleTestCase):
    def test_get(self):
        response = self.client.get("/")
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response["Content-Type"], "text/plain")
        self.assertEqual(response.content.decode(), "Welcome to example.com")

Now self.client.get() is only run once. Since this is the most expensive operation in these tests, they should be nearly three times faster!

Another way of looking at this is that the three assertions are really one assertion covering three different aspects of the same response. We are only lacking a concise way of checking all aspects at once. We could invent a custom assertion function to combine them, in which case the test would have a single assertion.

You should find a pragmatic balance between writing targeted tests and combining assertions for speed. A good idea is not to think about individual assertions but effects. For example, a “create” view often has two effects: creating the object in the database, and returning a response. It’s normally fine to test these together, but if the test becomes hard to read and maintain, then consider splitting them.

Write Mostly Unit Tests Not Integration Tests

You can divide tests into two categories: unit and integration. Unit tests check a component in isolation, while integration tests check multiple linked components together.

What you call a “component” can be a fuzzy proposition in Python, as it’s a fairly dynamic language. For example, if you add a decorator to a function, is that now two components?

Additionally, integration tests can also be divided into other categories depending on how much of the application they cover. When they cover “the whole system” they may be called “functional”, “end-to-end”, or “system tests”.

So maybe the division between unit and integration test is a spectrum. But this categorization can still be useful for understanding test speed.

Unit tests tend to be faster because they cover “less stuff”. They are also easier to maintain as changes in one component only affect their own unit tests.

Integration tests tend to be slower and harder to maintain. They are also harder to debug, since failures give you less information about which component is to blame. However, without integration tests you don’t verify that your whole system actually works.

A good mix is to aim for extensive unit tests, with a few integration tests on each user-facing function, for both happy and error paths. This is known as the testing pyramid:

The recommended split between the two categories varies by source. The original Google Testing Team recommendation was 70% unit, 20% integration, and 10% end-to-end tests.

There’s no perfect number, and it likely depends on what can be considered in your project’s code. But certainly, if your project currently overuses integration tests, you can speed them up by rewriting some as unit tests.

In Django projects, integration tests normally use the test client. This means creating a request, passing it through all the middleware, the view, maybe a template, then passing the response back through the same layers. This can be a lot of overhead compared to an equivalent unit test.

Let’s look at some examples of rewriting integration tests as unit tests.

Unit Tests for a Form

Forms are a great example of a component that can be easily unit tested. They accept a dictionary of values, validate it, and return either errors or cleaned data.

For an example, take this form:

from django import forms

from example.core.models import Book

class AddBookForm(forms.ModelForm):
    class Meta:
        model = Book
        fields = ["title"]

    def clean_title(self):
        title = self.cleaned_data["title"]
        if not title:
            return title

        if not title[0].isupper():
            self.add_error("title", "Should start with an uppercase letter")

        if title.endswith("."):
            self.add_error("title", "Should not end with a full stop")

        if "&" in title:
            self.add_error("title", "Use 'and' instead of '&'")

        return title

It has a few validation steps for the title field that we’d like to test in isolation.

For reference, here’s the corresponding view:

from django.shortcuts import redirect, render

from example.core.forms import AddBookForm

def add_book(request):
    if request.method == "POST":
        form = AddBookForm(request.POST)
        if form.is_valid():
            form.save()
            return redirect("/books/")
    else:
        form = AddBookForm()
    return render(request, "add_book.html", {"form": form})

Integration Tests

You can write integration tests for the form with the test client, hitting the view and checking for error messages in the responses’ HTML:

from http import HTTPStatus

from django.test import TestCase

class AddBookFormTests(TestCase):
    def test_title_starting_lowercase(self):
        response = self.client.post(
            "/books/add/", data={"title": "a lowercase title"}
        )

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(
            response, "Should start with an uppercase letter", html=True
        )

    def test_title_ending_full_stop(self):
        response = self.client.post(
            "/books/add/", data={"title": "A stopped title."}
        )

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(
            response, "Should not end with a full stop", html=True
        )

    def test_title_with_ampersand(self):
        response = self.client.post(
            "/books/add/", data={"title": "Dombey & Son"}
        )

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Use 'and' instead of '&'", html=True)

These tests work, but they have two flaws.

First, they have all of that integration test overhead. To check these error messages, we don’t really care about the details of HTTP or HTML. But here we have to check HTTP status codes and parse HTML with assertContains(..., html=True) in every test.

Second, they’re imprecise. The assertContains() calls check for error messages somewhere in the output, rather than directly related to the title field. If we had two fields with similar validation logic, these tests could accidentally pass because we used bad test data for the other field. We could rewrite the tests to inspect for a more precise HTML string, but that would couple them further to the details of form rendering.

Unit Tests

You can instead test the form directly:

from django.test import TestCase

from example.core.forms import AddBookForm

class AddBookFormTests(TestCase):
    def test_title_starting_lowercase(self):
        form = AddBookForm(data={"title": "a lowercase title"})

        self.assertEqual(
            form.errors["title"], ["Should start with an uppercase letter"]
        )

    def test_title_ending_full_stop(self):
        form = AddBookForm(data={"title": "A stopped title."})

        self.assertEqual(
            form.errors["title"], ["Should not end with a full stop"]
        )

    def test_title_with_ampersand(self):
        form = AddBookForm(data={"title": "Dombey & Son"})

        self.assertEqual(form.errors["title"], ["Use 'and' instead of '&'"])

These tests correct the two flaws. They’re faster because they simply pass in and read out dictionaries, with no need to touch anything related to HTTP or HTML. And they’re more precise because they directly inspect the errors for “title”, ignoring the other fields.

Note you’d still want to have some integration tests, to check that the view, form, and template work together:

from http import HTTPStatus

from django.test import TestCase

class AddBookViewTests(TestCase):
    def test_get(self):
        response = self.client.get("/books/add/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "<h1>Add Book</h1>", html=True)

    def test_post_success(self):
        response = self.client.post(
            "/books/add/", data={"title": "Dombey and Son"}
        )

        self.assertEqual(response.status_code, HTTPStatus.FOUND)
        self.assertEqual(response["Location"], "/books/")

    def test_post_error(self):
        response = self.client.post(
            "/books/add/", data={"title": "Dombey & Son"}
        )

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Use 'and' instead of '&'", html=True)

Given that the form is already fully tested, these view tests are sufficient as they provide full coverage of the only three paths through the view.

Unit Tests for a Management Command

Management commands are similar targets for moving some integration tests to unit tests. Logic tends to accumulate in handle(), meaning the only test is to run the whole command. It’s better to structure some logic in separate methods, for readability and testability.

Take this management command, that implements some of the same book title normalization rules as in the previous form:

from django.core.management.base import BaseCommand
from django.db.transaction import atomic

from example.core.models import Book

class Command(BaseCommand):
    help = "Normalize all Book titles"

    def add_arguments(self, parser):
        parser.add_argument(
            "--write",
            action="store_true",
            default=False,
            help="Actually edit the database",
        )

    @atomic
    def handle(self, *args, write, **kwargs):
        if not write:
            self.stdout.write("In dry run mode (--write not passed)")

        books = Book.objects.select_for_update()
        for book in books:
            book.title = self.normalize_title(book.title)
        if write:
            Book.objects.bulk_update(books, ["title"])
            self.stdout.write(f"Updated {len(books)} book(s)")

    def normalize_title(self, title):
        if not title:
            return "Unknown"

        if title[0].islower():
            title = title[0].upper() + title[1:]

        if title.endswith("."):
            title = title[:-1]

        title = title.replace("&", "and")

        return title

The logic inside normalize_title() has been separated from the handle() method, allowing us to use and test it in isolation.

Integration Tests

You can write integration tests for this command by creating data, running the command, and checking the data was correctly updated:

from io import StringIO

from django.core.management import call_command
from django.test import TestCase

from example.core.models import Book

class NormalizeBookTitlesTests(TestCase):
    def call_command(self, *args, **kwargs):
        out = StringIO()
        call_command(
            "normalize_book_titles",
            *args,
            stdout=out,
            stderr=StringIO(),
            **kwargs,
        )
        return out.getvalue()

    def test_dry_run(self):
        book_empty = Book.objects.create(title="")

        out = self.call_command()
        self.assertEqual(out, "In dry run mode (--write not passed)\n")

        book_empty.refresh_from_db()
        self.assertEqual(book_empty.title, "")

    def test_write_empty(self):
        book_empty = Book.objects.create(title="")

        out = self.call_command("--write")

        self.assertEqual(out, "Updated 1 book(s)\n")
        book_empty.refresh_from_db()
        self.assertEqual(book_empty.title, "Unknown")

    def test_write_lowercase(self):
        book_lowercase = Book.objects.create(title="lowercase")

        out = self.call_command("--write")

        self.assertEqual(out, "Updated 1 book(s)\n")
        book_lowercase.refresh_from_db()
        self.assertEqual(book_lowercase.title, "Lowercase")

    def test_write_full_stop(self):
        book_full_stop = Book.objects.create(title="Full Stop.")

        out = self.call_command("--write")

        self.assertEqual(out, "Updated 1 book(s)\n")
        book_full_stop.refresh_from_db()
        self.assertEqual(book_full_stop.title, "Full Stop")

    def test_write_ampersand(self):
        book_ampersand = Book.objects.create(title="Dombey & Son")

        out = self.call_command("--write")

        self.assertEqual(out, "Updated 1 book(s)\n")
        book_ampersand.refresh_from_db()
        self.assertEqual(book_ampersand.title, "Dombey and Son")

These tests capture the command output through the stdout argument to call_command(), covered later in Mock Output.

The visible repetition in writing and reading the Book instances to the database point to some overhead. There’s also the overhead of running call_command() to test only behaviour from normalize_title(). Seeing both of these nudges us to move some tests to unit tests.

Unit Tests

You can change some of these tests to directly test normalize_title() instead:

from io import StringIO

from django.core.management import call_command
from django.test import SimpleTestCase, TestCase

from example.core.management.commands.normalize_book_titles import Command
from example.core.models import Book

class NormalizeBookTitlesSimpleTests(SimpleTestCase):
    def test_normalize_empty(self):
        result = Command().normalize_title("")
        self.assertEqual(result, "Unknown")

    def test_normalize_lowercase(self):
        result = Command().normalize_title("lowercase")
        self.assertEqual(result, "Lowercase")

    def test_normalize_full_stop(self):
        result = Command().normalize_title("Full Stop.")
        self.assertEqual(result, "Full Stop")

    def test_normalize_ampersand(self):
        result = Command().normalize_title("Dombey & Son")
        self.assertEqual(result, "Dombey and Son")

class NormalizeBookTitlesTests(TestCase):
    def call_command(self, *args, **kwargs):
        call_command(
            "normalize_book_titles",
            *args,
            stdout=StringIO(),
            stderr=StringIO(),
            **kwargs,
        )

    def test_dry_run(self):
        book_empty = Book.objects.create(title="")

        self.call_command()

        book_empty.refresh_from_db()
        self.assertEqual(book_empty.title, "")

    def test_write(self):
        book_empty = Book.objects.create(title="")

        self.call_command("--write")

        book_empty.refresh_from_db()
        self.assertEqual(book_empty.title, "Unknown")

Because the new tests don’t touch the database, they use SimpleTestCase class. They’re also shorter as they don’t require any arrangement step.

There are two integration tests left, to cover the “dry run” and “write” pathways.

Unit Tests for a Middleware Class

On my blog I covered adding a middleware for a WWW redirect. This middleware redirects requests for “www.example.com” to the canonical domain “example.com”:

from django.http import HttpResponsePermanentRedirect

class WwwRedirectMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        host = request.get_host().partition(":")[0]
        if host == "www.example.com":
            return HttpResponsePermanentRedirect(
                "https://example.com" + request.path
            )
        return self.get_response(request)

Integration Tests

You can write integration tests for this middleware with the test client:

from http import HTTPStatus

from django.test import TestCase

class WwwRedirectMiddlewareTests(TestCase):
    def test_www_redirect(self):
        response = self.client.get("/home/", HTTP_HOST="www.example.com")

        self.assertEqual(response.status_code, HTTPStatus.MOVED_PERMANENTLY)
        self.assertEqual(response["Location"], "https://example.com/home/")

    def test_www_redirect_different_port(self):
        response = self.client.get("/home/", HTTP_HOST="www.example.com:8080")

        self.assertEqual(response.status_code, HTTPStatus.MOVED_PERMANENTLY)
        self.assertEqual(response["Location"], "https://example.com/home/")

    def test_non_redirect(self):
        response = self.client.get("/home/", HTTP_HOST="example.com")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Welcome to example.com")

These tests again have the overhead of the test client, plus the view. If the view at /home/ does a lot of database work, it can really slow down the tests.

Unit Tests

You can instead test this middleware in isolation:

from http import HTTPStatus

from django.http import HttpResponse
from django.test import RequestFactory, SimpleTestCase

from example.core.middleware import WwwRedirectMiddleware

class WwwRedirectMiddlewareTests(SimpleTestCase):
    request_factory = RequestFactory()

    middleware = WwwRedirectMiddleware(
        get_response=lambda request: HttpResponse("Hello World")
    )

    def test_www_redirect(self):
        request = self.request_factory.get(
            "/home/", HTTP_HOST="www.example.com"
        )

        response = self.middleware(request)

        self.assertEqual(response.status_code, HTTPStatus.MOVED_PERMANENTLY)
        self.assertEqual(response["Location"], "https://example.com/home/")

    def test_www_redirect_different_port(self):
        request = self.request_factory.get(
            "/home/", HTTP_HOST="www.example.com:8080"
        )

        response = self.middleware(request)

        self.assertEqual(response.status_code, HTTPStatus.MOVED_PERMANENTLY)
        self.assertEqual(response["Location"], "https://example.com/home/")

    def test_non_redirect(self):
        request = self.request_factory.get("/home/", HTTP_HOST="example.com")

        response = self.middleware(request)

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertEqual(response.content.decode(), "Hello World")

These tests use Django’s RequestFactory. This is similar to the test client, but instead of making requests, it only builds them. The middleware is constructed once on the class with a dummy view lambda as its get_response argument.

Each test builds a request, passes it through the middleware, and checks the response.

Because nothing in these tests touches the database, they use SimpleTestCase for an extra speed boost.

Unit Tests for a View

The documentation for RequestFactory describes its use in unit testing a view, in isolation from middleware and URL routing. I won’t repeat such an example here.

This technique can work, but it is a bit fragile. The documented example sets request.user inside the test, to emulate the behaviour of AuthenticationMiddleware. This emulation is a kind of mocking, which as covered later in Targeted Mocking, and should be done with caution. It can work fine for emulating one or two simple middlewares, but if you emulate more complicated middleware it will be fragile to changes and likely to become inaccurate.

I recommend sticking to testing view flows with integration tests, and instead moving view logic out into separate functions that can be unit tested. If particular middleware are slow and you want to avoid them, you can remove them in your Test Settings or by Overriding settings.

Use the Right TestCase Class

Django provides four test case classes. They have quite different performance profiles so it’s important to check you use the right one in each situation.

They are, fastest first:

SimpleTestCase - blocks all database access, so it doesn’t need to roll back any changes there.

TestCase - Rolls back database changes using transactions. These are applied to the whole test case and each test method.

TransactionTestCase - Rolls back database changes by flushing every table. It’s called this because it allows your tests to use transactions, rather than using transactions itself.

LiveServerTestCase - Based on TransactionTestCase for database rollback. It also launches a live server thread, and is mainly intended for testing with a browser-based tool like Selenium.

If you match the test case class you use to your tests’ requirements, you can see dramatic speed-ups. For example, I once sped up a client’s test suite three times by switching most of their tests from TransactionTestCase to TestCase.

As we saw in the previous section, when you’ve moved some tests from integration to unit, you can often “downsize” them to a faster test case. Most commonly this is when you move some database-touching tests into pure functional tests, and can move from TestCase to SimpleTestCase.

In general, to figure out which TestCase class is needed, you’ll need to inspect your tests to figure out which level of database access or rollback they need. Sometimes this is too hard, and you’ll want to use trial and error, seeing if you can change things without errors. I suggest you look at your slowest tests first.

Split Test Cases

Sometimes you have a bunch of tests for the same class, but only some of them need database access, or the ability to commit transactions. In this case, rather than sticking with a dogma of “one test case per component”, you should split your tests to use the appropriate base classes.

For example, take this model class:

from django.db import models
from django.db.models.functions import Length

# Register __length transform as per Length documentation
models.CharField.register_lookup(Length)

class Book(models.Model):
    title = models.CharField(max_length=200)

    class Meta:
        constraints = [
            models.CheckConstraint(
                check=models.Q(title__length__gte=1),
                name="%(app_label)s_%(class)s_title_length",
            )
        ]

    @property
    def short_title(self):
        if len(self.title) > 30:
            return self.title[:27] + "..."
        return self.title

There are two things to test: the database constraint and the property. Testing the constraint requires database access, but the property does not. Therefore you could split its tests in two appropriate cases, like so:

from django.db.utils import IntegrityError
from django.test import SimpleTestCase, TestCase

from example.core.models import Book

class BookTests(TestCase):
    def test_title_length_1_char_allowed(self):
        Book.objects.create(title="X")

    def test_title_length_not_allowed(self):
        msg = "CHECK constraint failed: core_book_title_length"
        with self.assertRaisesMessage(IntegrityError, msg):
            Book.objects.create(title="")

class BookSimpleTests(SimpleTestCase):
    def test_short_title_not_truncated(self):
        book = Book(title="War and Peace")
        self.assertEqual(book.short_title, "War and Peace")

    def test_short_title_truncated(self):
        book = Book(
            title=(
                "An Inquiry into the Nature and Causes of the Wealth of"
                + " Nations"
            )
        )
        self.assertEqual(book.short_title, "An Inquiry into the Nature ...")

The tests in BookTests can use the database to check the constraint, while those in BookSimpleTests are faster by avoiding the overhead of database transactions.

Create Multiple Custom Classes

One common pattern is to create a single custom test case class that inherits from TestCase or TransactionTestCase, and use that throughout the project. But this means all the projects’ tests forgo potential speed-ups from using the faster classes, such as SimpleTestCase. If you have such a custom test case class, you should move to the mixin pattern in Custom TestCase Classes, which allows you to use your own versions of each of Django’s classes.

Using unittest.TestCase

I sometimes see Django projects using unittest.TestCase for speed, because it has a lower overhead than SimpleTestCase. I prefer to stick with SimpleTestCase - its overhead is very low and unlikely to be noticeable when profiling. SimpleTestCase also blocks database access, which helps keep tests isolated, while unittest.TestCase allows arbitrary database interaction without any rollback.

TestCase Transaction Blockers

The rollback behaviour of TransactionTestCase (and by extension, LiveServerTestCase) is particularly slow. As it flushes every table, it takes time proportional to the number of models in your project. So, as your project grows, all your tests using TransactionTestCase get slower.

You can tame TransactionTestCase slightly with its available_apps feature, which is covered later in Advanced TestCase Features. But in general, it’s best to move tests to TestCase or SimpleTestCase.

You often use TransactionTestCase to allow transactions to commit. Here are a couple of common ways to rewrite your tests to avoid such commits.

Catching IntegrityError

Take this model:

from django.db import models
from django.db.models.functions import Length

# Register __length transform as per Length documentation
models.CharField.register_lookup(Length)

class Book(models.Model):
    title = models.CharField(max_length=200)

    class Meta:
        constraints = [
            models.CheckConstraint(
                check=models.Q(title__length__gte=1),
                name="%(app_label)s_%(class)s_title_length",
            )
        ]

It has a check constraint that should be tested.

You could write a test like so:

from django.db.utils import IntegrityError
from django.test import TransactionTestCase

from example.core.models import Book

class BookTests(TransactionTestCase):
    def test_title_length_not_allowed(self):
        msg = "CHECK constraint failed: core_book_title_length"
        with self.assertRaisesMessage(IntegrityError, msg):
            Book.objects.create(title="")

        self.assertEqual(Book.objects.count(), 0)

This test passes just fine, but with the slowness of TransactionTestCase.

Technically the assertion on Book.objects.count() is unnecessary - the IntegrityError indicates that the book wasn’t created. But it’s there to illustrate what happens when any query happens after IntegrityError. Imagine with more interesting models you’d want to make some other queries.

If you switch the test case to inherit from TestCase though, it fails:

from django.db.utils import IntegrityError
from django.test import TestCase

from example.core.models import Book

class BookTests(TestCase):
    def test_title_length_not_allowed(self):
        msg = "CHECK constraint failed: core_book_title_length"
        with self.assertRaisesMessage(IntegrityError, msg):
            Book.objects.create(title="")

        self.assertEqual(Book.objects.count(), 0)

The failure looks like:

$ python manage.py test example.core.tests.test_test_case_broken
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_title_length_not_allowed (example.core.tests.test_test_case_broken.BookTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/core/tests/test_test_case_broken.py", line 13, in test_title_length_not_allowed
    self.assertEqual(Book.objects.count(), 0)
  ...
  File "/.../site-packages/django/db/backends/base/base.py", line 448, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

----------------------------------------------------------------------
Ran 1 test in 0.008s

FAILED (errors=1)
Destroying test database for alias 'default'...

Django refuses to make a query from within a broken transaction, because the database would return an error or give invalid results. Our TestCase’s transaction was broken by the IntegrityError that we’re testing. The tests worked with TransactionTestCase because it doesn’t create a transaction.

The solution is to add an extra inner transaction (or savepoint) using atomic():

from django.db.transaction import atomic
from django.db.utils import IntegrityError
from django.test import TestCase

from example.core.models import Book

class BookTests(TestCase):
    def test_title_length_not_allowed(self):
        msg = "CHECK constraint failed: core_book_title_length"
        with atomic(), self.assertRaisesMessage(IntegrityError, msg):
            Book.objects.create(title="")

        self.assertEqual(Book.objects.count(), 0)

This test then passes. Great!

The new atomic() breaks with the IntegrityError, but then rolls back. The outer transaction is left alone and further queries are allowed. You can test any IntegrityError-raising code with TestCase by using an extra atomic() like this.

on_commit() Callbacks

Django’s on_commit() takes a callback function to execute when the current transaction commits. Since TestCase creates transactions around each test and rolls them back rather than committing them, any on_commit() callbacks enqueued do not execute.

Historically, Django’s documentation recommended switching tests for on_commit() callbacks to use TransactionTestCase, making such tests slower. But, from Django 3.2, TestCase has the TestCase.captureOnCommitCallbacks() method that allows us to test such callbacks without a speed penalty. This was contributed by myself, and is available for older Django versions in my package django-capture-on-commit-callbacks.

Let’s look at an example. Take this view:

from django.core.mail import send_mail
from django.db.transaction import on_commit
from django.shortcuts import redirect, render

from example.core.forms import AddBookForm

def add_book(request):
    if request.method == "POST":
        form = AddBookForm(request.POST)
        if form.is_valid():
            book = form.save()

            @on_commit
            def send_new_book_email():
                send_mail(
                    subject=f"New Book: {book.title}",
                    message=f"See https://example.com/book/{book.id}/",
                    from_email="bookbot@example.com",
                    recipient_list=["admin@example.com"],
                )

            return redirect("/books/")
    else:
        form = AddBookForm()
    return render(request, "add_book.html", {"form": form})

The view runs in a transaction because the project sets ATOMIC_REQUESTS to True. The email is sent within an on_commit() callback, so that if the transaction does not commit due to an error, the email is not sent.

We could test this view the slow way with a TransactionTestCase like so:

from http import HTTPStatus

from django.core import mail
from django.test import TransactionTestCase

from example.core.models import Book

class AddBookViewTests(TransactionTestCase):
    def test_post_success(self):
        response = self.client.post(
            "/books/add/", data={"title": "Dombey and Son"}
        )

        self.assertEqual(response.status_code, HTTPStatus.FOUND)
        self.assertEqual(response["Location"], "/books/")

        self.assertTrue(Book.objects.filter(title="Dombey and Son").exists())

        self.assertEqual(len(mail.outbox), 1)
        email = mail.outbox[0]
        self.assertEqual(email.subject, "New Book: Dombey and Son")

The test checks the response, the existence of the new Book in the database, and the email being sent. These assertions cover everything, but the test is unnecessarily slow due to TransactionTestCase.

We can rewrite the test with TestCase and use captureOnCommitCallbacks() like so:

from http import HTTPStatus

from django.core import mail
from django.test import TestCase

from example.core.models import Book

class AddBookViewTests(TestCase):
    def test_post_success(self):
        with self.captureOnCommitCallbacks() as callbacks:
            response = self.client.post(
                "/books/add/", data={"title": "Dombey and Son"}
            )

        self.assertEqual(response.status_code, HTTPStatus.FOUND)
        self.assertEqual(response["Location"], "/books/")
        self.assertTrue(Book.objects.filter(title="Dombey and Son").exists())

        self.assertEqual(len(callbacks), 1)
        callbacks[0]()

        self.assertEqual(len(mail.outbox), 1)
        email = mail.outbox[0]
        self.assertEqual(email.subject, "New Book: Dombey and Son")

(On Django < 3.2, we’d also need to add the mixin from django-capture-on-commit-callbacks, as per its documentation.)

The context manager captures the on_commit() callbacks enqueued while it’s active, and returns them in a list. The test then uses that list to assert that only one callback was enqueued, and calls the callback to send the email. Calling the callback within the test “fakes” the commit that it was expecting.

We can also pass execute=True to the context manager to call all the callbacks on its exit, saving us from writing code to call them:

from http import HTTPStatus

from django.core import mail
from django.test import TestCase

from example.core.models import Book

class AddBookViewTests(TestCase):
    def test_post_success(self):
        with self.captureOnCommitCallbacks(execute=True) as callbacks:
            response = self.client.post(
                "/books/add/", data={"title": "Dombey and Son"}
            )

        self.assertEqual(response.status_code, HTTPStatus.FOUND)
        self.assertEqual(response["Location"], "/books/")
        self.assertTrue(Book.objects.filter(title="Dombey and Son").exists())
        self.assertEqual(len(callbacks), 1)
        self.assertEqual(len(mail.outbox), 1)
        email = mail.outbox[0]
        self.assertEqual(email.subject, "New Book: Dombey and Son")

Nice!

Test Data

The more data we have, the more likely we are to drown in it.

—Nassim Nicholas Taleb, The Bed of Procrustes

Test data is inevitable. But the less of it you have, the faster your tests can complete, and the clearer they are.

Let’s look at some strategies for reducing the amount of test data you have, and the time spent loading it.

Avoid Fixture Files

Django’s TestCase.fixtures feature can help you write tests quickly (documentation). If you have some data in your development or production database, you can quickly write tests around it by creating a fixture file with python manage.py dumpdata.

However, tests using fixture files tend to get slower over time. I’ve never seen them work well, and they’re often recommended against in the Django community. The earliest recommendation against them that I’ve seen is in Carl Meyer’s Pycon US 2012 talk, and he even acknowledges his is the third talk that day to say this.

Fixture files are separate from the tests that use them. This makes it hard to determine which tests use which objects. The files tend to become “append only”, and only accumulate more data over time.

Additionally, when a new test needs a new object, it tends to be added to an existing file. This unnecessarily slows down all the other test cases using that file, as they all now load the new object, even though they don’t use it.

Finally, if there’s some data that most of your application depends on, using a fixture causes unnecessary reloading. It will be loaded and then rolled back for each test case, even when the next test case needs the exact same data.

For data you need in individual tests, you’re better off creating it in the test case or test method. You can simplify the creation of this data with factories, and keep it fast by using setUpTestData(). These are both covered later in this chapter.

For data that most of your application depends on, you can use one of the techniques I outlined in my blog post How to Add Database Modifications Beyond Migrations to Your Django Project. These will add the data during the migrate command, so it will be available throughout the test run - at least until your TransactionTestCases run and flush all tables.

Avoid Common Data in Custom TestCase Classes

It’s quite common for projects to define some always-created data in the setUp() or setUpTestData() methods of Custom TestCase Classes used throughout your project. This tends to have the same performance implications as using TestCase.fixtures, with the data being treated as “append-only” and individual tests using only a fraction of the created data. It can even be slower if the data is created in setUp() as this rebuilds it for every test.

It’s best to instead build the data inside the individual test case classes. To avoid repetition and verbosity, you can use factories.

Use Factories

Factories are functions or classes that provide shortcuts for creating data. They can take a lot of the pain out of creating data in your tests.

Take these tests:

from http import HTTPStatus

from django.test import TestCase

from example.core.models import Author, Book

class BookListViewTests(TestCase):
    def test_get(self):
        author1 = Author.objects.create(name="John Grisham")
        book1 = Book.objects.create(title="The Firm", author=author1)
        author2 = Author.objects.create(name="Marcus Aurelius")
        book2 = Book.objects.create(title="Meditations", author=author2)

        response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book1.title)
        self.assertContains(response, book2.title)

class BookDetailViewTests(TestCase):
    def test_get(self):
        author = Author.objects.create(name="John Grisham")
        book = Book.objects.create(title="The Firm", author=author)

        response = self.client.get(f"/book/{book.id}/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book.title)

They both need Books, and in order to create them, they also need to create Authors. The creation of Authors, even when they aren’t used in any assertions is a bit tedious. And as the models grow over time, more fields without defaults will need filling in, requiring modification of the tests.

Using factories solves these problems. You can either create your own factory functions, or use a factory library.

Factory Functions

Making your own factory functions is an easy way to start with factories, and is the most flexible. You can take the parameters you need and construct model instances as you need.

You can place them on your Custom TestCase Classes, or in a standalone module. Adding to your custom TestCase classes can be convenient, because it reduces the number of imports you need in tests. But it doesn’t scale well when you have lots of apps and models.

For example, here’s a custom TestCase mixin with factory methods to create Authors and Books:

from django import test

from example.core.models import Author, Book

class ExampleTestMixin:
    def make_author(self, *, name="Leo Tolstoy"):
        author, _created = Author.objects.get_or_create(name=name)
        return author

    def make_book(self, *, author=None, title="War and Peace"):
        if author is None:
            author = self.make_author()
        return Book.objects.create(author=author, title=title)

class SimpleTestCase(ExampleTestMixin, test.SimpleTestCase):
    pass

class TestCase(ExampleTestMixin, test.TestCase):
    pass

class TransactionTestCase(ExampleTestMixin, test.TransactionTestCase):
    pass

class LiveServerTestCase(ExampleTestMixin, test.LiveServerTestCase):
    pass

These functions use keyword arguments only (the standalone *). This keeps their API easier to extend as you add more fields, and makes the tests clear.

Note that create_author uses get_or_create(), because Author.name is a unique field.

You can rewrite the above tests to use the factory methods like so:

from http import HTTPStatus

from example.test import TestCase

class BookListViewTests(TestCase):
    def test_get(self):
        book1 = self.make_book(title="The Firm")
        book2 = self.make_book(title="Meditations")

        response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book1.title)
        self.assertContains(response, book2.title)

class BookDetailViewTests(TestCase):
    def test_get(self):
        book = self.make_book()

        response = self.client.get(f"/book/{book.id}/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book.title)

They’re shorter, and all mention of Authors has been removed, since they’re not used in the tests’ assertions. If we add another required field to Book, we can add it in only the factory function.

Factory Libraries

If you write several factory functions, you’ll realize they can get repetitive. You might also find yourself missing some useful functionality, such as the ability to make saving in the database optional.

To avoid re-inventing the wheel, you can use a factory library.

The two most popular (with Django support) are Model Bakery and Factory Boy. Model Bakery is easier to understand, but Factory Boy has more features. Because I have experience with it, I’ll demo Factory Boy here.

Factory Boy uses factory classes that map to their respective models. You give default values for fields, which can be other factories for relations. By convention, you store your factory classes in a factories.py next to your models.py.

For example, to copy the previous factory functions, you can add factories like this:

from factory import SubFactory
from factory.django import DjangoModelFactory

from example.core.models import Author, Book

class AuthorFactory(DjangoModelFactory):
    class Meta:
        model = Author
        django_get_or_create = ["name"]

    name = "Leo Tolstoy"

class BookFactory(DjangoModelFactory):
    class Meta:
        model = Book

    title = "War and Peace"
    author = SubFactory(AuthorFactory)

Your tests can then use them by calling their .create() method to create instances in the database:

from http import HTTPStatus

from django.test import TestCase

from example.core.factories import BookFactory

class BookListViewTests(TestCase):
    def test_get(self):
        book1 = BookFactory.create(
            title="The Firm", author__name="John Grisham"
        )
        book2 = BookFactory.create(
            title="Meditations", author__name="Marcus Aurelius"
        )

        response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book1.title)
        self.assertContains(response, book2.title)

class BookDetailViewTests(TestCase):
    def test_get(self):
        book = BookFactory.create()

        response = self.client.get(f"/book/{book.id}/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book.title)

The first test even sets the two authors’ names while creating the Book by passing a value through author__name.

Factory Boy has many more neat features, for example:

Saving in the database can be skipped by calling Factory.build()

Fields can be declared to use sequential or random data, so each instance is different.

Code can be shared for model inheritance, and you can have more than one factory per model.

It’s worth checking out its documentation. Also see my (slightly old) Factory Boy blog post.

Use setUpTestData

This is an underused feature of TestCase that can provide a valuable speed-up. I’ve often found it confers a 3x speedup, or more.

Recall that unittest.TestCase provides two setup hooks:

setUpClass() - a class method that runs once at the start of the test case class.

setUp() - an instance method that runs at the start of each test.

These are the main places for creating test data shared between tests. Objects we create and store on the class in setUpClass() are potentially a source of test isolation leakage, since tests could mutate them. Therefore those objects should be immutable, or add their own rollback behaviour.

Django’s TestCase adds another hook: the class method setUpTestData. Django runs this as part of the base setUpClass(), inside a class-level database transaction. setUpTestData() can create data that’s used in all tests, with changes rolled back between tests.

Django 3.2 improved the behaviour of setUpTestData() to perform in-memory rollback alongside the in-database rollback provided by the per-test transaction. On prior Django versions we need to use a package that backports this functionality.

We’ll cover that package shortly. But first, let’s examine exactly how setUpTestData() helps us write faster tests.

Example Usage

Take these simple tests:

from django.test import TestCase
from example.core.models import Book

class BookTests(TestCase):
    def setUp(self):
        self.book = Book.objects.create(title="Cyrus the Great")

    def test_id(self):
        self.assertIsInstance(self.book.id, int)

    def test_title(self):
        self.assertEqual(self.book.title, "Cyrus the Great")

They use the common pattern of creating some basic objects in setUp() and using those objects in each test. When run, the tests go through these steps:

Django enters a class-level transaction.

Django enters a test-level transaction for test_id().

setUp() creates a Book in the database.

test_id() runs.

Django rolls back the test-level transaction, removing the Book from the database.

Django enters a second test-level transaction for test_title().

setUp() again creates a new Book in the database.

test_title() runs.

Django again rolls back the test-level transaction, removing the Book from the database.

Django rolls back the class-level transaction, which didn’t contain any database changes.

There’s wasted effort creating the Book instance for each test.

Here’s how we can rewrite these tests using setUpTestData():

from django.test import TestCase
from example.core.models import Book

class BookTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.book = Book.objects.create(title="Cyrus the Great")

    def test_id(self):
        self.assertIsInstance(self.book.id, int)

    def test_title(self):
        self.assertEqual(self.book.title, "Cyrus the Great")

When run, they go through these steps:

Django enters a class-level transaction.

setUpTestData() runs, creating the Book in the database, and storing it on the class.

Django enters a test-level transaction for test_id().

test_id() runs.

Django rolls back the test-level transaction, undoing any changes to the Book in the database.

Django enters a second test-level transaction for test_title().

test_title() runs.

Django again rolls back the test-level transaction, undoing any changes to the Book in the database.

Django rolls back the class-level transaction, removing the Book from the database.

They now only store the Book in the database only once. While currently a small saving, with more objects or tests, this really adds up.

In-memory Rollback

Let’s return to the rollback issue.

Book.objects.create does two things: it creates the object in Python, and saves its data to the database. Django’s test-level transactions only undo the database changes, and do not affect the Python object. This could lead to model instance changes leaking from one test to the next.

Django 3.2 solved this by wrapping all objects assigned during setUpTestData(). The wrapper copies the underlying object with copy.deepcopy() when first read during a test. This means each test gets its own copies of such objects, so in-memory changes are effectively rolled back between tests.

On older Django versions, we can get the that same copying behaviour in the django-testdata package. This package was created by Simon Charette, before he went on to make the change in Django 3.2 (Ticket #31395).

To use django-testdata, we add its @wrap_testdata decorator on our setUpTestData() method. This will wrap all attributes assigned during the function with the deepcopy() behaviour:

from django.test import TestCase
from testdata import wrap_testdata

from example.core.models import Book

class BookTests(TestCase):
    @classmethod
    @wrap_testdata
    def setUpTestData(cls):
        cls.book = Book.objects.create(title="Cyrus the Great")

    def test_short_title_truncated(self):
        self.book.title = (
            "An Inquiry into the Nature and Causes of the Wealth of Nations"
        )
        self.book.save()
        self.assertEqual(
            self.book.short_title, "An Inquiry into the Nature ..."
        )

That’s all it takes! When upgrading to Django 3.2, we need only drop this decorator from any test cases using it.

Converting test cases to use setUpTestData()

Taking a test suite from using just setUp() to using setUpTestData() can be done fairly easily. We can modify one test case at a time to do so. For a step-by-step guide on converting test cases, see my blog post.

pytest-django Equivalent

There isn’t an easy way of achieving the same level of rollback using pytest test functions and fixtures. pytest-django has a long open issue: #514: How to use django_db mark at session fixture?. While there are some workarounds there, it’s definitely waiting for a contribution to improve it.

Optimize Database Queries

This applies not only to the creation of your test data, but everywhere you make queries, including in your application code and tests’ “Assert” steps. If you optimize your database queries, everything will run faster. And if you optimize them in your project code it’s a double win, as things will also be faster for your users.

Optimizing Django ORM queries is a big topic, enough for another book. But I hope the pointers here are enough to get you started.

The first place to start is, again, measurement. You can use the profilers covered in Measure! to discover code paths that make a lot of queries. There are also tools dedicated to performance profiling, such as django-debug-toolbar, APM solutions such as New Relic or Scout APM, and the test performance recorder django-perf-rec.

When you have some clue about the places your code makes the most queries, you can read Database access optimization in the Django documentation. This page has a lot of hints to work through. Even if you’re not actively looking to optimize your application right now, it’s worth reading it through to learn the best patterns for ORM usage.

Some of the most useful patterns for test code are:

Using select_related() / prefetch_related() appropriately.

Using bulk_create() to create multiple objects.

Understanding the lazy nature of QuerySets and when they are evaluated.

Advanced TestCase Features

TestCase and its relatives come with some advanced features for controlling the database. These only apply in particular situations, so may not be suitable for your project. Let’s look at them now to see what they do to performance.

available_apps

If you’re using TransactionTestCase, it does a lot of work at the end of each test to reset the database. Because it doesn’t know which database tables you may have altered, it flushes all of them.

In reality, tests tend to touch only a limited number of tables, mostly for models in the current app. By setting available_apps on a test case, you can limit the TransactionTestCase rollback to flush tables for a smaller selection of apps. This can work well if you have a lot of tests, a need to use TransactionTestCase, and your project is split into enough non-interacting apps.

Unfortunately, it comes with a lot of caveats, leading to non-isolated tests or different database states. These are called out in the documentation so I won’t repeat them here. It’s also a “private API”, meaning while it’s documented, it has no guarantee that it won’t change between versions.

Because it’s quite niche, I won’t go over an example here. If you think your project could use it, read up on the documentation, and check out how Django’s own test suite uses it.

reset_sequences

If reset_sequences is set to True, Django will reset the auto-increment ID sequences on all your tables between tests. As the documentation calls out, this is a very slow operation. You should remove it from all your test cases.

The only reason to keep it is having tests that depend on hard-coded ID’s, such as:

def test_book_view(self):
    Book.objects.create(title="The Theory of Everything")
    response = self.client.get("/books/1/")
    self.assertContains(response, "The Theory of Everything")

You should rewrite such tests to dynamically use auto-generated ID’s:

def test_book_view(self):
    book = Book.objects.create(title="The Theory of Everything")
    response = self.client.get(f"/books/{book.id}/")
    self.assertContains(response, book.title)

These tests are also clearer with less hard-coded strings, as they show how the test data relates to the expected values.

It’s possible to depend on auto-generated ID’s in more subtle ways than this. You should Check Your Tests Are Isolated to help prevent such dependencies.

databases / multi_db

Note

This section only applies to multi-database projects, when your DATABASES setting contains more than one entry.

On such projects, you can control which databases are available by setting the databases attribute on your TestCase classes.

(Prior to Django 2.2, this was controlled by the less fine-grained multi_db attribute. This was deprecated from Django 2.2, and removed in Django 3.1.)

It can be tempting to create your own custom TestCase classes for your whole project and set databases to contain all your databases. But if some of your tests don’t use all of your databases, this unnecessarily slows them down. For each database listed, Django creates and rolls back transactions per class and per test, so where the databases aren’t used these transactions are unnecessary.

Instead you should define it in much more limited fashion, normally on a class-by-class basis:

class FeatureOneTests(TestCase):

    databases = {"default"}

    ...

class FeatureTwoTests(TestCase):

    databases = {"default", "reporting"}

    ...

If only some of your apps use secondary databases, you could also use custom TestCase classes for those apps to set databases.

See more in the Django documentation at Multi-database support.

Targeted Mocking

Always be a first-rate version of yourself, instead of a second-rate version of somebody else.

—Judy Garland

Mocking is a useful tool for writing tests, but if it is imprecise it make your tests confusing, inaccurate, and hard to edit. Here are some of the safer, more targeted strategies for mocking that can help speed up your tests.

The Five Kinds of Mock Objects

Mocking is swapping a part of a system during tests for an alternative “mock” object. A mock object emulates the behaviour of the replaced component, whilst providing an advantage for testing, such as capturing data for assertions. Mocking tends to make tests faster because mock objects do less work, be it HTTP requests, database queries, or calculation. But this comes at a cost of possible test inaccuracy.

In Python, the name “mock object” has been popularized for many kinds of test object, thanks to the unittest.mock library. But in the wider programming literature, the name “test double” is used as a more generic term, as popularized by the classic unit testing book xUnit Test Patterns. The book identifies five types of test double:

Dummy Object - a simple object that the tested component requires, but will be ignored in the current test. For example, you might pass object() for a required but ignored parameter.

Test Stub - an object that is configured to return specific results to the tested component. For example, you can replace an HTTP API wrapper to return the result you require in your test.

Test Spy - an object that records indirect outputs, such as method calls, to allow assertions on them in the test. For example, if testing a view decorator, you might provide it a “spy” view object that records its calls, so you can assert they were made with the correct arguments.

Mock Object - a stub that’s focussed on capturing and directly making assertions on what the tested component does. For example, you might have a mock HTTP API wrapper that both returns results and verifies that the tested component makes certain requests.

Fake Object - a component swapped in for reasons other than making test assertions. For example, using faster in-memory alternatives for caching, file storage, and task queues (as covered in Easy Wins).

So, while “mock object” and “mocking” can mean something specific when discussing testing with developers in other languages, in Python we tend to use the terms more broadly. I’ll stick to that practice here, and call it “mocking” with “mock” objects, but keep in mind these different categories of test double. It’s useful when writing a test to think about what category of behaviour you’re looking for.

Additionally, the categories can be blurred, with mock objects acting in more than one of these capacities at once. This is especially the case in Python, since it’s a flexible language, and unittest.mock takes advantage of this flexibility.

The Danger With Mocking

Mock objects behave differently to the component they replace. The greater this difference in behaviour, the more likely the tested component behaves differently, and the more likely your tests are inaccurate. And inaccurate tests give you false confidence that your system works when it doesn’t.

How accurate your tests are depends on how close your mock objects’ behaviour is to the components they replace.

Hand-crafted mock objects that implement the same interface as the replaced component are the best. By implementing the same interface and similar behaviour, they can protect you a lot. It’s best they have their own tests that compare them to the real implementation for parity (how similar they are). Such mock objects are often called “fakes”, although that’s a bit of a different definition to the xUnit Test Patterns “fake object” definition.

Generic mock objects with broad, catch-all behaviour are the worst. If they allow almost any operation to succeed, you might even find it hard to write a test that fails!

Let’s look at how you can best use unittest.mock, and then some more specific mocking tools.

unittest.mock

unittest.mock has been part of the standard library since Python 3.3, and was available separately before then. It’s the “go-to” for mocking. It fulfils the two parts of mocking:

mock.patch() and other patchers provide ways to temporarily swap Python objects for mock objects.

Mock and its subclasses provide ways to construct flexible mock objects.

Let’s look at them in turn.

Patchers

Patchers target an object either by a direct reference, or through an import statement. For example, mock.patch.dict() adds or replaces dictionary keys temporarily. You can use it to temporarily replace an environment variable:

import os
from unittest import mock

from django.test import SimpleTestCase

class EnvironmentTests(SimpleTestCase):
    @mock.patch.dict(os.environ, {"DJANGO_ALLOW_ASYNC_UNSAFE": "true"})
    def test_that_requires_async_unsafe(self):
        ...

During test_that_requires_async_unsafe(), the environment variable DJANGO_ALLOW_ASYNC_UNSAFE will be set to "true".

Note

If your settings file reads environment variables, you won’t be able to replace them this way. The settings file runs only once, at import time. You should instead use Django’s @override_settings - see Mock Settings.

The most commonly used patcher is mock.patch(). It takes a string for its target, which it will use to import the object and replace the named attribute.

mock.patch() defaults to creating a MagicMock() for you, which we’ll see below. But you can change this by setting the new argument, which allows you to use a hand-crafted, more targeted mock object. For example, you can use it to temporarily override a module-level constant:

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core.models import Book

class BookListViewTests(TestCase):
    @mock.patch("example.core.views.BOOK_PAGINATION_COUNT", new=1)
    def test_pagination(self):
        book1 = Book.objects.create(title="The Firm")
        book2 = Book.objects.create(title="Meditations")

        response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book1.title)
        self.assertNotContains(response, book2.title)

You can use mock.patch.object() to patch the same things as mock.patch(), but with direct targeting:

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core import views
from example.core.models import Book

class BookListViewTests(TestCase):
    @mock.patch.object(views, "BOOK_PAGINATION_COUNT", new=1)
    def test_pagination(self):
        book1 = Book.objects.create(title="The Firm")
        book2 = Book.objects.create(title="Meditations")

        response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, book1.title)
        self.assertNotContains(response, book2.title)

This can reduce confusion around incorrectly targeted mocks. You do have to write an extra import statement, but that’s normally worth it.

Note

Python imports are not always intuitive, as they allow variables in several modules to refer to the same object. This can make it hard to correctly target mocks. I suggest reading Ned Batchelder’s post Why Your Mock Doesn’t Work to understand this problem.

Patchers are flexible and work as context managers, function decorators, class decorators, and standalone objects with start() and stop() methods. If you decorate a class with one, it will apply itself as a function decorator to all methods beginning with test_. For example, you can repeat the above, but for each test, with:

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core import views
from example.core.models import Book

@mock.patch.object(views, "BOOK_PAGINATION_COUNT", new=1)
class BookListViewTests(TestCase):
    def test_pagination(self):
        ...

This is convenient to avoid repetition.

Mock And Its Subclasses

Mock objects are very flexible. They’re intended to provide a lot of behaviour by default, which means they can replace nearly anything with little work. For example, any attribute access on a Mock() returns another child Mock():

>>> from unittest import mock
>>> mymock = mock.Mock()
>>> mymock
<Mock id='4394866960'>
>>> mymock.attribute
<Mock name='mock.attribute' id='4394847344'>

Note how the child has a name that contains which attribute access created it.

Mock()s are callable, accepting any arguments and returning another Mock():

>>> mymock(1, 2, 3)
<Mock name='mock()' id='4394844896'>

Much of this behaviour can be customized by setting extra attributes.

Mock() provides several attributes and methods for asserting what actions have taken place against it. For example, the mock_calls attribute provides a list of call objects that represent function calls:

>>> mymock.mock_calls
[call(1, 2, 3)]

The Subclasses

Mock’s subclasses provide extensions to its behaviour:

NonCallableMock is not callable.

MagicMock adds implementations for all the Python “magic” methods For example it defines __enter__() to allow its use as a context manager.

NonCallableMagicMock combines the above two classes.

PropertyMock can be used to replace @property definitions on classes.

AsyncMock can be used to replace asynchronous functions defined with async def.

When you create a patcher like mock.patch() without a new argument, it checks what its target is. If it’s replacing an asynchronous function, it uses an AsyncMock(), otherwise it uses a MagicMock().

The Danger of Catch-All Behaviour

Since Mock and its subclasses provide a lot of behaviour with little typing, they allow you to write tests quickly. But they’re risky since their catch-all behaviour can make them very mismatched with their target. Code that should fail passes, because nearly any operation on a Mock() succeeds.

For example, this test mocks out the Book model class (with a MagicMock(), the default replacement from the patcher):

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core import views

class BookListViewTests(TestCase):
    def test_get(self):
        with mock.patch.object(views, "Book"):
            response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)

The test passes, even though a major component is replaced. All the attribute accesses, function calls, slicing, and other operations succeed on the MagicMock() object and its children. If there was an obvious typo, such as the view using Book.ojets instead of Book.objects, the test would not discover this.

Because of this risk, Mock and its subclasses should be avoided where possible. Replacing with a hand-crafted mock class is better, even though it’s more work up front.

(Hint: you can build simple mock objects in one line with types.SimpleNamespace.)

If you are going to use Mock, you can make it more safe by using specifications.

Mock Specifications

Specifications, or “specs”, limit the available attributes and behaviour of a Mock to match the target object. There are two kinds - plain “specs” and “autospecs”. As we’ll see, you’ll normally want to use the latter, but let’s first look at the simpler “specs”.

spec is an argument to Mock() and its subclasses, and patchers that create Mock() objects. It restricts the available attributes on the Mock() object. It can be either a list of attribute names, or an object to copy the list of attribute names from. For example, to use it in the above tests, we would pass the original Book model class:

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core import views

class BookListViewTests(TestCase):
    def test_get(self):
        with mock.patch.object(views, "Book", spec=views.Book):
            response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)

The Mock() object would then be restricted to the list of attribute names defined on the Book class. If our view had the Book.ojets typo, the tests would fail:

$ python manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_main (example.core.tests.test_mock_mismatched_spec.MockTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/core/tests/test_mock_mismatched_spec.py", line 12, in test_main
    response = self.client.get("/books/")
  ...
  File "/.../example/core/views.py", line 9, in book_list_view
    books = Book.objets.all()[:BOOK_PAGINATION_COUNT]
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/unittest/mock.py", line 637, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'objets'

----------------------------------------------------------------------
Ran 1 test in 0.036s

FAILED (errors=1)
Destroying test database for alias 'default'...

There’s a stronger version of spec, spec_set, which also blocks unknown attributes from being set. But this is not always useful, since Python objects normally allow arbitrary attributes to be set. Several Django features even rely on this behaviour.

Unfortunately, spec doesn’t affect the Mock()’s children. For example, the above Mock() errors for typo’d attribute accesses on Book, but not for objects. You can see this by using a typo for objects.filter():

>>> from unittest import mock
>>> from example.core.models import Book
>>> mock_book = mock.Mock(spec=Book)
>>> mock_book.ojets
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/.../unittest/mock.py", line 637, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'ojets'
>>> mock_book.objects
<Mock name='mock.objects' id='4557671056'>
>>> mock_book.objects.filler(x=1)
<Mock name='mock.objects.filler()' id='4557671728'>
>>> mock_book.objects.filler(author__name='Mary Shelley')
<Mock name='mock.objects.filler()' id='4557671728'>

This is where autospeccing can help. An “autospec” is a spec based upon an object that follows its attributes as much as possible. They’re created with the create_autospec function, which determines the right Mock() subclass to use. For example, you can make one as above, and the typo’d child attribute access fails:

>>> mock_book = mock.create_autospec(Book)
>>> mock_book.objects.filter()
<MagicMock name='mock.objects.filter()' id='4501079472'>
>>> mock_book.objects.filler()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/.../unittest/mock.py", line 637, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'filler'

Autospeccing is not perfect. It doesn’t discover the return types of functions, so they return generic Mock() objects. For example, if you had typo’d a chained exclude() operation as exude(), that would incorrectly succeed:

>>> mock_book.objects.filter().exude()
<MagicMock name='mock.objects.filter().exude()' id='4501177104'>

That said, you should strive to use autospeccing every time you want to use Mock or its subclasses.

Autospeccing is conveniently built in to mock.patch() with the autospec argument:

from http import HTTPStatus
from unittest import mock

from django.test import TestCase

from example.core import views

class BookListViewTests(TestCase):
    def test_get(self):
        with mock.patch.object(views, "Book", autospec=True):
            response = self.client.get("/books/")

        self.assertEqual(response.status_code, HTTPStatus.OK)

Mock Settings

Sometimes we need to change a Django setting during a test function or class.

We can do this by assigning values directly to the django.conf.settings object, or using mock.patch.object() on it, but both these options have problems. First, direct assignments aren’t automatically undone, which can break test isolation. Second, neither option will trigger Django’s setting_changed signal, which means certain objects won’t update to reflect their settings.

Instead, we should use Django’s targeted mocking tools for settings, as documented in Overriding settings. The most flexible tool is @override_settings, which works as a class decorator, function decorator, and context manager. We can use it in all three forms like so:

from django.test import SimpleTestCase
from django.test.utils import override_settings

from example.core.views import get_list_items

@override_settings(PAGE_SIZE=10)
class ListTests(SimpleTestCase):
    def test_with_10(self):
        items = get_list_items()

        self.assertEqual(len(items), 10)

    @override_settings(PAGE_SIZE=20)
    def test_with_20(self):
        items = get_list_items()

        self.assertEqual(len(items), 20)

    def test_with_30(self):
        with override_settings(PAGE_SIZE=30):
            items = get_list_items()

        self.assertEqual(len(items), 30)

The other tools are @modify_settings, SimpleTestCase.settings(), and SimpleTestCase.modify_settings(). @modify_settings provides a way of mutating a setting without completely redefining it, for example appending to a setting that’s a list. The SimpleTestCase tools are shortcuts to use override_settings from within a test case without an extra import.

I normally use only @override_settings, since it’s clearer than @modify_settings and more flexible than the SimpleTestCase shortcuts.

pytest-django’s settings Fixture

pytest-django wraps up @override_settings in its settings fixture. We can use the fixture by directly assigning to it, and it will copy such assignments to Django’s settings:

from example.core.views import get_list_items

def test_with_10(settings):
    settings.PAGE_SIZE = 10

    items = get_list_items()

    assert len(items) == 10

All changes are reverted when the test function finishes.

Mock Output

This follows on from the output capturing described in Prevent Output.

You might want to make assertions your code’s output to stdout or stderr. There are several ways of capturing it in order to achieve this.

With StringIO()

Python’s io.StringIO acts as in-memory file-like object. Since sys.stdout and sys.stderr are file-like objects themselves, a StringIO makes a perfect mock object to replace them with. You can allow the tested component to write to it, then make assertions on the output it received.

For example, imagine you wanted to test a function that used print() to output a string to stdout. You could do that by using mock.patch.object() to replace sys.stdout temporarily with a StringIO():

import sys
from io import StringIO
from unittest import mock

from django.test import SimpleTestCase

from example.core.emailing import send_author_report

class SendAuthorReportTests(SimpleTestCase):
    def test_sent(self):
        out = StringIO()
        with mock.patch.object(sys, "stdout", new=out):
            send_author_report()

        self.assertEqual(out.getvalue(), "Sent!\n")

This works as long as the tested code uses import sys and references sys.stdout, rather than doing from sys import stdout. Thankfully import sys is the convention, and Python’s built-ins use that.

With pytest’s capsys Fixture

pytest provides fixtures that capture output and allow you to make assertions on it. These are documented in How to capture stdout/stderr output.

Normally, the one you want to use is capsys. If you’re writing function-based tests, you can use it directly, as per the example in the documentation. But if you’re using Django’s TestCase classes, you’ll need to wrap the fixture with the method described in mixing pytest fixtures with unittest. For example:

import pytest
from django.test import SimpleTestCase

from example.core.emailing import send_author_report

@pytest.fixture(scope="function")
def unittest_capsys(request, capsys):
    request.cls.capsys = capsys

@pytest.mark.usefixtures("unittest_capsys")
class SendAuthorReportTests(SimpleTestCase):
    def test_sent(self):
        send_author_report()

        captured = self.capsys.readouterr()
        self.assertEqual(captured.out, "Sent!\n")
        self.assertEqual(captured.err, "")

The capsys fixture is essentially a wrapper around StringIO() objects.

Management Commands

If you’re testing a management command, the ability to capture its output is built-in. Your command code should use the self.stdout and self.stderr proxies for its output, which Django uses to add extra output behaviour. Then when testing, you can replace the file-like object they proxy when calling call_command(). This is covered in the Django documentation.

Imagine you had to test this short management command:

from django.core.management import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        self.stdout.write("Hello!\n")

You can capture the output on stdout by passing a StringIO() to the stdout argument:

from io import StringIO

from django.core.management import call_command
from django.test import SimpleTestCase

class MyCommandTests(SimpleTestCase):
    def test_it(self):
        out = StringIO()

        call_command("hello", stdout=out)

        self.assertEqual(out.getvalue(), "Hello!\n")

You can do the same for stderr if required.

Mock Input

If you need to mock stdin you can use the same StringIO() technique as above.

Imagine you had to test this friendly management command:

from django.core.management import BaseCommand

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        self.stdout.write("What is your name? ", ending="")
        name = input()
        self.stdout.write(f"Hello {name}!\n")

You can mock the input by using mock.patch.object() to replace sys.stdin temporarily with a StringIO() initialized with the user’s input:

import sys
from io import StringIO
from unittest import mock

from django.core.management import call_command
from django.test import SimpleTestCase

class MyCommandTests(SimpleTestCase):
    def test_it(self):
        out = StringIO()

        with mock.patch.object(sys, "stdin", StringIO("Adam\n")):
            call_command("greeting", stdout=out)

        self.assertEqual(out.getvalue(), "What is your name? Hello Adam!\n")

(Note: management commands don’t have a stdin proxy, so there’s no stdin argument for call_command().)

Mock HTTP Requests

In this connected world, it’s likely your project makes HTTP requests, for scraping or communicating with API’s. When testing these features, you don’t want to rely on the third party service being available. You might have a few integration tests that make external requests (perhaps running only on CI), but most of your tests shouldn’t make requests. Ideally your normal test suite should pass in “aeroplane mode”, when your computer is disconnected from the internet.

The Python community has a number of targeted mocking libraries available to solve this problem. Let’s look at two libraries with different approaches: requests-mock and VCR.py.

requests-mock

requests-mock allows you to mock the popular requests library. It works by injecting itself as a fake “adapter” inside of requests, which makes its mock responses transparent. By being a targeted mock, it knows a lot about HTTP and provides many useful shortcuts.

(There’s a similar library called responses. It’s more popular, but I think requests-mock has a better API and deserves more exposure.)

Imagine you had this view which uses requests to call the Wikipedia Search API:

import requests
from django.shortcuts import render

from example.core.forms import WikipediaSearchForm

def wikipedia_search(request):
    form = WikipediaSearchForm(request.GET)

    search_results = []
    if form.is_valid():
        response = requests.get(
            "https://en.wikipedia.org/w/api.php",
            params={
                "format": "json",
                "action": "query",
                "list": "search",
                "srsearch": form.cleaned_data["query"],
            },
        )
        response.raise_for_status()
        search_results = response.json()["query"]["search"]

    return render(
        request,
        "wikipedia_search.html",
        {"form": form, "search_results": search_results},
    )

You can test it by using requests-mock’s Mocker() class, that decorates a test method:

from http import HTTPStatus

import requests_mock
from django.test import SimpleTestCase

class WikipediaSearchTests(SimpleTestCase):
    @requests_mock.Mocker()
    def test_query(self, mock_requests):
        mock_requests.get(
            (
                "https://en.wikipedia.org/w/api.php"
                + "?format=json"
                + "&action=query"
                + "&list=search"
                + "&srsearch=The%20Black%20Swan"
            ),
            json={"query": {"search": [{"title": "Black swan theory"}]}},
        )

        response = self.client.get("/", {"query": "The Black Swan"})

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Black swan theory")

The decorator adds an argument to the function that the test calls mock_requests. This is the Mocker() instance, which has an API similar to a requests.Session() but used for enqueueing mock responses. The test declares one expected URL with a response defined by the json argument, which is a shortcut for specifying a JSON body.

When the view runs its call to requests works as normal and requests-mock’s adapter returns the in-memory response without any network interaction.

Block All Outbound Requests

If you use a Mocker() and the tested code makes a request for which no mocked response matches, the adapter raises an exception:

$ python manage.py test example.core.tests.test_views_bad
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_query (example.core.tests.test_views_bad.WikipediaSearchTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../test_views_bad.py", line 8, in test_query
    response = self.client.get("/", {"query": "The Black Swan"})
...
  File "/.../requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/.../requests_mock/adapter.py", line 258, in send
    raise exceptions.NoMockAddress(request)
requests_mock.exceptions.NoMockAddress: No mock address: GET https://en.wikipedia.org/w/api.php?format=json&action=query&list=search&srsearch=The+Black+Swan

----------------------------------------------------------------------
Ran 1 test in 0.047s

FAILED (errors=1)

You can use this behaviour to block all outgoing requests in your tests by default. This will help you get your tests to run in “aeroplane mode”.

With Django’s Test Framework

You can use a custom test runner to do this:

import requests_mock
from django.test.runner import DiscoverRunner
from django.test.utils import override_settings

class ExampleTestRunner(DiscoverRunner):
    def run_tests(self, *args, **kwargs):
        with requests_mock.Mocker(), override_settings(**TEST_SETTINGS):
            return super().run_tests(*args, **kwargs)

TEST_SETTINGS = {
    "PASSWORD_HASHERS": ["django.contrib.auth.hashers.MD5PasswordHasher"],
}

This creates a Mocker() and activates it for the duration of the test run. Since Mocker() objects stack, tests can use their own to mock expected requests. Note I’ve added a use of @override_settings, as shown previously for test settings.

With pytest-django

You can create a session-scoped automatically-used fixture in your conftest.py:

import pytest
import requests_mock

@pytest.fixture(scope="session", autouse=True)
def session_requests_mock():
    with requests_mock.Mocker():
        yield

pytest starts this fixture at the start of the test run (the “session” scope), it starts the Mocker(), and pauses on the yield statement until the end. Nice.

VCR.py

VCR.py provides an alternative approach to mocking HTTP requests. It’s named after an ancient technology, the videocassette recorder (VCR), used for recording and replaying television. VCR.py works similarly - it records and replays HTTP requests and responses.

The first time you run a test using VCR.py, it allows HTTP requests to be made, recording the responses in a “cassette” file. From the second run onward, requests instead return mock responses, based on the contents of the cassette file.

This is a technique known as “snapshot testing”, which can be applied to other kinds of mocking. It can save you a lot of time writing tests, and increases accuracy because the tests run once against the external service. But it can often be hard to run tests against external services, especially without side effects. For example, you may use a service that is only accessible from your production network.

You can re-implement the Wikipedia search tests we looked at with requests-mock using VCR.py like so:

from http import HTTPStatus
from pathlib import Path

import vcr
from django.test import SimpleTestCase

module_dir = Path(__file__).parent
fixtures_dir = module_dir / "fixtures"

class WikipediaSearchTests(SimpleTestCase):
    @vcr.use_cassette(str(fixtures_dir / "test_query.yml"))
    def test_query(self):
        response = self.client.get("/", {"query": "The Black Swan"})

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Black swan theory")

It’s less code since we don’t need to declare which requests to mock, or anything about the responses. The use_cassette() decorator directs VCR.py to record all interactions in the wrapped function in the named cassette file. We’re using pathlib here to construct a path relative to the test module’s path.

The first time you run this test, it takes a noticeable amount of time as the request goes to the Wikipedia API:

 $ python manage.py test example.core.tests.test_views
 System check identified no issues (0 silenced).
 .
 ----------------------------------------------------------------------
 Ran 1 test in 0.513s

 OK

From the second time you run it, it’s much faster, since VCR.py returns the mock response:

 $ python manage.py test example.core.tests.test_views
 System check identified no issues (0 silenced).
 .
 ----------------------------------------------------------------------
 Ran 1 test in 0.025s

 OK

The cassette file is big as it records all details of the HTTP response, in case any of them affect the tested component. For this test, the file is 95 lines, including a binary encoded HTTP body. Here’s how it starts:

interactions:
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - gzip, deflate
      Connection:
      - keep-alive
      User-Agent:
      - python-requests/2.23.0
    method: GET
    uri: https://en.wikipedia.org/w/api.php?format=json&action=query&list=search&srsearch=The+Black+Swan
  response:
    body:
      string: !!binary |
        H4sIAAAAAAAAA81Ya2/bNhT9K4Q+DAngZNRb1roC6Tqsr3TD4mHA1n2gZMriLJEqScVwH/99h1Jd
        22viuEWGFciHUOLj6p57zzn0W69gtqxL1XYNt9zLPW/ilUpaIXuM3npGq6oy3Hq5T3ffeGfv3nnv

The request key specifies the request our code made, and the response key specifies response sent by the Wikipedia API. I’ve cropped off the rest of the response body, headers, and status.

If the tested component, test, or external service changes, the cassette file will need regenerating. This can be done by deleting it and running the tests again, when they’ll again make requests against the external service.

Prevent Recording on CI

VCR.py supports several “record modes”. The default, "once", describes the above behaviour: the first run records the cassette file, and future runs return mock responses based on it.

To check your tests are accurate, it’s advisable to run the tests without any possibility of recording. This can help you detect mistakes such as forgetting to commit a cassette file to your source control. The "none" record mode guards against this. It returns mock responses from existing cassette files but otherwise blocks requests.

A good workflow would be to use "once" during development, and "none" on CI. This makes it more convenient to develop tests, and the CI system checks that all cassette files exist and are complete.

You can do this by creating a custom instance of vcr.VCR() in your test extensions module:

import os
import vcr

on_ci = os.environ.get("CI", "") == "true"
vcr = vcr.VCR(record_mode=("none" if on_ci else "once"))

This uses the same “are we on CI?” check referred to previously in Skip Slow Tests Locally. Which environment variable to check for depends on your CI system.

Your tests should swap to use this instance:

from http import HTTPStatus
from pathlib import Path

from django.test import SimpleTestCase

from example.test import vcr

module_dir = Path(__file__).parent
fixtures_dir = module_dir / "fixtures"

class WikipediaSearchTests(SimpleTestCase):
    @vcr.use_cassette(str(fixtures_dir / "test_query.yml"))
    def test_query(self):
        response = self.client.get("/", {"query": "The Black Swan"})

        self.assertEqual(response.status_code, HTTPStatus.OK)
        self.assertContains(response, "Black swan theory")

By default this works as before. But if delete the cassette file and run the tests “in CI mode”, VCR.py blocks the request with a CannotOverwriteExistingCassetteException():

 $ CI=true python manage.py test example.core.tests.test_views_record_mode
 System check identified no issues (0 silenced).
 E
 ======================================================================
 ERROR: test_query (example.core.tests.test_views_record_mode.WikipediaSearchTests)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
   File "/.../site-packages/vcr/cassette.py", line 99, in __call__
     return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs)
   ...
   File "/.../example/core/tests/test_views_record_mode.py", line 16, in test_query
     response = self.client.get("/", {"query": "The Black Swan"})
   ...
   File "/.../vcr/stubs/__init__.py", line 231, in getresponse
     raise CannotOverwriteExistingCassetteException(
 vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/Users/chainz/Documents/Projects/12_targeted_mocking/book-speed-up-your-django-tests/projects/12_targeted_mocking/vcrpy/example/core/tests/fixtures/test_query.yml') in your current record mode ('none').
 No match for the request (<Request (GET) https://en.wikipedia.org/w/api.php?format=json&action=query&list=search&srsearch=The+Black+Swan>) was found.
 No similar requests, that have not been played, found.

 ----------------------------------------------------------------------
 Ran 1 test in 0.065s

 FAILED (errors=1)

The message explicitly tells you about the blocked request, and the traceback helps discover the test making it.

Mock Time

It’s likely your project has features that rely on changes to the current date and time (which I’ll call just the time). For example, you might have a feature where a free book may be claimed once per day. In code for these features, you can access the current time in one of two ways: either you pass it around as a variable, or you fetch it where needed through a function call.

If you pass around the time in a variable, your system will be easier to test, as tests can pass in different values for the current time to “mock” it. However, it’s not always practical or possible to pass around the current time. For example you can’t pass it into models from third-party Django packages.

Fetching the current time through a function call makes your project easier to write, and is used a lot in practice. However, to test it, that means you need to mock the functions that return the current time. This can get complicated.

In Python, there are many functions that return the current time, in different formats, in both the datetime and time modules. For example, time.time() returns a UNIX timestamp, and datetime.datetime.utcnow() returns a datetime object. On top of these functions, there are many popular wrappers, for example django.utils.timezone.now() which adds Django’s timezone support, or those provided by the arrow and Delorean libraries.

It’s possible to use unittest.mock to mock any of these individual functions, but it can quickly become inaccurate as each patcher only mocks a single function reference. This inaccuracy is multiplied by the many different functions that return the current time, in different formats, which all should return the same time. Due to the way Python’s imports work, it takes a lot of mocks to replace every instance of functions from datetime and time in a code path, and is sometimes impossible.

To solve this problem, you can use the time-machine library which I developed. This is my attempt to build a performant version of the popular freezegun (you can read more about the history in my post Introducing time-machine).

The library provides a mocking function, time_machine.travel(), that replaces the current time for all the standard library functions at once. It’s more performant than freezegun because it does its mocking at the C layer.

As an example, we can test the view for claiming a free book once per day by using time_machine.travel() to travel “into the future” like so:

import datetime as dt

import time_machine
from django.test import TestCase

from example.core.models import Book

class ClaimBookTests(TestCase):
    def setUp(self):
        self.book = Book.objects.create(
            last_giveaway_date=dt.datetime.utcnow().date()
        )

    def test_post_unsuccessful(self):
        response = self.client.post(f"/claim-book/{self.book.id}/")

        self.assertEqual(
            response.content.decode(),
            "Today's free copy has already been claimed.",
        )

    def test_post_successful(self):
        future = self.book.last_giveaway_date + dt.timedelta(days=1)

        with time_machine.travel(future):
            response = self.client.post(f"/claim-book/{self.book.id}/")

        self.assertEqual(
            response.content.decode(),
            "You have successfully claimed today's free copy!",
        )

Time travel: easy when you know how.

Outroduction

The reward of a thing well done, is to have done it.

—Ralph Waldo Emerson

Congratulations on making it through. I hope your tests are faster now, or at least you know where to start.

Further Reading

Eager to learn more about testing in Django? Great!

I have to start by recommending my own blog (again). See the top Django posts on the index and the list of all posts tagged Django.

In particular for tests, you should check out Getting a Django Application to 100% Coverage. 100% coverage means every line of your application is run at least once during tests. This is a great goal as it ensures there are fewer places bugs can hide. And it doesn’t slow down your tests - or at least the overhead is worth it.

Aside from that, you can find an index of high quality resources in the awesome-django list.

For blogs, subscribe to the Django Community blog post feed (RSS based, which you can convert to email using various services) and the Django News newsletter.

For more Django on testing, check out:

High Performance Django

Test-Driven Development with Python - Obey the Testing Goat

Two Scoops of Django

And if you haven’t, please contribute to Django! See the documentation section on getting started. You can see open testing-related tickets in the Trac ticket tracker by searching for “Component = Testing Framework”.

Thank You

Thank you spending time reading my book.

May your tests run fast, your development remain agile, and your sleep pass uninterrupted,

—Adam