A Practical Guide to Setuptools and Pyproject.toml

Rogier van der Geer/
18 March, 2022

About three years ago I wrote a blog post about using setup.py to set up your python projects. Since then a lot has changed, mostly due to PEP 517, PEP 518 and the introduction of the pyproject.toml file.

The goal of this file is to allow you to define what build tools are needed in order to build your package – no longer assuming it must be Setuptools. This makes it easier to use alternatives to Setuptools, which means that Setuptools does no longer have to be the one build tool that can do everything.

Nevertheless, I like the feature set of Setuptools, and would like to continue to use it in the foreseeable future. But while documentation for using for example Poetry or Flit together with a pyproject.toml is easy to find, it is more difficult to find similar documentation for Setuptools. So let me help you out.

The pyproject.toml file

The pyproject.toml defines what build tools are needed to build our package. This can be pretty simple:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

That is all there is to it! While alternatives like Poetry and Flit define the package information in this file as well, Setuptools places this in a separate file.

Of course, if you feel the need to restrict the version of Setuptools that can be used, you can do so like requires = ["setuptools>=40.6.0"].

The setup.cfg file

This is where all the configuration details of your package goes. These used to go in setup.py (although one could use setup.cfg also in the pre-PEP 517 era), but as you will see the declarative config used in the setup.cfg is more convenient.

Now let’s start with a basic example:

[metadata]
name = example
version = 0.1.0

[options]
packages = find:

This defines your package named example, with version 0.1.0. That is all you need!

Project Structure

The structure of your project should look like this:

/path/to/example/project/
    ├── example/             Python package directory.
    │   ├── __init__.py      This makes the directory a package.
    │   └── example.py       An example module.
    ├── pyproject.toml       Definition of build process of the package.
    ├── README.md            README with info of the project.
    └── setup.cfg            Configuration details of the python package.

Of course you can have other files or folders in the structure, such as a tests/ folder, a .gitignore or a LICENSE file, but these are not strictly required.

With the above in place, you can now build your package by running python -m build --wheel from the folder where the pyproject.toml resides. This should create a build/ folder as well as a dist/ folder (don’t forget to add these to your .gitignore, if they aren’t in there already!), in which you will find a wheel named something like example-0.1.0-py3-none-any.whl. That file contains your package, and this is all you need to distribute it. You can install this package anywhere by copying it to the relevant machine and running pip install example-0.1.0-py3-none-any.whl.

When developing you probably do not want to re-build and re-install the wheel every time you have made a change to the code, and for that you can use an editable install. This will install your package without packaging it into a file, but by referring to the source directory. You can do so by running pip install -e . from the directory where the pyproject.toml resides. Any changes you make to the source code will take immediate effect. But note that you may need to trigger your python to re-import the code, either by restarting your python session, or for example by using autoreload in a Jupyter notebook. Also note that any changes to the configuration of your package (i.e. changes to setup.cfg) will only take effect after re-installing it with pip install -e ..

Further options

Setuptools has a nice set of options that we can add. Let’s have a look at a few of those.

Dependencies

The dependencies of your package can be specified in the install_requires section of the options:

[options]
install_requires =
    pandas == 1.4.1
    PyYAML >= 6.0
    typer

If you have any dependencies that are only required in some cases, you can add them as extras_require:

[options.extras_require]
notebook = jupyter>=1.0.0, matplotlib
dev = 
    black==22.1.0
    flake8==4.0.1

These dependencies will only be installed if you ask for them, e.g. pip install -e ".[dev]" or pip install "example-0.1.0-py3-none-any.whl[dev,notebook]". Do not forget to quote the package name in those commands!

Entry points

If you have any functions in your package that you would like to expose to be used as a command-line utility, you can add them to the console_scripts entry points. For example, if you have a function called main in example.py, then adding this to your setup.cfg will allow users to run my-example-utility as a shell command:

[options.entry_points]
console_scripts = 
    my-example-utility = example.example:main

Or, if you use typer for example, and have a cli.py with the following contents:

from typer import Typer

app = Typer()

@app.command()
def hello():
    print("Hello.")

@app.command()
def bye(name: str):
    print(f"Bye {name}")

then adding

[options.entry_points]
console_scripts = 
    example-tool = example.cli:app

will allow you to run commands like example-tool hello and example-tool bye rogier.

A src/ layout

Many people prefer placing their python packages in a src/ folder in their project directory, which means having a project structure like this:

/path/to/example/project/
├── src/                     Source dir.
│   └── example/             Python package directory.
│       ├── __init__.py      This makes the directory a package.
│       └── example.py       Example module.
├── pyproject.toml           Definition of build process of the package.
├── README.md                README with info of the project.
└── setup.cfg                Configuration details of the python package.

You can do so by adding the following options to your package configuration:

[options]
package_dir=
    =src

[options.packages.find]
where=src

Package data

By default only .py files inside your package folder are added to wheels that you build. If you have other files that you would like to include, for example data that your package needs, you can add them as such:

[options]
zip_safe = True
include_package_data = True

[options.package_data]
example = data/schema.json, *.txt
* = README.md

This tells Setuptools to include the example/data/schema.json file, as well as any .txt files found in your example-package. It also tells it to include any README.md files in any package it can find (in case you have multiple packages in your project folder).

Typically python packages are installed as zip files, meaning that their source is not available as files on the file system. The zip_safe = True flag means that this is okay for your package. If you make use of __file__ attributes to find included data files in your package you will probably need to set zip_safe = False. But instead of doing that, please be kind to your users and consider using either importlib.resources or pkg_resources:

from json import load
from pkg_resources import resource_stream

def load_schema():
    return load(resource_stream("example", "data/schema.json"))

This will also work when your package is installed as a zip file.

Metadata

You might want to provide more information about your package than just a name and a version. Here is an example of some available options:

[metadata]
name = example
version = attr: example.__version__
author = You
author_email = your@email.address
url = https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml
description = Example package description
long_description = file: README.md
long_description_content_type = text/markdown
keywords = example, setuptools
license = BSD 3-Clause License
classifiers =
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3

Note that this example makes use of two special directives: the contents of the README.md are used as long description by using the file: directive, and the version of the package is read from the __version__ variable defined in the example/__init__.py. This last feat used to be pretty complicated when using setup.py.

Wrap up

If you want to make use of Setuptools and pyproject.toml to build your package, this should provide you with a good starting point:

pyproject.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

setup.cfg

[metadata]
name = example
version = attr: example.__version__
author = You
author_email = your@email.address
url = https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml
description = Example package description
long_description = file: README.md
long_description_content_type = text/markdown
keywords = example, setuptools
license = BSD 3-Clause License
classifiers =
    License :: OSI Approved :: BSD License
    Programming Language :: Python :: 3

[options]
packages = find:
zip_safe = True
include_package_data = True
install_requires =
    pandas == 1.4.1
    PyYAML >= 6.0
    typer

[options.entry_points]
console_scripts = 
    my-example-utility = example.example:main

[options.extras_require]
notebook = jupyter>=1.0.0, matplotlib
dev = 
    black==22.1.0
    flake8==4.0.1

[options.package_data]
example = data/schema.json, *.txt
* = README.md

Now build your project by running pip -m build . --wheel, or do an editable install with pip install -e .. You can find more details on the available options of the setup.cfg in the docs.

Subscribe to our newsletter

Stay up to date on the latest insights and best-practices by registering for the GoDataDriven newsletter.