Sometimes PyPI may lack some wheels for your environment:

  • the package doesn’t support your Python version yet
  • the package doesn’t support CPU architecture (hello Apple Silicon 👀)
  • the maintainer doesn’t upload wheel to PyPI

Having a good toolchain on your laptop is important and if you do, most packages will build just fine except if they need special dependencies. However, some packages may take a lot of time to compile and / or you may work with people that are not able to maintain their toolchain.

In these cases, one solution is to vendor these dependencies. Instead of relying on PyPI, you can build your own version of the wheels and host it somewhere. It can be your own PyPI server, or directly in your git(-lfs) repository.

For this example, we are going to use poetry as package manager and grpcio as the package lacking some wheels (1.47.2 is required by dagster 1.1.19, but mac arm64 wheel is not available).

Building the wheel

On an Apple Silicon mac with a working toolchain, download the source version of the package then build the wheel using the target python version.

# download and extract source code
wget https://files.pythonhosted.org/packages/f2/f1/165cb12e2601ffa55958579357577be92dcf2ec365c5d92cac0f26df02a6/grpcio-1.47.2.tar.gz
tar -xvf grpcio-1.47.2.tar.gz
cd grpcio-1.47.2

# set env variables required by grpcio on mac
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

# build
python setup.py build
python setup.py bdist_wheel

The wheel will be available in the dist folder.

Register the vendored wheel

Here is an example pyproject.toml for your project:

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = ""
authors = ["My Name <me@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"

grpcio = "==1.47.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Copy the wheel in your project directory, for example in a vendor folder:

.
├── poetry.lock
├── pyproject.toml
└── vendor
    └── grpcio-1.47.2-cp310-cp310-macosx_12_0_arm64.whl

Now edit your pyproject.toml:

[tool.poetry]
name = "myproject"
version = "0.1.0"
description = ""
authors = ["My Name <me@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"

grpcio = [
    {markers = "sys_platform != 'darwin' or platform_machine != 'arm64'", version = "==1.47.2", source = "pypi"},
    {markers = "sys_platform == 'darwin' and platform_machine == 'arm64'", file = "vendor/grpcio-1.47.2-cp310-cp310-macosx_12_0_arm64.whl"},
]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Don’t forget to regenerate your lock file with poetry lock then install your dependencies using poetry install:

  • your x86_64 linux machines will fetch the wheel on PyPI
  • your x86_64 mac machines will fetch the wheel on PyPI
  • your arm64 mac machines will use the vendored wheel distributed through git