macOS ARM builds on conda-forge

A new platform osx-arm64 has been added to the build matrix of conda-forge. osx-arm64 packages are built to run on upcoming macOS arm64 processors marketed as Apple Silicon. An installer for this platform can be found here.

This will install a conda environment with python and conda in it. Installed conda will be able to install packages like numpy, scipy. Currently there are about 100 packages out of 10000 packages pre-built for this platform.

All these packages are built on conda-forge's current macOS x86_64 infrastructure. In order to do so, we have made lots of changes to the infrastructure including, conda, conda-build, conda-smithy, constructor, conda-forge-ci-setup to facilitate cross-compiling which is the process of compiling a package that will run on a host platform (osx-arm64 in our case), with the compilation done on a build platform (osx-64 or linux-64 in our case).

osx-arm64 is the first conda platform that is completely bootstrapped using conda-build's cross-compiling facility. Previously, when adding a new platform, conda-build was built with an existing python and pip environment on the new platform. With cross-compiling, when the compilers and a sysroot is set up on a different platform, an existing conda-build installation (on osx-64 and linux-64 in this case) will be able to start building packages right away.

Cross-compiling builds for osx-arm64

In order to cross compile packages for osx-arm64 we need compilers. So, we first built clang=11.0.0.rc1 which has support for targetting osx-arm64. We also built compiler-rt=11.0.0.rc1 as a universal build support both osx-64 and osx-arm64.

Linker, archiver, otool, install_name_tool was built using the cctools-port project by Thomas Pöchtrager.

One issue we ran into was that the macOS 11.0 Big Sur Beta 7 required that all executables and shared libraries be ad-hoc signed which is signing without a verified signature. On suggestion of cctools-port developer we added support to cctools-port to sign these executables using ldid which can be used on Linux as well as macOS to sign.

Using these, the first cross compiled package we built was libcxx to facilitate C++ builds. For the osx-arm64 sysroot we used the MacOSX11.0.sdk already installed on Azure pipelines and Travis-CI. Due to licensing issues, we cannot distribute this, but it can be downloaded from the Apple developer website even on Linux.

With clang we have a C/C++ compiler, but lack a Fortran compiler. We used the GCC fork for darwin-arm64. First, a cross compiler (build == host != target) was built. Using that compiler, we built a cross-native compiler (build != host == target) which gave use the shared libraries like libgfortran.dylib.

We also added support for cross compiling rust programs to the rust packages in conda and installing rust_osx-64 on Linux will give you a compiler that will build packages for osx-64.

As we haven't done cross-compilation before, many packages needed to be updated. Most were trivial changes that we automated later on. These included getting a newer config.sub to identify the new autotools platform arm64-apple-darwin20.0.0, adding options to CMake with the environment variable CMAKE_ARGS to correctly set up the toolchain and recipes were update to use cmake ${CMAKE_ARGS} ... Running tests when building were also disabled by guarding commands like make check, make test, ctest with the env variable CONDA_BUILD_CROSS_COMPILATION.

Cross-compiling python extensions is quite tricky as distutils is not really setup to do this. Thanks to the project crossenv this is unofficially supported with a few quirks. With crossenv, we can run a python on the build machine (osx-64 or linux-64 in this case) that acts like it is on osx-arm64. crossenv monkey-patches a few functions like os.uname and sets up values like _PYTHON_SYSCONFIG_DATA to make python running on osx-64 or linux-64 behave like osx-arm64. One issue is that, monkey-patching sys.platform doesn't work and therefore if a python package in it's setup.py uses sys.platform to differentiate OSes this will lead to unintended consequences if you are cross-compiling from linux-64. Therefore, we have to use osx-64 as our build system when cross-compiling for osx-arm64. Note that packages using sysconfig.get_platform() will get the correct platform.

For creating an installer for conda, we needed a standalone conda executable to bootstrap the conda environment. For other platforms we relied on conda-standalone which is a standalone conda executable created using pyinstaller. Since pyinstaller does not support cross-compile, we decided to use micromamba as the bootstrapper and added features to micromamba so that it can function as the bootstrapper.

How to add a osx-arm64 build to a feedstock

All the below changes will be done by a bot and the packages the bot will send PRs to is determined by the list of packages at conda-forge-pinning and their dependences. If you would like to add support, please send a PR adding the feedstock name to the above list. After that PR is merged, you can monitor the status at conda-forge status-page and if a particular PR is stalled you can send a PR to the feedstock to fix it.

Following instructions are for when you want to add support manually.

Add the following to conda-forge.yml (on Linux or OSX),

build_platform:
  osx_arm64: osx_64
test: native_and_emulated

You can rerender using,

conda smithy rerender

For python packages, add one or more of the following to recipe/meta.yaml as needed, noting that you must only add numpy, cython, and/or pybind11 if they are used in host: as well,

requirements:
  build:
    - python                                 # [build_platform != target_platform]
    - cross-python_{{ target_platform }}     # [build_platform != target_platform]
    - cython                                 # [build_platform != target_platform]
    - numpy                                  # [build_platform != target_platform]
    - pybind11                               # [build_platform != target_platform]

For autotools package, add the following to recipe/meta.yaml,

requirements:
  build:
    - gnuconfig   # [unix]

and to recipe/build.sh,

# Get an updated config.sub and config.guess
cp $BUILD_PREFIX/share/gnuconfig/config.* .

For cmake packages, add the following to recipe/build.sh,

cmake ${CMAKE_ARGS} ..

For meson packages, add the following to recipe/build.sh,

meson ${MESON_ARGS} builddir/

note

Conda automatically creates a cross build definition file when cross-compiling, and adds the necessary argument to ${MESON_ARGS} to point meson to that file. ${MESON_ARGS} is only defined when cross-compiling, not for normal builds.

For rust packages, add the following to recipe/meta.yaml,

requirements:
  build:
    - {{ compiler('rust') }}

If there's a line like make check in recipe/build.sh that cannot be run when cross-compiling, do the following,

if [[ "$CONDA_BUILD_CROSS_COMPILATION" != "1" ]]; then
  make check
fi

After these changes, another rerendering might be required.

Some useful jinja variables,

1. build_platform - conda subdir for BUILD_PREFIX. eg: linux-64
2. target_platform - conda subdir for PREFIX. eg: osx-arm64

Some useful environment variables,

1. build_platform
2. target_platform
3. CONDA_BUILD_CROSS_COMPILATION - 1 if cross compiling
4. CMAKE_ARGS - arguments to pass to cmake
5. CC_FOR_BUILD - C compiler for build platform
6. CXX_FOR_BUILD - C++ compiler for build platform
7. HOST - a triplet for host passed to autoconf. eg: arm64-apple-darwin20.0.0
8. BUILD - a triplet for build passed to autoconf. eg: x86_64-conda-linux-gnu

Some useful configure options in conda-forge.yml

1. build_platform - a dictionary mapping build subdir to host subdir. eg:

   build_platform:
     osx_arm64: osx_64
     linux_ppc64le: linux_64
     linux_aarch64: linux_64

2. test_on_native_only - a boolean to turn off testing on cross compiling. If the tests don't require emulation (for eg: check that a file exists), then test_on_native_only: false will run the tests even when cross compiling.

Building locally

For building locally add the following in $HOME/conda_build_config.yaml.

SDKROOT:
  - /path/to/MacOSX11.0.sdk

After that, look for the config you want to run in .ci_support folder in the root of the feedstock For eg: .ci_support/osx_arm64_.yaml. Then run,

conda build recipe -m .ci_support/osx_arm64_.yaml -c conda-forge -c conda-forge/label/rust_dev

This should start a new build for osx-arm64.

Testing packages

In order to test packages intended to run on future Apple Silicon hardware, Apple provides a machine called Developer Transition Kit (DTK). Jonathan Helmus and Eli Rykoff has helped with testing these packages on DTKs. Thanks to Eli Rykoff, we are now running tests for these packages as a daily cron job which has led to finding several bugs in our cross compiling infrastructure and also bugs in our recipes.

To test cross compiled recipes, transfer the built conda package to the host and run,

conda build --test /path/to/package -c conda-forge

This work would not have been possible without the help of many people including the upstream maintainers of compiler infrastructure (which includes conda, conda-build, cctools, tapi, cctools-port, ldid, llvm, clang, compiler-rt, openmp, libcxx, crossenv, rust, gcc-darwin-arm64), conda-forge/help-osx-arm64 team including Matt Becker, Eli Rykoff and Uwe Korn who sent PRs to fix recipes, conda-forge/bot team and also all the conda-forge maintainers of the 100 feedstocks who reviewed and fixed PRs.

Isuru Fernando