This document describes the current design of multibuild in Exherbo, focussing mainly on what exheres authors need to know. By “multibuild”, we mean the ability for a package to build and install multiple versions of itself in different configurations. Right now this is used for Lua, PHP, Python and Ruby packages, which allows building against different ABIs (e.g. one package for both python 2.7 and 3.6).
The following example is gst-python-1.12.2.exheres-0 at the time of writing, with some of the detail trimmed out. It will be used to illustrate various aspects of the design.
# Copyright 2017 Rasmus Thomsen <Rasmus.thomsen@protonmail.com>
# Distributed under the terms of the GNU General Public License v2
# Based in part upon 'gst-python-0.10.22-r2.exheres-0', which is:
# Copyright 2009 Sterling X. Winter <sterling.winter@gmail.com>
require python [ blacklist=none ]
# LICENSE,HOMEPAGE etc.
PLATFORMS="~amd64 ~x86"
MYOPTIONS=""
DEPENDENCIES="
build:
virtual/pkg-config
build+run:
dev-libs/libxml2:2.0
gnome-bindings/pygobject:3[python_abis:*(-)?]
media-libs/gstreamer:1.0[>=1.12.0][gobject-introspection]
media-plugins/gst-plugins-base:1.0[>=1.12.0][gobject-introspection]
"
configure_one_multibuild() {
# Required for it to not install into /usr/lib/pythonx.x
econf \
--with-pygi-overrides-dir=$(python_get_sitedir)/gi/overrides/
}As described above, the multibuild infractructure is currently being used for Lua, PHP, Python and Ruby. This allows multiple axes of variation, such as building a package for multiple python versions (e.g. 2.7 and 3.6). Each of these axes is known as a “class”, and the possible configurations for each class are known as “targets”. In the example exheres above python.exlib is required and the exparam blacklist is set to none. This means that this package can be built for any Python ABI that currently is in the tree. By default python.exlib enables multibuilding, meaning that this package can be built for any combination of python_abis. One could build it only for Python 3.6, or for both 2.7 and 3.6 or for all available Python ABIs.
Users specify which configuration(s) they wish to build using the standard exheres options mechanism. Each multibuild class has its own suboption suffix, named as _abis preceded by the respective language it’s being used for in lower letters. In this example it’s python_abis, which has been defined by python.exlib.
If one python package depends on another one, then naturally the dependency needs to be built with at least all the variation used for the respective python package. This is expreessed in the dependencies using the [python_abis:*(-)?] syntax, described in Exheres for Smarties. The multibuild-specific elements are the * and the (-). The * is a shortcut meaning “all options under the specified suboption that are declared in this exheres”. In the example, since gst-python declares python_abis: 2.7 3.4 3.5 3.6 via python.exlib the dependency is equivalent to [python_abis:2.7(-)?][python_abis:3.4(-)?]
[python_abis:3.5(-)?] [python_abis:3.6(-)?]. The (-) means “do not consider the package matching the dependency if it does not declare at least one option under the specified suboption”, so in the example, if any version of pythongobject:3 didn’t have any python_abis options at all or not all variations requested for gst-python then it would not satisfy the dependency. Finally, the ? is used to mean that all targets enabled for this exheres must be enabled for the dependency, but targets disabled for this exheres can be either enabled or disabled.
easy-multibuild.exlibAs far as the exheres format is concerned, the package itself is responsible for building its code for each desired configuration – there is no magic applied by the package manager to do this, unless one requires an exlibs which does that. One such exlib is setup-py.exlib, which does most of the magic required for building python packages which use setup.py. setup-py.exlib requires python.exlib which in turn requires easy-multibuild.exlib. Since the build processes of most classes (languages) follow the same pattern of building easy-multibuild.exlib does most of the actual work required for multibuilding packages of various languages.
easy-multibuild.exlib takes an array exparam named classes to indicate which multibuild classes the package supports. Right now this can be lua,php,python or ruby, as described above. These packages usually simply require their language specific exlib in their exheres though, so this generally won’t be specified by anything but these exlibs.
easy-multibuild.exlib exports the src_configure, src_compile, src_test and src_install phase functions (and also src_unpack and src_prepare if the multiunpack exparam is specified as true – see below). Each of the exported phase functions calls the function ${EXHERES_PHASE}_one_multibuild if it is defined, or else default, once for each enabled target. (Note that ${EXHERES_PHASE} doesn’t include the src_ prefix, so the function names are configure_one_multibuild etc as in the example.) The default target for the class is run last – this is mainly important for the install phase, so that files not installed in a target-specific directory will be overwritten by the ones for the default. (This mainly applies to binaries – most other file types will either be the same for all targets anyway or should be installed in a target-specific location.) The example shows how to add custom code for each iteration of the configure phase. It’s still possible to define the src_* functions in exheres using multibuild, but the code inside these will only be called once. This can be useful for doing stuff that isn’t target specific, e.g. installing docs with emagicdocs.
Each call is made with the current directory set to a target-specific directory, namely ${WORKBASE}/${MULTIBUILD_CLASS}/${MULTIBUILD_TARGET}/ (with the latter two variables having the obvious values), so each target is kept separate. As of now easy-multibuild.exlib is almost exclusively used with multiunpack set to true. This means that the exlib will additionally export the src_unpack and src_prepare phases. Furthermore,the work exparam defaults to ${PNV} in this case. Using multiunpack will result in a complete copy of the work-tree to be unpacked and prepared for each multibuild target. If multiunpack is set to false the work-tree will be unpacked just once and ${WORKBASE}/${MULTIBUILD_CLASS}/${MULTIBUILD_TARGET}/ will only be used for the target specific object code. This was commonly used to multibuild C(++) code, but isn’t used anymore since native cross compiling has been introdued.
Each call is also made with various environment variables set which indicate how to build the software for the active configuration, defined by the profile. These are standard variables such as ${CHOST}, ${PYTHON}, etc, so for many packages using a conventional build system the exheres won’t need to do anything special to build the code for the correct target.
There can also be functions named ${EXHERES_PHASE}_prepare_one_multibuild – if these are defined they will be called after the environment variables have been set but before entering the target-specific directory. This mechanism is used to create the target-specific directory and to set ${ECONF_SOURCE} for the configure phase, so if you override configure_prepare_one_multibuild and still need this functionality you will need to replicate it in your function.
If multiple versions of a package’s code are being installed then they obviously need to be placed in different locations. For the python class, the files that vary between the targets are stored in /usr/${CTARGET}/lib/python${python_abi}. Other classes use similiar techniques to install multiple variations of the same package.
In the example $(python_get_sitedir), which is defined by python.exlib and returns /usr/${CTARGET}/lib/python${python_abi}, is called and passed to econf so that the target specific files are installed into the correct dir.
multibuild.exlibmultibuild.exlib is used to manage to target-specific variables defined in the profiles. In most cases it will be used via easy-multibuild.exlib (possibly with the multibuild_default_target function used explicitly as in the example), which in turn is mostly used via other exlibs. However, packages can use multibuild.exlib directly if they have requirements too complex for the latter.
(TODO: document the details here, I’m not too familiar with it and most readers won’t need it anyway.)
(TODO: document.)
Copyright 2011 David Leverton Copyright 2018 Rasmus Thomsen