| Message ID | 20231026125038.27985-1-william.vinnicombe@raspberrypi.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series |
|
| Related | show |
Could I request some reviews on this V2 of my python patch? It should now work with cross-compilation, as it still uses the old method for building the bindings when cross-compiling. Thanks, William On Thu, 26 Oct 2023 at 13:52, William Vinnicombe < william.vinnicombe@raspberrypi.com> wrote: > The existing meson.build file installs the bindings to an architecture > specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not > picked up by default python which only looks in the non architecture > specific libdir (eg /usr/local/lib/python3.11/). It also will always > build using the system python, rather than building using the same > python as meson is using. This prevents a user being able to build the > bindings for a different version of python, without changing their > system python to that version. > > Modify the build process to use the meson Python module to build the > python bindings targets, so it installs them to the correct directories > for python, and builds them for the version of python that meson is > running with. For cross-compiling, still use the previous method to > build the bindings, as the host machine version of python should be > used instead. > > Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> > --- > src/py/libcamera/meson.build | 60 +++++++++++++++++++++++++++--------- > 1 file changed, 45 insertions(+), 15 deletions(-) > > diff --git a/src/py/libcamera/meson.build b/src/py/libcamera/meson.build > index f58c7198..128793aa 100644 > --- a/src/py/libcamera/meson.build > +++ b/src/py/libcamera/meson.build > @@ -1,10 +1,25 @@ > # SPDX-License-Identifier: CC0-1.0 > > -py3_dep = dependency('python3', required : get_option('pycamera')) > - > -if not py3_dep.found() > - pycamera_enabled = false > - subdir_done() > +if meson.is_cross_build() > + py3_dep = dependency('python3', required : get_option('pycamera')) > + > + if not py3_dep.found() > + pycamera_enabled = false > + subdir_done() > + endif > +else > + py = import('python').find_installation('python3', required : > get_option('pycamera')) > + > + if not py.found() > + pycamera_enabled = false > + subdir_done() > + else > + py3_dep = py.dependency(required : get_option('pycamera')) > + if not py3_dep.found() > + pycamera_enabled = false > + subdir_done() > + endif > + endif > endif > > pybind11_dep = dependency('pybind11', required : get_option('pycamera')) > @@ -78,15 +93,24 @@ pycamera_args = [ > '-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT', > ] > > -destdir = get_option('libdir') / ('python' + py3_dep.version()) / > 'site-packages' / 'libcamera' > - > -pycamera = shared_module('_libcamera', > - pycamera_sources, > - install : true, > - install_dir : destdir, > - name_prefix : '', > - dependencies : pycamera_deps, > - cpp_args : pycamera_args) > +if meson.is_cross_build() > + destdir = get_option('libdir') / ('python' + py3_dep.version()) / > 'site-packages' / 'libcamera' > + > + pycamera = shared_module('_libcamera', > + pycamera_sources, > + install : true, > + install_dir : destdir, > + name_prefix : '', > + dependencies : pycamera_deps, > + cpp_args : pycamera_args) > +else > + pycamera = py.extension_module('_libcamera', > + pycamera_sources, > + install : true, > + subdir : 'libcamera', > + dependencies : pycamera_deps, > + cpp_args : pycamera_args) > +endif > > # Create symlinks from the build dir to the source dir so that we can use > the > # Python module directly from the build dir. > @@ -99,7 +123,13 @@ run_command('ln', '-fsrT', meson.current_source_dir() > / 'utils', > meson.current_build_dir() / 'utils', > check : true) > > -install_data(['__init__.py'], install_dir : destdir) > +if meson.is_cross_build() > + install_data(['__init__.py'], install_dir : destdir) > +else > + py.install_sources(['__init__.py'], > + subdir : 'libcamera', > + pure : false) > +endif > > # \todo Generate stubs when building. See > https://peps.python.org/pep-0484/#stub-files > # Note: Depends on pybind11-stubgen. To generate pylibcamera stubs: > -- > 2.39.2 > >
Hi, On 26/10/2023 15:50, William Vinnicombe via libcamera-devel wrote: > The existing meson.build file installs the bindings to an architecture > specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not For me, in buildroot, it would be "/usr/lib/python3.11/site-packages", so no arch part. On my PC, though, I get "/usr/lib/x86_64-linux-gnu/python3.10/site-packages/libcamera", which is not in the python search path. However, even if I did drop the arch part, it wouldn't be right for my PC as the python system search paths don't contain "site-packages", but "dist-packages": "/usr/lib/python3.10/dist-packages". Although there _is_ "site-packages" in the path, but for the user specific pip path: "/home/tomba/.local/lib/python3.10/site-packages"... > picked up by default python which only looks in the non architecture > specific libdir (eg /usr/local/lib/python3.11/). It also will always > build using the system python, rather than building using the same > python as meson is using. This prevents a user being able to build the Well... I don't know which one of those is better or worse =). And it's not system python (if that means the build system), it's the target python. I don't think it's a good idea to always use the same python as meson is using, and for cross-compiling that's totally wrong, of course. It should be perfectly fine for meson to always use the normal system python, while targeting some specific python version. In your version, shouldn't the import('python').find_installation() call be configurable, so that the user would tell meson where to look for the python installation? The docs say find_installation() can take e.g. a path to the python. > bindings for a different version of python, without changing their > system python to that version. How do you change the python version with your patch? > Modify the build process to use the meson Python module to build the > python bindings targets, so it installs them to the correct directories > for python, and builds them for the version of python that meson is > running with. For cross-compiling, still use the previous method to > build the bindings, as the host machine version of python should be > used instead. To clarify, if using your version (i.e. the "official" meson version) for finding python, the build goes totally bonkers as meson starts mixing up the build system's and target's libraries. I think it was less bonkers earlier, but now it looks to be totally broken (probably related to a meson bug Laurent found recently). In any case, this patch keeps the cross-compilation working for me, and a native build on my PC works too. So it's ugly, but, afaics, works. You could add some comments to the meson.build file, to explain why all this is being done. Tomi > Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> > --- > src/py/libcamera/meson.build | 60 +++++++++++++++++++++++++++--------- > 1 file changed, 45 insertions(+), 15 deletions(-) > > diff --git a/src/py/libcamera/meson.build b/src/py/libcamera/meson.build > index f58c7198..128793aa 100644 > --- a/src/py/libcamera/meson.build > +++ b/src/py/libcamera/meson.build > @@ -1,10 +1,25 @@ > # SPDX-License-Identifier: CC0-1.0 > > -py3_dep = dependency('python3', required : get_option('pycamera')) > - > -if not py3_dep.found() > - pycamera_enabled = false > - subdir_done() > +if meson.is_cross_build() > + py3_dep = dependency('python3', required : get_option('pycamera')) > + > + if not py3_dep.found() > + pycamera_enabled = false > + subdir_done() > + endif > +else > + py = import('python').find_installation('python3', required : get_option('pycamera')) > + > + if not py.found() > + pycamera_enabled = false > + subdir_done() > + else > + py3_dep = py.dependency(required : get_option('pycamera')) > + if not py3_dep.found() > + pycamera_enabled = false > + subdir_done() > + endif > + endif > endif > > pybind11_dep = dependency('pybind11', required : get_option('pycamera')) > @@ -78,15 +93,24 @@ pycamera_args = [ > '-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT', > ] > > -destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' > - > -pycamera = shared_module('_libcamera', > - pycamera_sources, > - install : true, > - install_dir : destdir, > - name_prefix : '', > - dependencies : pycamera_deps, > - cpp_args : pycamera_args) > +if meson.is_cross_build() > + destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' > + > + pycamera = shared_module('_libcamera', > + pycamera_sources, > + install : true, > + install_dir : destdir, > + name_prefix : '', > + dependencies : pycamera_deps, > + cpp_args : pycamera_args) > +else > + pycamera = py.extension_module('_libcamera', > + pycamera_sources, > + install : true, > + subdir : 'libcamera', > + dependencies : pycamera_deps, > + cpp_args : pycamera_args) > +endif > > # Create symlinks from the build dir to the source dir so that we can use the > # Python module directly from the build dir. > @@ -99,7 +123,13 @@ run_command('ln', '-fsrT', meson.current_source_dir() / 'utils', > meson.current_build_dir() / 'utils', > check : true) > > -install_data(['__init__.py'], install_dir : destdir) > +if meson.is_cross_build() > + install_data(['__init__.py'], install_dir : destdir) > +else > + py.install_sources(['__init__.py'], > + subdir : 'libcamera', > + pure : false) > +endif > > # \todo Generate stubs when building. See https://peps.python.org/pep-0484/#stub-files > # Note: Depends on pybind11-stubgen. To generate pylibcamera stubs:
Hello, Reviving this old thread after a weekend of rabbit hole exploration. On Thu, Nov 23, 2023 at 05:26:05PM +0200, 📷-dev wrote: > On 26/10/2023 15:50, William Vinnicombe via libcamera-devel wrote: > > The existing meson.build file installs the bindings to an architecture > > specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not > > For me, in buildroot, it would be "/usr/lib/python3.11/site-packages", > so no arch part. On my PC, though, I get > "/usr/lib/x86_64-linux-gnu/python3.10/site-packages/libcamera", which is > not in the python search path. However, even if I did drop the arch > part, it wouldn't be right for my PC as the python system search paths > don't contain "site-packages", but "dist-packages": > "/usr/lib/python3.10/dist-packages". Although there _is_ "site-packages" > in the path, but for the user specific pip path: > "/home/tomba/.local/lib/python3.10/site-packages"... "dist-packages" is Debian-specific. See [1] and [2]. [1] https://wiki.debian.org/Python#Deviations_from_upstream [2] https://stackoverflow.com/questions/9387928/whats-the-difference-between-dist-packages-and-site-packages The idea is to support coexistence of a Python interpreter installed from Debian packages, and the same version of the interpreter installed from source. The former would be installed in /usr and the latter in /usr/local, but they will both look in /usr/local/lib/python3.x/ to find site packages installed from source. The Debian interpreter looks in the custom dist-packages directory, and the one compiled from sources in the standard site-packages directory. That way, packages compiled for one of the interpreters won't conflict with packages compiled for the other. It seems a bit of a corner case. Another data point: on my Gentoo machine, libcamera currently installs Python packages in /usr/lib64/python3.13/site-packages, while Python looks for them in /usr/lib/python3.13/site-packages. > > picked up by default python which only looks in the non architecture > > specific libdir (eg /usr/local/lib/python3.11/). It also will always > > build using the system python, rather than building using the same > > python as meson is using. This prevents a user being able to build the > > Well... I don't know which one of those is better or worse =). And it's > not system python (if that means the build system), it's the target python. > > I don't think it's a good idea to always use the same python as meson is > using, and for cross-compiling that's totally wrong, of course. It > should be perfectly fine for meson to always use the normal system > python, while targeting some specific python version. > > In your version, shouldn't the import('python').find_installation() call > be configurable, so that the user would tell meson where to look for the > python installation? The docs say find_installation() can take e.g. a > path to the python. > > > bindings for a different version of python, without changing their > > system python to that version. > > How do you change the python version with your patch? This can be done by specifying the Python interpreter in a meson native file (the equivalent of a cross file for native builds). > > Modify the build process to use the meson Python module to build the > > python bindings targets, so it installs them to the correct directories > > for python, and builds them for the version of python that meson is > > running with. For cross-compiling, still use the previous method to > > build the bindings, as the host machine version of python should be > > used instead. > > To clarify, if using your version (i.e. the "official" meson version) > for finding python, the build goes totally bonkers as meson starts > mixing up the build system's and target's libraries. I think it was less > bonkers earlier, but now it looks to be totally broken (probably related > to a meson bug Laurent found recently). It wasn't entirely clear to me if the meson python module is meant to support cross-compilation. I have asked on the #mesonbuild IRC channel, and the information I have received from meson developers is that the python module, and the python_installation object returned by find_installation(), are meant to support compilation of Python extension modules, including both native and cross-compilation. The meson python module, on the other hand, is *not* meant to access the Python interpreter of the build machine, even if the find_installation() method can return an object that refers to that interpreter. The reason is that the build machine Python interpreter, as run for instance from the libcamera code generation scripts ("#!/usr/bin/env python3") does not necessarily match the Python interpreter for which the libcamera Python bindings extension module is compiled, even in the case of native compilation. Using find_installation() to check, for instance, the version of a Python module used by Sphinx to build the documentation, is wrong. I have submitted two patches to remove incorrect usage of the meson python module in libcamera. In the text below I'll use "build machine" and "host machine" in the meson sense, which translate to "host machine" and "target machine" for buildroot. The python_installation dependency() and extension_module() methods are interesting. In a cross-compilation environment, dependency() tries to find the Python libraries for the host machine, but searches for the exact same Python version as the Python interpreter returned by find_installation(). By default, find_installation() on my machine finds the build machine's native interpreter (v3.13), which doesn't match the Python version from my buildroot cross-compilation environment (v3.12). Meson fails to find the dependency(). I think the expected way to handle this is for find_installation() to return the *build* machine interpreter from the *cross-compilation* environment, which will likely be the same version as the *host* machine interpreter in the same cross-compilation environment. The meson documentation is very unclear about this though. Going down this rabbit hole deeper, I've set [binaries] python = br_host_dir / 'bin/python3' in my cross file. Meson picked the build machine Python installation from buildroot (in the buildroot "host" directory). This caused issues when building the libcamera documentation, as the sphinxcontrib.doxylink module wasn't installed in my buildroot environment, which the patches I mentioened above fixed. With that, py.dependency() found the Python dependency for the host, and py.extension_module() built the module correctly. py.install_sources() also behaved nearly as expected. Nearly :-( There are two issue I've run into. One of them is specific to cross-compilation, the other one isn't. The first issue is related to py.extension_module(). The method selects the appropriate file extension for the module, based on the Python interpreter configuration. Unfortunately, querying that configuration requires running Python code. With the cross file pointing to a Python interpreter that has been compiled for the build machine architecture, the result of the query will be the suffix expected on the build machine, not on the host machine. Cross-compiling from x86_64 to aarch64 on Gentoo, meson produces an aarch64 binary named _libcamera.cpython-313-x86_64-linux-gnu.so. This issue affects cross-compilation only. One workaround is to reference in the cross file a Python interpreter binary compiled for the host machine and wrapper in qemu. Given how this would impact all the libcamera users who cross-compile the project, I don't think it's a good idea. The second issue is related to install paths. Those are installation-dependent and distribution-dependent, and here too querying them involves running Python code (if anyone is curious, the information comes from the Python sysconfig module, while population sys.path at runtime is done in the site module). Even for native compilation, distribution-specific "interesting" behaviours (that some may call bugs) makes this complex. Meson tries to wrap all the dirty code that queries for those paths in a python_info.py helper script, to keep the rest of meson clean. It seems to work when installing libcamera to a /usr prefix, but breaks for instance on Debian when installing to /usr/local. I've tested usage of py.extension_module() and py.install_sources() with native compilation and the prefix set to /usr in the following environments: - Debian 11 (meson 1.2.0 from pip) Installing src/py/libcamera/_libcamera.cpython-39-x86_64-linux-gnu.so to /usr/lib/python3/dist-packages/libcamera Installing src/py/libcamera/__init__.py to /usr/lib/python3/dist-packages/libcamera - Debian 12 (meson 1.0.1) Installing src/py/libcamera/_libcamera.cpython-311-x86_64-linux-gnu.so to /usr/lib/python3/dist-packages/libcamera Installing src/py/libcamera/__init__.py to /usr/lib/python3/dist-packages/libcamera - Debian 13 (meson 1.7.0) Installing src/py/libcamera/_libcamera.cpython-313-x86_64-linux-gnu.so to /usr/lib/python3/dist-packages/libcamera Installing src/py/libcamera/__init__.py to /usr/lib/python3/dist-packages/libcamera - Gentoo (meson 1.7.2) Installing src/py/libcamera/_libcamera.cpython-313-x86_64-linux-gnu.so to /usr/lib/python3.13/site-packages/libcamera Installing src/py/libcamera/__init__.py to /usr/lib/python3.13/site-packages/libcamera The libcamera module can be imported successfully by the Python interpreter. This is a clear improvement over the currently situation. Setting the prefix to /usr/local gives "interesting" results on Debian. The files are installed in /usr/local/lib/python3/dist-packages/ instead of /usr/lib/python3/dist-packages/, but while the Debian Python interpreter is configured to look for system site packages in /usr/lib/python3/dist-packages/, it searches the local site packages in /usr/local/lib/python3.x/dist-packages/ (with x being the minor version). I don't know why this is the case, but it prevents Python from finding the libcamera module. This is not worse than the current behaviour though, which is also broken. I believe that using the meson python module (py.extension_module() and py.install_sources()) for native compilation, as proposed in this patch, is a good solution. It should fix the installation path issue when the prefix is set to /usr, and won't cause a regression when the prefix is set to /usr/local. Furthermore, the module introduces two meson configuration options (python.platlibdir and python.purelibdir) that allow overriding the installation path, so this will give us a workaround for installation in /usr/local, improving on the current situation. For cross-compilation, I think we should also switch to using the python module to get the dependency, as well as the installation path. This will require pointing the cross file to the build machine Python interpreter from the cross-compilation environment in some cases (manual compilation of libcamera against a Buildroot or Yocto environment, when the system Python version differs from the one used by Buildroot or Yocto), but will provide the ability to fix incorrect installation paths through a meson configuration option. PEP 739 [3], and the work in progress [4] in meson to implement support for it, will improve the situation and should fix the suffix issue in the cross compilation case. Installation paths will require more work, especially on Debian that uses very unconventional conventions, for both native and cross compilation. It should be possible to work around that issue in meson itself without PEP 739 or any additional PEP. [3] https://peps.python.org/pep-0739/ [4] https://github.com/mesonbuild/meson/pull/14657 > In any case, this patch keeps the cross-compilation working for me, and > a native build on my PC works too. So it's ugly, but, afaics, works. > > You could add some comments to the meson.build file, to explain why all > this is being done. > > > Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> > > --- > > src/py/libcamera/meson.build | 60 +++++++++++++++++++++++++++--------- > > 1 file changed, 45 insertions(+), 15 deletions(-) > > > > diff --git a/src/py/libcamera/meson.build b/src/py/libcamera/meson.build > > index f58c7198..128793aa 100644 > > --- a/src/py/libcamera/meson.build > > +++ b/src/py/libcamera/meson.build > > @@ -1,10 +1,25 @@ > > # SPDX-License-Identifier: CC0-1.0 > > > > -py3_dep = dependency('python3', required : get_option('pycamera')) > > - > > -if not py3_dep.found() > > - pycamera_enabled = false > > - subdir_done() > > +if meson.is_cross_build() > > + py3_dep = dependency('python3', required : get_option('pycamera')) > > + > > + if not py3_dep.found() > > + pycamera_enabled = false > > + subdir_done() > > + endif > > +else > > + py = import('python').find_installation('python3', required : get_option('pycamera')) > > + > > + if not py.found() > > + pycamera_enabled = false > > + subdir_done() > > + else > > + py3_dep = py.dependency(required : get_option('pycamera')) > > + if not py3_dep.found() > > + pycamera_enabled = false > > + subdir_done() > > + endif > > + endif > > endif This can be simplified to if meson.is_cross_build() py3_dep = dependency('python3', required : get_option('pycamera')) else py3 = import('python').find_installation('python3', required : get_option('pycamera')) if not py3.found() pycamera_enabled = false subdir_done() endif py3_dep = py.dependency(required : get_option('pycamera')) endif if not py3_dep.found() pycamera_enabled = false subdir_done() endif > > > > pybind11_dep = dependency('pybind11', required : get_option('pycamera')) > > @@ -78,15 +93,24 @@ pycamera_args = [ > > '-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT', > > ] > > > > -destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' > > - > > -pycamera = shared_module('_libcamera', > > - pycamera_sources, > > - install : true, > > - install_dir : destdir, > > - name_prefix : '', > > - dependencies : pycamera_deps, > > - cpp_args : pycamera_args) > > +if meson.is_cross_build() > > + destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' > > + > > + pycamera = shared_module('_libcamera', > > + pycamera_sources, > > + install : true, > > + install_dir : destdir, > > + name_prefix : '', > > + dependencies : pycamera_deps, > > + cpp_args : pycamera_args) > > +else > > + pycamera = py.extension_module('_libcamera', > > + pycamera_sources, > > + install : true, > > + subdir : 'libcamera', > > + dependencies : pycamera_deps, > > + cpp_args : pycamera_args) > > +endif > > > > # Create symlinks from the build dir to the source dir so that we can use the > > # Python module directly from the build dir. > > @@ -99,7 +123,13 @@ run_command('ln', '-fsrT', meson.current_source_dir() / 'utils', > > meson.current_build_dir() / 'utils', > > check : true) > > > > -install_data(['__init__.py'], install_dir : destdir) > > +if meson.is_cross_build() > > + install_data(['__init__.py'], install_dir : destdir) > > +else > > + py.install_sources(['__init__.py'], > > + subdir : 'libcamera', > > + pure : false) > > +endif You can merge this with the code above that also test for is_cross_build(). I'll submit a v3 of this patch. > > > > # \todo Generate stubs when building. See https://peps.python.org/pep-0484/#stub-files > > # Note: Depends on pybind11-stubgen. To generate pylibcamera stubs:
Hi, On 19/08/2025 04:10, Laurent Pinchart wrote: > Hello, > > Reviving this old thread after a weekend of rabbit hole exploration. > > On Thu, Nov 23, 2023 at 05:26:05PM +0200, 📷-dev wrote: >> On 26/10/2023 15:50, William Vinnicombe via libcamera-devel wrote: >>> The existing meson.build file installs the bindings to an architecture >>> specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not >> >> For me, in buildroot, it would be "/usr/lib/python3.11/site-packages", >> so no arch part. On my PC, though, I get >> "/usr/lib/x86_64-linux-gnu/python3.10/site-packages/libcamera", which is >> not in the python search path. However, even if I did drop the arch >> part, it wouldn't be right for my PC as the python system search paths >> don't contain "site-packages", but "dist-packages": >> "/usr/lib/python3.10/dist-packages". Although there _is_ "site-packages" >> in the path, but for the user specific pip path: >> "/home/tomba/.local/lib/python3.10/site-packages"... > > "dist-packages" is Debian-specific. See [1] and [2]. > > [1] https://wiki.debian.org/Python#Deviations_from_upstream > [2] https://stackoverflow.com/questions/9387928/whats-the-difference-between-dist-packages-and-site-packages > > The idea is to support coexistence of a Python interpreter installed > from Debian packages, and the same version of the interpreter installed > from source. The former would be installed in /usr and the latter in > /usr/local, but they will both look in /usr/local/lib/python3.x/ to find > site packages installed from source. The Debian interpreter looks in the > custom dist-packages directory, and the one compiled from sources in the > standard site-packages directory. That way, packages compiled for one of > the interpreters won't conflict with packages compiled for the other. It > seems a bit of a corner case. > > Another data point: on my Gentoo machine, libcamera currently installs > Python packages in /usr/lib64/python3.13/site-packages, while Python > looks for them in /usr/lib/python3.13/site-packages. > >>> picked up by default python which only looks in the non architecture >>> specific libdir (eg /usr/local/lib/python3.11/). It also will always >>> build using the system python, rather than building using the same >>> python as meson is using. This prevents a user being able to build the >> >> Well... I don't know which one of those is better or worse =). And it's >> not system python (if that means the build system), it's the target python. >> >> I don't think it's a good idea to always use the same python as meson is >> using, and for cross-compiling that's totally wrong, of course. It >> should be perfectly fine for meson to always use the normal system >> python, while targeting some specific python version. >> >> In your version, shouldn't the import('python').find_installation() call >> be configurable, so that the user would tell meson where to look for the >> python installation? The docs say find_installation() can take e.g. a >> path to the python. >> >>> bindings for a different version of python, without changing their >>> system python to that version. >> >> How do you change the python version with your patch? > > This can be done by specifying the Python interpreter in a meson native > file (the equivalent of a cross file for native builds). > >>> Modify the build process to use the meson Python module to build the >>> python bindings targets, so it installs them to the correct directories >>> for python, and builds them for the version of python that meson is >>> running with. For cross-compiling, still use the previous method to >>> build the bindings, as the host machine version of python should be >>> used instead. >> >> To clarify, if using your version (i.e. the "official" meson version) >> for finding python, the build goes totally bonkers as meson starts >> mixing up the build system's and target's libraries. I think it was less >> bonkers earlier, but now it looks to be totally broken (probably related >> to a meson bug Laurent found recently). > > It wasn't entirely clear to me if the meson python module is meant to > support cross-compilation. I have asked on the #mesonbuild IRC channel, > and the information I have received from meson developers is that the > python module, and the python_installation object returned by > find_installation(), are meant to support compilation of Python > extension modules, including both native and cross-compilation. > > The meson python module, on the other hand, is *not* meant to access the > Python interpreter of the build machine, even if the find_installation() > method can return an object that refers to that interpreter. The reason > is that the build machine Python interpreter, as run for instance from > the libcamera code generation scripts ("#!/usr/bin/env python3") does > not necessarily match the Python interpreter for which the libcamera > Python bindings extension module is compiled, even in the case of native > compilation. Using find_installation() to check, for instance, the > version of a Python module used by Sphinx to build the documentation, is > wrong. I have submitted two patches to remove incorrect usage of the > meson python module in libcamera. > > In the text below I'll use "build machine" and "host machine" in the > meson sense, which translate to "host machine" and "target machine" for > buildroot. > > The python_installation dependency() and extension_module() methods are > interesting. In a cross-compilation environment, dependency() tries to > find the Python libraries for the host machine, but searches for the > exact same Python version as the Python interpreter returned by > find_installation(). By default, find_installation() on my machine finds > the build machine's native interpreter (v3.13), which doesn't match the > Python version from my buildroot cross-compilation environment (v3.12). > Meson fails to find the dependency(). > > I think the expected way to handle this is for find_installation() to > return the *build* machine interpreter from the *cross-compilation* > environment, which will likely be the same version as the *host* machine > interpreter in the same cross-compilation environment. The meson > documentation is very unclear about this though. > > Going down this rabbit hole deeper, I've set Sigh, now I'm there too... I'm trying to collect my thoughts here, but I guess it's mostly reiterating what you've already covered in your emails. So, if my understanding about "py3 = import('python').find_installation('python3')" is right, based on what you've written and what I have been testing, py3 here is kind of a mess. It points to a build arch Python interpreter, but the dependencies you get from it are for the target machine. You can run the interpreter on the build machine, but if you build an extension module for it, you get something you can't run on the build machine. Well, it kind of matches what a compiler is, I guess. But I think this is the key: py3 should only be used when dealing with target. Never if you want to run python to, e.g. process some source code. This is what you fixed in the two patches you sent. And, as meson's python module is broken for cross-compiling, we can't use it as such for cross-compiling. You make that a bit better with the hacks in your 3 patch series. Tomi
On Wed, Aug 20, 2025 at 12:41:43PM +0300, Tomi Valkeinen wrote: > Hi, > > On 19/08/2025 04:10, Laurent Pinchart wrote: > > Hello, > > > > Reviving this old thread after a weekend of rabbit hole exploration. > > > > On Thu, Nov 23, 2023 at 05:26:05PM +0200, 📷-dev wrote: > >> On 26/10/2023 15:50, William Vinnicombe via libcamera-devel wrote: > >>> The existing meson.build file installs the bindings to an architecture > >>> specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not > >> > >> For me, in buildroot, it would be "/usr/lib/python3.11/site-packages", > >> so no arch part. On my PC, though, I get > >> "/usr/lib/x86_64-linux-gnu/python3.10/site-packages/libcamera", which is > >> not in the python search path. However, even if I did drop the arch > >> part, it wouldn't be right for my PC as the python system search paths > >> don't contain "site-packages", but "dist-packages": > >> "/usr/lib/python3.10/dist-packages". Although there _is_ "site-packages" > >> in the path, but for the user specific pip path: > >> "/home/tomba/.local/lib/python3.10/site-packages"... > > > > "dist-packages" is Debian-specific. See [1] and [2]. > > > > [1] https://wiki.debian.org/Python#Deviations_from_upstream > > [2] https://stackoverflow.com/questions/9387928/whats-the-difference-between-dist-packages-and-site-packages > > > > The idea is to support coexistence of a Python interpreter installed > > from Debian packages, and the same version of the interpreter installed > > from source. The former would be installed in /usr and the latter in > > /usr/local, but they will both look in /usr/local/lib/python3.x/ to find > > site packages installed from source. The Debian interpreter looks in the > > custom dist-packages directory, and the one compiled from sources in the > > standard site-packages directory. That way, packages compiled for one of > > the interpreters won't conflict with packages compiled for the other. It > > seems a bit of a corner case. > > > > Another data point: on my Gentoo machine, libcamera currently installs > > Python packages in /usr/lib64/python3.13/site-packages, while Python > > looks for them in /usr/lib/python3.13/site-packages. > > > >>> picked up by default python which only looks in the non architecture > >>> specific libdir (eg /usr/local/lib/python3.11/). It also will always > >>> build using the system python, rather than building using the same > >>> python as meson is using. This prevents a user being able to build the > >> > >> Well... I don't know which one of those is better or worse =). And it's > >> not system python (if that means the build system), it's the target python. > >> > >> I don't think it's a good idea to always use the same python as meson is > >> using, and for cross-compiling that's totally wrong, of course. It > >> should be perfectly fine for meson to always use the normal system > >> python, while targeting some specific python version. > >> > >> In your version, shouldn't the import('python').find_installation() call > >> be configurable, so that the user would tell meson where to look for the > >> python installation? The docs say find_installation() can take e.g. a > >> path to the python. > >> > >>> bindings for a different version of python, without changing their > >>> system python to that version. > >> > >> How do you change the python version with your patch? > > > > This can be done by specifying the Python interpreter in a meson native > > file (the equivalent of a cross file for native builds). > > > >>> Modify the build process to use the meson Python module to build the > >>> python bindings targets, so it installs them to the correct directories > >>> for python, and builds them for the version of python that meson is > >>> running with. For cross-compiling, still use the previous method to > >>> build the bindings, as the host machine version of python should be > >>> used instead. > >> > >> To clarify, if using your version (i.e. the "official" meson version) > >> for finding python, the build goes totally bonkers as meson starts > >> mixing up the build system's and target's libraries. I think it was less > >> bonkers earlier, but now it looks to be totally broken (probably related > >> to a meson bug Laurent found recently). > > > > It wasn't entirely clear to me if the meson python module is meant to > > support cross-compilation. I have asked on the #mesonbuild IRC channel, > > and the information I have received from meson developers is that the > > python module, and the python_installation object returned by > > find_installation(), are meant to support compilation of Python > > extension modules, including both native and cross-compilation. > > > > The meson python module, on the other hand, is *not* meant to access the > > Python interpreter of the build machine, even if the find_installation() > > method can return an object that refers to that interpreter. The reason > > is that the build machine Python interpreter, as run for instance from > > the libcamera code generation scripts ("#!/usr/bin/env python3") does > > not necessarily match the Python interpreter for which the libcamera > > Python bindings extension module is compiled, even in the case of native > > compilation. Using find_installation() to check, for instance, the > > version of a Python module used by Sphinx to build the documentation, is > > wrong. I have submitted two patches to remove incorrect usage of the > > meson python module in libcamera. > > > > In the text below I'll use "build machine" and "host machine" in the > > meson sense, which translate to "host machine" and "target machine" for > > buildroot. > > > > The python_installation dependency() and extension_module() methods are > > interesting. In a cross-compilation environment, dependency() tries to > > find the Python libraries for the host machine, but searches for the > > exact same Python version as the Python interpreter returned by > > find_installation(). By default, find_installation() on my machine finds > > the build machine's native interpreter (v3.13), which doesn't match the > > Python version from my buildroot cross-compilation environment (v3.12). > > Meson fails to find the dependency(). > > > > I think the expected way to handle this is for find_installation() to > > return the *build* machine interpreter from the *cross-compilation* > > environment, which will likely be the same version as the *host* machine > > interpreter in the same cross-compilation environment. The meson > > documentation is very unclear about this though. > > > > Going down this rabbit hole deeper, I've set > > Sigh, now I'm there too... I'm trying to collect my thoughts here, but I > guess it's mostly reiterating what you've already covered in your emails. > > So, if my understanding about "py3 = > import('python').find_installation('python3')" is right, based on what > you've written and what I have been testing, py3 here is kind of a mess. > It points to a build arch Python interpreter, but the dependencies you > get from it are for the target machine. You can run the interpreter on > the build machine, but if you build an extension module for it, you get > something you can't run on the build machine. > > Well, it kind of matches what a compiler is, I guess. But I think this > is the key: py3 should only be used when dealing with target. Never if > you want to run python to, e.g. process some source code. This is what > you fixed in the two patches you sent. > > And, as meson's python module is broken for cross-compiling, we can't > use it as such for cross-compiling. You make that a bit better with the > hacks in your 3 patch series. I'll take the "a bit better" as a big compliment in Python context :-) I think py.extension_module() is still the way to go for cross-compilation, but that will need to wait until PEP 739 support lands (and we'll have to continue using shared_module() until we stop supporting pre-PEP 739 versions of Python and meson). The patches I've sent go in that direction, so I'm relatively confident we're getting to a better state.
diff --git a/src/py/libcamera/meson.build b/src/py/libcamera/meson.build index f58c7198..128793aa 100644 --- a/src/py/libcamera/meson.build +++ b/src/py/libcamera/meson.build @@ -1,10 +1,25 @@ # SPDX-License-Identifier: CC0-1.0 -py3_dep = dependency('python3', required : get_option('pycamera')) - -if not py3_dep.found() - pycamera_enabled = false - subdir_done() +if meson.is_cross_build() + py3_dep = dependency('python3', required : get_option('pycamera')) + + if not py3_dep.found() + pycamera_enabled = false + subdir_done() + endif +else + py = import('python').find_installation('python3', required : get_option('pycamera')) + + if not py.found() + pycamera_enabled = false + subdir_done() + else + py3_dep = py.dependency(required : get_option('pycamera')) + if not py3_dep.found() + pycamera_enabled = false + subdir_done() + endif + endif endif pybind11_dep = dependency('pybind11', required : get_option('pycamera')) @@ -78,15 +93,24 @@ pycamera_args = [ '-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT', ] -destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' - -pycamera = shared_module('_libcamera', - pycamera_sources, - install : true, - install_dir : destdir, - name_prefix : '', - dependencies : pycamera_deps, - cpp_args : pycamera_args) +if meson.is_cross_build() + destdir = get_option('libdir') / ('python' + py3_dep.version()) / 'site-packages' / 'libcamera' + + pycamera = shared_module('_libcamera', + pycamera_sources, + install : true, + install_dir : destdir, + name_prefix : '', + dependencies : pycamera_deps, + cpp_args : pycamera_args) +else + pycamera = py.extension_module('_libcamera', + pycamera_sources, + install : true, + subdir : 'libcamera', + dependencies : pycamera_deps, + cpp_args : pycamera_args) +endif # Create symlinks from the build dir to the source dir so that we can use the # Python module directly from the build dir. @@ -99,7 +123,13 @@ run_command('ln', '-fsrT', meson.current_source_dir() / 'utils', meson.current_build_dir() / 'utils', check : true) -install_data(['__init__.py'], install_dir : destdir) +if meson.is_cross_build() + install_data(['__init__.py'], install_dir : destdir) +else + py.install_sources(['__init__.py'], + subdir : 'libcamera', + pure : false) +endif # \todo Generate stubs when building. See https://peps.python.org/pep-0484/#stub-files # Note: Depends on pybind11-stubgen. To generate pylibcamera stubs:
The existing meson.build file installs the bindings to an architecture specific libdir (eg /usr/local/lib/aarch64-linux-gnu/), which is not picked up by default python which only looks in the non architecture specific libdir (eg /usr/local/lib/python3.11/). It also will always build using the system python, rather than building using the same python as meson is using. This prevents a user being able to build the bindings for a different version of python, without changing their system python to that version. Modify the build process to use the meson Python module to build the python bindings targets, so it installs them to the correct directories for python, and builds them for the version of python that meson is running with. For cross-compiling, still use the previous method to build the bindings, as the host machine version of python should be used instead. Signed-off-by: William Vinnicombe <william.vinnicombe@raspberrypi.com> --- src/py/libcamera/meson.build | 60 +++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 15 deletions(-)