Message ID | 20250818121103.20073-2-laurent.pinchart@ideasonboard.com |
---|---|
State | Accepted |
Commit | 0cadad434540c5a1b41def7290c8ef16e0d3708a |
Headers | show |
Series |
|
Related | show |
Hi, On 18/08/2025 15:11, Laurent Pinchart wrote: > The meson python module's find_installation() method conveniently allows > testing if the interpreter provides a set of modules. We use it to check > for the presence of the modules required at build time. Unfortunately, > the meson python module is not meant to access the native Python > interpreter of the build machine, but the Python environment of the host > machine. Usage of find_installation() for this purpose is incorrect, it > may find modules in a different interpreter than the one used to run the > build scripts that use those modules. Same comments here as for the previous patch. And I'm not sure if "native" helps here or just confuses... You can't use non-native Python anyway. Well, you could, with qemu. In any case, with the desc changes for both patches: Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> I also wonder if there should be variables for the pythons, "host_py" and "build_py", that could be used in the meson files. Tomi > Replace find_installation() with a manual check using python3 directly. > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > --- > meson.build | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) > > diff --git a/meson.build b/meson.build > index d82dbaeb99ce..5e015692a22c 100644 > --- a/meson.build > +++ b/meson.build > @@ -287,9 +287,14 @@ run_command('ln', '-fsT', meson.project_source_root(), meson.project_build_root( > > configure_file(output : 'config.h', configuration : config_h) > > -# Check for python installation and modules. > -py_mod = import('python') > -py_mod.find_installation('python3', modules : py_modules) > +# Check for python modules. > +foreach module : py_modules > + result = run_command('python3', '-c' , 'import @0@'.format(module), > + capture : false, check : false) > + if result.returncode() != 0 > + error('Python module \'@0@\' not found'.format(module)) > + endif > +endforeach > > ## Summarise Configurations > summary({
On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: > On 18/08/2025 15:11, Laurent Pinchart wrote: > > The meson python module's find_installation() method conveniently allows > > testing if the interpreter provides a set of modules. We use it to check > > for the presence of the modules required at build time. Unfortunately, > > the meson python module is not meant to access the native Python > > interpreter of the build machine, but the Python environment of the host > > machine. Usage of find_installation() for this purpose is incorrect, it > > may find modules in a different interpreter than the one used to run the > > build scripts that use those modules. > > Same comments here as for the previous patch. And I'm not sure if > "native" helps here or just confuses... I'll replace "native" with "system". I'll rewrite the commit message as ---------- The meson python module's find_installation() method conveniently allows testing if the interpreter provides a set of modules. We use it to check for the presence of the modules required at build time. Unfortunately, the meson python module is not meant to access the system Python interpreter of the build machine, but the Python environment of the host machine. Usage of find_installation() for this purpose is incorrect, it may find modules in a different interpreter than the one used to run the build scripts that use those modules. For instance, when cross-compiling libcamera against a Buildroot environment, and pointing meson in the cross-file to the build machine Python interpreter from Buildroot, the find_installation() method will refer to the latter, while our code generation scripts that use the modules will use the former. Replace find_installation() with a manual check using python3 directly. ---------- > You can't use non-native Python > anyway. Well, you could, with qemu. That's one workaround for the issues we're experiencing when cross-compiling, but I fear users would scream at me if I wanted to make that mandatory :-) > In any case, with the desc changes for both patches: > > Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> > > I also wonder if there should be variables for the pythons, "host_py" > and "build_py", that could be used in the meson files. What would you set those variables to ? Something like build_py = find_program('python3') host_py = import('python').find_installation('python3') ? I think they would both refer to the host interpreter. > > Replace find_installation() with a manual check using python3 directly. > > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> > > --- > > meson.build | 11 ++++++++--- > > 1 file changed, 8 insertions(+), 3 deletions(-) > > > > diff --git a/meson.build b/meson.build > > index d82dbaeb99ce..5e015692a22c 100644 > > --- a/meson.build > > +++ b/meson.build > > @@ -287,9 +287,14 @@ run_command('ln', '-fsT', meson.project_source_root(), meson.project_build_root( > > > > configure_file(output : 'config.h', configuration : config_h) > > > > -# Check for python installation and modules. > > -py_mod = import('python') > > -py_mod.find_installation('python3', modules : py_modules) > > +# Check for python modules. > > +foreach module : py_modules > > + result = run_command('python3', '-c' , 'import @0@'.format(module), > > + capture : false, check : false) > > + if result.returncode() != 0 > > + error('Python module \'@0@\' not found'.format(module)) > > + endif > > +endforeach > > > > ## Summarise Configurations > > summary({
Hi, On 19/08/2025 12:14, Laurent Pinchart wrote: > On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: >> On 18/08/2025 15:11, Laurent Pinchart wrote: >>> The meson python module's find_installation() method conveniently allows >>> testing if the interpreter provides a set of modules. We use it to check >>> for the presence of the modules required at build time. Unfortunately, >>> the meson python module is not meant to access the native Python >>> interpreter of the build machine, but the Python environment of the host >>> machine. Usage of find_installation() for this purpose is incorrect, it >>> may find modules in a different interpreter than the one used to run the >>> build scripts that use those modules. >> >> Same comments here as for the previous patch. And I'm not sure if >> "native" helps here or just confuses... > > I'll replace "native" with "system". I'll rewrite the commit message as > > ---------- > The meson python module's find_installation() method conveniently allows > testing if the interpreter provides a set of modules. We use it to check > for the presence of the modules required at build time. Unfortunately, > the meson python module is not meant to access the system Python > interpreter of the build machine, but the Python environment of the host > machine. > > Usage of find_installation() for this purpose is incorrect, it may find > modules in a different interpreter than the one used to run the build > scripts that use those modules. For instance, when cross-compiling > libcamera against a Buildroot environment, and pointing meson in the > cross-file to the build machine Python interpreter from Buildroot, the > find_installation() method will refer to the latter, while our code > generation scripts that use the modules will use the former. > > Replace find_installation() with a manual check using python3 directly. > ---------- > >> You can't use non-native Python >> anyway. Well, you could, with qemu. > > That's one workaround for the issues we're experiencing when > cross-compiling, but I fear users would scream at me if I wanted to make > that mandatory :-) > >> In any case, with the desc changes for both patches: >> >> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> >> >> I also wonder if there should be variables for the pythons, "host_py" >> and "build_py", that could be used in the meson files. > > What would you set those variables to ? Something like > > build_py = find_program('python3') > host_py = import('python').find_installation('python3') > > ? I think they would both refer to the host interpreter. I'm confused... Didn't this this series just do that change? Switching from "import('python').find_installation('python3')" to (essentially) "find_program('python3')". And the point was that they are different? Oh, wait, now I'm actually confused also on the meaning of the words. If I'm not mistaken, in autoconf terminology host and target are the same here. We can't run the target interpreter. What do you mean with "host"? What we want is to distinguish between the system/distro python, and the build environment (buildroot's dev environment) python, right? Tomi
Hi again, On 19/08/2025 15:54, Tomi Valkeinen wrote: > Hi, > > On 19/08/2025 12:14, Laurent Pinchart wrote: >> On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: >>> On 18/08/2025 15:11, Laurent Pinchart wrote: >>>> The meson python module's find_installation() method conveniently allows >>>> testing if the interpreter provides a set of modules. We use it to check >>>> for the presence of the modules required at build time. Unfortunately, >>>> the meson python module is not meant to access the native Python >>>> interpreter of the build machine, but the Python environment of the host >>>> machine. Usage of find_installation() for this purpose is incorrect, it >>>> may find modules in a different interpreter than the one used to run the >>>> build scripts that use those modules. >>> >>> Same comments here as for the previous patch. And I'm not sure if >>> "native" helps here or just confuses... >> >> I'll replace "native" with "system". I'll rewrite the commit message as >> >> ---------- >> The meson python module's find_installation() method conveniently allows >> testing if the interpreter provides a set of modules. We use it to check >> for the presence of the modules required at build time. Unfortunately, >> the meson python module is not meant to access the system Python >> interpreter of the build machine, but the Python environment of the host >> machine. >> >> Usage of find_installation() for this purpose is incorrect, it may find >> modules in a different interpreter than the one used to run the build >> scripts that use those modules. For instance, when cross-compiling >> libcamera against a Buildroot environment, and pointing meson in the >> cross-file to the build machine Python interpreter from Buildroot, the >> find_installation() method will refer to the latter, while our code >> generation scripts that use the modules will use the former. >> >> Replace find_installation() with a manual check using python3 directly. >> ---------- >> >>> You can't use non-native Python >>> anyway. Well, you could, with qemu. >> >> That's one workaround for the issues we're experiencing when >> cross-compiling, but I fear users would scream at me if I wanted to make >> that mandatory :-) >> >>> In any case, with the desc changes for both patches: >>> >>> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> >>> >>> I also wonder if there should be variables for the pythons, "host_py" >>> and "build_py", that could be used in the meson files. >> >> What would you set those variables to ? Something like >> >> build_py = find_program('python3') >> host_py = import('python').find_installation('python3') >> >> ? I think they would both refer to the host interpreter. > > I'm confused... Didn't this this series just do that change? Switching > from "import('python').find_installation('python3')" to (essentially) > "find_program('python3')". And the point was that they are different? > > Oh, wait, now I'm actually confused also on the meaning of the words. If > I'm not mistaken, in autoconf terminology host and target are the same > here. We can't run the target interpreter. What do you mean with "host"? Ok, saw your another email. So in your terms, "build" refers to the build environment (buildroot's "host"), and "host" means the target (buildroot's "target"). We also have the "system" (the distro, outside the build env). Is this right? But... no... In your example above, find_program('python3') returns the system python, but you assign it to build_py. And the other one is the build_py. So: I'm even more confused now =). Tomi
On Tue, Aug 19, 2025 at 03:54:16PM +0300, Tomi Valkeinen wrote: > On 19/08/2025 12:14, Laurent Pinchart wrote: > > On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: > >> On 18/08/2025 15:11, Laurent Pinchart wrote: > >>> The meson python module's find_installation() method conveniently allows > >>> testing if the interpreter provides a set of modules. We use it to check > >>> for the presence of the modules required at build time. Unfortunately, > >>> the meson python module is not meant to access the native Python > >>> interpreter of the build machine, but the Python environment of the host > >>> machine. Usage of find_installation() for this purpose is incorrect, it > >>> may find modules in a different interpreter than the one used to run the > >>> build scripts that use those modules. > >> > >> Same comments here as for the previous patch. And I'm not sure if > >> "native" helps here or just confuses... > > > > I'll replace "native" with "system". I'll rewrite the commit message as > > > > ---------- > > The meson python module's find_installation() method conveniently allows > > testing if the interpreter provides a set of modules. We use it to check > > for the presence of the modules required at build time. Unfortunately, > > the meson python module is not meant to access the system Python > > interpreter of the build machine, but the Python environment of the host > > machine. > > > > Usage of find_installation() for this purpose is incorrect, it may find > > modules in a different interpreter than the one used to run the build > > scripts that use those modules. For instance, when cross-compiling > > libcamera against a Buildroot environment, and pointing meson in the > > cross-file to the build machine Python interpreter from Buildroot, the > > find_installation() method will refer to the latter, while our code > > generation scripts that use the modules will use the former. > > > > Replace find_installation() with a manual check using python3 directly. > > ---------- > > > >> You can't use non-native Python > >> anyway. Well, you could, with qemu. > > > > That's one workaround for the issues we're experiencing when > > cross-compiling, but I fear users would scream at me if I wanted to make > > that mandatory :-) > > > >> In any case, with the desc changes for both patches: > >> > >> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> > >> > >> I also wonder if there should be variables for the pythons, "host_py" > >> and "build_py", that could be used in the meson files. > > > > What would you set those variables to ? Something like > > > > build_py = find_program('python3') > > host_py = import('python').find_installation('python3') > > > > ? I think they would both refer to the host interpreter. > > I'm confused... Didn't this this series just do that change? Switching > from "import('python').find_installation('python3')" to (essentially) > "find_program('python3')". And the point was that they are different? find_program('python3') will essentially look for a binary called python3 in the search paths used by meson. It also takes the cross-file into account, if you have [binaries] python3 = /path/to/other/python3 that's what build_py would give you. The same will happen if you use find_program('python') and replace the 'python3' key with 'python' in the cross-file. import('python').find_installation('python3') will do something similar. Without any override in the cross-file, you'll get the same system Python as find_program('python3'). You can override this through the 'python' entry in the cross-file. Note that this has to be 'python', not 'python3', a 'python3' key in the cross-file will be ignored. This differs from find_program(). You can get the system Python interpreter with find_program('python3', native : true) I suppose we could indeed do that in a central location py3_build = find_program('python3', native : true) py3_host = import('python').find_installation('python3') > Oh, wait, now I'm actually confused also on the meaning of the words. If > I'm not mistaken, in autoconf terminology host and target are the same > here. We can't run the target interpreter. What do you mean with "host"? In meson terminology, build is the machine on which you build the software, host is the machine on which it runs, and target is the machine for which it will generate output. The target is mostly used when building compilers, you can for instance crosscompile gcc on a x86 machine, creating a binary that will run on arm64 and will generate output to be run on a riscv. For libcamera we only care about build and host architectures, target is irrelevant. Note that what meson calls build and host is respectively called host and target in buildroot. > What we want is to distinguish between the system/distro python, and the > build environment (buildroot's dev environment) python, right?
Hi, On 19/08/2025 16:30, Laurent Pinchart wrote: > On Tue, Aug 19, 2025 at 03:54:16PM +0300, Tomi Valkeinen wrote: >> On 19/08/2025 12:14, Laurent Pinchart wrote: >>> On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: >>>> On 18/08/2025 15:11, Laurent Pinchart wrote: >>>>> The meson python module's find_installation() method conveniently allows >>>>> testing if the interpreter provides a set of modules. We use it to check >>>>> for the presence of the modules required at build time. Unfortunately, >>>>> the meson python module is not meant to access the native Python >>>>> interpreter of the build machine, but the Python environment of the host >>>>> machine. Usage of find_installation() for this purpose is incorrect, it >>>>> may find modules in a different interpreter than the one used to run the >>>>> build scripts that use those modules. >>>> >>>> Same comments here as for the previous patch. And I'm not sure if >>>> "native" helps here or just confuses... >>> >>> I'll replace "native" with "system". I'll rewrite the commit message as >>> >>> ---------- >>> The meson python module's find_installation() method conveniently allows >>> testing if the interpreter provides a set of modules. We use it to check >>> for the presence of the modules required at build time. Unfortunately, >>> the meson python module is not meant to access the system Python >>> interpreter of the build machine, but the Python environment of the host >>> machine. >>> >>> Usage of find_installation() for this purpose is incorrect, it may find >>> modules in a different interpreter than the one used to run the build >>> scripts that use those modules. For instance, when cross-compiling >>> libcamera against a Buildroot environment, and pointing meson in the >>> cross-file to the build machine Python interpreter from Buildroot, the >>> find_installation() method will refer to the latter, while our code >>> generation scripts that use the modules will use the former. >>> >>> Replace find_installation() with a manual check using python3 directly. >>> ---------- >>> >>>> You can't use non-native Python >>>> anyway. Well, you could, with qemu. >>> >>> That's one workaround for the issues we're experiencing when >>> cross-compiling, but I fear users would scream at me if I wanted to make >>> that mandatory :-) >>> >>>> In any case, with the desc changes for both patches: >>>> >>>> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> >>>> >>>> I also wonder if there should be variables for the pythons, "host_py" >>>> and "build_py", that could be used in the meson files. >>> >>> What would you set those variables to ? Something like >>> >>> build_py = find_program('python3') >>> host_py = import('python').find_installation('python3') >>> >>> ? I think they would both refer to the host interpreter. >> >> I'm confused... Didn't this this series just do that change? Switching >> from "import('python').find_installation('python3')" to (essentially) >> "find_program('python3')". And the point was that they are different? > > find_program('python3') will essentially look for a binary called > python3 in the search paths used by meson. It also takes the cross-file > into account, if you have > > [binaries] > python3 = /path/to/other/python3 > > that's what build_py would give you. The same will happen if you use > find_program('python') and replace the 'python3' key with 'python' in > the cross-file. > > import('python').find_installation('python3') will do something similar. > Without any override in the cross-file, you'll get the same system > Python as find_program('python3'). You can override this through the > 'python' entry in the cross-file. Note that this has to be 'python', not > 'python3', a 'python3' key in the cross-file will be ignored. This > differs from find_program(). > > You can get the system Python interpreter with > > find_program('python3', native : true) > > I suppose we could indeed do that in a central location > > py3_build = find_program('python3', native : true) > py3_host = import('python').find_installation('python3') After some rabbit-holeing, I think I now understand your patches a bit better. Those two (py3_build and py3_host) are very different things: "py3_build" is a python command for running python scripts while building, whereas "py3_host" is a target definition for python3 extension modules. So py3_host (I don't like "host" name =) should only really be visible in a few specific meson files that are creating py extensions for the target. And "py3_build" could be set in the root meson.build, so that other meson files don't need to find the python interpreter themselves (and maybe get it wrong by using find_program('python3'), which returns the wrong interpreter). Maybe the "py3_host", which is "py3" currently in the src/py/ files can stay as "py3", and perhaps "py3_build" could be "py_cmd"? I think I have a headache from all this... =) Tomi
On Wed, Aug 20, 2025 at 12:51:39PM +0300, Tomi Valkeinen wrote: > Hi, > > On 19/08/2025 16:30, Laurent Pinchart wrote: > > On Tue, Aug 19, 2025 at 03:54:16PM +0300, Tomi Valkeinen wrote: > >> On 19/08/2025 12:14, Laurent Pinchart wrote: > >>> On Tue, Aug 19, 2025 at 11:42:49AM +0300, Tomi Valkeinen wrote: > >>>> On 18/08/2025 15:11, Laurent Pinchart wrote: > >>>>> The meson python module's find_installation() method conveniently allows > >>>>> testing if the interpreter provides a set of modules. We use it to check > >>>>> for the presence of the modules required at build time. Unfortunately, > >>>>> the meson python module is not meant to access the native Python > >>>>> interpreter of the build machine, but the Python environment of the host > >>>>> machine. Usage of find_installation() for this purpose is incorrect, it > >>>>> may find modules in a different interpreter than the one used to run the > >>>>> build scripts that use those modules. > >>>> > >>>> Same comments here as for the previous patch. And I'm not sure if > >>>> "native" helps here or just confuses... > >>> > >>> I'll replace "native" with "system". I'll rewrite the commit message as > >>> > >>> ---------- > >>> The meson python module's find_installation() method conveniently allows > >>> testing if the interpreter provides a set of modules. We use it to check > >>> for the presence of the modules required at build time. Unfortunately, > >>> the meson python module is not meant to access the system Python > >>> interpreter of the build machine, but the Python environment of the host > >>> machine. > >>> > >>> Usage of find_installation() for this purpose is incorrect, it may find > >>> modules in a different interpreter than the one used to run the build > >>> scripts that use those modules. For instance, when cross-compiling > >>> libcamera against a Buildroot environment, and pointing meson in the > >>> cross-file to the build machine Python interpreter from Buildroot, the > >>> find_installation() method will refer to the latter, while our code > >>> generation scripts that use the modules will use the former. > >>> > >>> Replace find_installation() with a manual check using python3 directly. > >>> ---------- > >>> > >>>> You can't use non-native Python > >>>> anyway. Well, you could, with qemu. > >>> > >>> That's one workaround for the issues we're experiencing when > >>> cross-compiling, but I fear users would scream at me if I wanted to make > >>> that mandatory :-) > >>> > >>>> In any case, with the desc changes for both patches: > >>>> > >>>> Reviewed-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> > >>>> > >>>> I also wonder if there should be variables for the pythons, "host_py" > >>>> and "build_py", that could be used in the meson files. > >>> > >>> What would you set those variables to ? Something like > >>> > >>> build_py = find_program('python3') > >>> host_py = import('python').find_installation('python3') > >>> > >>> ? I think they would both refer to the host interpreter. > >> > >> I'm confused... Didn't this this series just do that change? Switching > >> from "import('python').find_installation('python3')" to (essentially) > >> "find_program('python3')". And the point was that they are different? > > > > find_program('python3') will essentially look for a binary called > > python3 in the search paths used by meson. It also takes the cross-file > > into account, if you have > > > > [binaries] > > python3 = /path/to/other/python3 > > > > that's what build_py would give you. The same will happen if you use > > find_program('python') and replace the 'python3' key with 'python' in > > the cross-file. > > > > import('python').find_installation('python3') will do something similar. > > Without any override in the cross-file, you'll get the same system > > Python as find_program('python3'). You can override this through the > > 'python' entry in the cross-file. Note that this has to be 'python', not > > 'python3', a 'python3' key in the cross-file will be ignored. This > > differs from find_program(). > > > > You can get the system Python interpreter with > > > > find_program('python3', native : true) > > > > I suppose we could indeed do that in a central location > > > > py3_build = find_program('python3', native : true) > > py3_host = import('python').find_installation('python3') > > After some rabbit-holeing, I think I now understand your patches a bit > better. > > Those two (py3_build and py3_host) are very different things: > "py3_build" is a python command for running python scripts while > building, whereas "py3_host" is a target definition for python3 > extension modules. > > So py3_host (I don't like "host" name =) should only really be visible > in a few specific meson files that are creating py extensions for the > target. And "py3_build" could be set in the root meson.build, so that > other meson files don't need to find the python interpreter themselves > (and maybe get it wrong by using find_program('python3'), which returns > the wrong interpreter). > > Maybe the "py3_host", which is "py3" currently in the src/py/ files can > stay as "py3", and perhaps "py3_build" could be "py_cmd"? Keeping "py3" is fine with me as it means I won't have to change my patches :-) As for introducing py_cmd, there are two occurences of run_command('python3', ...) at the moment. Would you want to replace that ? > I think I have a headache from all this... =) Welcome to the club :-)
Hi, On 20/08/2025 13:00, Laurent Pinchart wrote: > As for introducing py_cmd, there are two occurences of > run_command('python3', ...) at the moment. Would you want to replace > that ? Ah, right, we don't (and didn't) do find_program('python3'). I assume run_command('python3', ...) should always do the right thing, so I think we can forget the py_cmd. Tomi
diff --git a/meson.build b/meson.build index d82dbaeb99ce..5e015692a22c 100644 --- a/meson.build +++ b/meson.build @@ -287,9 +287,14 @@ run_command('ln', '-fsT', meson.project_source_root(), meson.project_build_root( configure_file(output : 'config.h', configuration : config_h) -# Check for python installation and modules. -py_mod = import('python') -py_mod.find_installation('python3', modules : py_modules) +# Check for python modules. +foreach module : py_modules + result = run_command('python3', '-c' , 'import @0@'.format(module), + capture : false, check : false) + if result.returncode() != 0 + error('Python module \'@0@\' not found'.format(module)) + endif +endforeach ## Summarise Configurations summary({
The meson python module's find_installation() method conveniently allows testing if the interpreter provides a set of modules. We use it to check for the presence of the modules required at build time. Unfortunately, the meson python module is not meant to access the native Python interpreter of the build machine, but the Python environment of the host machine. Usage of find_installation() for this purpose is incorrect, it may find modules in a different interpreter than the one used to run the build scripts that use those modules. Replace find_installation() with a manual check using python3 directly. Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> --- meson.build | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)