[libcamera-ci,RFC,v1,3/3] Add job to run lc-compliance on the virtual pipeline handler
diff mbox series

Message ID 20241212181655.112958-3-barnabas.pocze@ideasonboard.com
State Superseded
Headers show
Series
  • [RFC,v1,1/3] Enable `UDMABUF` in the kernel
Related show

Commit Message

Barnabás Pőcze Dec. 12, 2024, 6:16 p.m. UTC 
Add a new job "lc-compliance:virtual" that uses the build artifacts
produced by the "build-test:debug" job to run lc-compliance in a
virtual machine on the "Virtual0" camera.

The `force_fallback_for=gtest` option is needed because `cpp_debustl`
makes ABI incompatible changes in the STL containers, so googletest
also needs to be compiled with these changes.

Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
---
 .gitlab-ci/test-lc-compliance.sh | 27 +++++++++++++++++++++++++++
 gitlab-ci.yml                    | 18 ++++++++++++++++--
 2 files changed, 43 insertions(+), 2 deletions(-)
 create mode 100755 .gitlab-ci/test-lc-compliance.sh

--
2.47.1

Comments

Laurent Pinchart Dec. 15, 2024, 7:21 p.m. UTC | #1 
Hi Barnabás,

Thank you for the patch.

On Thu, Dec 12, 2024 at 07:16:55PM +0100, Barnabás Pőcze wrote:
> Add a new job "lc-compliance:virtual" that uses the build artifacts
> produced by the "build-test:debug" job to run lc-compliance in a
> virtual machine on the "Virtual0" camera.
> 
> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
> makes ABI incompatible changes in the STL containers, so googletest
> also needs to be compiled with these changes.
> 
> Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
> ---
>  .gitlab-ci/test-lc-compliance.sh | 27 +++++++++++++++++++++++++++
>  gitlab-ci.yml                    | 18 ++++++++++++++++--
>  2 files changed, 43 insertions(+), 2 deletions(-)
>  create mode 100755 .gitlab-ci/test-lc-compliance.sh
> 
> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
> new file mode 100755
> index 0000000..e6697a6
> --- /dev/null
> +++ b/.gitlab-ci/test-lc-compliance.sh
> @@ -0,0 +1,27 @@
> +#!/bin/bash

Please add an SPDX comment block, see other scripts for examples.

> +
> +set -e
> +
> +source "$(dirname "$0")/lib.sh"
> +
> +libcamera_compliance() {
> +	echo "Running libcamera compliance tests in a qemu VM"
> +
> +	virtme-ng \
> +		--verbose \
> +		--skip-modules \
> +		--force-9p \
> +		--rwdir "$PWD/build" \
> +		--run /opt/linux/bzImage \
> +		--exec "LIBCAMERA_LOG_LEVELS=*:DEBUG ./build/src/apps/lc-compliance/lc-compliance -c Virtual0; echo \\\$? > ./build/.test-status"
> +
> +	local status=$(cat build/.test-status)
> +	echo "Test result exit state: $status"
> +	rm build/.test-status
> +
> +	if [[ $status != 0 ]] ; then
> +		exit $status
> +	fi
> +}
> +
> +run libcamera_compliance
> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
> index c7448b8..5e9e309 100644
> --- a/gitlab-ci.yml
> +++ b/gitlab-ci.yml
> @@ -382,9 +382,10 @@ build-test:debug:
>        -D cam=disabled
>        -D cpp_debugstl=true
>        -D documentation=disabled
> +      -D force_fallback_for=gtest
>        -D gstreamer=enabled
> -      -D lc-compliance=disabled
> -      -D pipelines=[]
> +      -D lc-compliance=enabled
> +      -D pipelines=virtual

      -D pipelines=[virtual]

but I think that's not strictly needed, as the virtual pipeline handler
should be selected automatically when the test option is enabled.

>        -D pycamera=enabled
>        -D qcam=disabled
>        -D test=true
> @@ -420,3 +421,16 @@ test-unit:
>    # artifacts:
>    #   reports:
>    #     junit: build/meson-logs/testlog.junit.xml
> +
> +lc-compliance:virtual:
> +  extends:
> +    - .fdo.distribution-image@debian
> +    - .libcamera-ci.debian:12
> +    - .libcamera-ci.scripts
> +  stage: test
> +  needs:
> +    - job: build-test:debug
> +  tags:
> +    - kvm
> +  script:
> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh

I think gtest support generating junit reports. Can we do that, and
package the report in artifacts ?
Barnabás Pőcze Dec. 16, 2024, 10:15 a.m. UTC | #2 
Hi


2024. 12. 15. 20:21 keltezéssel, Laurent Pinchart írta:
> Hi Barnabás,
> 
> Thank you for the patch.
> 
> On Thu, Dec 12, 2024 at 07:16:55PM +0100, Barnabás Pőcze wrote:
>> Add a new job "lc-compliance:virtual" that uses the build artifacts
>> produced by the "build-test:debug" job to run lc-compliance in a
>> virtual machine on the "Virtual0" camera.
>>
>> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
>> makes ABI incompatible changes in the STL containers, so googletest
>> also needs to be compiled with these changes.
>>
>> Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
>> ---
>>   .gitlab-ci/test-lc-compliance.sh | 27 +++++++++++++++++++++++++++
>>   gitlab-ci.yml                    | 18 ++++++++++++++++--
>>   2 files changed, 43 insertions(+), 2 deletions(-)
>>   create mode 100755 .gitlab-ci/test-lc-compliance.sh
>>
>> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
>> new file mode 100755
>> index 0000000..e6697a6
>> --- /dev/null
>> +++ b/.gitlab-ci/test-lc-compliance.sh
>> @@ -0,0 +1,27 @@
>> +#!/bin/bash
> 
> Please add an SPDX comment block, see other scripts for examples.
> 
>> +
>> +set -e
>> +
>> +source "$(dirname "$0")/lib.sh"
>> +
>> +libcamera_compliance() {
>> +	echo "Running libcamera compliance tests in a qemu VM"
>> +
>> +	virtme-ng \
>> +		--verbose \
>> +		--skip-modules \
>> +		--force-9p \
>> +		--rwdir "$PWD/build" \
>> +		--run /opt/linux/bzImage \
>> +		--exec "LIBCAMERA_LOG_LEVELS=*:DEBUG ./build/src/apps/lc-compliance/lc-compliance -c Virtual0; echo \\\$? > ./build/.test-status"
>> +
>> +	local status=$(cat build/.test-status)
>> +	echo "Test result exit state: $status"
>> +	rm build/.test-status
>> +
>> +	if [[ $status != 0 ]] ; then
>> +		exit $status
>> +	fi
>> +}
>> +
>> +run libcamera_compliance
>> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
>> index c7448b8..5e9e309 100644
>> --- a/gitlab-ci.yml
>> +++ b/gitlab-ci.yml
>> @@ -382,9 +382,10 @@ build-test:debug:
>>         -D cam=disabled
>>         -D cpp_debugstl=true
>>         -D documentation=disabled
>> +      -D force_fallback_for=gtest
>>         -D gstreamer=enabled
>> -      -D lc-compliance=disabled
>> -      -D pipelines=[]
>> +      -D lc-compliance=enabled
>> +      -D pipelines=virtual
> 
>        -D pipelines=[virtual]

I am fairly certain `pipelines=virtual` should work, or is this
a coding style preference? As far as I see tings are not consistent
unfortunately, `build-libcamera-cros.sh` does not use the full
array notation, while the other occurrences do.


> 
> but I think that's not strictly needed, as the virtual pipeline handler
> should be selected automatically when the test option is enabled.

You're right. So should I remove it?


> 
>>         -D pycamera=enabled
>>         -D qcam=disabled
>>         -D test=true
>> @@ -420,3 +421,16 @@ test-unit:
>>     # artifacts:
>>     #   reports:
>>     #     junit: build/meson-logs/testlog.junit.xml
>> +
>> +lc-compliance:virtual:
>> +  extends:
>> +    - .fdo.distribution-image@debian
>> +    - .libcamera-ci.debian:12
>> +    - .libcamera-ci.scripts
>> +  stage: test
>> +  needs:
>> +    - job: build-test:debug
>> +  tags:
>> +    - kvm
>> +  script:
>> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
> 
> I think gtest support generating junit reports. Can we do that, and
> package the report in artifacts ?
> 

Indeed, it can:
https://gitlab.freedesktop.org/pobrn/libcamera/-/pipelines/1330595/test_report?job_name=lc-compliance%3Avirtual


Regards,
Barnabás Pőcze
Laurent Pinchart Dec. 16, 2024, 11:06 a.m. UTC | #3 
On Mon, Dec 16, 2024 at 11:15:33AM +0100, Barnabás Pőcze wrote:
> 2024. 12. 15. 20:21 keltezéssel, Laurent Pinchart írta:
> > On Thu, Dec 12, 2024 at 07:16:55PM +0100, Barnabás Pőcze wrote:
> >> Add a new job "lc-compliance:virtual" that uses the build artifacts
> >> produced by the "build-test:debug" job to run lc-compliance in a
> >> virtual machine on the "Virtual0" camera.
> >>
> >> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
> >> makes ABI incompatible changes in the STL containers, so googletest
> >> also needs to be compiled with these changes.
> >>
> >> Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
> >> ---
> >>   .gitlab-ci/test-lc-compliance.sh | 27 +++++++++++++++++++++++++++
> >>   gitlab-ci.yml                    | 18 ++++++++++++++++--
> >>   2 files changed, 43 insertions(+), 2 deletions(-)
> >>   create mode 100755 .gitlab-ci/test-lc-compliance.sh
> >>
> >> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
> >> new file mode 100755
> >> index 0000000..e6697a6
> >> --- /dev/null
> >> +++ b/.gitlab-ci/test-lc-compliance.sh
> >> @@ -0,0 +1,27 @@
> >> +#!/bin/bash
> > 
> > Please add an SPDX comment block, see other scripts for examples.
> > 
> >> +
> >> +set -e
> >> +
> >> +source "$(dirname "$0")/lib.sh"
> >> +
> >> +libcamera_compliance() {
> >> +	echo "Running libcamera compliance tests in a qemu VM"
> >> +
> >> +	virtme-ng \
> >> +		--verbose \
> >> +		--skip-modules \
> >> +		--force-9p \
> >> +		--rwdir "$PWD/build" \
> >> +		--run /opt/linux/bzImage \
> >> +		--exec "LIBCAMERA_LOG_LEVELS=*:DEBUG ./build/src/apps/lc-compliance/lc-compliance -c Virtual0; echo \\\$? > ./build/.test-status"
> >> +
> >> +	local status=$(cat build/.test-status)
> >> +	echo "Test result exit state: $status"
> >> +	rm build/.test-status
> >> +
> >> +	if [[ $status != 0 ]] ; then
> >> +		exit $status
> >> +	fi
> >> +}
> >> +
> >> +run libcamera_compliance
> >> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
> >> index c7448b8..5e9e309 100644
> >> --- a/gitlab-ci.yml
> >> +++ b/gitlab-ci.yml
> >> @@ -382,9 +382,10 @@ build-test:debug:
> >>         -D cam=disabled
> >>         -D cpp_debugstl=true
> >>         -D documentation=disabled
> >> +      -D force_fallback_for=gtest
> >>         -D gstreamer=enabled
> >> -      -D lc-compliance=disabled
> >> -      -D pipelines=[]
> >> +      -D lc-compliance=enabled
> >> +      -D pipelines=virtual
> > 
> >        -D pipelines=[virtual]
> 
> I am fairly certain `pipelines=virtual` should work, or is this
> a coding style preference?

It's a style preference, as the option is an array. I should have made
that clearer, sorry.

> As far as I see tings are not consistent
> unfortunately, `build-libcamera-cros.sh` does not use the full
> array notation, while the other occurrences do.

Maybe something to fix ? :-)

> > but I think that's not strictly needed, as the virtual pipeline handler
> > should be selected automatically when the test option is enabled.
> 
> You're right. So should I remove it?

I'm fine dropping it (or to be exact keeping "-D pipelines=[]").

> >>         -D pycamera=enabled
> >>         -D qcam=disabled
> >>         -D test=true
> >> @@ -420,3 +421,16 @@ test-unit:
> >>     # artifacts:
> >>     #   reports:
> >>     #     junit: build/meson-logs/testlog.junit.xml
> >> +
> >> +lc-compliance:virtual:
> >> +  extends:
> >> +    - .fdo.distribution-image@debian
> >> +    - .libcamera-ci.debian:12
> >> +    - .libcamera-ci.scripts
> >> +  stage: test
> >> +  needs:
> >> +    - job: build-test:debug
> >> +  tags:
> >> +    - kvm
> >> +  script:
> >> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
> > 
> > I think gtest support generating junit reports. Can we do that, and
> > package the report in artifacts ?
> 
> Indeed, it can:
> https://gitlab.freedesktop.org/pobrn/libcamera/-/pipelines/1330595/test_report?job_name=lc-compliance%3Avirtual
Kieran Bingham Dec. 16, 2024, 11:11 a.m. UTC | #4 
Quoting Laurent Pinchart (2024-12-16 11:06:10)
> On Mon, Dec 16, 2024 at 11:15:33AM +0100, Barnabás Pőcze wrote:
> > 2024. 12. 15. 20:21 keltezéssel, Laurent Pinchart írta:
> > > On Thu, Dec 12, 2024 at 07:16:55PM +0100, Barnabás Pőcze wrote:
> > >> Add a new job "lc-compliance:virtual" that uses the build artifacts
> > >> produced by the "build-test:debug" job to run lc-compliance in a
> > >> virtual machine on the "Virtual0" camera.
> > >>
> > >> The `force_fallback_for=gtest` option is needed because `cpp_debustl`
> > >> makes ABI incompatible changes in the STL containers, so googletest
> > >> also needs to be compiled with these changes.
> > >>
> > >> Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
> > >> ---
> > >>   .gitlab-ci/test-lc-compliance.sh | 27 +++++++++++++++++++++++++++
> > >>   gitlab-ci.yml                    | 18 ++++++++++++++++--
> > >>   2 files changed, 43 insertions(+), 2 deletions(-)
> > >>   create mode 100755 .gitlab-ci/test-lc-compliance.sh
> > >>
> > >> diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
> > >> new file mode 100755
> > >> index 0000000..e6697a6
> > >> --- /dev/null
> > >> +++ b/.gitlab-ci/test-lc-compliance.sh
> > >> @@ -0,0 +1,27 @@
> > >> +#!/bin/bash
> > > 
> > > Please add an SPDX comment block, see other scripts for examples.
> > > 
> > >> +
> > >> +set -e
> > >> +
> > >> +source "$(dirname "$0")/lib.sh"
> > >> +
> > >> +libcamera_compliance() {
> > >> +  echo "Running libcamera compliance tests in a qemu VM"
> > >> +
> > >> +  virtme-ng \
> > >> +          --verbose \
> > >> +          --skip-modules \
> > >> +          --force-9p \
> > >> +          --rwdir "$PWD/build" \
> > >> +          --run /opt/linux/bzImage \
> > >> +          --exec "LIBCAMERA_LOG_LEVELS=*:DEBUG ./build/src/apps/lc-compliance/lc-compliance -c Virtual0; echo \\\$? > ./build/.test-status"
> > >> +
> > >> +  local status=$(cat build/.test-status)
> > >> +  echo "Test result exit state: $status"
> > >> +  rm build/.test-status
> > >> +
> > >> +  if [[ $status != 0 ]] ; then
> > >> +          exit $status
> > >> +  fi
> > >> +}
> > >> +
> > >> +run libcamera_compliance
> > >> diff --git a/gitlab-ci.yml b/gitlab-ci.yml
> > >> index c7448b8..5e9e309 100644
> > >> --- a/gitlab-ci.yml
> > >> +++ b/gitlab-ci.yml
> > >> @@ -382,9 +382,10 @@ build-test:debug:
> > >>         -D cam=disabled
> > >>         -D cpp_debugstl=true
> > >>         -D documentation=disabled
> > >> +      -D force_fallback_for=gtest
> > >>         -D gstreamer=enabled
> > >> -      -D lc-compliance=disabled
> > >> -      -D pipelines=[]
> > >> +      -D lc-compliance=enabled
> > >> +      -D pipelines=virtual
> > > 
> > >        -D pipelines=[virtual]
> > 
> > I am fairly certain `pipelines=virtual` should work, or is this
> > a coding style preference?
> 
> It's a style preference, as the option is an array. I should have made
> that clearer, sorry.
> 
> > As far as I see tings are not consistent
> > unfortunately, `build-libcamera-cros.sh` does not use the full
> > array notation, while the other occurrences do.
> 
> Maybe something to fix ? :-)
> 
> > > but I think that's not strictly needed, as the virtual pipeline handler
> > > should be selected automatically when the test option is enabled.
> > 
> > You're right. So should I remove it?
> 
> I'm fine dropping it (or to be exact keeping "-D pipelines=[]").
> 
> > >>         -D pycamera=enabled
> > >>         -D qcam=disabled
> > >>         -D test=true
> > >> @@ -420,3 +421,16 @@ test-unit:
> > >>     # artifacts:
> > >>     #   reports:
> > >>     #     junit: build/meson-logs/testlog.junit.xml
> > >> +
> > >> +lc-compliance:virtual:
> > >> +  extends:
> > >> +    - .fdo.distribution-image@debian
> > >> +    - .libcamera-ci.debian:12
> > >> +    - .libcamera-ci.scripts
> > >> +  stage: test
> > >> +  needs:
> > >> +    - job: build-test:debug
> > >> +  tags:
> > >> +    - kvm
> > >> +  script:
> > >> +    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh
> > > 
> > > I think gtest support generating junit reports. Can we do that, and
> > > package the report in artifacts ?
> > 
> > Indeed, it can:
> > https://gitlab.freedesktop.org/pobrn/libcamera/-/pipelines/1330595/test_report?job_name=lc-compliance%3Avirtual

Oh - wow - Is there any way we can get the CTS results pulled into this
in the future too ?

--
Kieran

Patch
diff mbox series 

diff --git a/.gitlab-ci/test-lc-compliance.sh b/.gitlab-ci/test-lc-compliance.sh
new file mode 100755
index 0000000..e6697a6
--- /dev/null
+++ b/.gitlab-ci/test-lc-compliance.sh
@@ -0,0 +1,27 @@ 
+#!/bin/bash
+
+set -e
+
+source "$(dirname "$0")/lib.sh"
+
+libcamera_compliance() {
+	echo "Running libcamera compliance tests in a qemu VM"
+
+	virtme-ng \
+		--verbose \
+		--skip-modules \
+		--force-9p \
+		--rwdir "$PWD/build" \
+		--run /opt/linux/bzImage \
+		--exec "LIBCAMERA_LOG_LEVELS=*:DEBUG ./build/src/apps/lc-compliance/lc-compliance -c Virtual0; echo \\\$? > ./build/.test-status"
+
+	local status=$(cat build/.test-status)
+	echo "Test result exit state: $status"
+	rm build/.test-status
+
+	if [[ $status != 0 ]] ; then
+		exit $status
+	fi
+}
+
+run libcamera_compliance
diff --git a/gitlab-ci.yml b/gitlab-ci.yml
index c7448b8..5e9e309 100644
--- a/gitlab-ci.yml
+++ b/gitlab-ci.yml
@@ -382,9 +382,10 @@  build-test:debug:
       -D cam=disabled
       -D cpp_debugstl=true
       -D documentation=disabled
+      -D force_fallback_for=gtest
       -D gstreamer=enabled
-      -D lc-compliance=disabled
-      -D pipelines=[]
+      -D lc-compliance=enabled
+      -D pipelines=virtual
       -D pycamera=enabled
       -D qcam=disabled
       -D test=true
@@ -420,3 +421,16 @@  test-unit:
   # artifacts:
   #   reports:
   #     junit: build/meson-logs/testlog.junit.xml
+
+lc-compliance:virtual:
+  extends:
+    - .fdo.distribution-image@debian
+    - .libcamera-ci.debian:12
+    - .libcamera-ci.scripts
+  stage: test
+  needs:
+    - job: build-test:debug
+  tags:
+    - kvm
+  script:
+    - $CI_PROJECT_DIR/.gitlab-ci/test-lc-compliance.sh