[v2,2/4] virtual: Support more common pixel formats
diff mbox series

Message ID 20261009194007.23175-3-robert.mader@collabora.com
State Superseded
Headers show
Series
  • virtual: Support more common pixel formats
Related show

Commit Message

Robert Mader Oct. 9, 2026, 7:40 p.m. UTC 
While NV12 is the most common native format for hardware video encoders,
YUV420 is the equivalent for most software video encoders. Thus
supporting both is great for testing.

Similarly having some RGB formats with and without alpha can be very
useful, especially as the swISP is currently limited to such formats.

This is a polished version of a hack that was used to help debugging and
testing https://webrtc-review.googlesource.com/c/src/+/491680 - and so
helpful with that that I'd like to see it upstream.

Signed-off-by: Robert Mader <robert.mader@collabora.com>
---
 .../pipeline/virtual/frame_generator.h        |  3 +-
 .../virtual/image_frame_generator.cpp         | 55 ++++++++++++++++---
 .../pipeline/virtual/image_frame_generator.h  |  2 +-
 .../virtual/test_pattern_generator.cpp        | 46 ++++++++++++++--
 .../pipeline/virtual/test_pattern_generator.h |  2 +-
 src/libcamera/pipeline/virtual/virtual.cpp    | 41 ++++++++++----
 6 files changed, 123 insertions(+), 26 deletions(-)

Comments

Kieran Bingham Oct. 11, 2026, 5:44 p.m. UTC | #1 
Quoting Robert Mader (2026-10-09 20:40:05)
> While NV12 is the most common native format for hardware video encoders,
> YUV420 is the equivalent for most software video encoders. Thus
> supporting both is great for testing.
> 
> Similarly having some RGB formats with and without alpha can be very
> useful, especially as the swISP is currently limited to such formats.
> 
> This is a polished version of a hack that was used to help debugging and
> testing https://webrtc-review.googlesource.com/c/src/+/491680 - and so
> helpful with that that I'd like to see it upstream.

probably lives under the ---.

I'm a bit confused, and haven't fully understood the code paths below. I
see conversions from ARGB to NV12 and then NV12 to other formats.

What's the relationship between all these conversions ?

Is the TPG generating ARGB, then converting to NV12 to then convert to
ARGB when requested ?

> 
> Signed-off-by: Robert Mader <robert.mader@collabora.com>
> ---
>  .../pipeline/virtual/frame_generator.h        |  3 +-
>  .../virtual/image_frame_generator.cpp         | 55 ++++++++++++++++---
>  .../pipeline/virtual/image_frame_generator.h  |  2 +-
>  .../virtual/test_pattern_generator.cpp        | 46 ++++++++++++++--
>  .../pipeline/virtual/test_pattern_generator.h |  2 +-
>  src/libcamera/pipeline/virtual/virtual.cpp    | 41 ++++++++++----
>  6 files changed, 123 insertions(+), 26 deletions(-)
> 
> diff --git a/src/libcamera/pipeline/virtual/frame_generator.h b/src/libcamera/pipeline/virtual/frame_generator.h
> index a0658c45cd82..fd34a91a5e50 100644
> --- a/src/libcamera/pipeline/virtual/frame_generator.h
> +++ b/src/libcamera/pipeline/virtual/frame_generator.h
> @@ -9,6 +9,7 @@
>  
>  #include <libcamera/framebuffer.h>
>  #include <libcamera/geometry.h>
> +#include <libcamera/stream.h>
>  
>  namespace libcamera {
>  
> @@ -19,7 +20,7 @@ public:
>  
>         virtual void configure(const Size &size) = 0;
>  
> -       virtual int generateFrame(const Size &size,
> +       virtual int generateFrame(const StreamConfiguration &cfg,
>                                   const FrameBuffer *buffer) = 0;
>  
>  protected:
> diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.cpp b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
> index 1853bddeca27..5c56db96d8e2 100644
> --- a/src/libcamera/pipeline/virtual/image_frame_generator.cpp
> +++ b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
> @@ -14,7 +14,9 @@
>  
>  #include <libcamera/framebuffer.h>
>  
> +#include <libcamera/formats.h>
>  #include "libcamera/internal/dma_buf_allocator.h"
> +#include "libcamera/internal/formats.h"
>  #include "libcamera/internal/mapped_framebuffer.h"
>  
>  #include "libyuv/convert.h"
> @@ -124,7 +126,7 @@ void ImageFrameGenerator::configure(const Size &size)
>         }
>  }
>  
> -int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buffer)
> +int ImageFrameGenerator::generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer)
>  {

Passing a config seems fine.

>         ASSERT(!scaledFrameDatas_.empty());
>  
> @@ -141,16 +143,55 @@ int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
>                                             MappedFrameBuffer::MapFlag::Write);
>  
>         const auto &planes = mappedFrameBuffer.planes();
> +       auto size = cfg.size;
> +       const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>  
>         /* Loop only around the number of images available */
>         frameIndex_ %= imageFrameDatas_.size();
>  
>         /* Write the scaledY and scaledUV to the mapped frame buffer */
> -       libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(), size.width,
> -                        scaledFrameDatas_[frameIndex_].UV.get(), size.width,
> -                        planes[0].data(), size.width,
> -                        planes[1].data(), size.width,
> -                        size.width, size.height);
> +       int ret;
> +       switch (cfg.pixelFormat) {
> +       case formats::NV12:
> +               ret = libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(),
> +                                      size.width,
> +                                      scaledFrameDatas_[frameIndex_].UV.get(),
> +                                      size.width,
> +                                      planes[0].data(),
> +                                      info.stride(size.width, 0),
> +                                      planes[1].data(),
> +                                      info.stride(size.width, 1),
> +                                      size.width, size.height);
> +               break;
> +       case formats::YUV420:
> +               ret = libyuv::NV12ToI420(scaledFrameDatas_[frameIndex_].Y.get(),
> +                                        size.width,
> +                                        scaledFrameDatas_[frameIndex_].UV.get(),
> +                                        size.width,
> +                                        planes[0].data(),
> +                                        info.stride(size.width, 0),
> +                                        planes[1].data(),
> +                                        info.stride(size.width, 1),
> +                                        planes[2].data(),
> +                                        info.stride(size.width, 2),
> +                                        size.width, size.height);
> +               break;
> +       case formats::ABGR8888:
> +       case formats::XBGR8888:
> +               ret = libyuv::NV12ToARGB(scaledFrameDatas_[frameIndex_].Y.get(),
> +                                        size.width,
> +                                        scaledFrameDatas_[frameIndex_].UV.get(),
> +                                        size.width,
> +                                        planes[0].data(),
> +                                        info.stride(size.width, 0),
> +                                        size.width, size.height);
> +               break;
> +       default:
> +               LOG(Virtual, Error) << "Unsupported format";
> +               return -1;
> +       }
> +       if (ret != 0)
> +               LOG(Virtual, Error) << "Conversation failed with " << ret;
>  
>         /* Proceed to the next image every 4 frames */
>         /* \todo Consider setting the frameRepeat in the config file  */
> @@ -158,7 +199,7 @@ int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
>         if (parameter_ % frameRepeat == 0)
>                 frameIndex_++;
>  
> -       return 0;
> +       return ret;
>  }
>  
>  /*
> diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.h b/src/libcamera/pipeline/virtual/image_frame_generator.h
> index 42a077ba803c..19b6fe38654e 100644
> --- a/src/libcamera/pipeline/virtual/image_frame_generator.h
> +++ b/src/libcamera/pipeline/virtual/image_frame_generator.h
> @@ -37,7 +37,7 @@ private:
>         };
>  
>         void configure(const Size &size) override;
> -       int generateFrame(const Size &size, const FrameBuffer *buffer) override;
> +       int generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer) override;
>  
>         std::vector<ImageFrameData> imageFrameDatas_;
>         std::vector<ImageFrameData> scaledFrameDatas_;
> diff --git a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
> index 7b53e8b32327..4cde32176661 100644
> --- a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
> +++ b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
> @@ -11,7 +11,9 @@
>  
>  #include <libcamera/base/log.h>
>  
> +#include <libcamera/formats.h>
>  #include "libcamera/internal/dma_buf_allocator.h"
> +#include "libcamera/internal/formats.h"
>  #include "libcamera/internal/mapped_framebuffer.h"
>  
>  #include <libyuv/convert_from_argb.h>
> @@ -42,7 +44,7 @@ LOG_DECLARE_CATEGORY(Virtual)
>  
>  static const unsigned int kARGBSize = 4;
>  
> -int TestPatternGenerator::generateFrame(const Size &size,
> +int TestPatternGenerator::generateFrame(const StreamConfiguration &cfg,
>                                         const FrameBuffer *buffer)
>  {
>         /*
> @@ -58,16 +60,48 @@ int TestPatternGenerator::generateFrame(const Size &size,
>                                             MappedFrameBuffer::MapFlag::Write);
>  
>         const auto &planes = mappedFrameBuffer.planes();
> +       const auto size = cfg.size;
> +       const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>  
>         rotateLeft1Column<kARGBSize>(size, template_.get());
>  
>         /* Convert the template_ to the frame buffer */
> -       int ret = libyuv::ARGBToNV12(template_.get(), size.width * kARGBSize,
> -                                    planes[0].data(), size.width,
> -                                    planes[1].data(), size.width,
> -                                    size.width, size.height);
> +       int ret;
> +       switch (cfg.pixelFormat) {
> +       case formats::NV12:
> +               ret = libyuv::ARGBToNV12(template_.get(),
> +                                        size.width * kARGBSize,
> +                                        planes[0].data(),
> +                                        info.stride(size.width, 0),
> +                                        planes[1].data(),
> +                                        info.stride(size.width, 1),
> +                                        size.width, size.height);
> +               break;
> +       case formats::YUV420:
> +               ret = libyuv::ARGBToI420(template_.get(),
> +                                        size.width * kARGBSize,
> +                                        planes[0].data(),
> +                                        info.stride(size.width, 0),
> +                                        planes[1].data(),
> +                                        info.stride(size.width, 1),
> +                                        planes[2].data(),
> +                                        info.stride(size.width, 2),
> +                                        size.width, size.height);
> +               break;
> +       case formats::ABGR8888:
> +       case formats::XBGR8888:
> +               ret = libyuv::ARGBCopy(template_.get(),
> +                                      size.width * kARGBSize,
> +                                      planes[0].data(),
> +                                      info.stride(size.width, 0),
> +                                      size.width, size.height);
> +               break;
> +       default:
> +               LOG(Virtual, Error) << "Unsupported format";
> +               return -1;
> +       }
>         if (ret != 0)
> -               LOG(Virtual, Error) << "ARGBToNV12() failed with " << ret;
> +               LOG(Virtual, Error) << "Conversation failed with " << ret;
>  
>         return ret;
>  }
> diff --git a/src/libcamera/pipeline/virtual/test_pattern_generator.h b/src/libcamera/pipeline/virtual/test_pattern_generator.h
> index 2a51bd31abdf..9f72d652ab2c 100644
> --- a/src/libcamera/pipeline/virtual/test_pattern_generator.h
> +++ b/src/libcamera/pipeline/virtual/test_pattern_generator.h
> @@ -24,7 +24,7 @@ enum class TestPattern : char {
>  class TestPatternGenerator : public FrameGenerator
>  {
>  public:
> -       int generateFrame(const Size &size, const FrameBuffer *buffer) override;
> +       int generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer) override;
>  
>  protected:
>         /* Buffer of test pattern template */
> diff --git a/src/libcamera/pipeline/virtual/virtual.cpp b/src/libcamera/pipeline/virtual/virtual.cpp
> index e25dd85a7391..7ed2d0b7f596 100644
> --- a/src/libcamera/pipeline/virtual/virtual.cpp
> +++ b/src/libcamera/pipeline/virtual/virtual.cpp
> @@ -47,6 +47,13 @@ LOG_DEFINE_CATEGORY(Virtual)
>  
>  namespace {
>  
> +constexpr PixelFormat outputFormats[] = {
> +       formats::NV12,
> +       formats::YUV420,
> +       formats::ABGR8888,
> +       formats::XBGR8888,
> +};
> +
>  uint64_t currentTimestamp()
>  {
>         const auto now = std::chrono::steady_clock::now();
> @@ -148,7 +155,7 @@ void VirtualCameraData::processRequest(Request *request)
>                                 found = true;
>  
>                                 if (streamConfig.frameGenerator->generateFrame(
> -                                           stream->configuration().size, buffer))
> +                                           stream->configuration(), buffer))
>                                         fmd.status = FrameMetadata::Status::FrameError;
>  
>                                 bufferCompleted.emit(buffer);
> @@ -202,17 +209,31 @@ CameraConfiguration::Status VirtualCameraConfiguration::validate()
>                         adjusted = true;
>                 }
>  
> -               if (cfg.pixelFormat != formats::NV12) {
> -                       cfg.pixelFormat = formats::NV12;
> +               if (std::find(std::begin(outputFormats),
> +                             std::end(outputFormats),
> +                             cfg.pixelFormat) == std::end(outputFormats)) {
> +                       cfg.pixelFormat = outputFormats[0];
>                         status = Adjusted;
>                         adjusted = true;
>                 }
>  
> -               if (cfg.colorSpace != ColorSpace::Smpte170m) {
> -                       cfg.colorSpace = ColorSpace::Smpte170m;
> +               const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
> +               std::optional<ColorSpace> colorSpace;
> +               switch (info.colourEncoding) {
> +               case PixelFormatInfo::ColourEncodingRGB:
> +                       colorSpace = ColorSpace::Srgb;
> +                       break;
> +               case PixelFormatInfo::ColourEncodingYUV:
> +                       colorSpace = ColorSpace::Smpte170m;
> +                       break;
> +               default:
> +                       ASSERT(false);
> +               }
> +               if (cfg.colorSpace && cfg.colorSpace != colorSpace) {
>                         status = Adjusted;
>                         adjusted = true;
>                 }
> +               cfg.colorSpace = colorSpace;
>  
>                 if (validateColorSpaces() == Adjusted) {
>                         status = Adjusted;
> @@ -223,7 +244,6 @@ CameraConfiguration::Status VirtualCameraConfiguration::validate()
>                         LOG(Virtual, Info)
>                                 << "Stream configuration adjusted to " << cfg.toString();
>  
> -               const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>                 cfg.stride = info.stride(cfg.size.width, 0, 1);
>                 cfg.frameSize = info.frameSize(cfg.size, 1);
>  
> @@ -275,12 +295,13 @@ PipelineHandlerVirtual::generateConfiguration(Camera *camera,
>                 }
>  
>                 std::map<PixelFormat, std::vector<SizeRange>> streamFormats;
> -               PixelFormat pixelFormat = formats::NV12;
> -               streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
> -                                                data->config_.maxResolutionSize } };
> +               for (auto pixelFormat : outputFormats) {
> +                       streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
> +                                                        data->config_.maxResolutionSize } };
> +               }
>                 StreamFormats formats(streamFormats);
>                 StreamConfiguration cfg(formats);
> -               cfg.pixelFormat = pixelFormat;
> +               cfg.pixelFormat = outputFormats[0];
>                 cfg.size = data->config_.maxResolutionSize;
>                 cfg.bufferCount = VirtualCameraConfiguration::kBufferCount;
>                 cfg.colorSpace = ColorSpace::Smpte170m;
> -- 
> 2.56.0
>
Robert Mader Oct. 11, 2026, 9:59 p.m. UTC | #2 
Hi,

On 11.10.26 19:44, Kieran Bingham wrote:
> Quoting Robert Mader (2026-10-09 20:40:05)
>> While NV12 is the most common native format for hardware video encoders,
>> YUV420 is the equivalent for most software video encoders. Thus
>> supporting both is great for testing.
>>
>> Similarly having some RGB formats with and without alpha can be very
>> useful, especially as the swISP is currently limited to such formats.
>>
>> This is a polished version of a hack that was used to help debugging and
>> testinghttps://webrtc-review.googlesource.com/c/src/+/491680 - and so
>> helpful with that that I'd like to see it upstream.
> probably lives under the ---.
True
>
> I'm a bit confused, and haven't fully understood the code paths below. I
> see conversions from ARGB to NV12 and then NV12 to other formats.
>
> What's the relationship between all these conversions ?
>
> Is the TPG generating ARGB, then converting to NV12 to then convert to
> ARGB when requested ?

There are two case:

 1. In TestPatternGenerator the pattern is generated as ARGB and then
    either copied (ARGBCopy()) or converted to NV12 or YUV420
    (ARGBToNV12() / ARGBToI420()). The ARGBToNV12() case was
    pre-existent, the patch adds the ARGBCopy() and ARGBToI420() cases.
 2. In ImageFrameGenerator in create() MJPG files are read and decoded /
    stored in original size as NV12 (MJPGToNV12()). In configure()
    scaled copies for the requested size are made (NV12Scale()).
    Finally, in generateFrame() the scaled buffers get copied again to
    the output buffers (NV12Copy()). The patch only changes the last
    step, replacing NV12Copy() with NV12ToI420() or NV12ToARGB() where
    appropriate. This approach is arguably not optimal for performance -
    but the one with the smallest amount of changes and the easiest to
    maintain (and expand) going forward. If there are strong concerns
    about performance we could replace the MJPGToNV12() and NV12Scale()
    with YUV420 and ARGB versions, so generateFrame() can use
    YUV420Copy() and ARGBCopy(). Not sure if its worth the additional
    complexity, though.

Regards

>
>> Signed-off-by: Robert Mader<robert.mader@collabora.com>
>> ---
>>   .../pipeline/virtual/frame_generator.h        |  3 +-
>>   .../virtual/image_frame_generator.cpp         | 55 ++++++++++++++++---
>>   .../pipeline/virtual/image_frame_generator.h  |  2 +-
>>   .../virtual/test_pattern_generator.cpp        | 46 ++++++++++++++--
>>   .../pipeline/virtual/test_pattern_generator.h |  2 +-
>>   src/libcamera/pipeline/virtual/virtual.cpp    | 41 ++++++++++----
>>   6 files changed, 123 insertions(+), 26 deletions(-)
>>
>> diff --git a/src/libcamera/pipeline/virtual/frame_generator.h b/src/libcamera/pipeline/virtual/frame_generator.h
>> index a0658c45cd82..fd34a91a5e50 100644
>> --- a/src/libcamera/pipeline/virtual/frame_generator.h
>> +++ b/src/libcamera/pipeline/virtual/frame_generator.h
>> @@ -9,6 +9,7 @@
>>   
>>   #include <libcamera/framebuffer.h>
>>   #include <libcamera/geometry.h>
>> +#include <libcamera/stream.h>
>>   
>>   namespace libcamera {
>>   
>> @@ -19,7 +20,7 @@ public:
>>   
>>          virtual void configure(const Size &size) = 0;
>>   
>> -       virtual int generateFrame(const Size &size,
>> +       virtual int generateFrame(const StreamConfiguration &cfg,
>>                                    const FrameBuffer *buffer) = 0;
>>   
>>   protected:
>> diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.cpp b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
>> index 1853bddeca27..5c56db96d8e2 100644
>> --- a/src/libcamera/pipeline/virtual/image_frame_generator.cpp
>> +++ b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
>> @@ -14,7 +14,9 @@
>>   
>>   #include <libcamera/framebuffer.h>
>>   
>> +#include <libcamera/formats.h>
>>   #include "libcamera/internal/dma_buf_allocator.h"
>> +#include "libcamera/internal/formats.h"
>>   #include "libcamera/internal/mapped_framebuffer.h"
>>   
>>   #include "libyuv/convert.h"
>> @@ -124,7 +126,7 @@ void ImageFrameGenerator::configure(const Size &size)
>>          }
>>   }
>>   
>> -int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buffer)
>> +int ImageFrameGenerator::generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer)
>>   {
> Passing a config seems fine.
>
>>          ASSERT(!scaledFrameDatas_.empty());
>>   
>> @@ -141,16 +143,55 @@ int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
>>                                              MappedFrameBuffer::MapFlag::Write);
>>   
>>          const auto &planes = mappedFrameBuffer.planes();
>> +       auto size = cfg.size;
>> +       const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>>   
>>          /* Loop only around the number of images available */
>>          frameIndex_ %= imageFrameDatas_.size();
>>   
>>          /* Write the scaledY and scaledUV to the mapped frame buffer */
>> -       libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(), size.width,
>> -                        scaledFrameDatas_[frameIndex_].UV.get(), size.width,
>> -                        planes[0].data(), size.width,
>> -                        planes[1].data(), size.width,
>> -                        size.width, size.height);
>> +       int ret;
>> +       switch (cfg.pixelFormat) {
>> +       case formats::NV12:
>> +               ret = libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(),
>> +                                      size.width,
>> +                                      scaledFrameDatas_[frameIndex_].UV.get(),
>> +                                      size.width,
>> +                                      planes[0].data(),
>> +                                      info.stride(size.width, 0),
>> +                                      planes[1].data(),
>> +                                      info.stride(size.width, 1),
>> +                                      size.width, size.height);
>> +               break;
>> +       case formats::YUV420:
>> +               ret = libyuv::NV12ToI420(scaledFrameDatas_[frameIndex_].Y.get(),
>> +                                        size.width,
>> +                                        scaledFrameDatas_[frameIndex_].UV.get(),
>> +                                        size.width,
>> +                                        planes[0].data(),
>> +                                        info.stride(size.width, 0),
>> +                                        planes[1].data(),
>> +                                        info.stride(size.width, 1),
>> +                                        planes[2].data(),
>> +                                        info.stride(size.width, 2),
>> +                                        size.width, size.height);
>> +               break;
>> +       case formats::ABGR8888:
>> +       case formats::XBGR8888:
>> +               ret = libyuv::NV12ToARGB(scaledFrameDatas_[frameIndex_].Y.get(),
>> +                                        size.width,
>> +                                        scaledFrameDatas_[frameIndex_].UV.get(),
>> +                                        size.width,
>> +                                        planes[0].data(),
>> +                                        info.stride(size.width, 0),
>> +                                        size.width, size.height);
>> +               break;
>> +       default:
>> +               LOG(Virtual, Error) << "Unsupported format";
>> +               return -1;
>> +       }
>> +       if (ret != 0)
>> +               LOG(Virtual, Error) << "Conversation failed with " << ret;
>>   
>>          /* Proceed to the next image every 4 frames */
>>          /* \todo Consider setting the frameRepeat in the config file  */
>> @@ -158,7 +199,7 @@ int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
>>          if (parameter_ % frameRepeat == 0)
>>                  frameIndex_++;
>>   
>> -       return 0;
>> +       return ret;
>>   }
>>   
>>   /*
>> diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.h b/src/libcamera/pipeline/virtual/image_frame_generator.h
>> index 42a077ba803c..19b6fe38654e 100644
>> --- a/src/libcamera/pipeline/virtual/image_frame_generator.h
>> +++ b/src/libcamera/pipeline/virtual/image_frame_generator.h
>> @@ -37,7 +37,7 @@ private:
>>          };
>>   
>>          void configure(const Size &size) override;
>> -       int generateFrame(const Size &size, const FrameBuffer *buffer) override;
>> +       int generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer) override;
>>   
>>          std::vector<ImageFrameData> imageFrameDatas_;
>>          std::vector<ImageFrameData> scaledFrameDatas_;
>> diff --git a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
>> index 7b53e8b32327..4cde32176661 100644
>> --- a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
>> +++ b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
>> @@ -11,7 +11,9 @@
>>   
>>   #include <libcamera/base/log.h>
>>   
>> +#include <libcamera/formats.h>
>>   #include "libcamera/internal/dma_buf_allocator.h"
>> +#include "libcamera/internal/formats.h"
>>   #include "libcamera/internal/mapped_framebuffer.h"
>>   
>>   #include <libyuv/convert_from_argb.h>
>> @@ -42,7 +44,7 @@ LOG_DECLARE_CATEGORY(Virtual)
>>   
>>   static const unsigned int kARGBSize = 4;
>>   
>> -int TestPatternGenerator::generateFrame(const Size &size,
>> +int TestPatternGenerator::generateFrame(const StreamConfiguration &cfg,
>>                                          const FrameBuffer *buffer)
>>   {
>>          /*
>> @@ -58,16 +60,48 @@ int TestPatternGenerator::generateFrame(const Size &size,
>>                                              MappedFrameBuffer::MapFlag::Write);
>>   
>>          const auto &planes = mappedFrameBuffer.planes();
>> +       const auto size = cfg.size;
>> +       const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>>   
>>          rotateLeft1Column<kARGBSize>(size, template_.get());
>>   
>>          /* Convert the template_ to the frame buffer */
>> -       int ret = libyuv::ARGBToNV12(template_.get(), size.width * kARGBSize,
>> -                                    planes[0].data(), size.width,
>> -                                    planes[1].data(), size.width,
>> -                                    size.width, size.height);
>> +       int ret;
>> +       switch (cfg.pixelFormat) {
>> +       case formats::NV12:
>> +               ret = libyuv::ARGBToNV12(template_.get(),
>> +                                        size.width * kARGBSize,
>> +                                        planes[0].data(),
>> +                                        info.stride(size.width, 0),
>> +                                        planes[1].data(),
>> +                                        info.stride(size.width, 1),
>> +                                        size.width, size.height);
>> +               break;
>> +       case formats::YUV420:
>> +               ret = libyuv::ARGBToI420(template_.get(),
>> +                                        size.width * kARGBSize,
>> +                                        planes[0].data(),
>> +                                        info.stride(size.width, 0),
>> +                                        planes[1].data(),
>> +                                        info.stride(size.width, 1),
>> +                                        planes[2].data(),
>> +                                        info.stride(size.width, 2),
>> +                                        size.width, size.height);
>> +               break;
>> +       case formats::ABGR8888:
>> +       case formats::XBGR8888:
>> +               ret = libyuv::ARGBCopy(template_.get(),
>> +                                      size.width * kARGBSize,
>> +                                      planes[0].data(),
>> +                                      info.stride(size.width, 0),
>> +                                      size.width, size.height);
>> +               break;
>> +       default:
>> +               LOG(Virtual, Error) << "Unsupported format";
>> +               return -1;
>> +       }
>>          if (ret != 0)
>> -               LOG(Virtual, Error) << "ARGBToNV12() failed with " << ret;
>> +               LOG(Virtual, Error) << "Conversation failed with " << ret; return ret; } diff --git 
>> a/src/libcamera/pipeline/virtual/test_pattern_generator.h 
>> b/src/libcamera/pipeline/virtual/test_pattern_generator.h index 
>> 2a51bd31abdf..9f72d652ab2c 100644 --- 
>> a/src/libcamera/pipeline/virtual/test_pattern_generator.h +++ 
>> b/src/libcamera/pipeline/virtual/test_pattern_generator.h @@ -24,7 
>> +24,7 @@ enum class TestPattern : char { class TestPatternGenerator : 
>> public FrameGenerator { public: - int generateFrame(const Size &size, 
>> const FrameBuffer *buffer) override; + int generateFrame(const 
>> StreamConfiguration &cfg, const FrameBuffer *buffer) override; 
>> protected: /* Buffer of test pattern template */ diff --git 
>> a/src/libcamera/pipeline/virtual/virtual.cpp 
>> b/src/libcamera/pipeline/virtual/virtual.cpp index 
>> e25dd85a7391..7ed2d0b7f596 100644 --- 
>> a/src/libcamera/pipeline/virtual/virtual.cpp +++ 
>> b/src/libcamera/pipeline/virtual/virtual.cpp @@ -47,6 +47,13 @@ 
>> LOG_DEFINE_CATEGORY(Virtual) namespace { +constexpr PixelFormat 
>> outputFormats[] = { + formats::NV12, + formats::YUV420, + 
>> formats::ABGR8888, + formats::XBGR8888, +}; + uint64_t 
>> currentTimestamp() { const auto now = 
>> std::chrono::steady_clock::now(); @@ -148,7 +155,7 @@ void 
>> VirtualCameraData::processRequest(Request *request) found = true; if 
>> (streamConfig.frameGenerator->generateFrame(
>> -                                           stream->configuration().size, buffer))
>> +                                           stream->configuration(), buffer))
>>                                          fmd.status = FrameMetadata::Status::FrameError;
>>   
>>                                  bufferCompleted.emit(buffer);
>> @@ -202,17 +209,31 @@ CameraConfiguration::Status VirtualCameraConfiguration::validate()
>>                          adjusted = true;
>>                  }
>>   
>> -               if (cfg.pixelFormat != formats::NV12) {
>> -                       cfg.pixelFormat = formats::NV12;
>> +               if (std::find(std::begin(outputFormats),
>> +                             std::end(outputFormats),
>> +                             cfg.pixelFormat) == std::end(outputFormats)) {
>> +                       cfg.pixelFormat = outputFormats[0];
>>                          status = Adjusted;
>>                          adjusted = true;
>>                  }
>>   
>> -               if (cfg.colorSpace != ColorSpace::Smpte170m) {
>> -                       cfg.colorSpace = ColorSpace::Smpte170m;
>> +               const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>> +               std::optional<ColorSpace> colorSpace;
>> +               switch (info.colourEncoding) {
>> +               case PixelFormatInfo::ColourEncodingRGB:
>> +                       colorSpace = ColorSpace::Srgb;
>> +                       break;
>> +               case PixelFormatInfo::ColourEncodingYUV:
>> +                       colorSpace = ColorSpace::Smpte170m;
>> +                       break;
>> +               default:
>> +                       ASSERT(false);
>> +               }
>> +               if (cfg.colorSpace && cfg.colorSpace != colorSpace) {
>>                          status = Adjusted;
>>                          adjusted = true;
>>                  }
>> +               cfg.colorSpace = colorSpace;
>>   
>>                  if (validateColorSpaces() == Adjusted) {
>>                          status = Adjusted;
>> @@ -223,7 +244,6 @@ CameraConfiguration::Status VirtualCameraConfiguration::validate()
>>                          LOG(Virtual, Info)
>>                                  << "Stream configuration adjusted to " << cfg.toString();
>>   
>> -               const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
>>                  cfg.stride = info.stride(cfg.size.width, 0, 1);
>>                  cfg.frameSize = info.frameSize(cfg.size, 1);
>>   
>> @@ -275,12 +295,13 @@ PipelineHandlerVirtual::generateConfiguration(Camera *camera,
>>                  }
>>   
>>                  std::map<PixelFormat, std::vector<SizeRange>> streamFormats;
>> -               PixelFormat pixelFormat = formats::NV12;
>> -               streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
>> -                                                data->config_.maxResolutionSize } };
>> +               for (auto pixelFormat : outputFormats) {
>> +                       streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
>> +                                                        data->config_.maxResolutionSize } };
>> +               }
>>                  StreamFormats formats(streamFormats);
>>                  StreamConfiguration cfg(formats);
>> -               cfg.pixelFormat = pixelFormat;
>> +               cfg.pixelFormat = outputFormats[0];
>>                  cfg.size = data->config_.maxResolutionSize;
>>                  cfg.bufferCount = VirtualCameraConfiguration::kBufferCount;
>>                  cfg.colorSpace = ColorSpace::Smpte170m;
>> -- 
>> 2.56.0
>>

Patch
diff mbox series 

diff --git a/src/libcamera/pipeline/virtual/frame_generator.h b/src/libcamera/pipeline/virtual/frame_generator.h
index a0658c45cd82..fd34a91a5e50 100644
--- a/src/libcamera/pipeline/virtual/frame_generator.h
+++ b/src/libcamera/pipeline/virtual/frame_generator.h
@@ -9,6 +9,7 @@ 
 
 #include <libcamera/framebuffer.h>
 #include <libcamera/geometry.h>
+#include <libcamera/stream.h>
 
 namespace libcamera {
 
@@ -19,7 +20,7 @@  public:
 
 	virtual void configure(const Size &size) = 0;
 
-	virtual int generateFrame(const Size &size,
+	virtual int generateFrame(const StreamConfiguration &cfg,
 				  const FrameBuffer *buffer) = 0;
 
 protected:
diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.cpp b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
index 1853bddeca27..5c56db96d8e2 100644
--- a/src/libcamera/pipeline/virtual/image_frame_generator.cpp
+++ b/src/libcamera/pipeline/virtual/image_frame_generator.cpp
@@ -14,7 +14,9 @@ 
 
 #include <libcamera/framebuffer.h>
 
+#include <libcamera/formats.h>
 #include "libcamera/internal/dma_buf_allocator.h"
+#include "libcamera/internal/formats.h"
 #include "libcamera/internal/mapped_framebuffer.h"
 
 #include "libyuv/convert.h"
@@ -124,7 +126,7 @@  void ImageFrameGenerator::configure(const Size &size)
 	}
 }
 
-int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buffer)
+int ImageFrameGenerator::generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer)
 {
 	ASSERT(!scaledFrameDatas_.empty());
 
@@ -141,16 +143,55 @@  int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
 					    MappedFrameBuffer::MapFlag::Write);
 
 	const auto &planes = mappedFrameBuffer.planes();
+	auto size = cfg.size;
+	const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
 
 	/* Loop only around the number of images available */
 	frameIndex_ %= imageFrameDatas_.size();
 
 	/* Write the scaledY and scaledUV to the mapped frame buffer */
-	libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(), size.width,
-			 scaledFrameDatas_[frameIndex_].UV.get(), size.width,
-			 planes[0].data(), size.width,
-			 planes[1].data(), size.width,
-			 size.width, size.height);
+	int ret;
+	switch (cfg.pixelFormat) {
+	case formats::NV12:
+		ret = libyuv::NV12Copy(scaledFrameDatas_[frameIndex_].Y.get(),
+				       size.width,
+				       scaledFrameDatas_[frameIndex_].UV.get(),
+				       size.width,
+				       planes[0].data(),
+				       info.stride(size.width, 0),
+				       planes[1].data(),
+				       info.stride(size.width, 1),
+				       size.width, size.height);
+		break;
+	case formats::YUV420:
+		ret = libyuv::NV12ToI420(scaledFrameDatas_[frameIndex_].Y.get(),
+					 size.width,
+					 scaledFrameDatas_[frameIndex_].UV.get(),
+					 size.width,
+					 planes[0].data(),
+					 info.stride(size.width, 0),
+					 planes[1].data(),
+					 info.stride(size.width, 1),
+					 planes[2].data(),
+					 info.stride(size.width, 2),
+					 size.width, size.height);
+		break;
+	case formats::ABGR8888:
+	case formats::XBGR8888:
+		ret = libyuv::NV12ToARGB(scaledFrameDatas_[frameIndex_].Y.get(),
+					 size.width,
+					 scaledFrameDatas_[frameIndex_].UV.get(),
+					 size.width,
+					 planes[0].data(),
+					 info.stride(size.width, 0),
+					 size.width, size.height);
+		break;
+	default:
+		LOG(Virtual, Error) << "Unsupported format";
+		return -1;
+	}
+	if (ret != 0)
+		LOG(Virtual, Error) << "Conversation failed with " << ret;
 
 	/* Proceed to the next image every 4 frames */
 	/* \todo Consider setting the frameRepeat in the config file  */
@@ -158,7 +199,7 @@  int ImageFrameGenerator::generateFrame(const Size &size, const FrameBuffer *buff
 	if (parameter_ % frameRepeat == 0)
 		frameIndex_++;
 
-	return 0;
+	return ret;
 }
 
 /*
diff --git a/src/libcamera/pipeline/virtual/image_frame_generator.h b/src/libcamera/pipeline/virtual/image_frame_generator.h
index 42a077ba803c..19b6fe38654e 100644
--- a/src/libcamera/pipeline/virtual/image_frame_generator.h
+++ b/src/libcamera/pipeline/virtual/image_frame_generator.h
@@ -37,7 +37,7 @@  private:
 	};
 
 	void configure(const Size &size) override;
-	int generateFrame(const Size &size, const FrameBuffer *buffer) override;
+	int generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer) override;
 
 	std::vector<ImageFrameData> imageFrameDatas_;
 	std::vector<ImageFrameData> scaledFrameDatas_;
diff --git a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
index 7b53e8b32327..4cde32176661 100644
--- a/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
+++ b/src/libcamera/pipeline/virtual/test_pattern_generator.cpp
@@ -11,7 +11,9 @@ 
 
 #include <libcamera/base/log.h>
 
+#include <libcamera/formats.h>
 #include "libcamera/internal/dma_buf_allocator.h"
+#include "libcamera/internal/formats.h"
 #include "libcamera/internal/mapped_framebuffer.h"
 
 #include <libyuv/convert_from_argb.h>
@@ -42,7 +44,7 @@  LOG_DECLARE_CATEGORY(Virtual)
 
 static const unsigned int kARGBSize = 4;
 
-int TestPatternGenerator::generateFrame(const Size &size,
+int TestPatternGenerator::generateFrame(const StreamConfiguration &cfg,
 					const FrameBuffer *buffer)
 {
 	/*
@@ -58,16 +60,48 @@  int TestPatternGenerator::generateFrame(const Size &size,
 					    MappedFrameBuffer::MapFlag::Write);
 
 	const auto &planes = mappedFrameBuffer.planes();
+	const auto size = cfg.size;
+	const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
 
 	rotateLeft1Column<kARGBSize>(size, template_.get());
 
 	/* Convert the template_ to the frame buffer */
-	int ret = libyuv::ARGBToNV12(template_.get(), size.width * kARGBSize,
-				     planes[0].data(), size.width,
-				     planes[1].data(), size.width,
-				     size.width, size.height);
+	int ret;
+	switch (cfg.pixelFormat) {
+	case formats::NV12:
+		ret = libyuv::ARGBToNV12(template_.get(),
+					 size.width * kARGBSize,
+					 planes[0].data(),
+					 info.stride(size.width, 0),
+					 planes[1].data(),
+					 info.stride(size.width, 1),
+					 size.width, size.height);
+		break;
+	case formats::YUV420:
+		ret = libyuv::ARGBToI420(template_.get(),
+					 size.width * kARGBSize,
+					 planes[0].data(),
+					 info.stride(size.width, 0),
+					 planes[1].data(),
+					 info.stride(size.width, 1),
+					 planes[2].data(),
+					 info.stride(size.width, 2),
+					 size.width, size.height);
+		break;
+	case formats::ABGR8888:
+	case formats::XBGR8888:
+		ret = libyuv::ARGBCopy(template_.get(),
+				       size.width * kARGBSize,
+				       planes[0].data(),
+				       info.stride(size.width, 0),
+				       size.width, size.height);
+		break;
+	default:
+		LOG(Virtual, Error) << "Unsupported format";
+		return -1;
+	}
 	if (ret != 0)
-		LOG(Virtual, Error) << "ARGBToNV12() failed with " << ret;
+		LOG(Virtual, Error) << "Conversation failed with " << ret;
 
 	return ret;
 }
diff --git a/src/libcamera/pipeline/virtual/test_pattern_generator.h b/src/libcamera/pipeline/virtual/test_pattern_generator.h
index 2a51bd31abdf..9f72d652ab2c 100644
--- a/src/libcamera/pipeline/virtual/test_pattern_generator.h
+++ b/src/libcamera/pipeline/virtual/test_pattern_generator.h
@@ -24,7 +24,7 @@  enum class TestPattern : char {
 class TestPatternGenerator : public FrameGenerator
 {
 public:
-	int generateFrame(const Size &size, const FrameBuffer *buffer) override;
+	int generateFrame(const StreamConfiguration &cfg, const FrameBuffer *buffer) override;
 
 protected:
 	/* Buffer of test pattern template */
diff --git a/src/libcamera/pipeline/virtual/virtual.cpp b/src/libcamera/pipeline/virtual/virtual.cpp
index e25dd85a7391..7ed2d0b7f596 100644
--- a/src/libcamera/pipeline/virtual/virtual.cpp
+++ b/src/libcamera/pipeline/virtual/virtual.cpp
@@ -47,6 +47,13 @@  LOG_DEFINE_CATEGORY(Virtual)
 
 namespace {
 
+constexpr PixelFormat outputFormats[] = {
+	formats::NV12,
+	formats::YUV420,
+	formats::ABGR8888,
+	formats::XBGR8888,
+};
+
 uint64_t currentTimestamp()
 {
 	const auto now = std::chrono::steady_clock::now();
@@ -148,7 +155,7 @@  void VirtualCameraData::processRequest(Request *request)
 				found = true;
 
 				if (streamConfig.frameGenerator->generateFrame(
-					    stream->configuration().size, buffer))
+					    stream->configuration(), buffer))
 					fmd.status = FrameMetadata::Status::FrameError;
 
 				bufferCompleted.emit(buffer);
@@ -202,17 +209,31 @@  CameraConfiguration::Status VirtualCameraConfiguration::validate()
 			adjusted = true;
 		}
 
-		if (cfg.pixelFormat != formats::NV12) {
-			cfg.pixelFormat = formats::NV12;
+		if (std::find(std::begin(outputFormats),
+			      std::end(outputFormats),
+			      cfg.pixelFormat) == std::end(outputFormats)) {
+			cfg.pixelFormat = outputFormats[0];
 			status = Adjusted;
 			adjusted = true;
 		}
 
-		if (cfg.colorSpace != ColorSpace::Smpte170m) {
-			cfg.colorSpace = ColorSpace::Smpte170m;
+		const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
+		std::optional<ColorSpace> colorSpace;
+		switch (info.colourEncoding) {
+		case PixelFormatInfo::ColourEncodingRGB:
+			colorSpace = ColorSpace::Srgb;
+			break;
+		case PixelFormatInfo::ColourEncodingYUV:
+			colorSpace = ColorSpace::Smpte170m;
+			break;
+		default:
+			ASSERT(false);
+		}
+		if (cfg.colorSpace && cfg.colorSpace != colorSpace) {
 			status = Adjusted;
 			adjusted = true;
 		}
+		cfg.colorSpace = colorSpace;
 
 		if (validateColorSpaces() == Adjusted) {
 			status = Adjusted;
@@ -223,7 +244,6 @@  CameraConfiguration::Status VirtualCameraConfiguration::validate()
 			LOG(Virtual, Info)
 				<< "Stream configuration adjusted to " << cfg.toString();
 
-		const PixelFormatInfo &info = PixelFormatInfo::info(cfg.pixelFormat);
 		cfg.stride = info.stride(cfg.size.width, 0, 1);
 		cfg.frameSize = info.frameSize(cfg.size, 1);
 
@@ -275,12 +295,13 @@  PipelineHandlerVirtual::generateConfiguration(Camera *camera,
 		}
 
 		std::map<PixelFormat, std::vector<SizeRange>> streamFormats;
-		PixelFormat pixelFormat = formats::NV12;
-		streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
-						 data->config_.maxResolutionSize } };
+		for (auto pixelFormat : outputFormats) {
+			streamFormats[pixelFormat] = { { data->config_.minResolutionSize,
+							 data->config_.maxResolutionSize } };
+		}
 		StreamFormats formats(streamFormats);
 		StreamConfiguration cfg(formats);
-		cfg.pixelFormat = pixelFormat;
+		cfg.pixelFormat = outputFormats[0];
 		cfg.size = data->config_.maxResolutionSize;
 		cfg.bufferCount = VirtualCameraConfiguration::kBufferCount;
 		cfg.colorSpace = ColorSpace::Smpte170m;