[v11,0/8] Enable raw streams with software ISP

23906 mbox series

This makes raw streams working again in ‘simple’ pipeline when software
ISP is enabled for the given device.  At most one raw stream and one
processed stream (possibly both at once) are supported.

An example ‘cam’ invocation requesting a raw stream rather than a debayered stream:

  cam -c1 -C8 -s role=raw,width=1920,height=1080 -Ffile#.raw

Or for both raw and processed streams:

  cam -c1 -C8 -Ffile# \
    -s role=viewfinder,width=1920,height=1080,pixelformat=RGB888 \
    -s role=raw,width=3280,height=2464,pixelformat=SRGGB8

When only a raw stream is requested, there are no exposure/gain
adjustments applied.  This could be improved in future, once software
ISP gets a mechanism to gather image statistics without processing and
using them to make the adjustments, or once manual exposure controls are
added to software ISP.  In the meantime, exposure must be changed
externally.

Changes in v11:
- Use rawStream_ pointer to a raw stream rather than rawRequested_ and
  processedRequested_ flags, as suggested by Umang.
- Selection of pipeline configurations reworked, partially incorporating
  Umang’s ideas and permitting adjustments of raw stream configurations
  if they don’t match the processed stream or what’s available from the
  sensor.
- Tested-by tags removed from here because the changes are substantial.
  v11 works better for me than both v10 and Umang’s RFC but it requires
  independent testing.

Changes in v10:
- Missing initialisation of swIspEnabled_ added.
- "Add plain output configurations" patch dropped.  Raw configurations
  can be built implicitly from capture parameters.  Related adjustments
  in "Validate raw stream configurations".
- "Identify requested stream roles" patch dropped and merged into
  followup patches; camera configuration is updated in validate() and
  not in generateConfiguration() now.
- A commit title improved as suggested by Umang.

Changes in v9:
- Fix of calling a wrong output buffer allocator when both raw and
  processed streams are used.

Changes in v8:
- A missing `status = Adjusted' added to
  SimpleCameraConfiguration::validate.
- Code comments regarding raw colour space for raw requested roles
  added.
- Dropped raw format helpers.

Changes in v7:
- Rebased on current master.

Changes in v6:
- An unnecessary copy of formats avoided.
- rawRequested_ and processedrequested_ set to false when there are no
  roles.
- rawRequested_ and processedrequested_ updated in
  SimpleCameraConfiguration::validate().

Changes in v5:
- Possible temporary segfault in the patch adding plain output
  configurations avoided.
- PixelFormatInfo::isRaw() helper added.
- SimplePipelineHandler::setUpFormatSizes replaced with a lambda
  function.
- SimpleCameraData::{rawRequested_,processedRequested_} are set in
  SimpleCameraConfiguration first and copied to SimpleCameraData only
  after successful configure().

Changes in v4:
- Broken range pruning due to passing a value rather than a reference
  fixed.
- New common function isFormatRaw introduced.
- The patch assigning colour spaces in the simple pipeline, previously
  posted separately, included in this series, as the first patch.  It
  can still be handled separately; in any case the rest sort of depends
  on it.
- Setting metadataRequired to false where needed to prevent freezes and
  assertion errors; related to metadata reporting support merged to
  master since v3.

Changes in v3:
- Significantly reworked, with both functional and clarity improvements.
  The level of guesswork and confusion is hopefully reduced enough now
  to drop the RFC prefix.
- The number of streams is set to 2 only with software ISP.
- SimpleCameraData::pipeConfig_ nullptr check patch dropped.
- PPM/raw file output patch dropped from this series.  Let’s handle this
  separately as the patch series is already complex enough.

Changes in v2:
- Completely reworked.
- Extended to be able to produce a raw stream together with a processed
  stream.

Milan Zamazal (8):
  libcamera: software_isp: Assign colour spaces in configurations
  libcamera: simple: Set the number of software ISP streams to 2
  libcamera: simple: Exclude raw configurations from output conversions
  libcamera: simple: Handle processed and raw formats separately
  libcamera: simple: Validate raw stream configurations
  libcamera: simple: Don't enforce conversion with an added raw stream
  libcamera: simple: Require metadata only when software ISP is used
  libcamera: simple: Make raw streams working

 src/libcamera/pipeline/simple/simple.cpp | 283 ++++++++++++++++++-----
 1 file changed, 219 insertions(+), 64 deletions(-)

On Wed, Jul 23, 2025 at 08:08:05PM +0200, Milan Zamazal wrote:
> This makes raw streams working again in ‘simple’ pipeline when software
> ISP is enabled for the given device.  At most one raw stream and one
> processed stream (possibly both at once) are supported.
> 
> An example ‘cam’ invocation requesting a raw stream rather than a debayered stream:
> 
>   cam -c1 -C8 -s role=raw,width=1920,height=1080 -Ffile#.raw
> 
> Or for both raw and processed streams:
> 
>   cam -c1 -C8 -Ffile# \
>     -s role=viewfinder,width=1920,height=1080,pixelformat=RGB888 \
>     -s role=raw,width=3280,height=2464,pixelformat=SRGGB8
> 
> When only a raw stream is requested, there are no exposure/gain
> adjustments applied.  This could be improved in future, once software
> ISP gets a mechanism to gather image statistics without processing and
> using them to make the adjustments, or once manual exposure controls are
> added to software ISP.  In the meantime, exposure must be changed
> externally.

It's a pretty important feature, but let's go one step at a time. When
implementing support for this, I'd like to try and enable both manual
and auto mode.

> Changes in v11:
> - Use rawStream_ pointer to a raw stream rather than rawRequested_ and
>   processedRequested_ flags, as suggested by Umang.
> - Selection of pipeline configurations reworked, partially incorporating
>   Umang’s ideas and permitting adjustments of raw stream configurations
>   if they don’t match the processed stream or what’s available from the
>   sensor.
> - Tested-by tags removed from here because the changes are substantial.
>   v11 works better for me than both v10 and Umang’s RFC but it requires
>   independent testing.
> 
> Changes in v10:
> - Missing initialisation of swIspEnabled_ added.
> - "Add plain output configurations" patch dropped.  Raw configurations
>   can be built implicitly from capture parameters.  Related adjustments
>   in "Validate raw stream configurations".
> - "Identify requested stream roles" patch dropped and merged into
>   followup patches; camera configuration is updated in validate() and
>   not in generateConfiguration() now.
> - A commit title improved as suggested by Umang.
> 
> Changes in v9:
> - Fix of calling a wrong output buffer allocator when both raw and
>   processed streams are used.
> 
> Changes in v8:
> - A missing `status = Adjusted' added to
>   SimpleCameraConfiguration::validate.
> - Code comments regarding raw colour space for raw requested roles
>   added.
> - Dropped raw format helpers.
> 
> Changes in v7:
> - Rebased on current master.
> 
> Changes in v6:
> - An unnecessary copy of formats avoided.
> - rawRequested_ and processedrequested_ set to false when there are no
>   roles.
> - rawRequested_ and processedrequested_ updated in
>   SimpleCameraConfiguration::validate().
> 
> Changes in v5:
> - Possible temporary segfault in the patch adding plain output
>   configurations avoided.
> - PixelFormatInfo::isRaw() helper added.
> - SimplePipelineHandler::setUpFormatSizes replaced with a lambda
>   function.
> - SimpleCameraData::{rawRequested_,processedRequested_} are set in
>   SimpleCameraConfiguration first and copied to SimpleCameraData only
>   after successful configure().
> 
> Changes in v4:
> - Broken range pruning due to passing a value rather than a reference
>   fixed.
> - New common function isFormatRaw introduced.
> - The patch assigning colour spaces in the simple pipeline, previously
>   posted separately, included in this series, as the first patch.  It
>   can still be handled separately; in any case the rest sort of depends
>   on it.
> - Setting metadataRequired to false where needed to prevent freezes and
>   assertion errors; related to metadata reporting support merged to
>   master since v3.
> 
> Changes in v3:
> - Significantly reworked, with both functional and clarity improvements.
>   The level of guesswork and confusion is hopefully reduced enough now
>   to drop the RFC prefix.
> - The number of streams is set to 2 only with software ISP.
> - SimpleCameraData::pipeConfig_ nullptr check patch dropped.
> - PPM/raw file output patch dropped from this series.  Let’s handle this
>   separately as the patch series is already complex enough.
> 
> Changes in v2:
> - Completely reworked.
> - Extended to be able to produce a raw stream together with a processed
>   stream.
> 
> Milan Zamazal (8):
>   libcamera: software_isp: Assign colour spaces in configurations
>   libcamera: simple: Set the number of software ISP streams to 2
>   libcamera: simple: Exclude raw configurations from output conversions
>   libcamera: simple: Handle processed and raw formats separately
>   libcamera: simple: Validate raw stream configurations
>   libcamera: simple: Don't enforce conversion with an added raw stream
>   libcamera: simple: Require metadata only when software ISP is used
>   libcamera: simple: Make raw streams working
> 
>  src/libcamera/pipeline/simple/simple.cpp | 283 ++++++++++++++++++-----
>  1 file changed, 219 insertions(+), 64 deletions(-)