[{"id":38686,"web_url":"https://patchwork.libcamera.org/comment/38686/","msgid":"<177788003794.3468922.528411873495754823@ping.linuxembedded.co.uk>","date":"2026-05-04T07:33:57","subject":"Re: [RFC PATCH v1 0/3] libcamera: pipeline: virtual: Add raw Bayer\n\tframe support","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Max,\n\nThanks for posting these patches - very interesting work.\n\nI haven't reviewed the patches yet, but the CI has just run on this\nseries and failed the build-history rule.\n\nhttps://gitlab.freedesktop.org/camera/libcamera/-/jobs/98905270\n\nWe always maintain a bisectable tree, so each commit has to compile\nfully and independently.\n\nI usually use something like:\n \"git rebase -i origin/main -x ninja -C build\" \n\nwhich will compile each commit locally for me when I'm testing.\n\n--\nRegards\n\nKieran\n\nQuoting Max Bretschneider (2026-05-01 11:52:02)\n> The virtual pipeline handler currently only produces NV12 output, either\n> from test patterns or JPEG files. This means that SoftISP, which\n> processes raw Bayer data, cannot be tested without physical camera\n> hardware. A first step to achieving this would be to support raw streams\n> in the virtual pipeline handler. This series introduces a RawFrameGenerator\n> that reads binary raw Bayer files from disk and serves them through the virtual\n> pipeline. The introduced changes are opt-in through the YAML configuration. If\n> virtual cameras are configured with raw_frames, the new code path is\n> chosen, paths for the existing test_patterns and frames are unaffected.\n> \n> The SoftISP part of this patch's motivation is complemented by David Plowman's\n> series, which introduced the CameraSensorMemory class\n> (https://patchwork.libcamera.org/patch/25468/), since SoftISP needs an\n> actual CameraSensor for correct construction. The idea is that in the\n> future it can be used together to serve Raw frames through the\n> virtual pipeline and process them with SoftISP, for example for CI tests or more easily reproducible benchmarks.\n> \n> Testing: I have used `dd` to create zero filled raw frames to\n> test things such as format negotiation, buffer sizing, general frame\n> flow, config parsing (both right/wrong, e.g. rejecting invalid input)\n> and bit depth handling. I've written a small bash script to essentialy\n> automate this for all n : { 8, 10, 12, 14, 16 } and the four Bayer\n> formats in accordance to `src/libcamera/formats.yaml`. I then used numpy to\n> test the generateFrame() functionality by generating a known frame with\n> content, capturing it through the pipeline to disk with the cam application\n> and then reading the output back again to verify that all the pixels match\n> the expected values.\n> \n> I noticed that the pipeline handler components currently have no\n> dedicated test coverage, if I missed them or if there is some preferred\n> approach for adding additional testing I'd be open to any suggestions.\n> For now I can also share my scripts if thats wanted.\n> \n> Ran `git clang-format` as well as the `checkstyle.py`. I've run the\n> Meson tests both with and without the changes and get the same:\n> Ok: 51\n> Expected Fail: 1\n> Fail: 1\n> Unexpected Pass: 0\n> Skipped: 29\n> Timeout: 0\n> \n> Open questions:\n> - The sensor properties set in match() are currently hardcoded, and I've\n> left them with a \\todo. Especially since we've also discussed adding\n> controls such as Exposure and AnalogueGain in \"[RFC] Decouple\n> SoftwareISP from CameraSensor for virtual pipeline testing\": Should I\n> move these directly to the yaml as well or proceed differently?\n> - I've written a lambda to derive the pixel format, but at this point it\n> has gotten quite large. Maybe this could also be a 2D lookup table or\n> something else that is cleaner. On that lambda also: all cases are\n> covered I think, but the default case is 16, I wasn't sure if 16\n> should also be made an explicit case and something more sensible\n> should be chosen for the default.\n> - Between the raw_frames and the frames path, the file collection logic\n> is now also duplicated. Should I add a helper here to change this?\n> - Currently every frame is read into framesDatas_ immediately at\n> startup. Depending on the number and the size of the raw files, this\n> could take up quite a lot of memory. For the use cases I thought of\n> (e.g. CI) this would be fine, since the number of files would probably\n> be comparatively small there. Still I wanted to bring it up.\n> - We weren't sure if the SPDX license header is fine like this. For the\n> moment I've just placed my intern company mail, is this fine as is?\n> \n> Documentation:\n> I've added documentation according to the coding-style.rst,\n> if my comments are not sufficient I'd be happy to add more of course.\n> \n> AI Disclosure:\n> I've used DeepWiki to familiarize myself with the libcamera repository\n> and to point me at reference implementations and patterns to use as orientation.\n> \n> Thank you all for your time!\n> \n> Max Bretschneider (3):\n> libcamera: pipeline: virtual: Add RawFrameGenerator\n> libcamera: pipeline: virtual: Add raw_frames config parser support\n> libcamera: pipeline: virtual: Support StreamRole::Raw and Bayer\n> formats\n> \n> .../pipeline/virtual/config_parser.cpp | 100 ++++++++++++-\n> src/libcamera/pipeline/virtual/meson.build | 1 +\n> .../pipeline/virtual/raw_frame_generator.cpp | 132 ++++++++++++++++++\n> .../pipeline/virtual/raw_frame_generator.h | 47 +++++++\n> src/libcamera/pipeline/virtual/virtual.cpp | 128 ++++++++++++++---\n> src/libcamera/pipeline/virtual/virtual.h | 3 +-\n> 6 files changed, 387 insertions(+), 24 deletions(-)\n> create mode 100644 src/libcamera/pipeline/virtual/raw_frame_generator.cpp\n> create mode 100644 src/libcamera/pipeline/virtual/raw_frame_generator.h\n> \n> --\n> 2.43.0\n> \n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BD15DBDCB5\n\tfor ;\n\tMon, 4 May 2026 07:34:03 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id BC86662DC4;\n\tMon, 4 May 2026 09:34:02 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 920AF6271A\n\tfor ;\n\tMon, 4 May 2026 09:34:00 +0200 (CEST)","from monstersaurus.ideasonboard.com\n\t(cpc89244-aztw30-2-0-cust6594.18-1.cable.virginm.net [86.31.185.195])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DE2FA8F;\n\tMon, 4 May 2026 09:33:58 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"u94LtByj\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1777880038;\n\tbh=rHyC2EdR4RBjfyvCr0swsUdsckAfhgcFFtqJMmjG9uM=;\n\th=In-Reply-To:References:Subject:From:Cc:To:Date:From;\n\tb=u94LtByjfzstoma4f8wGXLzPDG7OXhDl0K6DUW0AN75kxefVq/C88gV7DgIUNUfNY\n\t/h4vPZCe42XlkR6cMnLYwmNIZ5LJIMbmJyOdawLaeJ0ZbGjX2IoDJTP3vq1V+q42S5\n\tK6kJbzb5xWPRUyzg9pYkgPtwqVLPguRb/pmxW9dQ=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20260501105137.439519-1-maxbretschneider@protonmail.com>","References":"<20260501105137.439519-1-maxbretschneider@protonmail.com>","Subject":"Re: [RFC PATCH v1 0/3] libcamera: pipeline: virtual: Add raw Bayer\n\tframe support","From":"Kieran Bingham ","Cc":"Max Bretschneider ","To":"Max Bretschneider ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Mon, 04 May 2026 08:33:57 +0100","Message-ID":"<177788003794.3468922.528411873495754823@ping.linuxembedded.co.uk>","User-Agent":"alot/0.9.1","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":38706,"web_url":"https://patchwork.libcamera.org/comment/38706/","msgid":"","date":"2026-05-04T09:19:53","subject":"Re: [RFC PATCH v1 0/3] libcamera: pipeline: virtual: Add raw Bayer\n\tframe support","submitter":{"id":266,"url":"https://patchwork.libcamera.org/api/people/266/","name":"Max Bretschneider","email":"maxbretschneider@protonmail.com"},"content":"Hello Kieran, \n\nSorry for the oversight and thanks for the hint, one change got split awkwardly when I was rebasing. I have rebased it again using your suggestion, each commit should now compile independently. \n\nRegards, Max. \n\nKieran Bingham schrieb am Montag, 4. Mai 2026 um 9:34 vorm.:\n\n> Hi Max,\n> \n> Thanks for posting these patches - very interesting work.\n> \n> I haven't reviewed the patches yet, but the CI has just run on this\n> series and failed the build-history rule.\n> \n> https://gitlab.freedesktop.org/camera/libcamera/-/jobs/98905270\n> \n> We always maintain a bisectable tree, so each commit has to compile\n> fully and independently.\n> \n> I usually use something like:\n> \"git rebase -i origin/main -x ninja -C build\"\n> \n> which will compile each commit locally for me when I'm testing.\n> \n> --\n> Regards\n> \n> Kieran\n> \n> Quoting Max Bretschneider (2026-05-01 11:52:02)\n> > The virtual pipeline handler currently only produces NV12 output, either\n> > from test patterns or JPEG files. This means that SoftISP, which\n> > processes raw Bayer data, cannot be tested without physical camera\n> > hardware. A first step to achieving this would be to support raw streams\n> > in the virtual pipeline handler. This series introduces a RawFrameGenerator\n> > that reads binary raw Bayer files from disk and serves them through the virtual\n> > pipeline. The introduced changes are opt-in through the YAML configuration. If\n> > virtual cameras are configured with raw_frames, the new code path is\n> > chosen, paths for the existing test_patterns and frames are unaffected.\n> >\n> > The SoftISP part of this patch's motivation is complemented by David Plowman's\n> > series, which introduced the CameraSensorMemory class\n> > (https://patchwork.libcamera.org/patch/25468/), since SoftISP needs an\n> > actual CameraSensor for correct construction. The idea is that in the\n> > future it can be used together to serve Raw frames through the\n> > virtual pipeline and process them with SoftISP, for example for CI tests or more easily reproducible benchmarks.\n> >\n> > Testing: I have used `dd` to create zero filled raw frames to\n> > test things such as format negotiation, buffer sizing, general frame\n> > flow, config parsing (both right/wrong, e.g. rejecting invalid input)\n> > and bit depth handling. I've written a small bash script to essentialy\n> > automate this for all n : { 8, 10, 12, 14, 16 } and the four Bayer\n> > formats in accordance to `src/libcamera/formats.yaml`. I then used numpy to\n> > test the generateFrame() functionality by generating a known frame with\n> > content, capturing it through the pipeline to disk with the cam application\n> > and then reading the output back again to verify that all the pixels match\n> > the expected values.\n> >\n> > I noticed that the pipeline handler components currently have no\n> > dedicated test coverage, if I missed them or if there is some preferred\n> > approach for adding additional testing I'd be open to any suggestions.\n> > For now I can also share my scripts if thats wanted.\n> >\n> > Ran `git clang-format` as well as the `checkstyle.py`. I've run the\n> > Meson tests both with and without the changes and get the same:\n> > Ok: 51\n> > Expected Fail: 1\n> > Fail: 1\n> > Unexpected Pass: 0\n> > Skipped: 29\n> > Timeout: 0\n> >\n> > Open questions:\n> > - The sensor properties set in match() are currently hardcoded, and I've\n> > left them with a \\todo. Especially since we've also discussed adding\n> > controls such as Exposure and AnalogueGain in \"[RFC] Decouple\n> > SoftwareISP from CameraSensor for virtual pipeline testing\": Should I\n> > move these directly to the yaml as well or proceed differently?\n> > - I've written a lambda to derive the pixel format, but at this point it\n> > has gotten quite large. Maybe this could also be a 2D lookup table or\n> > something else that is cleaner. On that lambda also: all cases are\n> > covered I think, but the default case is 16, I wasn't sure if 16\n> > should also be made an explicit case and something more sensible\n> > should be chosen for the default.\n> > - Between the raw_frames and the frames path, the file collection logic\n> > is now also duplicated. Should I add a helper here to change this?\n> > - Currently every frame is read into framesDatas_ immediately at\n> > startup. Depending on the number and the size of the raw files, this\n> > could take up quite a lot of memory. For the use cases I thought of\n> > (e.g. CI) this would be fine, since the number of files would probably\n> > be comparatively small there. Still I wanted to bring it up.\n> > - We weren't sure if the SPDX license header is fine like this. For the\n> > moment I've just placed my intern company mail, is this fine as is?\n> >\n> > Documentation:\n> > I've added documentation according to the coding-style.rst,\n> > if my comments are not sufficient I'd be happy to add more of course.\n> >\n> > AI Disclosure:\n> > I've used DeepWiki to familiarize myself with the libcamera repository\n> > and to point me at reference implementations and patterns to use as orientation.\n> >\n> > Thank you all for your time!\n> >\n> > Max Bretschneider (3):\n> > libcamera: pipeline: virtual: Add RawFrameGenerator\n> > libcamera: pipeline: virtual: Add raw_frames config parser support\n> > libcamera: pipeline: virtual: Support StreamRole::Raw and Bayer\n> > formats\n> >\n> > .../pipeline/virtual/config_parser.cpp | 100 ++++++++++++-\n> > src/libcamera/pipeline/virtual/meson.build | 1 +\n> > .../pipeline/virtual/raw_frame_generator.cpp | 132 ++++++++++++++++++\n> > .../pipeline/virtual/raw_frame_generator.h | 47 +++++++\n> > src/libcamera/pipeline/virtual/virtual.cpp | 128 ++++++++++++++---\n> > src/libcamera/pipeline/virtual/virtual.h | 3 +-\n> > 6 files changed, 387 insertions(+), 24 deletions(-)\n> > create mode 100644 src/libcamera/pipeline/virtual/raw_frame_generator.cpp\n> > create mode 100644 src/libcamera/pipeline/virtual/raw_frame_generator.h\n> >\n> > --\n> > 2.43.0\n> >\n> >\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BE76BBE173\n\tfor ;\n\tMon, 4 May 2026 09:20:03 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id B31286302A;\n\tMon, 4 May 2026 11:20:02 +0200 (CEST)","from mail-4325.protonmail.ch (mail-4325.protonmail.ch\n\t[185.70.43.25])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 76DD463022\n\tfor ;\n\tMon, 4 May 2026 11:20:01 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (2048-bit key;\n\tunprotected) header.d=protonmail.com header.i=@protonmail.com\n\theader.b=\"SLHrloMm\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com;\n\ts=protonmail3; t=1777886399; x=1778145599;\n\tbh=aXR5uCL4K8JkkPV6qaI+g3Z+HARDxyQM8RBcQAKp5Q4=;\n\th=Date:To:From:Subject:Message-ID:In-Reply-To:References:\n\tFeedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID:\n\tMessage-ID:BIMI-Selector;\n\tb=SLHrloMmVACZzuUjICQZw1kbXcsCFSwlvLFOvlk8+WHcNomziaTA0OWEHMm8HKwTB\n\tmQ/bHdanCZgnTTPMU9w1M64bnctPYfHFCP5SLSBdv6BqhY7iRzo+gbVgh9TSZxgA9i\n\txz+qrZZKO6gBiw8OaXnLMSGMg0mCwhjpVouraF7dLmrWlU8lNgnHHFY43E36GbP80q\n\tlByvn4VbQBy84VJEOZmWtH0WacbcmzrqTMVNc8ZpfFPm2LMqUFHEPZ0ENLJ62kCT3O\n\tRwraV5DE7OJnbFDtmAgXE5/YMFXQjtP0vGTSfivl808fGgzwcTL1cuT5WMgJ45J3Ez\n\tFFaMFZ+2TxEMg==","Date":"Mon, 04 May 2026 09:19:53 +0000","To":"Kieran Bingham ,\n\t\"libcamera-devel@lists.libcamera.org\"\n\t","From":"Max Bretschneider ","Subject":"Re: [RFC PATCH v1 0/3] libcamera: pipeline: virtual: Add raw Bayer\n\tframe support","Message-ID":"","In-Reply-To":"<177788003794.3468922.528411873495754823@ping.linuxembedded.co.uk>","References":"<20260501105137.439519-1-maxbretschneider@protonmail.com>\n\t<177788003794.3468922.528411873495754823@ping.linuxembedded.co.uk>","Feedback-ID":"122687743:user:proton","X-Pm-Message-ID":"f9ab5cf855c4b9fca4470a9b54d34c8f949a006d","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Transfer-Encoding":"quoted-printable","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]