From patchwork Wed Aug 4 07:33:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siyuan Fan X-Patchwork-Id: 13189 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 657EEC3236 for ; Wed, 4 Aug 2021 07:34:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7BF8468822; Wed, 4 Aug 2021 09:34:05 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=foxmail.com header.i=@foxmail.com header.b="ADVV/K1S"; dkim-atps=neutral Received: from out162-62-58-211.mail.qq.com (out162-62-58-211.mail.qq.com [162.62.58.211]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id EF45668810 for ; Wed, 4 Aug 2021 09:33:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1628062434; bh=bjp+htc5ywvGucIThHbIlSty4pziM5OG38ZKNJgVSDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ADVV/K1SL0MK94+nrRvwsW5UL0g6HRnprYPnJhOgMXKzJ/uyyvLwAJSQrx7Tx1nQC lJzBqq0lFGkwbc+9vzdp7s9KPW+rIddxidAWCq+Mj/Gcpk/xac+w3kXsOq5aPYF132 95YJ28LwYV2y2DZcB82xYHyw5tvJc2NrSJUuqLJ8= Received: from localhost.localdomain ([123.126.82.9]) by newxmesmtplogicsvrsza5.qq.com (NewEsmtp) with SMTP id 8722689A; Wed, 04 Aug 2021 15:33:50 +0800 X-QQ-mid: xmsmtpt1628062431tgzczbxa0 Message-ID: X-QQ-XMAILINFO: ONalwM1aM/hi0gK1Uu1MxmqXNSAy/G+LXtilntyUerbhZ40hqZbN/ozEWm2PI6 EelJosgv80hynUarzVmM7B4NtoswZoWxVwZMWguTs2lf6L9dUa39jZaMJL1h0m1buS/r9qsxq9ql IJUe//39PcBB7K3zrOpnj8IUk5JnzJaIRhNEQgIYPp/BQ73zNrVp1ngj+gc6dGlVcHUzE4y6icle bjsSiPnrOVOimbqvN3IgdYr9eCybeekfrp4msW+G2muEfcFR6yJt6SPCezRNcIDuaAAlrh91hsEF 4h/BbQzMFbfWNMR3qWWIaCpMNCgo7pDYlHGlk2jhgDArnU3PhSFhmqgRFziQlQKaPsqJ0qdY3IlC PL1Lzyx9v76NTmigxKCQ12jIsbY3heSms0EufSLyhNNnv0JbDfYr073Jk2bVNEoENo0Gp0GNwl15 BUCbQDd8Re54eIfkm62uRot8ivSODsTSbX+8zDPE2X6JZM1YD5mvJXScyFHMI7DQym1OsMa6BA7g 5/J3WTqP+jqu+H0v+ECVYg8co4Ck0f7srHGSK7xPGikqzsVakg3CtG4isEFE0IqJTZb8Zyx7+nAi Fm9sIPVPZWsTdczCT4IfHKvH8VL8qNqQr9wIBARQD3/OVNRybb3RmMFOIpxj2C86q++Sc0MbZjYx is/WDikg1gSvfNcZUsmvLOnoZZ96gajzH3JJgrsTQoTPW5wZWg/5Q3sOJy7LWI0iz7It7AknvRrG GCaVrue4TwlTASxh46KbTGQkjFnABzSR2+ZfnA498rW6xfrCoTtw2QCg4OPuEgfus7zGbx3BFVij jAh3zp8O3OS2CSHwTbqtxRcXRf2hTeRFRbsqzJqiw0hsTj+K+MSrROxzaCFxQWRaIUkEAk6fWX1J Pn/LXnqRQSnpD1v/du/Vmj4CmwviQI0sV1QzwuHgzZSSusZDF0/5nN8o56ej0t0PV9F/uIjSaFE4 c6FR0ZVLZOTprnNGlphseT6DsD2nMvodnauGG35rUC5KdKg4dZjw== From: Siyuan Fan To: libcamera-devel@lists.libcamera.org Date: Wed, 4 Aug 2021 08:33:45 +0100 X-OQ-MSGID: <20210804073347.1368-2-siyuan.fan@foxmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210804073347.1368-1-siyuan.fan@foxmail.com> References: <20210804073347.1368-1-siyuan.fan@foxmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v1 1/3] pipeline: isp: The software ISP-based pipeline handler X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Fan Siyuan Add class PipelineHandlerISP to connect three parts, including application, Camera Sensor and ISP processing. The process is roughly as follows. The application requests the format and size of rgb image(e.x. RGB888 640x480), then pipeline handler configures the format and size of raw image (e.x. BGGR10 640X480) and calls V4L2VideoDevice::allocateBuffers and V4L2VideoDevice::queueBuffer to queue raw buffer. When bufferReady signal is emitted, which means that the raw buffer is filled with data captured by camera sensor. Later, pipeline handler calls the bufferReady function to pass the raw buffer to class isp::processing and the raw buffer is processed by isp pipeline. When processing done, the rgb buffer is exported to application by PipelineHandlerISP::exportFrameBuffers. Signed-off-by: Fan Siyuan --- src/libcamera/pipeline/isp/isp.cpp | 323 +++++++++++++++++++++++++++++ 1 file changed, 323 insertions(+) create mode 100644 src/libcamera/pipeline/isp/isp.cpp diff --git a/src/libcamera/pipeline/isp/isp.cpp b/src/libcamera/pipeline/isp/isp.cpp new file mode 100644 index 00000000..e5fbd536 --- /dev/null +++ b/src/libcamera/pipeline/isp/isp.cpp @@ -0,0 +1,323 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Siyuan Fan + * + * isp.cpp - The software ISP-based pipeline handler + */ + +#include "isp_processing.h" + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "libcamera/base/thread.h" +#include "libcamera/internal/device_enumerator.h" +#include "libcamera/internal/media_device.h" +#include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/v4l2_videodevice.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(ISP) + +class ISPCameraData : public CameraData +{ +public: + ISPCameraData(PipelineHandler *pipe, MediaDevice *media) + : CameraData(pipe), media_(media), video_(nullptr) + { + } + + ~ISPCameraData() + { + delete video_; + delete rawBuffers; + } + + int init(); + void bufferReady(FrameBuffer *buffer); + void ISPCompleted(FrameBuffer *buffer); + + Stream stream_; + ISP isp_; + Thread threadISP_; + int width; + int height; + + MediaDevice *media_; + V4L2VideoDevice *video_; + std::vector> *tempCopyExportBuffers; + std::vector> *rawBuffers; + std::map bufferPair; +}; + + +class ISPCameraConfiguration : public CameraConfiguration +{ +public: + ISPCameraConfiguration(); + + Status validate() override; +}; + +class PipelineHandlerISP : public PipelineHandler +{ +public: + PipelineHandlerISP(CameraManager *manager); + + CameraConfiguration *generateConfiguration(Camera *camera, + const StreamRoles &roles) override; + int configure(Camera *camera, CameraConfiguration *config) override; + + int exportFrameBuffers(Camera *camera, Stream *stream, + std::vector> *buffers) override; + + int start(Camera *camera, const ControlList *controls) override; + void stop(Camera *camera) override; + + int queueRequestDevice(Camera *camera, Request *request) override; + + bool match(DeviceEnumerator *enumerator) override; + +private: + ISPCameraData *cameraData(const Camera *camera) + { + return static_cast( + PipelineHandler::cameraData(camera)); + } +}; + +ISPCameraConfiguration::ISPCameraConfiguration() + : CameraConfiguration() +{ +} + +CameraConfiguration::Status ISPCameraConfiguration::validate() +{ + Status status = Valid; + + return status; +} + +PipelineHandlerISP::PipelineHandlerISP(CameraManager *manager) + : PipelineHandler(manager) +{ +} + +CameraConfiguration *PipelineHandlerISP::generateConfiguration(Camera *camera, + const StreamRoles &roles) +{ + ISPCameraData *data = cameraData(camera); + CameraConfiguration *config = new ISPCameraConfiguration(); + + if (roles.empty()) + return config; + + std::map> v4l2Formats = + data->video_->formats(); + std::map> deviceFormats; + std::transform(v4l2Formats.begin(), v4l2Formats.end(), + std::inserter(deviceFormats, deviceFormats.begin()), + [&](const decltype(v4l2Formats)::value_type &format) { + return decltype(deviceFormats)::value_type{ + format.first.toPixelFormat(), + format.second + }; + }); + + StreamFormats formats(deviceFormats); + StreamConfiguration cfg(formats); + + cfg.pixelFormat = formats::RGB888; + cfg.size = { 640, 480 }; + cfg.bufferCount = 4; + + config->addConfiguration(cfg); + + config->validate(); + + return config; +} + +int PipelineHandlerISP::configure(Camera *camera, CameraConfiguration *config) +{ + ISPCameraData *data = cameraData(camera); + StreamConfiguration &cfg = config->at(0); + + V4L2VideoDevice::Formats fmts = data->video_->formats(); + V4L2PixelFormat v4l2Format = fmts.begin()->first; + + V4L2DeviceFormat format = {}; + format.fourcc = v4l2Format; + format.size = cfg.size; + + data->width = format.size.width; + data->height = format.size.height; + + int ret = data->video_->setFormat(&format); + if (ret) + return ret; + + cfg.setStream(&data->stream_); + cfg.stride = format.planes[0].bpl; + + return 0; +} + +int PipelineHandlerISP::exportFrameBuffers(Camera *camera, Stream *stream, + std::vector> *buffers) +{ + unsigned int count = stream->configuration().bufferCount; + ISPCameraData *data = cameraData(camera); + + for (unsigned int i = 0; i < count; i++) { + std::string name = "frame-" + std::to_string(i); + const int isp_fd = memfd_create(name.c_str(), 0); + int ret = ftruncate(isp_fd, data->width * data->height * 3); + if (ret < 0) { + LOG(ISP, Error) + << "truncate: " + << strerror(-ret); + return ret; + } + FileDescriptor temp = FileDescriptor(std::move(isp_fd)); + + FrameBuffer::Plane rgbPlane; + rgbPlane.fd = std::move(temp); + rgbPlane.length = data->width * data->height * 3; + + std::vector planes; + planes.push_back(std::move(rgbPlane)); + std::unique_ptr buffer = std::make_unique(std::move(planes)); + buffers->push_back(std::move(buffer)); + } + + data->tempCopyExportBuffers = buffers; + + return count; + +} + +int PipelineHandlerISP::start(Camera *camera, [[maybe_unused]] const ControlList *controls) +{ + ISPCameraData *data = cameraData(camera); + unsigned int count = data->stream_.configuration().bufferCount; + + data->rawBuffers = new std::vector>; + int ret = data->video_->allocateBuffers(count, data->rawBuffers); + if (ret < 0) { + LOG(ISP, Error) << strerror(-ret); + return ret; + } + + for (unsigned int i = 0; i < count; i++) + data->bufferPair[data->tempCopyExportBuffers->at(i).get()] = data->rawBuffers->at(i).get(); + + ret = data->video_->streamOn(); + if (ret < 0) { + data->video_->releaseBuffers(); + return ret; + } + + data->threadISP_.start(); + + return 0; +} + +void PipelineHandlerISP::stop(Camera *camera) +{ + ISPCameraData *data = cameraData(camera); + + data->threadISP_.exit(); + data->threadISP_.wait(); + + data->video_->streamOff(); + data->video_->releaseBuffers(); +} + +int PipelineHandlerISP::queueRequestDevice(Camera *camera, Request *request) +{ + ISPCameraData *data = cameraData(camera); + FrameBuffer *rgbBuffer = request->findBuffer(&data->stream_); + + if (!rgbBuffer) { + LOG(ISP, Error) << "Attempt to queue request with invalid stream"; + return -ENOENT; + } + + int ret = data->video_->queueBuffer(data->bufferPair[rgbBuffer]); + if (ret < 0) + return ret; + + return 0; +} + +bool PipelineHandlerISP::match(DeviceEnumerator *enumerator) +{ + DeviceMatch unicam("unicam"); + + unicam.add("unicam-embedded"); + unicam.add("unicam-image"); + + MediaDevice *unicam_ = acquireMediaDevice(enumerator, unicam); + if (!unicam_) { + LOG(ISP, Debug) << "unicam Device not found"; + return false; + } + + LOG(ISP, Debug) << "unicam Device Identified"; + + std::unique_ptr data = std::make_unique(this, unicam_); + + if(data->init()) return false; + + std::set streams{&data->stream_}; + std::shared_ptr camera = Camera::create(this, data->video_->deviceName(), streams); + registerCamera(std::move(camera), std::move(data)); + + return true; +} + + +void ISPCameraData::ISPCompleted(FrameBuffer *buffer) +{ + Request *request = buffer->request(); + + pipe_->completeBuffer(request, buffer); + pipe_->completeRequest(request); + +} + +void ISPCameraData::bufferReady(FrameBuffer *buffer) +{ + for (std::map::iterator it = bufferPair.begin(); it != bufferPair.end(); it++) { + if (it->second == buffer) + isp_.invokeMethod(&ISP::processing, ConnectionTypeQueued, buffer, it->first, width, height); + } + +} + +int ISPCameraData::init() +{ + video_ = new V4L2VideoDevice(media_->getEntityByName("unicam-image")); + if (video_->open()) + return -ENODEV; + + video_->bufferReady.connect(this, &ISPCameraData::bufferReady); + + isp_.moveToThread(&threadISP_); + isp_.ispCompleted.connect(this, &ISPCameraData::ISPCompleted); + + return 0; +} + +REGISTER_PIPELINE_HANDLER(PipelineHandlerISP) + +} /* namespace libcamera */ From patchwork Wed Aug 4 07:33:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siyuan Fan X-Patchwork-Id: 13190 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1450EC3235 for ; Wed, 4 Aug 2021 07:34:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DE6B168888; Wed, 4 Aug 2021 09:34:05 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=foxmail.com header.i=@foxmail.com header.b="hlsL2Qzz"; dkim-atps=neutral Received: from out162-62-57-87.mail.qq.com (out162-62-57-87.mail.qq.com [162.62.57.87]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E24AC6880F for ; Wed, 4 Aug 2021 09:33:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1628062436; bh=cDfBCdx3KpPINQMCVlKW7BBE9zwzOZtJCpQ0x30VTuk=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hlsL2QzzK/f8rmndVqFkqUcEuZpskrihaewpQWTO/sJ5X7XQaEz7ohGOEGIQSP41A RfQR5mwDmd71AxhUrD9+WSUAMh1RN9f9LFfOM1/S9mXPqAbK5Ml8NhBjK29eO+5bGE ZalpwYCFinVkOAKlecG5XLc3oVXrSlpxrDB8av/U= Received: from localhost.localdomain ([123.126.82.9]) by newxmesmtplogicsvrsza5.qq.com (NewEsmtp) with SMTP id 8722689A; Wed, 04 Aug 2021 15:33:50 +0800 X-QQ-mid: xmsmtpt1628062433t8qqkh1qy Message-ID: X-QQ-XMAILINFO: NOIkHYnr7VzdUfjOSQ3+N4cGBQZfJu/bkiCYSdew0OTWpOt/XF/zQ+2atIL2kM RGbBxeaoAoaqfoR11xZ60/2ixO8qog6BLpGNOK6+0J78Qd2Ko50EVeoBooEMfJEmZZoukE1exdif VjOYN/JNyJIbCfJxpXcHBo/Ox2MCyOJmTpfRdv4U5fosXXtx3f8WE1HbKqNDeqSzXMxc3bNINdUo xlahi6k8MgWyB9DTbvV65ktsaEdSobubtpOVKNFzAHSgvATWhrP1LhDm3Tb3euddRtBHPc411yyC 9seUn5IyrbVSZIISmIHMjtDhplOM9bop8pZydw86ldFLpMuyQYyqRYGt7pTxYZNto4qTMHxSL0pX aECtFn7Y0TO5+3VSgde+l3nRGNouWI883ggYBtUZIuY2Hi/mevRNJT4AZ/8oCwpnPfBKREIv/bxv 9AK3UABBcQeXflZDk0nDufPsdxwmWUElXnnN/dZCY/RLZxkvbQ7KEN/BJur76L2TDpXqZW9O2WqW l9Nxx6SP4e0Yb8A5cazfTvVS3FQ1uk9ojg2qsiJbXv4Di73aBGatX72/9obDG4HirPZ6sAWaBnqQ ozOfUGF3jaJngGcMTkYP6Z5GJJh//yLTUJK99VVXgdreroKjXKAeAodhL84W6HUyx+6yhDGxtZgm rD42cXlUB0aaisB0l24BJhodNPphsH0Aap5ylJu4ACWZHOYBnL47SIMxYgZt980MQ2pcvfMc5Uin wy0PAgcsJWtORCNmWVFVIN3Ldm1ht16ouGS2YLTQbKZH8EaJUMSQzOckpPQGiXgoJbOPMRliarG/ wX+FAjW2ke4PViy7r7ErtcvTTZq6KLceXbbOVif0HA0A7lGXnMBx5/IfSVRaa+nz/fl6+F7ujAPf 65LpT1/qVdWke4GP5Q14QzVwPaqdD3Qr15wdsxsARIK6MJRWDSXmGuwI9TUfebufgWdVVsyM6xAT p/+AGwEhgYSkohI93r0sskoeKhmBicyplbEbxg9qyJamKi6d/aMg== From: Siyuan Fan To: libcamera-devel@lists.libcamera.org Date: Wed, 4 Aug 2021 08:33:46 +0100 X-OQ-MSGID: <20210804073347.1368-3-siyuan.fan@foxmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210804073347.1368-1-siyuan.fan@foxmail.com> References: <20210804073347.1368-1-siyuan.fan@foxmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v1 2/3] pipeline: isp: The software ISP class X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Fan Siyuan Add the software ISP class only CPU-based. So far the basic alogrithm includes black level correct, bilinear demosaic interpolation, auto white balance(gray world), tone mapping(auto contrast), gamma correct. bilateral filter, 10bit to 8bit compression. Signed-off-by: Fan Siyuan --- src/libcamera/pipeline/isp/isp_processing.cpp | 593 ++++++++++++++++++ src/libcamera/pipeline/isp/isp_processing.h | 62 ++ 2 files changed, 655 insertions(+) create mode 100644 src/libcamera/pipeline/isp/isp_processing.cpp create mode 100644 src/libcamera/pipeline/isp/isp_processing.h diff --git a/src/libcamera/pipeline/isp/isp_processing.cpp b/src/libcamera/pipeline/isp/isp_processing.cpp new file mode 100644 index 00000000..94e8081d --- /dev/null +++ b/src/libcamera/pipeline/isp/isp_processing.cpp @@ -0,0 +1,593 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Siyuan Fan + * + * isp_processing.cpp - The software ISP class + */ + +#include "isp_processing.h" + +#include +#include +#include +#include +#include + +#include + +#include "libcamera/base/log.h" + +namespace libcamera{ + +LOG_DECLARE_CATEGORY(ISP) + +void ISP::autoContrast(uint16_t *data, float lowCut, float highCut, int width, int height) +{ + int blue, gr, gb, red; + int histBlue[1024] = {0}, histGb[1024] = {0}, histGr[1024] = {0}, histRed[1024] = {0}; + + int index; + for (int i = 0; i < height; i++) { + index = i * width; + for (int j = 0; j < width; j++) { + if (i % 2 == 0 && j % 2 == 0) { + blue = data[index]; + histBlue[blue]++; + } + else if ((i % 2 == 0 && j % 2 == 1)) { + gb = data[index]; + histGb[gb]++; + } + else if ((i % 2 == 1 && j % 2 == 0)) { + gr = data[index]; + histGr[gr]++; + } + else { + red = data[index]; + histRed[red]++; + } + index++; + } + } + + int pixelAmount = width * height; + int sum = 0; + int minBlue; + for (int i = 0; i < 1024; i++){ + sum = sum + histBlue[i]; + if (sum >= pixelAmount * lowCut * 0.01) { + minBlue = i; + break; + } + } + + sum = 0; + int maxBlue; + for (int i = 1023; i >= 0; i--){ + sum = sum + histBlue[i]; + if (sum >= pixelAmount * highCut * 0.01) { + maxBlue = i; + break; + } + } + + sum = 0; + int minGb; + for (int i = 0; i < 1024; i++){ + sum = sum + histGb[i]; + if (sum >= pixelAmount * lowCut * 0.01) { + minGb = i; + break; + } + } + + sum = 0; + int maxGb; + for (int i = 1023; i >= 0; i--){ + sum = sum + histGb[i]; + if (sum >= pixelAmount * highCut * 0.01) { + maxGb = i; + break; + } + } + + sum = 0; + int minGr; + for (int i = 0; i < 1024; i++){ + sum = sum + histGr[i]; + if (sum >= pixelAmount * lowCut * 0.01) { + minGr = i; + break; + } + } + + sum = 0; + int maxGr; + for (int i = 1023; i >= 0; i--){ + sum = sum + histGr[i]; + if (sum >= pixelAmount * highCut * 0.01) { + maxGr = i; + break; + } + } + + sum = 0; + int minRed; + for (int i = 0; i < 1024; i++){ + sum = sum + histRed[i]; + if (sum >= pixelAmount * lowCut * 0.01) { + minRed = i; + break; + } + } + + sum = 0; + int maxRed; + for (int i = 1023; i >= 0; i--){ + sum = sum + histRed[i]; + if (sum >= pixelAmount * highCut * 0.01) { + maxRed = i; + break; + } + } + + int blueMap[1024]; + float norb = 1.0 / (maxBlue - minBlue); + for (int i = 0; i < 1024; i++) { + if (i < minBlue) { + blueMap[i] = 0; + } + else if (i > maxBlue) { + blueMap[i] = 1023; + } + else { + blueMap[i] = (i - minBlue) * norb * 1023; + } + if (blueMap[i] > 1023) blueMap[i] = 1023; + } + + int gbMap[1024]; + float norgb = 1.0 / (maxGb - minGb); + for (int i = 0; i < 1024; i++) { + if (i < minGb) { + gbMap[i] = 0; + } + else if (i > maxGb) { + gbMap[i] = 1023; + } + else { + gbMap[i] = (i - minGb) * norgb * 1023; + } + if (gbMap[i] > 1023) gbMap[i] = 1023; + } + + int grMap[1024]; + float norgr = 1.0 / (maxGr - minGr); + for (int i = 0; i < 1024; i++) { + if (i < minGr) { + grMap[i] = 0; + } + else if (i > maxGr) { + grMap[i] = 1023; + } + else { + grMap[i] = (i - minGr) * norgr * 1023; + } + if (grMap[i] > 1023) grMap[i] = 1023; + } + + int redMap[1024]; + float norr = 1.0 / (maxRed - minRed); + for (int i = 0; i < 1024; i++) { + if (i < minRed) { + redMap[i] = 0; + } + else if (i > maxRed) { + redMap[i] = 1023; + } + else{ + redMap[i] = (i - minRed) * norr * 1023; + } + if (redMap[i] > 1023) redMap[i] = 1023; + } + + for (int i = 0;i < height; i++) { + for (int j = 0; j < width; j++){ + index = i * width; + if (i % 2 == 0 && j % 2 == 0) { + data[index] = blueMap[data[index]]; + } + else if (i % 2 == 0 && j % 2 == 1) { + data[index] = gbMap[data[index]]; + } + else if (i % 2 == 1 && j % 2 == 0) { + data[index] = grMap[data[index]]; + } + else { + data[index] = redMap[data[index]]; + } + index++; + } + } +} + +void ISP::blackLevelCorrect(uint16_t *data, uint16_t offset, int width, int height) +{ + int len = width * height; + for(int i = 0; i < len; i++) { + if (data[i] < offset){ + data[i] = 0; + } + else { + data[i] -= offset; + } + } +} + +void ISP::readChannels(uint16_t *data, uint16_t *R, uint16_t *G, uint16_t *B, + int width, int height) +{ + int index; + for (int i = 0; i < height; i++) { + index = i * width; + for (int j = 0; j < width; j++) { + if (i % 2 == 0 && j % 2 == 0) { + B[index] = data[index]; + } + else if ((i % 2 == 0 && j % 2 == 1) || (i % 2 == 1 && j % 2 == 0)){ + G[index] = data[index]; + } + else { + R[index] = data[index]; + } + index++; + } + } +} + +void ISP::firstPixelInsert(uint16_t *src, uint16_t *dst, int width, int height) +{ + int index; + for (int i = 0; i < height; i++) { + index = i * width; + for (int j = 0; j < width; j++){ + if (i % 2 == 0 && j % 2 == 1) { + if (j == (width - 1)) { + dst[index] = src[index - 1]; + } + else { + dst[index] = (src[index - 1] + + src[index + 1]) >> 1; + } + } + + if (i % 2 == 1 && j % 2 == 0) { + if(i == height - 1) { + dst[index] = src[index - width]; + } + else { + dst[index] = (src[index - width]+ + src[index + width]) >> 1; + } + } + + if (i % 2 == 1 && j % 2 == 1) { + if (j < width - 1 && i < height - 1) { + dst[index] = (src[index - width - 1] + + src[index - width + 1] + + src[index + width - 1] + + src[index + width + 1]) >> 2; + } + else if (i == height - 1 && j < width - 1) { + dst[index] = (src[index - width - 1] + + src[index - width + 1]) >> 1; + } + else if (i < height - 1 && j == width - 1) { + dst[index] = (src[index - width - 1] + + src[index + width - 1]) >> 1; + } + else { + dst[index] = src[index - width - 1]; + } + } + index++; + } + } +} + +void ISP::twoPixelInsert(uint16_t *src, uint16_t *dst, int width, int height) +{ + int index; + for (int i = 0; i < height; i++) { + index = i * width; + for (int j = 0; j < width; j++) { + if (i == 0 && j == 0) { + dst[index] = (src[index + width] + + src[index + 1]) >> 1; + } + else if (i == 0 && j > 0 && j % 2 == 0) { + dst[index] = (src[index - 1] + + src[index + width] + + src[index + 1]) / 3; + } + else if (i > 0 && j == 0 && i % 2 == 0) { + dst[index] = (src[index - width] + + src[index + 1] + + src[index + width]) / 3; + } + else if (i == (height - 1) && j < (width - 1) && j % 2 == 1) { + dst[index] = (src[index - 1] + + src[index - width] + + src[index + 1]) / 3; + } + else if (i < (height - 1) && j == (width - 1) && i % 2 == 1) { + dst[index] = (src[index - width] + + src[index - 1] + + src[index + width]) / 3; + } + else if (i == (height - 1) && j == (width - 1)) { + dst[index] = (src[index - width] + + src[index - 1]) >> 1; + } + else if ((i % 2 == 0 && j % 2 == 0) || (i % 2 == 1 && j % 2 == 1)) { + dst[index] = (src[index - 1] + + src[index + 1] + + src[index - width] + + src[index + width]) / 4; + } + index++; + } + } +} + +void ISP::lastPixelInsert(uint16_t *src, uint16_t *dst, int width, int height) +{ + int index; + for (int i = 0; i < height; i++) { + index = i * width; + for (int j = 0; j < width; j++){ + if (i % 2 == 1 && j % 2 == 0) { + if (j == 0) { + dst[index] = src[index + 1]; + } + else { + dst[index] = (src[index - 1] + + src[index + 1]) >> 1; + } + } + + if (i % 2 == 0 && j % 2 == 1) { + if(i == 0) { + dst[index] = src[index + width]; + } + else { + dst[index] = (src[index - width]+ + src[index + width]) >> 1; + } + } + + if (i % 2 == 0 && j % 2 == 0) { + if (i > 0 && j > 0) { + dst[index] = (src[index - width - 1] + + src[index - width + 1] + + src[index + width - 1] + + src[index + width + 1]) >> 2; + } + else if (i == 0 && j > 0) { + dst[index] = (src[index + width - 1] + + src[index + width + 1]) >> 1; + } + else if (i > 0 && j == 0) { + dst[index] = (src[index - width + 1] + + src[index + width + 1]) >> 1; + } + else { + dst[index] = src[index + width + 1]; + } + } + index++; + } + } +} + +void ISP::demosaic(uint16_t *data, uint16_t *R, uint16_t *G, uint16_t *B, + int width, int height) +{ + firstPixelInsert(data, B, width, height); + twoPixelInsert(data, G, width, height); + lastPixelInsert(data, R, width, height); +} + +void ISP::autoWhiteBalance(uint16_t *R, uint16_t *G, uint16_t *B, int width, int height) +{ + float aveB = 0, aveG = 0, aveR = 0; + float Kb, Kg, Kr; + + for (int i = 0; i < width * height; i++) { + aveB += 1.0 * B[i]; + aveG += 1.0 * G[i]; + aveR += 1.0 * R[i]; + } + + aveB *= (1.0 / (width * height)); + aveG *= (1.0 / (width * height)); + aveR *= (1.0 / (width * height)); + + Kr = (aveB + aveG + aveR) / aveR * (1.0 / 3.0); + Kg = (aveB + aveG + aveR) / aveG * (1.0 / 3.0); + Kb = (aveB + aveG + aveR) / aveB * (1.0 / 3.0); + + for (int i = 0; i < width * height; i++) { + B[i] = B[i] * Kb; + G[i] = G[i] * Kg; + R[i] = R[i] * Kr; + + if (R[i] > 1023) R[i] = 1023; + if (G[i] > 1023) G[i] = 1023; + if (R[i] > 1023) B[i] = 1023; + } +} + +void ISP::gammaCorrect(uint16_t *R, uint16_t *G, uint16_t *B, float val, + int width, int height) +{ + float nor = 1.0 / 1023.0; + float gamma = 1.0 / val; + for (int i = 0; i < width * height; i++) { + R[i] = pow(R[i] * nor, gamma) * 1023; + G[i] = pow(G[i] * nor, gamma) * 1023; + B[i] = pow(B[i] * nor, gamma) * 1023; + + if (R[i] > 1023) R[i] = 1023; + if (G[i] > 1023) G[i] = 1023; + if (B[i] > 1023) B[i] = 1023; + } +} + +void ISP::compress_10bit_to_8bit (uint16_t *src, uint8_t *dst, int width, int height) +{ + for (int i = 0; i < width * height; i++) { + dst[i] = src[i] >> 2 & 0xff; + } +} + +float ISP::distance(int x, int y, int i, int j) +{ + return float(sqrt(pow(x - i, 2) + pow(y - j, 2))); +} + +double ISP::gaussian(float x, double sigma) +{ + return exp(-(pow(x, 2)) / (2 * pow(sigma, 2))) / (2 * 3.1415926 * pow(sigma, 2)); +} + +void ISP::bilateralFilter(uint16_t *R, uint16_t *G, uint16_t *B, + int diameter, double sigmaI, + double sigmaS, int width, int height) +{ + for (int i = 2; i < height - 2; i++) { + for (int j = 2; j < width - 2; j++) { + double iFiltered = 0; + double wp = 0; + int neighbor_x = 0; + int neighbor_y = 0; + int half = diameter / 2; + + for (int k = 0; k < diameter; k++) { + for (int l = 0; l < diameter; l++) { + neighbor_x = i - (half - k); + neighbor_y = j - (half - l); + double gi = gaussian(R[neighbor_x * width + neighbor_y] - R[i * width +j], sigmaI); + double gs = gaussian(distance(i, j, neighbor_x, neighbor_y), sigmaS); + double w = gi * gs; + iFiltered = iFiltered + R[neighbor_x * width + neighbor_y] * w; + wp = wp + w; + } + } + + iFiltered = iFiltered / wp; + R[i * width + j] = iFiltered; + } + } + + for (int i = 2; i < height - 2; i++) { + for (int j = 2; j < width - 2; j++) { + double iFiltered = 0; + double wp = 0; + int neighbor_x = 0; + int neighbor_y = 0; + int half = diameter / 2; + + for (int k = 0; k < diameter; k++) { + for (int l = 0; l < diameter; l++) { + neighbor_x = i - (half - k); + neighbor_y = j - (half - l); + double gi = gaussian(G[neighbor_x * width + neighbor_y] - G[i * width +j], sigmaI); + double gs = gaussian(distance(i, j, neighbor_x, neighbor_y), sigmaS); + double w = gi * gs; + iFiltered = iFiltered + G[neighbor_x * width + neighbor_y] * w; + wp = wp + w; + } + } + + iFiltered = iFiltered / wp; + G[i * width + j] = iFiltered; + } + } + + for (int i = 2; i < height - 2; i++) { + for (int j = 2; j < width - 2; j++) { + double iFiltered = 0; + double wp = 0; + int neighbor_x = 0; + int neighbor_y = 0; + int half = diameter / 2; + + for (int k = 0; k < diameter; k++) { + for (int l = 0; l < diameter; l++) { + neighbor_x = i - (half - k); + neighbor_y = j - (half - l); + double gi = gaussian(B[neighbor_x * width + neighbor_y] - B[i * width +j], sigmaI); + double gs = gaussian(distance(i, j, neighbor_x, neighbor_y), sigmaS); + double w = gi * gs; + iFiltered = iFiltered + B[neighbor_x * width + neighbor_y] * w; + wp = wp + w; + } + } + + iFiltered = iFiltered / wp; + B[i * width + j] = iFiltered; + } + } +} + +void ISP::processing(FrameBuffer *srcBuffer, FrameBuffer *dstBuffer, int width, int height) +{ + uint8_t *rgb_buf; + uint16_t *rawData; + uint16_t *rgbData = new std::uint16_t[width * height * 3]; + + uint16_t *rData = rgbData; + uint16_t *gData = rData + width * height; + uint16_t *bData = gData + width * height; + memset(rgbData, 0x0, width * height * 3); + + const FrameBuffer::Plane &plane = srcBuffer->planes()[0]; + rawData = (uint16_t *)mmap(NULL, plane.length, PROT_READ|PROT_WRITE, MAP_SHARED, plane.fd.fd(), 0); + if (rawData == MAP_FAILED) { + LOG(ISP, Error) << "Read raw data failed"; + ispCompleted.emit(dstBuffer); + } + + blackLevelCorrect(rawData, 16, width, height); + readChannels(rawData, rData, gData, bData, width, height); + demosaic(rawData, rData, gData, bData, width, height); + autoWhiteBalance(rData, gData, bData, width, height); + autoContrast(rData, 0.01, 0.01, width, height); + autoContrast(gData, 0.01, 0.01, width, height); + autoContrast(bData, 0.01, 0.01, width, height); + gammaCorrect(rData, gData, bData, 2.2, width, height); + //bilateralFilter(rData, gData, bData, 5, 24.0, 32.0); + + const FrameBuffer::Plane &rgbPlane = dstBuffer->planes()[0]; + rgb_buf = (uint8_t *)mmap(NULL, rgbPlane.length, PROT_READ|PROT_WRITE, MAP_SHARED, rgbPlane.fd.fd(), 0); + if (rgb_buf == MAP_FAILED) { + LOG(ISP, Error) << "Read rgb data failed"; + ispCompleted.emit(dstBuffer); + } + + uint8_t *rData_8 = rgb_buf; + uint8_t *gData_8 = rData_8 + width * height; + uint8_t *bData_8 = gData_8 + width * height; + + compress_10bit_to_8bit(rData, rData_8, width, height); + compress_10bit_to_8bit(gData, gData_8, width, height); + compress_10bit_to_8bit(bData, bData_8, width, height); + + delete[] rgbData; + + ispCompleted.emit(dstBuffer); +} + +} /* namespace libcamera */ + diff --git a/src/libcamera/pipeline/isp/isp_processing.h b/src/libcamera/pipeline/isp/isp_processing.h new file mode 100644 index 00000000..b80b4218 --- /dev/null +++ b/src/libcamera/pipeline/isp/isp_processing.h @@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2021, Siyuan Fan + * + * isp_processing.h - The software ISP class + */ +#ifndef __LIBCAMERA_PIPELINE_ISP_PROCESSING_H__ +#define __LIBCAMERA_PIPELINE_ISP_PROCESSING_H__ + +#include + +#include "libcamera/base/signal.h" +#include "libcamera/base/object.h" + +namespace libcamera{ + +using std::uint16_t; +using std::uint8_t; + +class ISP : public Object +{ +public: + void autoContrast(uint16_t *data, float lowCut, float highCut, int width, int height); + + void blackLevelCorrect(uint16_t *data, uint16_t offset, int width, int height); + + void readChannels(uint16_t *data, uint16_t *R, uint16_t *G, uint16_t *B, + int width, int height); + + void firstPixelInsert(uint16_t *src, uint16_t *dst, int width, int height); + + void twoPixelInsert(uint16_t *src, uint16_t *dst, int width, int height); + + void lastPixelInsert(uint16_t *src, uint16_t *dst, int width, int height); + + void demosaic(uint16_t *data, uint16_t *R, uint16_t *G, uint16_t *B, + int width, int height); + + void autoWhiteBalance(uint16_t *R, uint16_t *G, uint16_t *B, int width, int height); + + void gammaCorrect(uint16_t *R, uint16_t *G, uint16_t *B, float val, + int width, int height); + + float distance(int x, int y, int i, int j); + + double gaussian(float x, double sigma); + + void bilateralFilter(uint16_t *R, uint16_t *G, uint16_t *B, + int diameter, double sigmaI, double sigmaS, + int width, int height); + + void compress_10bit_to_8bit(uint16_t *src, uint8_t *dst, int width, int height); + + void processing(FrameBuffer *srcBuffer, FrameBuffer *dstBuffer, int width, int height); + + Signal ispCompleted; + +}; + +} + +#endif /* __LIBCAMERA_PIPELINE_ISP_PROCESSING_H__ */ From patchwork Wed Aug 4 07:33:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siyuan Fan X-Patchwork-Id: 13191 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3FF13C3237 for ; Wed, 4 Aug 2021 07:34:07 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 401AD6884E; Wed, 4 Aug 2021 09:34:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=foxmail.com header.i=@foxmail.com header.b="QkNcMF+R"; dkim-atps=neutral Received: from out203-205-251-59.mail.qq.com (out203-205-251-59.mail.qq.com [203.205.251.59]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 1CB3D6880F for ; Wed, 4 Aug 2021 09:34:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com; s=s201512; t=1628062437; bh=/5+11qVgW1w/IzBqZLx5eQf/lveJkC5V3kg/uDJDd4I=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=QkNcMF+RE3mpIvAsQzHfHWe5AK6AmghUgjP6CIMdQZDf47yel8yGG/C2EVoUw74s6 3vA4U0iH+bPJkNImPIiy3UdzMCs9rbetWjFn7jqHFw+y5Tq00D2J4rmjgZWWP+NxWQ iGjphWQj+Cq26JavdyWrtAZAYu2o92NXHFJS+cxE= Received: from localhost.localdomain ([123.126.82.9]) by newxmesmtplogicsvrsza5.qq.com (NewEsmtp) with SMTP id 8722689A; Wed, 04 Aug 2021 15:33:50 +0800 X-QQ-mid: xmsmtpt1628062435tkgmlc0bz Message-ID: X-QQ-XMAILINFO: OUyBsK7uGFiXg3G5qi7CWQjGbNlwm4Goo20Kqkqvvk5whp6YLbr0V5CrouH8JE kyHFqt5wCkdAICX+YG2uBj7XEPceHq+yx87WY9WEyFlOmwkFulbZIg/vYctgFtUWCyKidOZJQ+Bc f11OyEtxlc/GRe/o1PyD9jUH9f9ls0L021HLyZ6Id1tuIVoVd4D3BFa1cc9Z22AOGs36bCwF3FUM Q1rqcXgAFlIP4X0KA0/8XcrdJtrAaXrrLykinL3OvbHpkw6WKkbYmfBLsqtYDoTg0n2rBAmqwFVM vP4ZYzhyCj3+IhGdGy9Dd58oVhlhMHgW1pXR/6WEL7z8oDJaeXDGfCQkXnQqFF5U37YL3yzCH84A XPUUV3+u4/R8faehNvnLC6Shoi4IZZJdsU762HJYrYGoPx9bYD0/HqLvTM/x7BFN0fRLjuymA4P3 YptPopOb08zgqJHhsd1/F5utjx2iqjKPK6FMGJz5estyedv9OfakdSzs978z9vnBvEw3plrAI37B V1b/vMSkBl4KoeCKrLL3Uxn+XhT/1UJ30NQ7BLgIoLwoHl3V8ji6xoayvd9/ZZpFsPHf8s2YTN3p LZX4JbxCRrUqS9fRDRtbhcJ4O0RnPohczFf7ZbRaWe9JIIztipA/Ilwz10352LZpLt8D2tGz4DB5 OFfD8zx1AaPpYoIQ9tZYuvbbXnVIJREJW0OyQWj6kdYYAnfSB0v1lk94aDSphslUXVFYx0NHw3kF F3B6kyOnpoyOiJR6P9V7rdN9klb9J3Jl2sbG63C1e2+Xf+a5H5eSP1C3t6YkRrX9An8m0ZvPTseF 8orVWZ8kpoDU870hKh4aUKHkKkzfrO5zb9Jhkk9ewJ3c3tCsEbYap8prWN+AysKTy4mzl0Dbhe0S URvx8EEsLnX6dTxFNpRwKjqmrihCSTDLesZ3Gqg0RLrRcders3nSxFT5r6KDwwEpgF8g5xrvnD From: Siyuan Fan To: libcamera-devel@lists.libcamera.org Date: Wed, 4 Aug 2021 08:33:47 +0100 X-OQ-MSGID: <20210804073347.1368-4-siyuan.fan@foxmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210804073347.1368-1-siyuan.fan@foxmail.com> References: <20210804073347.1368-1-siyuan.fan@foxmail.com> MIME-Version: 1.0 Subject: [libcamera-devel] [RFC PATCH v1 3/3] pipeline: isp: All meson configure files X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" From: Fan Siyuan Add all meson configure file the software ISP module needs. After configuration completed, we can enable the software ISP. Signed-off-by: Fan Siyuan --- meson_options.txt | 2 +- src/libcamera/pipeline/isp/meson.build | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) create mode 100644 src/libcamera/pipeline/isp/meson.build diff --git a/meson_options.txt b/meson_options.txt index 2c80ad8b..03110587 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -37,7 +37,7 @@ option('lc-compliance', option('pipelines', type : 'array', - choices : ['ipu3', 'raspberrypi', 'rkisp1', 'simple', 'uvcvideo', 'vimc'], + choices : ['ipu3', 'raspberrypi', 'rkisp1', 'simple', 'uvcvideo', 'vimc', 'isp'], description : 'Select which pipeline handlers to include') option('qcam', diff --git a/src/libcamera/pipeline/isp/meson.build b/src/libcamera/pipeline/isp/meson.build new file mode 100644 index 00000000..562b83b5 --- /dev/null +++ b/src/libcamera/pipeline/isp/meson.build @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: CC0-1.0 + +libcamera_sources += files([ + 'isp.cpp', + 'isp_processing.cpp', +])