{"id":13295,"url":"https://patchwork.libcamera.org/api/1.1/patches/13295/?format=json","web_url":"https://patchwork.libcamera.org/patch/13295/","project":{"id":1,"url":"https://patchwork.libcamera.org/api/1.1/projects/1/?format=json","name":"libcamera","link_name":"libcamera","list_id":"libcamera_core","list_email":"libcamera-devel@lists.libcamera.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"","date":"2021-08-11T06:12:55","name":"[libcamera-devel,RFC,v2,1/4] pipeline: isp: The software ISP-based pipeline handler","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"ec5d919eaa725c5b805b0fb71d9187d03ece9777","submitter":{"id":88,"url":"https://patchwork.libcamera.org/api/1.1/people/88/?format=json","name":"Siyuan Fan","email":"siyuan.fan@foxmail.com"},"delegate":null,"mbox":"https://patchwork.libcamera.org/patch/13295/mbox/","series":[{"id":2336,"url":"https://patchwork.libcamera.org/api/1.1/series/2336/?format=json","web_url":"https://patchwork.libcamera.org/project/libcamera/list/?series=2336","date":"2021-08-11T06:12:54","name":"pipeline: isp: The software ISP module","version":2,"mbox":"https://patchwork.libcamera.org/series/2336/mbox/"}],"comments":"https://patchwork.libcamera.org/api/patches/13295/comments/","check":"pending","checks":"https://patchwork.libcamera.org/api/patches/13295/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id EA3ACC3241\n\tfor ;\n\tWed, 11 Aug 2021 06:13:36 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 16B6B68888;\n\tWed, 11 Aug 2021 08:13:36 +0200 (CEST)","from out203-205-221-173.mail.qq.com (out203-205-221-173.mail.qq.com\n\t[203.205.221.173])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id D68D660261\n\tfor ;\n\tWed, 11 Aug 2021 08:13:29 +0200 (CEST)","from localhost.localdomain ([123.126.82.8])\n\tby newxmesmtplogicsvrszc9.qq.com (NewEsmtp) with SMTP\n\tid 3569928F; Wed, 11 Aug 2021 14:13:22 +0800"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=foxmail.com header.i=@foxmail.com\n\theader.b=\"e+7iYt7R\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=foxmail.com;\n\ts=s201512; t=1628662405;\n\tbh=bKjfddsYLO/vZgypLTtEZ9UOa5oMmmI3W8CdpeQXtD4=;\n\th=From:To:Cc:Subject:Date:In-Reply-To:References;\n\tb=e+7iYt7R2nTM7PybMi1Rwm//paCSPdeeJLb/C1Xr6XKVj4rRt7EWka9bOmuBWQGL8\n\teapo4zRXvyYFNOXlQouUbsswrDDJ9JbavfoGzdOsc9f/IWOk+q/zMDLtL84LAiFjLI\n\tS8Oi/+o7lGxCue0JVofOolrRaB69L2aZQzCMbAxg=","X-QQ-mid":"xmsmtpt1628662404tiw8dybl6","Message-ID":"","X-QQ-XMAILINFO":"OYpbVsTx4C81v8OvB++dVdoZgfaYp5yEeKh9IJvO7g3AYn7XuArdghl0aFh7uo\n\tCQb78NSt7Ofkji0GDVUvgXugAGw5m0ip477dAWz0ZS7WgSZ7OQcgyKtbcCNiy6k1YNqZ5QXD0LfQ\n\tutPdAggbiqnqFpo2yozuMx+Z9dDdPSZBB0HNC+PbDQ7oqrT7LR+y1hp7mxqCO8Dqt9y7rXJ5ST/2\n\t1N7UUguqDX+gwTXvWj0aPQuHuZbab3Dl2u7SMFUThKZLqY56X108LTyJ/yxBBUISiMy29RFpqEzV\n\tlZ9TmW4sQev+sBxelE6QMy6kupHUrvBANzflnPs1Cbg084CAesBGnswj2lUZHcdMLk76uEm+Bai+\n\tH7LMKqaPI9tTAKJ20j0AiVXZTgq1Qet0p7XLz3okQ4s+WPIsLp0/swBKsQ4PMH4LCGNLkLttrMPj\n\taVyIHJaXk7/QF0zYbVBQGZUQF4BIQS1RnApI1BK2Z9gOETkwdjh4RGDc0jiiqFk8ajL1tUGAy8Ht\n\tZYTArmrMu6FLZ4TwA2zDX8wbcYja0dKOHJ1TGefkz1+VBMFAL655HX1XaRMSU9t/G7m0heb4Y0T3\n\toS6KSADp1ZE1O6EJea6YeqNPpGAhx0ZmQ9+0tr82m+19hW1F5AzH5Yx1WqS6ghjDitOVa04if+Pd\n\tDo4Xd1TQXyKxdBuFqeU/O6Cutl4BTOlAHghwbNsbjNt43CCBdkbplurR0scet6s2Rf74TeYs40qE\n\ttvvS+76FIV+I0bpLL0HK4wSI+kueQ6AQsobvgpV1zJqrRlMRU9WeN/R/74jsW+kydXKPNZlWEdSQ\n\tY6yS+dHjXQSIbJ7I4uEE16bz626CAjzXit7xyw2V09mC6HrKMaedpGBKPIpsvRwdYOws79HdkKUo\n\tVs7LTq6Nsyqp9ZZ0beWSmZiS9w9HNdwcOdVIR/Wc6bNpa6Cfon1UlXNrejEi3gyecZJIdmhAzliV\n\thuWhAlKPB46f7tsWK2jVfNREpa8zREm7ex3vQDenU=","From":"Siyuan Fan ","To":"libcamera-devel@lists.libcamera.org","Date":"Wed, 11 Aug 2021 07:12:55 +0100","X-OQ-MSGID":"<20210811061258.7421-2-siyuan.fan@foxmail.com>","X-Mailer":"git-send-email 2.20.1","In-Reply-To":"<20210811061258.7421-1-siyuan.fan@foxmail.com>","References":"<20210811061258.7421-1-siyuan.fan@foxmail.com>","MIME-Version":"1.0","Content-Transfer-Encoding":"8bit","Subject":"[libcamera-devel] [RFC PATCH v2 1/4] pipeline: isp: The software\n\tISP-based pipeline handler","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "},"content":"From: Fan Siyuan \n\n Changes in V2:\n -fix the raw and rgb data flow based queue model in pipeline handler\n -move buffer alloc and thread to ISP class\n -Fill metadata information in dstBuffer\n\nSigned-off-by: Fan Siyuan \n---\n src/libcamera/pipeline/isp/isp.cpp | 306 +++++++++++++++++++++++++++++\n 1 file changed, 306 insertions(+)\n create mode 100644 src/libcamera/pipeline/isp/isp.cpp","diff":"diff --git a/src/libcamera/pipeline/isp/isp.cpp b/src/libcamera/pipeline/isp/isp.cpp\nnew file mode 100644\nindex 00000000..c6b7808c\n--- /dev/null\n+++ b/src/libcamera/pipeline/isp/isp.cpp\n@@ -0,0 +1,306 @@\n+/* SPDX-License-Identifier: LGPL-2.1-or-later */\n+/*\n+ * Copyright (C) 2021, Siyuan Fan \n+ *\n+ * isp.cpp - The software ISP-based pipeline handler\n+ */\n+\n+#include \"../../swisp/isp.h\"\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \n+#include \n+#include \n+#include \n+\n+#include \"libcamera/internal/device_enumerator.h\"\n+#include \"libcamera/internal/media_device.h\"\n+#include \"libcamera/internal/pipeline_handler.h\"\n+#include \"libcamera/internal/v4l2_videodevice.h\"\n+\n+namespace libcamera {\n+\n+LOG_DEFINE_CATEGORY(ISP)\n+\n+class ISPCameraData : public CameraData\n+{\n+public:\n+ ISPCameraData(PipelineHandler *pipe, MediaDevice *media)\n+ : CameraData(pipe), media_(media), video_(nullptr)\n+ {\n+ }\n+\n+ ~ISPCameraData()\n+ {\n+ delete video_;\n+ }\n+\n+ int init();\n+ void bufferReady(FrameBuffer *buffer);\n+ void ISPCompleted(FrameBuffer *buffer);\n+\n+ Stream stream_;\n+ CPU_ISP isp_;\n+ int width_;\n+ int height_;\n+\n+ std::vector> rawBuffers_;\n+ std::vector rawQueueBuffers_;\n+ std::vector rgbQueueBuffers_;\n+\n+ MediaDevice *media_;\n+ V4L2VideoDevice *video_;\n+};\n+\n+\n+class ISPCameraConfiguration : public CameraConfiguration\n+{\n+public:\n+ ISPCameraConfiguration();\n+\n+ Status validate() override;\n+};\n+\n+class PipelineHandlerISP : public PipelineHandler\n+{\n+public:\n+ PipelineHandlerISP(CameraManager *manager);\n+\n+ CameraConfiguration *generateConfiguration(Camera *camera,\n+ const StreamRoles &roles) override;\n+ int configure(Camera *camera, CameraConfiguration *config) override;\n+\n+ int exportFrameBuffers(Camera *camera, Stream *stream,\n+ std::vector> *buffers) override;\n+\n+ int start(Camera *camera, const ControlList *controls) override;\n+ void stop(Camera *camera) override;\n+\n+ int queueRequestDevice(Camera *camera, Request *request) override;\n+\n+ bool match(DeviceEnumerator *enumerator) override;\n+\n+private:\n+ ISPCameraData *cameraData(const Camera *camera)\n+ {\n+ return static_cast(\n+ PipelineHandler::cameraData(camera));\n+ }\n+};\n+\n+ISPCameraConfiguration::ISPCameraConfiguration()\n+ : CameraConfiguration()\n+{\n+}\n+\n+CameraConfiguration::Status ISPCameraConfiguration::validate()\n+{\n+ Status status = Valid;\n+\n+ return status;\n+}\n+\n+PipelineHandlerISP::PipelineHandlerISP(CameraManager *manager)\n+ : PipelineHandler(manager)\n+{\n+}\n+\n+CameraConfiguration *PipelineHandlerISP::generateConfiguration(Camera *camera,\n+ const StreamRoles &roles)\n+{\n+ ISPCameraData *data = cameraData(camera);\n+ CameraConfiguration *config = new ISPCameraConfiguration();\n+\n+ if (roles.empty())\n+ return config;\n+\n+ std::map> v4l2Formats =\n+ data->video_->formats();\n+ std::map> deviceFormats;\n+ std::transform(v4l2Formats.begin(), v4l2Formats.end(),\n+ std::inserter(deviceFormats, deviceFormats.begin()),\n+ [&](const decltype(v4l2Formats)::value_type &format) {\n+ return decltype(deviceFormats)::value_type{\n+ format.first.toPixelFormat(),\n+ format.second\n+ };\n+ });\n+\n+ StreamFormats formats(deviceFormats);\n+ StreamConfiguration cfg(formats);\n+\n+ cfg.pixelFormat = formats::RGB888;\n+ cfg.size = { 640, 480 };\n+ cfg.bufferCount = 4;\n+\n+ config->addConfiguration(cfg);\n+\n+ config->validate();\n+\n+ return config;\n+}\n+\n+int PipelineHandlerISP::configure(Camera *camera, CameraConfiguration *config)\n+{\n+ ISPCameraData *data = cameraData(camera);\n+ StreamConfiguration &cfg = config->at(0);\n+\n+ V4L2VideoDevice::Formats fmts = data->video_->formats();\n+ V4L2PixelFormat v4l2Format = fmts.begin()->first;\n+\n+ V4L2DeviceFormat format = {};\n+ format.fourcc = v4l2Format;\n+ format.size = cfg.size;\n+\n+ data->width_ = format.size.width;\n+ data->height_ = format.size.height;\n+\n+ int ret = data->video_->setFormat(&format);\n+ if (ret)\n+ return ret;\n+\n+ cfg.setStream(&data->stream_);\n+ cfg.stride = format.planes[0].bpl;\n+\n+ return 0;\n+}\n+\n+int PipelineHandlerISP::exportFrameBuffers(Camera *camera, Stream *stream,\n+ std::vector> *buffers)\n+{\n+ unsigned int count = stream->configuration().bufferCount;\n+ ISPCameraData *data = cameraData(camera);\n+\n+ count = data->isp_.exportBuffers(buffers, count, data->width_, data->height_);\n+\n+ return count; \n+\n+}\n+\n+int PipelineHandlerISP::start(Camera *camera, [[maybe_unused]] const ControlList *controls)\n+{\n+ ISPCameraData *data = cameraData(camera);\n+ unsigned int count = data->stream_.configuration().bufferCount;\n+\n+ int ret = data->video_->allocateBuffers(count, &data->rawBuffers_);\n+ if (ret < 0) {\n+ LOG(ISP, Error) << strerror(-ret);\n+ return ret;\n+ }\n+\n+ for (unsigned int i = 0; i < count; i++) {\n+ data->rawQueueBuffers_.push_back(data->rawBuffers_[i].get());\n+ }\n+ \n+\n+ ret = data->video_->streamOn();\n+ if (ret < 0) {\n+ data->video_->releaseBuffers();\n+ return ret;\n+ }\n+\n+ data->isp_.startThreadISP();\n+\n+ return 0;\n+}\n+\n+void PipelineHandlerISP::stop(Camera *camera)\n+{\n+ ISPCameraData *data = cameraData(camera);\n+\n+ if (!(data->rawBuffers_.empty())) {\n+ data->rawBuffers_.clear();\n+ }\n+\n+ data->isp_.stopThreadISP();\n+\n+ data->video_->streamOff();\n+ data->video_->releaseBuffers();\n+}\n+\n+int PipelineHandlerISP::queueRequestDevice(Camera *camera, Request *request)\n+{\n+ ISPCameraData *data = cameraData(camera);\n+ FrameBuffer *rgbBuffer = request->findBuffer(&data->stream_);\n+ data->rgbQueueBuffers_.push_back(rgbBuffer);\n+ \n+ if (!rgbBuffer) {\n+ LOG(ISP, Error) << \"Attempt to queue request with invalid stream\";\n+ return -ENOENT;\n+ }\n+\n+ int ret = data->video_->queueBuffer(data->rawQueueBuffers_[0]);\n+ if (ret < 0)\n+ return ret;\n+ FrameBuffer *temp = data->rawQueueBuffers_[0];\n+ data->rawQueueBuffers_.erase(data->rawQueueBuffers_.begin());\n+ data->rawQueueBuffers_.push_back(std::move(temp));\n+\n+ return 0;\n+}\n+\n+bool PipelineHandlerISP::match(DeviceEnumerator *enumerator)\n+{\n+ DeviceMatch unicam(\"unicam\");\n+\n+ unicam.add(\"unicam-embedded\");\n+ unicam.add(\"unicam-image\");\n+\n+ MediaDevice *unicam_ = acquireMediaDevice(enumerator, unicam);\n+ if (!unicam_) {\n+ LOG(ISP, Debug) << \"unicam Device not found\";\n+ return false;\n+ }\n+\n+ LOG(ISP, Debug) << \"unicam Device Identified\";\n+\n+ std::unique_ptr data = std::make_unique(this, unicam_);\n+\n+ if(data->init()) return false;\n+\n+ std::set streams{&data->stream_};\n+ std::shared_ptr camera = Camera::create(this, data->video_->deviceName(), streams);\n+ registerCamera(std::move(camera), std::move(data));\n+\n+ return true;\n+}\n+\n+\n+void ISPCameraData::ISPCompleted(FrameBuffer *buffer)\n+{\n+ Request *request = buffer->request();\n+\n+ pipe_->completeBuffer(request, buffer);\n+ pipe_->completeRequest(request);\n+\n+}\n+\n+void ISPCameraData::bufferReady(FrameBuffer *buffer)\n+{\n+ LOG(ISP, Debug) << rgbQueueBuffers_[0]->planes()[0].fd.fd();\n+ isp_.invokeMethod(&CPU_ISP::processing, ConnectionTypeQueued, buffer, rgbQueueBuffers_[0], width_, height_);\n+ rgbQueueBuffers_.erase(rgbQueueBuffers_.begin());\n+ rgbQueueBuffers_.shrink_to_fit();\n+ \n+}\n+\n+int ISPCameraData::init()\n+{\n+ video_ = new V4L2VideoDevice(media_->getEntityByName(\"unicam-image\"));\n+ if (video_->open())\n+ return -ENODEV;\n+\n+ video_->bufferReady.connect(this, &ISPCameraData::bufferReady);\n+ isp_.ispCompleted.connect(this, &ISPCameraData::ISPCompleted);\n+\n+ return 0;\n+}\n+\n+REGISTER_PIPELINE_HANDLER(PipelineHandlerISP)\n+\n+} /* namespace libcamera */\n","prefixes":["libcamera-devel","RFC","v2","1/4"]}