From patchwork Mon Sep 27 12:33:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Plowman X-Patchwork-Id: 13953 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EE36DBDC71 for ; Mon, 27 Sep 2021 12:33:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2B4C46919B; Mon, 27 Sep 2021 14:33:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=raspberrypi.com header.i=@raspberrypi.com header.b="G1lB8Gg/"; dkim-atps=neutral Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id D6FD569191 for ; Mon, 27 Sep 2021 14:33:33 +0200 (CEST) Received: by mail-wr1-x430.google.com with SMTP id t18so51726229wrb.0 for ; Mon, 27 Sep 2021 05:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raspberrypi.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s644ahboex6q+shoMOYNj6OgxWUg8OUuJkbKuV6OFsQ=; b=G1lB8Gg/8hcjY8nLI1dJLiELnZQ/kvLaryfinkAROrfZrLpvpacj41dxA8RoJt7Ej9 7D2X/W4pJHpHj9zaWjekOpmjHVo76HcV9r2vkVDGxelSPDwJ6V7PLJfEa2TKbN4xnzZv 637hkUvhxZ5+9KFai64AUweAxUjm6iFEKAHgINfFbici0Els+yR1DP90tB3f7AqTPoHf O8vWYTte52IuMa4X7Lt9rJjtV4E3nsdTjUAvty6wa6lyevAeTdnpidymV6OfOUK+tl+A qfUuXw+idZ8fU6btgoX3iuYdZQtlMhxUFTkD230HaPm5s8jA6Dxwnb9KEP1DdBUbM+2u A19g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s644ahboex6q+shoMOYNj6OgxWUg8OUuJkbKuV6OFsQ=; b=ZhXQrhJjfacq+6p0a4W4FFnvTLfPMWRZPogWhiJHcDQNb2hFyOVFhGDyRTGCGLu3cY 2VtPVvjNJS+5aSpCNT5U58P8xK+ZltN6zVTimh72xfrAUDvnr8tFY3x+fSz0x+uVa5sR HqoqVeX4BGY+KmsVUVW2czE4YVvZ1+SlVC1kpFISvYXUDTP973BpQwcAr6SKytqP5i1Y Ip1ks3bvbYtiFl34FS3/S4Okoy4xFaE3HqEu7u8XZikTKWNfeWZ0tqpDwty7GC3SleKW Z397QlrtNhBh0B46DbCJLGZuor5zqkOL5iL+IO/cKXyRB4y9QCvTpPXE/liYu9UKiBdI djcw== X-Gm-Message-State: AOAM531KdFtFmrQkAakHRPJJPWfl5he0DC/MfKH62ajqq/TABOw/C16v WjnO/Pp78QoMoGk9PSRdslGlIdyZosxTU4gv X-Google-Smtp-Source: ABdhPJznuqsg+KO1+cVrfXfy8YQFStL/o+s4AXI9756wfwP6lr5GBCRe2YPatDKZkxW7SzaTHXM/Sw== X-Received: by 2002:a5d:4eca:: with SMTP id s10mr27953478wrv.255.1632746013310; Mon, 27 Sep 2021 05:33:33 -0700 (PDT) Received: from pi4-davidp.pitowers.org ([2a00:1098:3142:14:1ce1:9965:4328:89c4]) by smtp.gmail.com with ESMTPSA id r9sm16285110wru.2.2021.09.27.05.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 05:33:32 -0700 (PDT) From: David Plowman To: libcamera-devel@lists.libcamera.org Date: Mon, 27 Sep 2021 13:33:27 +0100 Message-Id: <20210927123327.14554-4-david.plowman@raspberrypi.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210927123327.14554-1-david.plowman@raspberrypi.com> References: <20210927123327.14554-1-david.plowman@raspberrypi.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH v2 3/3] libcamera: pipeline: raspberrypi: Support colour spaces X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" The Raspberry Pi pipeline handler now sets colour spaces correctly. In generateConfiguration() is sets them to reasonable default values based on the stream role. Note how video recording streams use the "VIDEO" YCbCr encoding, which will later be fixed up according to the requested resolution. validate() and configure() check the colour space is valid, and also force all (non-raw) output streams to share the same colour space (which is a hardware restriction). Finally, the "VIDEO" YCbCr encoding is corrected by configure() to be one of REC601, REC709 or REC2020. Signed-off-by: David Plowman --- .../pipeline/raspberrypi/raspberrypi.cpp | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 0bdfa727..33ab49d6 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -281,6 +281,24 @@ RPiCameraConfiguration::RPiCameraConfiguration(const RPiCameraData *data) { } +static bool validateColorSpace(ColorSpace &colorSpace) +{ + const std::vector validColorSpaces = { + ColorSpace::JFIF, + ColorSpace::SMPTE170M, + ColorSpace::REC709, + ColorSpace::REC2020, + ColorSpace::VIDEO, + }; + auto it = std::find_if(validColorSpaces.begin(), validColorSpaces.end(), + [&colorSpace](const ColorSpace &item) { return colorSpace == item; }); + if (it != validColorSpaces.end()) + return true; + + colorSpace = ColorSpace::JFIF; + return false; +} + CameraConfiguration::Status RPiCameraConfiguration::validate() { Status status = Valid; @@ -346,6 +364,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() unsigned int rawCount = 0, outCount = 0, count = 0, maxIndex = 0; std::pair outSize[2]; Size maxSize; + ColorSpace colorSpace; /* colour space for all non-raw output streams */ for (StreamConfiguration &cfg : config_) { if (isRaw(cfg.pixelFormat)) { /* @@ -354,6 +373,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() */ V4L2VideoDevice::Formats fmts = data_->unicam_[Unicam::Image].dev()->formats(); V4L2DeviceFormat sensorFormat = findBestMode(fmts, cfg.size); + sensorFormat.colorSpace = ColorSpace::RAW; int ret = data_->unicam_[Unicam::Image].dev()->tryFormat(&sensorFormat); if (ret) return Invalid; @@ -392,6 +412,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() if (maxSize < cfg.size) { maxSize = cfg.size; maxIndex = outCount; + colorSpace = cfg.colorSpace; } outCount++; } @@ -405,6 +426,10 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() } } + /* Ensure the output colour space (if present) is one we handle. */ + if (outCount) + validateColorSpace(colorSpace); + /* * Now do any fixups needed. For the two ISP outputs, one stream must be * equal or smaller than the other in all dimensions. @@ -446,10 +471,26 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() status = Adjusted; } + /* Output streams must share the same colour space. */ + if (cfg.colorSpace != colorSpace) { + LOG(RPI, Warning) << "Output stream " << (i == maxIndex ? 0 : 1) + << " colour space changed"; + cfg.colorSpace = colorSpace; + status = Adjusted; + } + V4L2DeviceFormat format; format.fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat); format.size = cfg.size; + /* + * Request a sensible colour space. Note that "VIDEO" isn't a real + * encoding, so substitute something else sensible. + */ + format.colorSpace = colorSpace; + if (format.colorSpace.encoding == ColorSpace::Encoding::VIDEO) + format.colorSpace.encoding = ColorSpace::Encoding::REC601; + int ret = dev->tryFormat(&format); if (ret) return Invalid; @@ -475,6 +516,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, V4L2DeviceFormat sensorFormat; unsigned int bufferCount; PixelFormat pixelFormat; + ColorSpace colorSpace; V4L2VideoDevice::Formats fmts; Size size; @@ -490,6 +532,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, fmts = data->unicam_[Unicam::Image].dev()->formats(); sensorFormat = findBestMode(fmts, size); pixelFormat = sensorFormat.fourcc.toPixelFormat(); + colorSpace = ColorSpace::RAW; ASSERT(pixelFormat.isValid()); bufferCount = 2; rawCount++; @@ -501,6 +544,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, /* Return the largest sensor resolution. */ size = data->sensor_->resolution(); bufferCount = 1; + colorSpace = ColorSpace::JFIF; outCount++; break; @@ -517,6 +561,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, pixelFormat = formats::YUV420; size = { 1920, 1080 }; bufferCount = 4; + colorSpace = ColorSpace::VIDEO; outCount++; break; @@ -525,6 +570,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, pixelFormat = formats::ARGB8888; size = { 800, 600 }; bufferCount = 4; + colorSpace = ColorSpace::JFIF; outCount++; break; @@ -554,6 +600,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, StreamConfiguration cfg(formats); cfg.size = size; cfg.pixelFormat = pixelFormat; + cfg.colorSpace = colorSpace; cfg.bufferCount = bufferCount; config->addConfiguration(cfg); } @@ -575,6 +622,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) Size maxSize, sensorSize; unsigned int maxIndex = 0; bool rawStream = false; + ColorSpace colorSpace; /* colour space for all non-raw output streams */ /* * Look for the RAW stream (if given) size as well as the largest @@ -593,14 +641,40 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) } else { if (cfg.size > maxSize) { maxSize = config->at(i).size; + colorSpace = config->at(i).colorSpace; maxIndex = i; } } } + if (maxSize.isNull()) { + /* + * No non-raw streams, so some will get made below. Doesn't matter + * what colour space we assign to them. + */ + colorSpace = ColorSpace::JFIF; + } else { + /* Make sure we can handle this colour space. */ + validateColorSpace(colorSpace); + + /* + * The "VIDEO" colour encoding means that we choose one of REC601, + * REC709 or REC2020 automatically according to the resolution. + */ + if (colorSpace.encoding == ColorSpace::Encoding::VIDEO) { + if (maxSize.width >= 3840) + colorSpace.encoding = ColorSpace::Encoding::REC2020; + else if (maxSize.width >= 1280) + colorSpace.encoding = ColorSpace::Encoding::REC709; + else + colorSpace.encoding = ColorSpace::Encoding::REC601; + } + } + /* First calculate the best sensor mode we can use based on the user request. */ V4L2VideoDevice::Formats fmts = data->unicam_[Unicam::Image].dev()->formats(); V4L2DeviceFormat sensorFormat = findBestMode(fmts, rawStream ? sensorSize : maxSize); + sensorFormat.colorSpace = ColorSpace::RAW; /* * Unicam image output format. The ISP input format gets set at start, @@ -638,11 +712,18 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) StreamConfiguration &cfg = config->at(i); if (isRaw(cfg.pixelFormat)) { + cfg.colorSpace = ColorSpace::RAW; cfg.setStream(&data->unicam_[Unicam::Image]); data->unicam_[Unicam::Image].setExternal(true); continue; } + /* All other streams share the same colour space. */ + if (cfg.colorSpace != colorSpace) { + LOG(RPI, Warning) << "Stream " << i << " colour space changed"; + cfg.colorSpace = colorSpace; + } + /* The largest resolution gets routed to the ISP Output 0 node. */ RPi::Stream *stream = i == maxIndex ? &data->isp_[Isp::Output0] : &data->isp_[Isp::Output1]; @@ -650,6 +731,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) V4L2PixelFormat fourcc = V4L2PixelFormat::fromPixelFormat(cfg.pixelFormat); format.size = cfg.size; format.fourcc = fourcc; + format.colorSpace = cfg.colorSpace; LOG(RPI, Debug) << "Setting " << stream->name() << " to " << format.toString(); @@ -689,6 +771,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) format = {}; format.size = maxSize; format.fourcc = V4L2PixelFormat::fromPixelFormat(formats::YUV420); + format.colorSpace = colorSpace; ret = data->isp_[Isp::Output0].dev()->setFormat(&format); if (ret) { LOG(RPI, Error) @@ -718,6 +801,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config) const Size limit = maxDimensions.boundedToAspectRatio(format.size); output1Format.size = (format.size / 2).boundedTo(limit).alignedDownTo(2, 2); + output1Format.colorSpace = colorSpace; LOG(RPI, Debug) << "Setting ISP Output1 (internal) to " << output1Format.toString();