[{"id":13626,"web_url":"https://patchwork.libcamera.org/comment/13626/","msgid":"<20201106133659.GD3195686@oden.dyn.berto.se>","date":"2020-11-06T13:36:59","subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","submitter":{"id":5,"url":"https://patchwork.libcamera.org/api/people/5/","name":"Niklas Söderlund","email":"niklas.soderlund@ragnatech.se"},"content":"Hi Laurent,\n\nThanks for this work!\n\nOn 2020-11-03 04:05:20 +0200, Laurent Pinchart wrote:\n> Introduce a pipeline model that lists the operations applied by the\n> camera pipeline. This is a first step towards defining explicitly how\n> the camrera processes images, and how the libcamera controls affect the\n> processing.\n> \n> The initial list of operations is meant to be expanded, and possibly\n> refactored (a block diagram should also be considered to make this\n> easier to read). How the controls affect the pipeline is largely missing\n> at this stage, with only a short explanation of the digital zoom to show\n> how this is meant to be documented. More documentation will be added\n> over time.\n> \n> Signed-off-by: Laurent Pinchart \n\nReviewed-by: Niklas Söderlund \n\n> ---\n> src/libcamera/camera.cpp | 92 +++++++++++++++++++++++++++++++++-------\n> 1 file changed, 76 insertions(+), 16 deletions(-)\n> \n> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp\n> index dffbd6bd5a10..eff999ec322a 100644\n> --- a/src/libcamera/camera.cpp\n> +++ b/src/libcamera/camera.cpp\n> @@ -23,22 +23,82 @@\n> * \\file camera.h\n> * \\brief Camera device handling\n> *\n> - * At the core of libcamera is the camera device, combining one image source\n> - * with processing hardware able to provide one or multiple image streams. The\n> - * Camera class represents a camera device.\n> - *\n> - * A camera device contains a single image source, and separate camera device\n> - * instances relate to different image sources. For instance, a phone containing\n> - * front and back image sensors will be modelled with two camera devices, one\n> - * for each sensor. When multiple streams can be produced from the same image\n> - * source, all those streams are guaranteed to be part of the same camera\n> - * device.\n> - *\n> - * While not sharing image sources, separate camera devices can share other\n> - * system resources, such as an ISP. For this reason camera device instances may\n> - * not be fully independent, in which case usage restrictions may apply. For\n> - * instance, a phone with a front and a back camera device may not allow usage\n> - * of the two devices simultaneously.\n> + * \\page camera-model Camera Model\n> + *\n> + * libcamera acts as a middleware between applications and camera hardware. It\n> + * provides a solution to an unsolvable problem: reconciling applications,\n> + * which need to run on different systems without dealing with device-specific\n> + * details, and camera hardware, which exhibits a wide variety of features,\n> + * limitations and architecture variations. In order to do so, it creates an\n> + * abstract camera model that hides the camera hardware from applications. The\n> + * model is designed to strike the right balance between genericity, to please\n> + * generic applications, and flexibililty, to expose even the most specific\n> + * hardware features to the most demanding applications.\n> + *\n> + * In libcamera, a Camera is defined as a device that can capture frames\n> + * continuously from a camera sensor and store them in memory. If supported by\n> + * the device and desired by the application, the camera may store each\n> + * captured frame in multiple copies, possibly in different formats and sizes.\n> + * Each of these memory outputs of the camera is called a Stream.\n> + *\n> + * A camera contains a single image source, and separate camera instances\n> + * relate to different image sources. For instance, a phone containing front\n> + * and back image sensors will be modelled with two cameras, one for each\n> + * sensor. When multiple streams can be produced from the same image source,\n> + * all those streams are guaranteed to be part of the same camera.\n> + *\n> + * While not sharing image sources, separate cameras can share other system\n> + * resources, such as ISPs. For this reason camera instances may not be fully\n> + * independent, in which case usage restrictions may apply. For instance, a\n> + * phone with a front and a back camera may not allow usage of the two cameras\n> + * simultaneously.\n> + *\n> + * The camera model defines an implicit pipeline, whose input is the camera\n> + * sensor, and whose outputs are the streams. Along the pipeline, the frames\n> + * produced by the camera sensor are transformed by the camera into a format\n> + * suitable for applications, with image processing that improves the quality\n> + * of the captured frames. The camera exposes a set of controls that\n> + * application may use to manually control the processing steps. This\n> + * high-level camera model is the minimum baseline that all cameras must\n> + * conform to.\n> + *\n> + * \\section camera-pipeline-model Pipeline Model\n> + *\n> + * Camera hardware differs in the supported image processing operations and the\n> + * order in which they are applied. The libcamera pipelines abstract the\n> + * hardware differences and expose a logical view of the processing operations\n> + * with a fixed order. This offers low-level control of those operations to\n> + * applications, while keeping application code generic.\n> + *\n> + * Starting from the camera sensor, a pipeline applies the following\n> + * operations, in that order.\n> + *\n> + * - Pixel exposure\n> + * - Analog to digital conversion and readout\n> + * - Black level subtraction\n> + * - Defective pixel correction\n> + * - Lens shading correction\n> + * - Spatial noise filtering\n> + * - Per-channel gains (white balance)\n> + * - Demosaicing (color filter array interpolation)\n> + * - Color correction matrix (typically RGB to RGB)\n> + * - Gamma correction\n> + * - Color space transformation (typically RGB to YUV)\n> + * - Cropping\n> + * - Scaling\n> + *\n> + * Not all cameras implement all operations, and they are not necessarily\n> + * implemented in the above order at the hardware level. The libcamera pipeline\n> + * handlers translate the pipeline model to the real hardware configuration.\n> + *\n> + * \\subsection digital-zoom Digital Zoom\n> + *\n> + * Digital zoom is implemented as a combination of the cropping and scaling\n> + * stages of the pipeline. Cropping is controlled explicitly through the\n> + * controls::ScalerCrop control, while scalling is controlled implicitly based\n> + * on the crop rectangle and the output stream size. The crop rectangle is\n> + * expressed relatively to the full pixel array size and indicates how the field\n> + * of view is affected by the pipeline.\n> */\n> \n> namespace libcamera {\n> -- \n> Regards,\n> \n> Laurent Pinchart\n> \n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 17C5CBDB89\n\tfor ;\n\tFri, 6 Nov 2020 13:37:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 8D14D62D3F;\n\tFri, 6 Nov 2020 14:37:04 +0100 (CET)","from mail-lf1-x143.google.com (mail-lf1-x143.google.com\n\t[IPv6:2a00:1450:4864:20::143])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 72A8362B8E\n\tfor ;\n\tFri, 6 Nov 2020 14:37:02 +0100 (CET)","by mail-lf1-x143.google.com with SMTP id z21so641240lfe.12\n\tfor ;\n\tFri, 06 Nov 2020 05:37:02 -0800 (PST)","from localhost (h-209-203.A463.priv.bahnhof.se. [155.4.209.203])\n\tby smtp.gmail.com with ESMTPSA id\n\tb18sm166655lfp.89.2020.11.06.05.37.00\n\t(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n\tFri, 06 Nov 2020 05:37:00 -0800 (PST)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n\tunprotected) header.d=ragnatech-se.20150623.gappssmtp.com\n\theader.i=@ragnatech-se.20150623.gappssmtp.com\n\theader.b=\"KmaiYOFL\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=ragnatech-se.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:content-transfer-encoding:in-reply-to;\n\tbh=xRrIIWR2xR6vKHCufn3rtozZbnMp29xKNh+mcqTrHA4=;\n\tb=KmaiYOFLJVupnNlvbiWIyglfInmDk2RFqsDZDKoE6IH3lStgFD95KHtN2C9k8/jWrE\n\tuT6+QPytgnkh9403x5wDno7GOE6iMBrX4IOb30R3a/XD0jbeLbFGtQxKCdPm9sKqzTUe\n\t4TItW+BkiR8hWBdvfxlCOTPgrSJAnWoXCtp3aiMjgnqvNqWjdgbn3BBO8otPvP5HyJM5\n\tlP0Bo8NgsG7K5hl93XODPqROFjU017tsk5UjmvRIJEOe3RY80IESkXT+19j46J15i41f\n\tPVZ7gxGM2y9hnO/cn20RR4tDBeYgfaT+d5KcPIiFmUCqhWo67hpXnw2sqG7RdSxwft5M\n\tXhRg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:content-transfer-encoding\n\t:in-reply-to;\n\tbh=xRrIIWR2xR6vKHCufn3rtozZbnMp29xKNh+mcqTrHA4=;\n\tb=FNiRkEZxV4NJrmlZEgjVna2oNaDmORYV67N7Kj5JBU5UP1kVe/pabovZVJ7c+tlvEI\n\t8h3z78/3pGJmooKnEe7ftsUbnbQnXZ4lBepsptvxIgSrrk4aMYwMmpafgjU7tnVDKIWT\n\tEaIG5JtP89f+OgDpGwQeVv7yOd2AG0FlLNd0gg51YU3tqGpaY1wqPZFseCj0IM7EPMcS\n\tsIA8vDxre+QWiSMx1qFZ1ZsxodJkXLC3gc+gwchfezYsZKkV0U2Lk+zLatRjFH8lBQKH\n\txmzrcBfxumbMJN8NdGoy+NfVqSllZJYknIzrZB/AuUCA75oYSeeP3pQ8zpMavBBpelmU\n\t4M8w==","X-Gm-Message-State":"AOAM532WMBbZBtK+Kpo91V+jVz9JiseB2p+laZDmP1J5iKRAmrsW4sQ6\n\tpnf0e/zdApfHDZx3D2B4MgLAyA==","X-Google-Smtp-Source":"ABdhPJyml2OZJfKroqmmNzxAliVH02R+RSiOT68GKmlScrorToFx84mnN1fxPnZwcIOpurJ63shkAQ==","X-Received":"by 2002:ac2:5e34:: with SMTP id o20mr847870lfg.337.1604669821595;\n\tFri, 06 Nov 2020 05:37:01 -0800 (PST)","Date":"Fri, 6 Nov 2020 14:36:59 +0100","From":"Niklas =?iso-8859-1?q?S=F6derlund?= ","To":"Laurent Pinchart ","Message-ID":"<20201106133659.GD3195686@oden.dyn.berto.se>","References":"<20201103020520.19354-1-laurent.pinchart@ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201103020520.19354-1-laurent.pinchart@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"iso-8859-1\"","Content-Transfer-Encoding":"quoted-printable","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13651,"web_url":"https://patchwork.libcamera.org/comment/13651/","msgid":"","date":"2020-11-09T14:44:35","subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 03/11/2020 02:05, Laurent Pinchart wrote:\n> Introduce a pipeline model that lists the operations applied by the\n> camera pipeline. This is a first step towards defining explicitly how\n> the camrera processes images, and how the libcamera controls affect the\n\ns/camrera/camera/\n\n> processing.\n> \n> The initial list of operations is meant to be expanded, and possibly\n> refactored (a block diagram should also be considered to make this\n> easier to read). How the controls affect the pipeline is largely missing\n> at this stage, with only a short explanation of the digital zoom to show\n> how this is meant to be documented. More documentation will be added\n> over time.\n> \n> Signed-off-by: Laurent Pinchart \n> ---\n> src/libcamera/camera.cpp | 92 +++++++++++++++++++++++++++++++++-------\n> 1 file changed, 76 insertions(+), 16 deletions(-)\n> \n> diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp\n> index dffbd6bd5a10..eff999ec322a 100644\n> --- a/src/libcamera/camera.cpp\n> +++ b/src/libcamera/camera.cpp\n> @@ -23,22 +23,82 @@\n> * \\file camera.h\n> * \\brief Camera device handling\n> *\n> - * At the core of libcamera is the camera device, combining one image source\n> - * with processing hardware able to provide one or multiple image streams. The\n> - * Camera class represents a camera device.\n> - *\n> - * A camera device contains a single image source, and separate camera device\n> - * instances relate to different image sources. For instance, a phone containing\n> - * front and back image sensors will be modelled with two camera devices, one\n> - * for each sensor. When multiple streams can be produced from the same image\n> - * source, all those streams are guaranteed to be part of the same camera\n> - * device.\n> - *\n> - * While not sharing image sources, separate camera devices can share other\n> - * system resources, such as an ISP. For this reason camera device instances may\n> - * not be fully independent, in which case usage restrictions may apply. For\n> - * instance, a phone with a front and a back camera device may not allow usage\n> - * of the two devices simultaneously.\n> + * \\page camera-model Camera Model\n> + *\n> + * libcamera acts as a middleware between applications and camera hardware. It\n> + * provides a solution to an unsolvable problem: reconciling applications,\n\nAre we really solving an unsolvable problem? Doesn't that by definition\nmean it's solvable ? :-)\n\n\n> + * which need to run on different systems without dealing with device-specific\n> + * details, and camera hardware, which exhibits a wide variety of features,\n> + * limitations and architecture variations. In order to do so, it creates an\n> + * abstract camera model that hides the camera hardware from applications. The\n> + * model is designed to strike the right balance between genericity, to please\n> + * generic applications, and flexibililty, to expose even the most specific\n\ns/flexibililty/flexibility/\n\n\n> + * hardware features to the most demanding applications.\n> + *\n> + * In libcamera, a Camera is defined as a device that can capture frames\n> + * continuously from a camera sensor and store them in memory. If supported by\n> + * the device and desired by the application, the camera may store each\n> + * captured frame in multiple copies, possibly in different formats and sizes.\n> + * Each of these memory outputs of the camera is called a Stream.\n> + *\n> + * A camera contains a single image source, and separate camera instances\n> + * relate to different image sources. For instance, a phone containing front\n> + * and back image sensors will be modelled with two cameras, one for each\n> + * sensor. When multiple streams can be produced from the same image source,\n> + * all those streams are guaranteed to be part of the same camera.\n> + *\n> + * While not sharing image sources, separate cameras can share other system\n> + * resources, such as ISPs. For this reason camera instances may not be fully\n> + * independent, in which case usage restrictions may apply. For instance, a\n> + * phone with a front and a back camera may not allow usage of the two cameras\n> + * simultaneously.\n> + *\n> + * The camera model defines an implicit pipeline, whose input is the camera\n> + * sensor, and whose outputs are the streams. Along the pipeline, the frames\n> + * produced by the camera sensor are transformed by the camera into a format\n> + * suitable for applications, with image processing that improves the quality\n> + * of the captured frames. The camera exposes a set of controls that\n> + * application may use to manually control the processing steps. This\n\ns/application/applications/\n\n> + * high-level camera model is the minimum baseline that all cameras must\n> + * conform to.\n> + *\n> + * \\section camera-pipeline-model Pipeline Model\n> + *\n> + * Camera hardware differs in the supported image processing operations and the\n> + * order in which they are applied. The libcamera pipelines abstract the\n> + * hardware differences and expose a logical view of the processing operations\n> + * with a fixed order. This offers low-level control of those operations to\n> + * applications, while keeping application code generic.\n> + *\n> + * Starting from the camera sensor, a pipeline applies the following\n> + * operations, in that order.\n> + *\n> + * - Pixel exposure\n> + * - Analog to digital conversion and readout\n> + * - Black level subtraction\n> + * - Defective pixel correction\n> + * - Lens shading correction\n> + * - Spatial noise filtering\n> + * - Per-channel gains (white balance)\n> + * - Demosaicing (color filter array interpolation)\n> + * - Color correction matrix (typically RGB to RGB)\n> + * - Gamma correction\n> + * - Color space transformation (typically RGB to YUV)\n> + * - Cropping\n> + * - Scaling\n\nDo we really define that they happen in that order? Are there\nimplications if pipelines \"don't\"\n\n> + *\n> + * Not all cameras implement all operations, and they are not necessarily\n> + * implemented in the above order at the hardware level. The libcamera pipeline\n> + * handlers translate the pipeline model to the real hardware configuration.\n\ner ... ok ... I'm not (yet) quite sure of the relevance of stating the\norder of operations if the pipelines can still do what they like ;-)\n\n\n> + *\n> + * \\subsection digital-zoom Digital Zoom\n> + *\n> + * Digital zoom is implemented as a combination of the cropping and scaling\n> + * stages of the pipeline. Cropping is controlled explicitly through the\n> + * controls::ScalerCrop control, while scalling is controlled implicitly based\n\ns/scalling/scaling/\n\n> + * on the crop rectangle and the output stream size. The crop rectangle is\n> + * expressed relatively to the full pixel array size and indicates how the field\n> + * of view is affected by the pipeline.\n> */\n\nWith the small fixups above:\n\nReviewed-by: Kieran Bingham \n\n> \n> namespace libcamera {\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 26AB8BDB89\n\tfor ;\n\tMon, 9 Nov 2020 14:44:41 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7574063090;\n\tMon, 9 Nov 2020 15:44:40 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 4744062D3E\n\tfor ;\n\tMon, 9 Nov 2020 15:44:39 +0100 (CET)","from [192.168.0.20]\n\t(cpc89244-aztw30-2-0-cust3082.18-1.cable.virginm.net [86.31.172.11])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id A281FB2B;\n\tMon, 9 Nov 2020 15:44:38 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"JcFPTov6\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1604933078;\n\tbh=XiZt2oSLi+Km3ffKuSVpyNNuSlX7XusgcLJQkB3FIeQ=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=JcFPTov6UY7qT7D61+I/3JLGwmYLWa4aQPi4yIhkNCzxSbEmGKaXlDMseZxNOpd3x\n\ts8PEP4xhiIjBmrTDnpqG7rKBbSvKklmbu+REehdKPxdbEaQjAdEDLZ8ewX6t5l7w4Z\n\tlt6Ed2Xy7HAXqWnr/g1PDG4SVJhiRuODdbfyYoj4=","To":"Laurent Pinchart ,\n\tlibcamera-devel@lists.libcamera.org","References":"<20201103020520.19354-1-laurent.pinchart@ideasonboard.com>","From":"Kieran Bingham ","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"","Date":"Mon, 9 Nov 2020 14:44:35 +0000","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101\n\tThunderbird/68.10.0","MIME-Version":"1.0","In-Reply-To":"<20201103020520.19354-1-laurent.pinchart@ideasonboard.com>","Content-Language":"en-GB","Subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Reply-To":"kieran.bingham@ideasonboard.com","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13840,"web_url":"https://patchwork.libcamera.org/comment/13840/","msgid":"<20201124085242.GE4592@pendragon.ideasonboard.com>","date":"2020-11-24T08:52:42","subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Kieran,\n\nOn Mon, Nov 09, 2020 at 02:44:35PM +0000, Kieran Bingham wrote:\n> On 03/11/2020 02:05, Laurent Pinchart wrote:\n> > Introduce a pipeline model that lists the operations applied by the\n> > camera pipeline. This is a first step towards defining explicitly how\n> > the camrera processes images, and how the libcamera controls affect the\n> \n> s/camrera/camera/\n> \n> > processing.\n> > \n> > The initial list of operations is meant to be expanded, and possibly\n> > refactored (a block diagram should also be considered to make this\n> > easier to read). How the controls affect the pipeline is largely missing\n> > at this stage, with only a short explanation of the digital zoom to show\n> > how this is meant to be documented. More documentation will be added\n> > over time.\n> > \n> > Signed-off-by: Laurent Pinchart \n> > ---\n> > src/libcamera/camera.cpp | 92 +++++++++++++++++++++++++++++++++-------\n> > 1 file changed, 76 insertions(+), 16 deletions(-)\n> > \n> > diff --git a/src/libcamera/camera.cpp b/src/libcamera/camera.cpp\n> > index dffbd6bd5a10..eff999ec322a 100644\n> > --- a/src/libcamera/camera.cpp\n> > +++ b/src/libcamera/camera.cpp\n> > @@ -23,22 +23,82 @@\n> > * \\file camera.h\n> > * \\brief Camera device handling\n> > *\n> > - * At the core of libcamera is the camera device, combining one image source\n> > - * with processing hardware able to provide one or multiple image streams. The\n> > - * Camera class represents a camera device.\n> > - *\n> > - * A camera device contains a single image source, and separate camera device\n> > - * instances relate to different image sources. For instance, a phone containing\n> > - * front and back image sensors will be modelled with two camera devices, one\n> > - * for each sensor. When multiple streams can be produced from the same image\n> > - * source, all those streams are guaranteed to be part of the same camera\n> > - * device.\n> > - *\n> > - * While not sharing image sources, separate camera devices can share other\n> > - * system resources, such as an ISP. For this reason camera device instances may\n> > - * not be fully independent, in which case usage restrictions may apply. For\n> > - * instance, a phone with a front and a back camera device may not allow usage\n> > - * of the two devices simultaneously.\n> > + * \\page camera-model Camera Model\n> > + *\n> > + * libcamera acts as a middleware between applications and camera hardware. It\n> > + * provides a solution to an unsolvable problem: reconciling applications,\n> \n> Are we really solving an unsolvable problem? Doesn't that by definition\n> mean it's solvable ? :-)\n\nIt's magiiiiic ;-)\n\n> > + * which need to run on different systems without dealing with device-specific\n> > + * details, and camera hardware, which exhibits a wide variety of features,\n> > + * limitations and architecture variations. In order to do so, it creates an\n> > + * abstract camera model that hides the camera hardware from applications. The\n> > + * model is designed to strike the right balance between genericity, to please\n> > + * generic applications, and flexibililty, to expose even the most specific\n> \n> s/flexibililty/flexibility/\n> \n> > + * hardware features to the most demanding applications.\n> > + *\n> > + * In libcamera, a Camera is defined as a device that can capture frames\n> > + * continuously from a camera sensor and store them in memory. If supported by\n> > + * the device and desired by the application, the camera may store each\n> > + * captured frame in multiple copies, possibly in different formats and sizes.\n> > + * Each of these memory outputs of the camera is called a Stream.\n> > + *\n> > + * A camera contains a single image source, and separate camera instances\n> > + * relate to different image sources. For instance, a phone containing front\n> > + * and back image sensors will be modelled with two cameras, one for each\n> > + * sensor. When multiple streams can be produced from the same image source,\n> > + * all those streams are guaranteed to be part of the same camera.\n> > + *\n> > + * While not sharing image sources, separate cameras can share other system\n> > + * resources, such as ISPs. For this reason camera instances may not be fully\n> > + * independent, in which case usage restrictions may apply. For instance, a\n> > + * phone with a front and a back camera may not allow usage of the two cameras\n> > + * simultaneously.\n> > + *\n> > + * The camera model defines an implicit pipeline, whose input is the camera\n> > + * sensor, and whose outputs are the streams. Along the pipeline, the frames\n> > + * produced by the camera sensor are transformed by the camera into a format\n> > + * suitable for applications, with image processing that improves the quality\n> > + * of the captured frames. The camera exposes a set of controls that\n> > + * application may use to manually control the processing steps. This\n> \n> s/application/applications/\n> \n> > + * high-level camera model is the minimum baseline that all cameras must\n> > + * conform to.\n> > + *\n> > + * \\section camera-pipeline-model Pipeline Model\n> > + *\n> > + * Camera hardware differs in the supported image processing operations and the\n> > + * order in which they are applied. The libcamera pipelines abstract the\n> > + * hardware differences and expose a logical view of the processing operations\n> > + * with a fixed order. This offers low-level control of those operations to\n> > + * applications, while keeping application code generic.\n> > + *\n> > + * Starting from the camera sensor, a pipeline applies the following\n> > + * operations, in that order.\n> > + *\n> > + * - Pixel exposure\n> > + * - Analog to digital conversion and readout\n> > + * - Black level subtraction\n> > + * - Defective pixel correction\n> > + * - Lens shading correction\n> > + * - Spatial noise filtering\n> > + * - Per-channel gains (white balance)\n> > + * - Demosaicing (color filter array interpolation)\n> > + * - Color correction matrix (typically RGB to RGB)\n> > + * - Gamma correction\n> > + * - Color space transformation (typically RGB to YUV)\n> > + * - Cropping\n> > + * - Scaling\n> \n> Do we really define that they happen in that order? Are there\n> implications if pipelines \"don't\"\n> \n> > + *\n> > + * Not all cameras implement all operations, and they are not necessarily\n> > + * implemented in the above order at the hardware level. The libcamera pipeline\n> > + * handlers translate the pipeline model to the real hardware configuration.\n> \n> er ... ok ... I'm not (yet) quite sure of the relevance of stating the\n> order of operations if the pipelines can still do what they like ;-)\n\nThe pipeline model defines a logical view of the pipeline, which\nmandates operations to be applied in the above order from the point of\nview of an external observer. The hardware implementation may differ,\nbut the effect needs to match the pipeline model.\n\n> > + *\n> > + * \\subsection digital-zoom Digital Zoom\n> > + *\n> > + * Digital zoom is implemented as a combination of the cropping and scaling\n> > + * stages of the pipeline. Cropping is controlled explicitly through the\n> > + * controls::ScalerCrop control, while scalling is controlled implicitly based\n> \n> s/scalling/scaling/\n> \n> > + * on the crop rectangle and the output stream size. The crop rectangle is\n> > + * expressed relatively to the full pixel array size and indicates how the field\n> > + * of view is affected by the pipeline.\n> > */\n> \n> With the small fixups above:\n> \n> Reviewed-by: Kieran Bingham \n> \n> > \n> > namespace libcamera {","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BE3A4BE08A\n\tfor ;\n\tTue, 24 Nov 2020 08:52:50 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 273F9633F4;\n\tTue, 24 Nov 2020 09:52:50 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A7EED60333\n\tfor ;\n\tTue, 24 Nov 2020 09:52:49 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 1654F292;\n\tTue, 24 Nov 2020 09:52:49 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"p7Dm+djy\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1606207969;\n\tbh=rkO9kq0M3wvghJydo9GZ+Z/OG9ATrNX+t5XGF6JtftE=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=p7Dm+djyF7ldv2cqHfwI0QiQ6GOYPuWk/1P6Qv6ohtG9DfQHHe+GSsCPbRuXQZiC+\n\tAkWG4YEFjAIrS1r8ssNqvtyVef9m2bKYJv3WadN80v9+jJUaFCvtfA/decfkmIE0vh\n\tEcr5u9WZ8ad52nR/c4MdQxyq5HgmTmbJdx9ynNDU=","Date":"Tue, 24 Nov 2020 10:52:42 +0200","From":"Laurent Pinchart ","To":"Kieran Bingham ","Message-ID":"<20201124085242.GE4592@pendragon.ideasonboard.com>","References":"<20201103020520.19354-1-laurent.pinchart@ideasonboard.com>\n\t","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"","Subject":"Re: [libcamera-devel] [PATCH] libcamera: camera: Document the\n\tcamera and pipeline model","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]