[{"id":3135,"web_url":"https://patchwork.libcamera.org/comment/3135/","msgid":"<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>","date":"2019-11-25T10:57:53","subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 23/11/2019 12:43, Laurent Pinchart wrote:\n> When a fatal error occurs the program aborts, and all the logger\n> provides is the location of the line that caused the error. Extend this\n> with a full backtrace to help debugging.\n> \n> The backtrace is generated using the backtrace() call, a GNU extension\n> to the C library. It is available in glibc and uClibc but not in musl.\n> Test for availability of the function to condition compilation of the\n> backtrace printing. Implementing backtrace support with musl is an\n> exercise left to the reader if desired.\n\nNice, I like features like this. I had this as a TODO note in one of my\nbranches, so I'm glad you've got to it first!.\n\nHave you looked at libSegFault as well perhaps by any chance?\n\nI'm wondering if something like the following might be a useful addition\nin this topic:\n\n+if cc.find_library('SegFault', required: false).found()\n+    message('Found libSegFault')\n+    common_arguments += [ '-lSegFault', ]\n+endif\n+\n\nAlthough - also possibly conditioned on being a debug build.\n\nThis works nicely for me so far, so:\n\n  Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n\nIt might be argued that the LogOutput changes required here could be\nsplit to clarify them, as it took a bit of parsing to understand what\nhad to be moved and why. I don't see much requirement to split it, but\nit might be nice to add a description to the changelog to help\nunderstanding.\n\n>From my parsing - something like:\n\nThe LogOutput class is extended to support writing string messages\ndirectly to the output. Strings written directly will be considered as\nLogDebug messages when written to the Syslog.\n\n(Adapting that for correctness if you do decide to add it of course)\n\n  Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n\n\n\n> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n> ---\n>  meson.build           |  4 ++\n>  src/libcamera/log.cpp | 93 ++++++++++++++++++++++++++++++++++---------\n>  2 files changed, 79 insertions(+), 18 deletions(-)\n> \n> diff --git a/meson.build b/meson.build\n> index 0a222ba96dcb..634488589a46 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -26,6 +26,10 @@ libcamera_version = libcamera_git_version.split('+')[0]\n>  cc = meson.get_compiler('c')\n>  config_h = configuration_data()\n>  \n> +if cc.has_header_symbol('execinfo.h', 'backtrace')\n> +    config_h.set('HAVE_BACKTRACE', 1)\n> +endif\n> +\n>  if cc.has_header_symbol('stdlib.h', 'secure_getenv', prefix : '#define _GNU_SOURCE')\n>      config_h.set('HAVE_SECURE_GETENV', 1)\n>  endif\n> diff --git a/src/libcamera/log.cpp b/src/libcamera/log.cpp\n> index 50f345b98c74..f4eb8c11adc3 100644\n> --- a/src/libcamera/log.cpp\n> +++ b/src/libcamera/log.cpp\n> @@ -7,6 +7,9 @@\n>  \n>  #include \"log.h\"\n>  \n> +#if HAVE_BACKTRACE\n> +#include <execinfo.h>\n> +#endif\n>  #include <fstream>\n>  #include <iostream>\n>  #include <list>\n> @@ -108,10 +111,11 @@ public:\n>  \n>  \tbool isValid() const;\n>  \tvoid write(const LogMessage &msg);\n> +\tvoid write(const std::string &msg);\n>  \n>  private:\n> -\tvoid writeSyslog(const LogMessage &msg);\n> -\tvoid writeStream(const LogMessage &msg);\n> +\tvoid writeSyslog(LogSeverity severity, const std::string &msg);\n> +\tvoid writeStream(const std::string &msg);\n>  \n>  \tstd::ostream *stream_;\n>  \tLoggingTarget target_;\n> @@ -180,34 +184,55 @@ bool LogOutput::isValid() const\n>   * \\param[in] msg Message to write\n>   */\n>  void LogOutput::write(const LogMessage &msg)\n> +{\n> +\tstd::string str;\n> +\n> +\tswitch (target_) {\n> +\tcase LoggingTargetSyslog:\n> +\t\tstr = std::string(log_severity_name(msg.severity())) + \" \"\n> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> +\t\t    + msg.msg();\n> +\t\twriteSyslog(msg.severity(), str);\n> +\t\tbreak;\n> +\tcase LoggingTargetStream:\n> +\tcase LoggingTargetFile:\n> +\t\tstr = \"[\" + utils::time_point_to_string(msg.timestamp()) + \"]\"\n> +\t\t    + log_severity_name(msg.severity()) + \" \"\n> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> +\t\t    + msg.msg();\n> +\t\twriteStream(str);\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tbreak;\n> +\t}\n> +}\n> +\n> +/**\n> + * \\brief Write string to log output\n> + * \\param[in] str String to write\n> + */\n> +void LogOutput::write(const std::string &str)\n>  {\n>  \tswitch (target_) {\n>  \tcase LoggingTargetSyslog:\n> -\t\twriteSyslog(msg);\n> +\t\twriteSyslog(LogDebug, str);\n>  \t\tbreak;\n>  \tcase LoggingTargetStream:\n>  \tcase LoggingTargetFile:\n> -\t\twriteStream(msg);\n> +\t\twriteStream(str);\n>  \t\tbreak;\n>  \tdefault:\n>  \t\tbreak;\n>  \t}\n>  }\n>  \n> -void LogOutput::writeSyslog(const LogMessage &msg)\n> +void LogOutput::writeSyslog(LogSeverity severity, const std::string &str)\n>  {\n> -\tstd::string str = std::string(log_severity_name(msg.severity())) + \" \" +\n> -\t      \t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> -\t\t\t  msg.msg();\n> -\tsyslog(log_severity_to_syslog(msg.severity()), \"%s\", str.c_str());\n> +\tsyslog(log_severity_to_syslog(severity), \"%s\", str.c_str());\n>  }\n>  \n> -void LogOutput::writeStream(const LogMessage &msg)\n> +void LogOutput::writeStream(const std::string &str)\n>  {\n> -\tstd::string str = \"[\" + utils::time_point_to_string(msg.timestamp()) +\n> -\t\t\t  \"]\" + log_severity_name(msg.severity()) + \" \" +\n> -\t\t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> -\t\t\t  msg.msg();\n>  \tstream_->write(str.c_str(), str.size());\n>  \tstream_->flush();\n>  }\n> @@ -223,6 +248,7 @@ public:\n>  \tstatic Logger *instance();\n>  \n>  \tvoid write(const LogMessage &msg);\n> +\tvoid backtrace();\n>  \n>  \tint logSetFile(const char *path);\n>  \tint logSetStream(std::ostream *stream);\n> @@ -240,9 +266,6 @@ private:\n>  \tvoid registerCategory(LogCategory *category);\n>  \tvoid unregisterCategory(LogCategory *category);\n>  \n> -\tvoid writeSyslog(const LogMessage &msg);\n> -\tvoid writeStream(const LogMessage &msg);\n> -\n>  \tstd::unordered_set<LogCategory *> categories_;\n>  \tstd::list<std::pair<std::string, LogSeverity>> levels_;\n>  \n> @@ -370,6 +393,38 @@ void Logger::write(const LogMessage &msg)\n>  \toutput->write(msg);\n>  }\n>  \n> +/**\n> + * \\brief Write a backtrace to the log\n> + */\n> +void Logger::backtrace()\n> +{\n> +#if HAVE_BACKTRACE\n\n\nWill this still be usable/desirable on release builds when\nfunction/symbol names are possibly unavailable?\n\nIf not - then we might want to make this:\n\n\t#if HAVE_BACKTRACE && !defined(NDEBUG)\n\nBut I'm not entirely averse to having the backtrace in regardless in\nfact, as it will likely backtrace all the way into the calling\napplication (which should have symbols if it's being developed) and\ncould add value in that scenario regardless.\n\n\n\n> +\tstd::shared_ptr<LogOutput> output = std::atomic_load(&output_);\n> +\tif (!output)\n> +\t\treturn;\n> +\n> +\tvoid *buffer[32];\n> +\tint num_entries = ::backtrace(buffer, ARRAY_SIZE(buffer));\n> +\tchar **strings = backtrace_symbols(buffer, num_entries);\n> +\tif (!strings)\n> +\t\treturn;\n> +\n> +\tstd::ostringstream msg;\n> +\tmsg << \"Backtrace:\" << std::endl;\n> +\n> +\t/*\n> +\t * Skip the first two entries that correspond to this method and\n> +\t * ~LogMessage().\n> +\t */\n> +\tfor (int i = 2; i < num_entries; ++i)\n> +\t\tmsg << strings[i] << std::endl;\n> +\n> +\toutput->write(msg.str());\n> +\n> +\tfree(strings);\n> +#endif\n> +}\n> +\n>  /**\n>   * \\brief Set the log file\n>   * \\param[in] path Full path to the log file\n> @@ -783,8 +838,10 @@ LogMessage::~LogMessage()\n>  \tif (severity_ >= category_.severity())\n>  \t\tLogger::instance()->write(*this);\n>  \n> -\tif (severity_ == LogSeverity::LogFatal)\n> +\tif (severity_ == LogSeverity::LogFatal) {\n> +\t\tLogger::instance()->backtrace();\n>  \t\tstd::abort();\n> +\t}\n>  }\n>  \n>  /**\n>","headers":{"Return-Path":"<kieran.bingham@ideasonboard.com>","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id A14C360BFF\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tMon, 25 Nov 2019 11:57:57 +0100 (CET)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 160AD729;\n\tMon, 25 Nov 2019 11:57:57 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1574679477;\n\tbh=gg3Wgb9NjeBwnANRvFFP0oTgqbNrXZhk3Yl2I8RZiKU=;\n\th=Reply-To:Subject:To:References:From:Date:In-Reply-To:From;\n\tb=qNpYVKLWTit1dOAMX1zIkrM3SuOkjJr12F8ptdwCJx9Cq6foTqMbn3d7Txq8pEbOU\n\tc+riWjV7ztuEOV90zTq/tmPFCguui2GXGaAXfBDC21mxyPr6fX4DyuL572Snfza0TV\n\t7W6qsCY1dufF7ewtmGmhOgcCJnIxoNIklhuwFUnM=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>,\n\tlibcamera-devel@lists.libcamera.org","References":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>","From":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Openpgp":"preference=signencrypt","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>","Date":"Mon, 25 Nov 2019 10:57:53 +0000","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101\n\tThunderbird/60.9.0","MIME-Version":"1.0","In-Reply-To":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","X-List-Received-Date":"Mon, 25 Nov 2019 10:57:58 -0000"}},{"id":3137,"web_url":"https://patchwork.libcamera.org/comment/3137/","msgid":"<20191126020148.GA4705@pendragon.ideasonboard.com>","date":"2019-11-26T02:01:48","subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Kieran,\n\nOn Mon, Nov 25, 2019 at 10:57:53AM +0000, Kieran Bingham wrote:\n> On 23/11/2019 12:43, Laurent Pinchart wrote:\n> > When a fatal error occurs the program aborts, and all the logger\n> > provides is the location of the line that caused the error. Extend this\n> > with a full backtrace to help debugging.\n> > \n> > The backtrace is generated using the backtrace() call, a GNU extension\n> > to the C library. It is available in glibc and uClibc but not in musl.\n> > Test for availability of the function to condition compilation of the\n> > backtrace printing. Implementing backtrace support with musl is an\n> > exercise left to the reader if desired.\n> \n> Nice, I like features like this. I had this as a TODO note in one of my\n> branches, so I'm glad you've got to it first!.\n> \n> Have you looked at libSegFault as well perhaps by any chance?\n\nI wasn't aware of it, thanks for the pointer.\n\n> I'm wondering if something like the following might be a useful addition\n> in this topic:\n> \n> +if cc.find_library('SegFault', required: false).found()\n> +    message('Found libSegFault')\n> +    common_arguments += [ '-lSegFault', ]\n> +endif\n> +\n> \n> Although - also possibly conditioned on being a debug build.\n\nIn this particular case it wouldn't have helped, as I could only\nreproduce the issue with release builds. I thus wonder if we shouldn't\nLD_PRELOAD libSegFault.so instead when running tests, like we use\nvalgrind on tests from time to time. Or would libasan be a better option\nin that case ?\n\nWhat I'm trying to figure out is how much overlap there is between this\npatch and libSegFault or libasan.\n\n> This works nicely for me so far, so:\n> \n>   Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n> \n> It might be argued that the LogOutput changes required here could be\n> split to clarify them, as it took a bit of parsing to understand what\n> had to be moved and why. I don't see much requirement to split it, but\n> it might be nice to add a description to the changelog to help\n> understanding.\n> \n> From my parsing - something like:\n> \n> The LogOutput class is extended to support writing string messages\n> directly to the output. Strings written directly will be considered as\n> LogDebug messages when written to the Syslog.\n> \n> (Adapting that for correctness if you do decide to add it of course)\n\nText added, thank you.\n\n> \n>   Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n> \n> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n> > ---\n> >  meson.build           |  4 ++\n> >  src/libcamera/log.cpp | 93 ++++++++++++++++++++++++++++++++++---------\n> >  2 files changed, 79 insertions(+), 18 deletions(-)\n> > \n> > diff --git a/meson.build b/meson.build\n> > index 0a222ba96dcb..634488589a46 100644\n> > --- a/meson.build\n> > +++ b/meson.build\n> > @@ -26,6 +26,10 @@ libcamera_version = libcamera_git_version.split('+')[0]\n> >  cc = meson.get_compiler('c')\n> >  config_h = configuration_data()\n> >  \n> > +if cc.has_header_symbol('execinfo.h', 'backtrace')\n> > +    config_h.set('HAVE_BACKTRACE', 1)\n> > +endif\n> > +\n> >  if cc.has_header_symbol('stdlib.h', 'secure_getenv', prefix : '#define _GNU_SOURCE')\n> >      config_h.set('HAVE_SECURE_GETENV', 1)\n> >  endif\n> > diff --git a/src/libcamera/log.cpp b/src/libcamera/log.cpp\n> > index 50f345b98c74..f4eb8c11adc3 100644\n> > --- a/src/libcamera/log.cpp\n> > +++ b/src/libcamera/log.cpp\n> > @@ -7,6 +7,9 @@\n> >  \n> >  #include \"log.h\"\n> >  \n> > +#if HAVE_BACKTRACE\n> > +#include <execinfo.h>\n> > +#endif\n> >  #include <fstream>\n> >  #include <iostream>\n> >  #include <list>\n> > @@ -108,10 +111,11 @@ public:\n> >  \n> >  \tbool isValid() const;\n> >  \tvoid write(const LogMessage &msg);\n> > +\tvoid write(const std::string &msg);\n> >  \n> >  private:\n> > -\tvoid writeSyslog(const LogMessage &msg);\n> > -\tvoid writeStream(const LogMessage &msg);\n> > +\tvoid writeSyslog(LogSeverity severity, const std::string &msg);\n> > +\tvoid writeStream(const std::string &msg);\n> >  \n> >  \tstd::ostream *stream_;\n> >  \tLoggingTarget target_;\n> > @@ -180,34 +184,55 @@ bool LogOutput::isValid() const\n> >   * \\param[in] msg Message to write\n> >   */\n> >  void LogOutput::write(const LogMessage &msg)\n> > +{\n> > +\tstd::string str;\n> > +\n> > +\tswitch (target_) {\n> > +\tcase LoggingTargetSyslog:\n> > +\t\tstr = std::string(log_severity_name(msg.severity())) + \" \"\n> > +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> > +\t\t    + msg.msg();\n> > +\t\twriteSyslog(msg.severity(), str);\n> > +\t\tbreak;\n> > +\tcase LoggingTargetStream:\n> > +\tcase LoggingTargetFile:\n> > +\t\tstr = \"[\" + utils::time_point_to_string(msg.timestamp()) + \"]\"\n> > +\t\t    + log_severity_name(msg.severity()) + \" \"\n> > +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> > +\t\t    + msg.msg();\n> > +\t\twriteStream(str);\n> > +\t\tbreak;\n> > +\tdefault:\n> > +\t\tbreak;\n> > +\t}\n> > +}\n> > +\n> > +/**\n> > + * \\brief Write string to log output\n> > + * \\param[in] str String to write\n> > + */\n> > +void LogOutput::write(const std::string &str)\n> >  {\n> >  \tswitch (target_) {\n> >  \tcase LoggingTargetSyslog:\n> > -\t\twriteSyslog(msg);\n> > +\t\twriteSyslog(LogDebug, str);\n> >  \t\tbreak;\n> >  \tcase LoggingTargetStream:\n> >  \tcase LoggingTargetFile:\n> > -\t\twriteStream(msg);\n> > +\t\twriteStream(str);\n> >  \t\tbreak;\n> >  \tdefault:\n> >  \t\tbreak;\n> >  \t}\n> >  }\n> >  \n> > -void LogOutput::writeSyslog(const LogMessage &msg)\n> > +void LogOutput::writeSyslog(LogSeverity severity, const std::string &str)\n> >  {\n> > -\tstd::string str = std::string(log_severity_name(msg.severity())) + \" \" +\n> > -\t      \t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> > -\t\t\t  msg.msg();\n> > -\tsyslog(log_severity_to_syslog(msg.severity()), \"%s\", str.c_str());\n> > +\tsyslog(log_severity_to_syslog(severity), \"%s\", str.c_str());\n> >  }\n> >  \n> > -void LogOutput::writeStream(const LogMessage &msg)\n> > +void LogOutput::writeStream(const std::string &str)\n> >  {\n> > -\tstd::string str = \"[\" + utils::time_point_to_string(msg.timestamp()) +\n> > -\t\t\t  \"]\" + log_severity_name(msg.severity()) + \" \" +\n> > -\t\t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> > -\t\t\t  msg.msg();\n> >  \tstream_->write(str.c_str(), str.size());\n> >  \tstream_->flush();\n> >  }\n> > @@ -223,6 +248,7 @@ public:\n> >  \tstatic Logger *instance();\n> >  \n> >  \tvoid write(const LogMessage &msg);\n> > +\tvoid backtrace();\n> >  \n> >  \tint logSetFile(const char *path);\n> >  \tint logSetStream(std::ostream *stream);\n> > @@ -240,9 +266,6 @@ private:\n> >  \tvoid registerCategory(LogCategory *category);\n> >  \tvoid unregisterCategory(LogCategory *category);\n> >  \n> > -\tvoid writeSyslog(const LogMessage &msg);\n> > -\tvoid writeStream(const LogMessage &msg);\n> > -\n> >  \tstd::unordered_set<LogCategory *> categories_;\n> >  \tstd::list<std::pair<std::string, LogSeverity>> levels_;\n> >  \n> > @@ -370,6 +393,38 @@ void Logger::write(const LogMessage &msg)\n> >  \toutput->write(msg);\n> >  }\n> >  \n> > +/**\n> > + * \\brief Write a backtrace to the log\n> > + */\n> > +void Logger::backtrace()\n> > +{\n> > +#if HAVE_BACKTRACE\n> \n> Will this still be usable/desirable on release builds when\n> function/symbol names are possibly unavailable?\n> \n> If not - then we might want to make this:\n> \n> \t#if HAVE_BACKTRACE && !defined(NDEBUG)\n> \n> But I'm not entirely averse to having the backtrace in regardless in\n> fact, as it will likely backtrace all the way into the calling\n> application (which should have symbols if it's being developed) and\n> could add value in that scenario regardless.\n\nI think it's still desirable, as I'm using it with release builds :-)\n\n> > +\tstd::shared_ptr<LogOutput> output = std::atomic_load(&output_);\n> > +\tif (!output)\n> > +\t\treturn;\n> > +\n> > +\tvoid *buffer[32];\n> > +\tint num_entries = ::backtrace(buffer, ARRAY_SIZE(buffer));\n> > +\tchar **strings = backtrace_symbols(buffer, num_entries);\n> > +\tif (!strings)\n> > +\t\treturn;\n> > +\n> > +\tstd::ostringstream msg;\n> > +\tmsg << \"Backtrace:\" << std::endl;\n> > +\n> > +\t/*\n> > +\t * Skip the first two entries that correspond to this method and\n> > +\t * ~LogMessage().\n> > +\t */\n> > +\tfor (int i = 2; i < num_entries; ++i)\n> > +\t\tmsg << strings[i] << std::endl;\n> > +\n> > +\toutput->write(msg.str());\n> > +\n> > +\tfree(strings);\n> > +#endif\n> > +}\n> > +\n> >  /**\n> >   * \\brief Set the log file\n> >   * \\param[in] path Full path to the log file\n> > @@ -783,8 +838,10 @@ LogMessage::~LogMessage()\n> >  \tif (severity_ >= category_.severity())\n> >  \t\tLogger::instance()->write(*this);\n> >  \n> > -\tif (severity_ == LogSeverity::LogFatal)\n> > +\tif (severity_ == LogSeverity::LogFatal) {\n> > +\t\tLogger::instance()->backtrace();\n> >  \t\tstd::abort();\n> > +\t}\n> >  }\n> >  \n> >  /**","headers":{"Return-Path":"<laurent.pinchart@ideasonboard.com>","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7E30460C33\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tTue, 26 Nov 2019 03:02:00 +0100 (CET)","from pendragon.ideasonboard.com (unknown [104.132.253.101])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 381B3554;\n\tTue, 26 Nov 2019 03:01:57 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1574733719;\n\tbh=YA95PxQOqZgvs9vMysD3ZUqqyknfxlPAT1ZTLL9di2M=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=hDxyg8usCcJSwHgIf5qUUIGunBU0ugt1CzVrLRQuQP+oNyqz1XDn2OrWu0QK/TwoS\n\tfFn7u2dbMv8JoLzUyIRRtZU+S4Wh0wW//SvvpT6IFc9nakHYi2rEza03yDuOdXU6x+\n\t5sucXtsqAVDv9cZFdAfc2xgVUccgQrEEKiV0iwi8=","Date":"Tue, 26 Nov 2019 04:01:48 +0200","From":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>","To":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Cc":"libcamera-devel@lists.libcamera.org","Message-ID":"<20191126020148.GA4705@pendragon.ideasonboard.com>","References":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>\n\t<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>","User-Agent":"Mutt/1.10.1 (2018-07-13)","Subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","X-List-Received-Date":"Tue, 26 Nov 2019 02:02:00 -0000"}},{"id":3138,"web_url":"https://patchwork.libcamera.org/comment/3138/","msgid":"<1b79482c-e8ca-4a18-743f-0252e55a30de@ideasonboard.com>","date":"2019-11-26T09:55:12","subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 26/11/2019 02:01, Laurent Pinchart wrote:\n> Hi Kieran,\n> \n> On Mon, Nov 25, 2019 at 10:57:53AM +0000, Kieran Bingham wrote:\n>> On 23/11/2019 12:43, Laurent Pinchart wrote:\n>>> When a fatal error occurs the program aborts, and all the logger\n>>> provides is the location of the line that caused the error. Extend this\n>>> with a full backtrace to help debugging.\n>>>\n>>> The backtrace is generated using the backtrace() call, a GNU extension\n>>> to the C library. It is available in glibc and uClibc but not in musl.\n>>> Test for availability of the function to condition compilation of the\n>>> backtrace printing. Implementing backtrace support with musl is an\n>>> exercise left to the reader if desired.\n>>\n>> Nice, I like features like this. I had this as a TODO note in one of my\n>> branches, so I'm glad you've got to it first!.\n>>\n>> Have you looked at libSegFault as well perhaps by any chance?\n> \n> I wasn't aware of it, thanks for the pointer.\n> \n>> I'm wondering if something like the following might be a useful addition\n>> in this topic:\n>>\n>> +if cc.find_library('SegFault', required: false).found()\n>> +    message('Found libSegFault')\n>> +    common_arguments += [ '-lSegFault', ]\n>> +endif\n>> +\n>>\n>> Although - also possibly conditioned on being a debug build.\n> \n> In this particular case it wouldn't have helped, as I could only\n> reproduce the issue with release builds. I thus wonder if we shouldn't\n> LD_PRELOAD libSegFault.so instead when running tests, like we use\n> valgrind on tests from time to time. Or would libasan be a better option\n> in that case ?\n> \n> What I'm trying to figure out is how much overlap there is between this\n> patch and libSegFault or libasan.\n\nWhen I tried to test out libsegfault with the above snippet, I found\nlibasan (which I have manually installed on my system) caught the traps\nfirst. I think libasan probably has more checks rather than just\ntrapping on SIGSEGV so has more value, but the key difference is\nlibsegfault is part that being part of the std gnu library, libsegfault\nprobably has more potential users.\n\nI'm still a bit bemused that I didn't have libasan explicitly enabled\nfor my build, so I suspect meson has enabled it 'because it was there'\nbut I need to get that confirmed rather than just being a theory.\n\nI think the difference is libSegFault will help reporting /where/ a\nfailure occured when it happens, whereas libasan is more likely to help\ncatch bugs earlier in the development cycle. (Note that we can\nexplicitly enable libasan functionality with meson with something like\nthe following snippets:\n\n\n... meson.build\nindex b2d52363ccae..aecace8ac556 100644\n@@ -3,10 +3,13 @@ project('libcamera', 'c', 'cpp',\n     version : '0.0.0',\n     default_options : [\n         'werror=true',\n         'warning_level=2',\n         'cpp_std=c++11',\n+        'b_sanitize=address,undefined',\n+        #'b_sanitize=thread',\n+        #'b_sanitize=memory',\n     ],\n     license : 'LGPL 2.1+')\n\nThe thread and memory sanitizers are mutually exclusive against the\naddress,undefined-behaviour sanitiser - so it might be useful to use\nthis in some automated test setup with three separate builds with each\nof them enabled.\n\nAnyway, all of the above is slightly unrelated to this patch, so is only\ndiscussion - as this patch reports more information on our own internal\nFatal log-level which is a great addition IMO...\n\n\n>> This works nicely for me so far, so:\n>>\n>>   Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n>>\n>> It might be argued that the LogOutput changes required here could be\n>> split to clarify them, as it took a bit of parsing to understand what\n>> had to be moved and why. I don't see much requirement to split it, but\n>> it might be nice to add a description to the changelog to help\n>> understanding.\n>>\n>> From my parsing - something like:\n>>\n>> The LogOutput class is extended to support writing string messages\n>> directly to the output. Strings written directly will be considered as\n>> LogDebug messages when written to the Syslog.\n>>\n>> (Adapting that for correctness if you do decide to add it of course)\n> \n> Text added, thank you.\n> \n>>\n>>   Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n>>\n>>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n>>> ---\n>>>  meson.build           |  4 ++\n>>>  src/libcamera/log.cpp | 93 ++++++++++++++++++++++++++++++++++---------\n>>>  2 files changed, 79 insertions(+), 18 deletions(-)\n>>>\n>>> diff --git a/meson.build b/meson.build\n>>> index 0a222ba96dcb..634488589a46 100644\n>>> --- a/meson.build\n>>> +++ b/meson.build\n>>> @@ -26,6 +26,10 @@ libcamera_version = libcamera_git_version.split('+')[0]\n>>>  cc = meson.get_compiler('c')\n>>>  config_h = configuration_data()\n>>>  \n>>> +if cc.has_header_symbol('execinfo.h', 'backtrace')\n>>> +    config_h.set('HAVE_BACKTRACE', 1)\n>>> +endif\n>>> +\n>>>  if cc.has_header_symbol('stdlib.h', 'secure_getenv', prefix : '#define _GNU_SOURCE')\n>>>      config_h.set('HAVE_SECURE_GETENV', 1)\n>>>  endif\n>>> diff --git a/src/libcamera/log.cpp b/src/libcamera/log.cpp\n>>> index 50f345b98c74..f4eb8c11adc3 100644\n>>> --- a/src/libcamera/log.cpp\n>>> +++ b/src/libcamera/log.cpp\n>>> @@ -7,6 +7,9 @@\n>>>  \n>>>  #include \"log.h\"\n>>>  \n>>> +#if HAVE_BACKTRACE\n>>> +#include <execinfo.h>\n>>> +#endif\n>>>  #include <fstream>\n>>>  #include <iostream>\n>>>  #include <list>\n>>> @@ -108,10 +111,11 @@ public:\n>>>  \n>>>  \tbool isValid() const;\n>>>  \tvoid write(const LogMessage &msg);\n>>> +\tvoid write(const std::string &msg);\n>>>  \n>>>  private:\n>>> -\tvoid writeSyslog(const LogMessage &msg);\n>>> -\tvoid writeStream(const LogMessage &msg);\n>>> +\tvoid writeSyslog(LogSeverity severity, const std::string &msg);\n>>> +\tvoid writeStream(const std::string &msg);\n>>>  \n>>>  \tstd::ostream *stream_;\n>>>  \tLoggingTarget target_;\n>>> @@ -180,34 +184,55 @@ bool LogOutput::isValid() const\n>>>   * \\param[in] msg Message to write\n>>>   */\n>>>  void LogOutput::write(const LogMessage &msg)\n>>> +{\n>>> +\tstd::string str;\n>>> +\n>>> +\tswitch (target_) {\n>>> +\tcase LoggingTargetSyslog:\n>>> +\t\tstr = std::string(log_severity_name(msg.severity())) + \" \"\n>>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n>>> +\t\t    + msg.msg();\n>>> +\t\twriteSyslog(msg.severity(), str);\n>>> +\t\tbreak;\n>>> +\tcase LoggingTargetStream:\n>>> +\tcase LoggingTargetFile:\n>>> +\t\tstr = \"[\" + utils::time_point_to_string(msg.timestamp()) + \"]\"\n>>> +\t\t    + log_severity_name(msg.severity()) + \" \"\n>>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n>>> +\t\t    + msg.msg();\n>>> +\t\twriteStream(str);\n>>> +\t\tbreak;\n>>> +\tdefault:\n>>> +\t\tbreak;\n>>> +\t}\n>>> +}\n>>> +\n>>> +/**\n>>> + * \\brief Write string to log output\n>>> + * \\param[in] str String to write\n>>> + */\n>>> +void LogOutput::write(const std::string &str)\n>>>  {\n>>>  \tswitch (target_) {\n>>>  \tcase LoggingTargetSyslog:\n>>> -\t\twriteSyslog(msg);\n>>> +\t\twriteSyslog(LogDebug, str);\n>>>  \t\tbreak;\n>>>  \tcase LoggingTargetStream:\n>>>  \tcase LoggingTargetFile:\n>>> -\t\twriteStream(msg);\n>>> +\t\twriteStream(str);\n>>>  \t\tbreak;\n>>>  \tdefault:\n>>>  \t\tbreak;\n>>>  \t}\n>>>  }\n>>>  \n>>> -void LogOutput::writeSyslog(const LogMessage &msg)\n>>> +void LogOutput::writeSyslog(LogSeverity severity, const std::string &str)\n>>>  {\n>>> -\tstd::string str = std::string(log_severity_name(msg.severity())) + \" \" +\n>>> -\t      \t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n>>> -\t\t\t  msg.msg();\n>>> -\tsyslog(log_severity_to_syslog(msg.severity()), \"%s\", str.c_str());\n>>> +\tsyslog(log_severity_to_syslog(severity), \"%s\", str.c_str());\n>>>  }\n>>>  \n>>> -void LogOutput::writeStream(const LogMessage &msg)\n>>> +void LogOutput::writeStream(const std::string &str)\n>>>  {\n>>> -\tstd::string str = \"[\" + utils::time_point_to_string(msg.timestamp()) +\n>>> -\t\t\t  \"]\" + log_severity_name(msg.severity()) + \" \" +\n>>> -\t\t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n>>> -\t\t\t  msg.msg();\n>>>  \tstream_->write(str.c_str(), str.size());\n>>>  \tstream_->flush();\n>>>  }\n>>> @@ -223,6 +248,7 @@ public:\n>>>  \tstatic Logger *instance();\n>>>  \n>>>  \tvoid write(const LogMessage &msg);\n>>> +\tvoid backtrace();\n>>>  \n>>>  \tint logSetFile(const char *path);\n>>>  \tint logSetStream(std::ostream *stream);\n>>> @@ -240,9 +266,6 @@ private:\n>>>  \tvoid registerCategory(LogCategory *category);\n>>>  \tvoid unregisterCategory(LogCategory *category);\n>>>  \n>>> -\tvoid writeSyslog(const LogMessage &msg);\n>>> -\tvoid writeStream(const LogMessage &msg);\n>>> -\n>>>  \tstd::unordered_set<LogCategory *> categories_;\n>>>  \tstd::list<std::pair<std::string, LogSeverity>> levels_;\n>>>  \n>>> @@ -370,6 +393,38 @@ void Logger::write(const LogMessage &msg)\n>>>  \toutput->write(msg);\n>>>  }\n>>>  \n>>> +/**\n>>> + * \\brief Write a backtrace to the log\n>>> + */\n>>> +void Logger::backtrace()\n>>> +{\n>>> +#if HAVE_BACKTRACE\n>>\n>> Will this still be usable/desirable on release builds when\n>> function/symbol names are possibly unavailable?\n>>\n>> If not - then we might want to make this:\n>>\n>> \t#if HAVE_BACKTRACE && !defined(NDEBUG)\n>>\n>> But I'm not entirely averse to having the backtrace in regardless in\n>> fact, as it will likely backtrace all the way into the calling\n>> application (which should have symbols if it's being developed) and\n>> could add value in that scenario regardless.\n> \n> I think it's still desirable, as I'm using it with release builds :-)\n\n\nExcellent - I thought a release build would have stripped out the\nrequired symbol names.\n\nI certainly see some benefit to having this anyway.\n\nIt will be even more beneficial if we later make Fatal 'non-fatal' in\nrelease builds, as there we really will want as much debug information\nto be reportable as possible, and Fatal will then give us a route to say\n\n \"This is reallllly bad, we're trying to continue - but please report\nthis issue\"\n\n(And in debug builds, I very much prefer the fail-fast abort on a fatal).\n\n--\nKieran\n\n\n\n\n>>> +\tstd::shared_ptr<LogOutput> output = std::atomic_load(&output_);\n>>> +\tif (!output)\n>>> +\t\treturn;\n>>> +\n>>> +\tvoid *buffer[32];\n>>> +\tint num_entries = ::backtrace(buffer, ARRAY_SIZE(buffer));\n>>> +\tchar **strings = backtrace_symbols(buffer, num_entries);\n>>> +\tif (!strings)\n>>> +\t\treturn;\n>>> +\n>>> +\tstd::ostringstream msg;\n>>> +\tmsg << \"Backtrace:\" << std::endl;\n>>> +\n>>> +\t/*\n>>> +\t * Skip the first two entries that correspond to this method and\n>>> +\t * ~LogMessage().\n>>> +\t */\n>>> +\tfor (int i = 2; i < num_entries; ++i)\n>>> +\t\tmsg << strings[i] << std::endl;\n>>> +\n>>> +\toutput->write(msg.str());\n>>> +\n>>> +\tfree(strings);\n>>> +#endif\n>>> +}\n>>> +\n>>>  /**\n>>>   * \\brief Set the log file\n>>>   * \\param[in] path Full path to the log file\n>>> @@ -783,8 +838,10 @@ LogMessage::~LogMessage()\n>>>  \tif (severity_ >= category_.severity())\n>>>  \t\tLogger::instance()->write(*this);\n>>>  \n>>> -\tif (severity_ == LogSeverity::LogFatal)\n>>> +\tif (severity_ == LogSeverity::LogFatal) {\n>>> +\t\tLogger::instance()->backtrace();\n>>>  \t\tstd::abort();\n>>> +\t}\n>>>  }\n>>>  \n>>>  /**\n>","headers":{"Return-Path":"<kieran.bingham@ideasonboard.com>","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9B63A60C33\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tTue, 26 Nov 2019 10:55:16 +0100 (CET)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id B7F0E554;\n\tTue, 26 Nov 2019 10:55:15 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1574762116;\n\tbh=K3To73SwQKhR5QCeTVkEJO0eMQZvhRlHPAViW+BeA+c=;\n\th=Reply-To:Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=D9DxC5Co2UVv3OtTYqe6YJclX7O9+sNiQ3f9SgeVuYDVl3W9LaeSvPS6EJcrUlFge\n\tr1IQd0qp73Jo6B+DuHhQXV/q4cd+g+t1EFticH6Ixn81R5O6XbNF+sytpHG5f6P43V\n\tFy214FFlU4Mho7O0w1BBmsWal5oJw0cvyRwEaVqY=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>","Cc":"libcamera-devel@lists.libcamera.org","References":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>\n\t<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>\n\t<20191126020148.GA4705@pendragon.ideasonboard.com>","From":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Openpgp":"preference=signencrypt","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<1b79482c-e8ca-4a18-743f-0252e55a30de@ideasonboard.com>","Date":"Tue, 26 Nov 2019 09:55:12 +0000","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101\n\tThunderbird/60.9.0","MIME-Version":"1.0","In-Reply-To":"<20191126020148.GA4705@pendragon.ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","X-List-Received-Date":"Tue, 26 Nov 2019 09:55:16 -0000"}},{"id":3419,"web_url":"https://patchwork.libcamera.org/comment/3419/","msgid":"<20200111030724.GX4859@pendragon.ideasonboard.com>","date":"2020-01-11T03:07:24","subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Kieran,\n\nOn Tue, Nov 26, 2019 at 09:55:12AM +0000, Kieran Bingham wrote:\n> On 26/11/2019 02:01, Laurent Pinchart wrote:\n> > On Mon, Nov 25, 2019 at 10:57:53AM +0000, Kieran Bingham wrote:\n> >> On 23/11/2019 12:43, Laurent Pinchart wrote:\n> >>> When a fatal error occurs the program aborts, and all the logger\n> >>> provides is the location of the line that caused the error. Extend this\n> >>> with a full backtrace to help debugging.\n> >>>\n> >>> The backtrace is generated using the backtrace() call, a GNU extension\n> >>> to the C library. It is available in glibc and uClibc but not in musl.\n> >>> Test for availability of the function to condition compilation of the\n> >>> backtrace printing. Implementing backtrace support with musl is an\n> >>> exercise left to the reader if desired.\n> >>\n> >> Nice, I like features like this. I had this as a TODO note in one of my\n> >> branches, so I'm glad you've got to it first!.\n> >>\n> >> Have you looked at libSegFault as well perhaps by any chance?\n> > \n> > I wasn't aware of it, thanks for the pointer.\n> > \n> >> I'm wondering if something like the following might be a useful addition\n> >> in this topic:\n> >>\n> >> +if cc.find_library('SegFault', required: false).found()\n> >> +    message('Found libSegFault')\n> >> +    common_arguments += [ '-lSegFault', ]\n> >> +endif\n> >> +\n> >>\n> >> Although - also possibly conditioned on being a debug build.\n> > \n> > In this particular case it wouldn't have helped, as I could only\n> > reproduce the issue with release builds. I thus wonder if we shouldn't\n> > LD_PRELOAD libSegFault.so instead when running tests, like we use\n> > valgrind on tests from time to time. Or would libasan be a better option\n> > in that case ?\n> > \n> > What I'm trying to figure out is how much overlap there is between this\n> > patch and libSegFault or libasan.\n> \n> When I tried to test out libsegfault with the above snippet, I found\n> libasan (which I have manually installed on my system) caught the traps\n> first. I think libasan probably has more checks rather than just\n> trapping on SIGSEGV so has more value, but the key difference is\n> libsegfault is part that being part of the std gnu library, libsegfault\n> probably has more potential users.\n> \n> I'm still a bit bemused that I didn't have libasan explicitly enabled\n> for my build, so I suspect meson has enabled it 'because it was there'\n> but I need to get that confirmed rather than just being a theory.\n\nHave you had a chance to investigate this ?\n\n> I think the difference is libSegFault will help reporting /where/ a\n> failure occured when it happens, whereas libasan is more likely to help\n> catch bugs earlier in the development cycle. (Note that we can\n> explicitly enable libasan functionality with meson with something like\n> the following snippets:\n> \n> \n> ... meson.build\n> index b2d52363ccae..aecace8ac556 100644\n> @@ -3,10 +3,13 @@ project('libcamera', 'c', 'cpp',\n>      version : '0.0.0',\n>      default_options : [\n>          'werror=true',\n>          'warning_level=2',\n>          'cpp_std=c++11',\n> +        'b_sanitize=address,undefined',\n> +        #'b_sanitize=thread',\n> +        #'b_sanitize=memory',\n>      ],\n>      license : 'LGPL 2.1+')\n> \n> The thread and memory sanitizers are mutually exclusive against the\n> address,undefined-behaviour sanitiser - so it might be useful to use\n> this in some automated test setup with three separate builds with each\n> of them enabled.\n\nAgreed.\n\n> Anyway, all of the above is slightly unrelated to this patch, so is only\n> discussion - as this patch reports more information on our own internal\n> Fatal log-level which is a great addition IMO...\n> \n> >> This works nicely for me so far, so:\n> >>\n> >>   Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n> >>\n> >> It might be argued that the LogOutput changes required here could be\n> >> split to clarify them, as it took a bit of parsing to understand what\n> >> had to be moved and why. I don't see much requirement to split it, but\n> >> it might be nice to add a description to the changelog to help\n> >> understanding.\n> >>\n> >> From my parsing - something like:\n> >>\n> >> The LogOutput class is extended to support writing string messages\n> >> directly to the output. Strings written directly will be considered as\n> >> LogDebug messages when written to the Syslog.\n> >>\n> >> (Adapting that for correctness if you do decide to add it of course)\n> > \n> > Text added, thank you.\n> > \n> >>\n> >>   Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n> >>\n> >>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n> >>> ---\n> >>>  meson.build           |  4 ++\n> >>>  src/libcamera/log.cpp | 93 ++++++++++++++++++++++++++++++++++---------\n> >>>  2 files changed, 79 insertions(+), 18 deletions(-)\n> >>>\n> >>> diff --git a/meson.build b/meson.build\n> >>> index 0a222ba96dcb..634488589a46 100644\n> >>> --- a/meson.build\n> >>> +++ b/meson.build\n> >>> @@ -26,6 +26,10 @@ libcamera_version = libcamera_git_version.split('+')[0]\n> >>>  cc = meson.get_compiler('c')\n> >>>  config_h = configuration_data()\n> >>>  \n> >>> +if cc.has_header_symbol('execinfo.h', 'backtrace')\n> >>> +    config_h.set('HAVE_BACKTRACE', 1)\n> >>> +endif\n> >>> +\n> >>>  if cc.has_header_symbol('stdlib.h', 'secure_getenv', prefix : '#define _GNU_SOURCE')\n> >>>      config_h.set('HAVE_SECURE_GETENV', 1)\n> >>>  endif\n> >>> diff --git a/src/libcamera/log.cpp b/src/libcamera/log.cpp\n> >>> index 50f345b98c74..f4eb8c11adc3 100644\n> >>> --- a/src/libcamera/log.cpp\n> >>> +++ b/src/libcamera/log.cpp\n> >>> @@ -7,6 +7,9 @@\n> >>>  \n> >>>  #include \"log.h\"\n> >>>  \n> >>> +#if HAVE_BACKTRACE\n> >>> +#include <execinfo.h>\n> >>> +#endif\n> >>>  #include <fstream>\n> >>>  #include <iostream>\n> >>>  #include <list>\n> >>> @@ -108,10 +111,11 @@ public:\n> >>>  \n> >>>  \tbool isValid() const;\n> >>>  \tvoid write(const LogMessage &msg);\n> >>> +\tvoid write(const std::string &msg);\n> >>>  \n> >>>  private:\n> >>> -\tvoid writeSyslog(const LogMessage &msg);\n> >>> -\tvoid writeStream(const LogMessage &msg);\n> >>> +\tvoid writeSyslog(LogSeverity severity, const std::string &msg);\n> >>> +\tvoid writeStream(const std::string &msg);\n> >>>  \n> >>>  \tstd::ostream *stream_;\n> >>>  \tLoggingTarget target_;\n> >>> @@ -180,34 +184,55 @@ bool LogOutput::isValid() const\n> >>>   * \\param[in] msg Message to write\n> >>>   */\n> >>>  void LogOutput::write(const LogMessage &msg)\n> >>> +{\n> >>> +\tstd::string str;\n> >>> +\n> >>> +\tswitch (target_) {\n> >>> +\tcase LoggingTargetSyslog:\n> >>> +\t\tstr = std::string(log_severity_name(msg.severity())) + \" \"\n> >>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> >>> +\t\t    + msg.msg();\n> >>> +\t\twriteSyslog(msg.severity(), str);\n> >>> +\t\tbreak;\n> >>> +\tcase LoggingTargetStream:\n> >>> +\tcase LoggingTargetFile:\n> >>> +\t\tstr = \"[\" + utils::time_point_to_string(msg.timestamp()) + \"]\"\n> >>> +\t\t    + log_severity_name(msg.severity()) + \" \"\n> >>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n> >>> +\t\t    + msg.msg();\n> >>> +\t\twriteStream(str);\n> >>> +\t\tbreak;\n> >>> +\tdefault:\n> >>> +\t\tbreak;\n> >>> +\t}\n> >>> +}\n> >>> +\n> >>> +/**\n> >>> + * \\brief Write string to log output\n> >>> + * \\param[in] str String to write\n> >>> + */\n> >>> +void LogOutput::write(const std::string &str)\n> >>>  {\n> >>>  \tswitch (target_) {\n> >>>  \tcase LoggingTargetSyslog:\n> >>> -\t\twriteSyslog(msg);\n> >>> +\t\twriteSyslog(LogDebug, str);\n> >>>  \t\tbreak;\n> >>>  \tcase LoggingTargetStream:\n> >>>  \tcase LoggingTargetFile:\n> >>> -\t\twriteStream(msg);\n> >>> +\t\twriteStream(str);\n> >>>  \t\tbreak;\n> >>>  \tdefault:\n> >>>  \t\tbreak;\n> >>>  \t}\n> >>>  }\n> >>>  \n> >>> -void LogOutput::writeSyslog(const LogMessage &msg)\n> >>> +void LogOutput::writeSyslog(LogSeverity severity, const std::string &str)\n> >>>  {\n> >>> -\tstd::string str = std::string(log_severity_name(msg.severity())) + \" \" +\n> >>> -\t      \t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> >>> -\t\t\t  msg.msg();\n> >>> -\tsyslog(log_severity_to_syslog(msg.severity()), \"%s\", str.c_str());\n> >>> +\tsyslog(log_severity_to_syslog(severity), \"%s\", str.c_str());\n> >>>  }\n> >>>  \n> >>> -void LogOutput::writeStream(const LogMessage &msg)\n> >>> +void LogOutput::writeStream(const std::string &str)\n> >>>  {\n> >>> -\tstd::string str = \"[\" + utils::time_point_to_string(msg.timestamp()) +\n> >>> -\t\t\t  \"]\" + log_severity_name(msg.severity()) + \" \" +\n> >>> -\t\t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n> >>> -\t\t\t  msg.msg();\n> >>>  \tstream_->write(str.c_str(), str.size());\n> >>>  \tstream_->flush();\n> >>>  }\n> >>> @@ -223,6 +248,7 @@ public:\n> >>>  \tstatic Logger *instance();\n> >>>  \n> >>>  \tvoid write(const LogMessage &msg);\n> >>> +\tvoid backtrace();\n> >>>  \n> >>>  \tint logSetFile(const char *path);\n> >>>  \tint logSetStream(std::ostream *stream);\n> >>> @@ -240,9 +266,6 @@ private:\n> >>>  \tvoid registerCategory(LogCategory *category);\n> >>>  \tvoid unregisterCategory(LogCategory *category);\n> >>>  \n> >>> -\tvoid writeSyslog(const LogMessage &msg);\n> >>> -\tvoid writeStream(const LogMessage &msg);\n> >>> -\n> >>>  \tstd::unordered_set<LogCategory *> categories_;\n> >>>  \tstd::list<std::pair<std::string, LogSeverity>> levels_;\n> >>>  \n> >>> @@ -370,6 +393,38 @@ void Logger::write(const LogMessage &msg)\n> >>>  \toutput->write(msg);\n> >>>  }\n> >>>  \n> >>> +/**\n> >>> + * \\brief Write a backtrace to the log\n> >>> + */\n> >>> +void Logger::backtrace()\n> >>> +{\n> >>> +#if HAVE_BACKTRACE\n> >>\n> >> Will this still be usable/desirable on release builds when\n> >> function/symbol names are possibly unavailable?\n> >>\n> >> If not - then we might want to make this:\n> >>\n> >> \t#if HAVE_BACKTRACE && !defined(NDEBUG)\n> >>\n> >> But I'm not entirely averse to having the backtrace in regardless in\n> >> fact, as it will likely backtrace all the way into the calling\n> >> application (which should have symbols if it's being developed) and\n> >> could add value in that scenario regardless.\n> > \n> > I think it's still desirable, as I'm using it with release builds :-)\n> \n> \n> Excellent - I thought a release build would have stripped out the\n> required symbol names.\n> \n> I certainly see some benefit to having this anyway.\n> \n> It will be even more beneficial if we later make Fatal 'non-fatal' in\n> release builds, as there we really will want as much debug information\n> to be reportable as possible, and Fatal will then give us a route to say\n> \n>  \"This is reallllly bad, we're trying to continue - but please report\n> this issue\"\n> \n> (And in debug builds, I very much prefer the fail-fast abort on a fatal).\n> \n> >>> +\tstd::shared_ptr<LogOutput> output = std::atomic_load(&output_);\n> >>> +\tif (!output)\n> >>> +\t\treturn;\n> >>> +\n> >>> +\tvoid *buffer[32];\n> >>> +\tint num_entries = ::backtrace(buffer, ARRAY_SIZE(buffer));\n> >>> +\tchar **strings = backtrace_symbols(buffer, num_entries);\n> >>> +\tif (!strings)\n> >>> +\t\treturn;\n> >>> +\n> >>> +\tstd::ostringstream msg;\n> >>> +\tmsg << \"Backtrace:\" << std::endl;\n> >>> +\n> >>> +\t/*\n> >>> +\t * Skip the first two entries that correspond to this method and\n> >>> +\t * ~LogMessage().\n> >>> +\t */\n> >>> +\tfor (int i = 2; i < num_entries; ++i)\n> >>> +\t\tmsg << strings[i] << std::endl;\n> >>> +\n> >>> +\toutput->write(msg.str());\n> >>> +\n> >>> +\tfree(strings);\n> >>> +#endif\n> >>> +}\n> >>> +\n> >>>  /**\n> >>>   * \\brief Set the log file\n> >>>   * \\param[in] path Full path to the log file\n> >>> @@ -783,8 +838,10 @@ LogMessage::~LogMessage()\n> >>>  \tif (severity_ >= category_.severity())\n> >>>  \t\tLogger::instance()->write(*this);\n> >>>  \n> >>> -\tif (severity_ == LogSeverity::LogFatal)\n> >>> +\tif (severity_ == LogSeverity::LogFatal) {\n> >>> +\t\tLogger::instance()->backtrace();\n> >>>  \t\tstd::abort();\n> >>> +\t}\n> >>>  }\n> >>>  \n> >>>  /**","headers":{"Return-Path":"<laurent.pinchart@ideasonboard.com>","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 536CB6045A\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tSat, 11 Jan 2020 04:07:39 +0100 (CET)","from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi\n\t[81.175.216.236])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id AD64A9B1;\n\tSat, 11 Jan 2020 04:07:38 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1578712058;\n\tbh=pLEUZ/spFwW6eh0TZn9qV+F4kfbbWx1e0YlcVfKBtPk=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=QwMArEXTotirgCvrrjiXQpaVsPpNCajaP//1dMnH7f7yfqiP0dpWXQmefYtlubQUS\n\t/GpjPLGp5Z/dc79gbC8EWRPpYq0W3wukp4DYgdtXHpLgVoXLPRJfbAck8xBUHxvXWM\n\trmW+cht9XIXt586fv1F/A19HFonYUHAZTQu0l3Wg=","Date":"Sat, 11 Jan 2020 05:07:24 +0200","From":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>","To":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Cc":"libcamera-devel@lists.libcamera.org","Message-ID":"<20200111030724.GX4859@pendragon.ideasonboard.com>","References":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>\n\t<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>\n\t<20191126020148.GA4705@pendragon.ideasonboard.com>\n\t<1b79482c-e8ca-4a18-743f-0252e55a30de@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<1b79482c-e8ca-4a18-743f-0252e55a30de@ideasonboard.com>","User-Agent":"Mutt/1.10.1 (2018-07-13)","Subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","X-List-Received-Date":"Sat, 11 Jan 2020 03:07:39 -0000"}},{"id":3566,"web_url":"https://patchwork.libcamera.org/comment/3566/","msgid":"<76ddf46c-95fb-b848-2985-d28f67af647e@ideasonboard.com>","date":"2020-01-22T15:17:27","subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","submitter":{"id":4,"url":"https://patchwork.libcamera.org/api/people/4/","name":"Kieran Bingham","email":"kieran.bingham@ideasonboard.com"},"content":"Hi Laurent,\n\nOn 11/01/2020 03:07, Laurent Pinchart wrote:\n> Hi Kieran,\n> \n> On Tue, Nov 26, 2019 at 09:55:12AM +0000, Kieran Bingham wrote:\n>> On 26/11/2019 02:01, Laurent Pinchart wrote:\n>>> On Mon, Nov 25, 2019 at 10:57:53AM +0000, Kieran Bingham wrote:\n>>>> On 23/11/2019 12:43, Laurent Pinchart wrote:\n>>>>> When a fatal error occurs the program aborts, and all the logger\n>>>>> provides is the location of the line that caused the error. Extend this\n>>>>> with a full backtrace to help debugging.\n>>>>>\n>>>>> The backtrace is generated using the backtrace() call, a GNU extension\n>>>>> to the C library. It is available in glibc and uClibc but not in musl.\n>>>>> Test for availability of the function to condition compilation of the\n>>>>> backtrace printing. Implementing backtrace support with musl is an\n>>>>> exercise left to the reader if desired.\n>>>>\n>>>> Nice, I like features like this. I had this as a TODO note in one of my\n>>>> branches, so I'm glad you've got to it first!.\n>>>>\n>>>> Have you looked at libSegFault as well perhaps by any chance?\n>>>\n>>> I wasn't aware of it, thanks for the pointer.\n>>>\n>>>> I'm wondering if something like the following might be a useful addition\n>>>> in this topic:\n>>>>\n>>>> +if cc.find_library('SegFault', required: false).found()\n>>>> +    message('Found libSegFault')\n>>>> +    common_arguments += [ '-lSegFault', ]\n>>>> +endif\n>>>> +\n>>>>\n>>>> Although - also possibly conditioned on being a debug build.\n>>>\n>>> In this particular case it wouldn't have helped, as I could only\n>>> reproduce the issue with release builds. I thus wonder if we shouldn't\n>>> LD_PRELOAD libSegFault.so instead when running tests, like we use\n>>> valgrind on tests from time to time. Or would libasan be a better option\n>>> in that case ?\n>>>\n>>> What I'm trying to figure out is how much overlap there is between this\n>>> patch and libSegFault or libasan.\n>>\n>> When I tried to test out libsegfault with the above snippet, I found\n>> libasan (which I have manually installed on my system) caught the traps\n>> first. I think libasan probably has more checks rather than just\n>> trapping on SIGSEGV so has more value, but the key difference is\n>> libsegfault is part that being part of the std gnu library, libsegfault\n>> probably has more potential users.\n>>\n>> I'm still a bit bemused that I didn't have libasan explicitly enabled\n>> for my build, so I suspect meson has enabled it 'because it was there'\n>> but I need to get that confirmed rather than just being a theory.\n> \n> Have you had a chance to investigate this ?\n\nI expect it was because I hadn't cleaned my build at the time.\n\nMuch as we've recently seen how changes to these default options\n(cpp_std=c++11) does not cause a reconfigure (and as they are 'default\noptions' nor would I in fact expect them to).\n\nOk, so I can confirm, the libasan will not be used unless\nb_sanitize=address,undefined is added to the configure options.\n\nMeson doesn't do anything to jsut \"use it because it's there\".\n\n--\nKieran\n\n\n\n>> I think the difference is libSegFault will help reporting /where/ a\n>> failure occured when it happens, whereas libasan is more likely to help\n>> catch bugs earlier in the development cycle. (Note that we can\n>> explicitly enable libasan functionality with meson with something like\n>> the following snippets:\n>>\n>>\n>> ... meson.build\n>> index b2d52363ccae..aecace8ac556 100644\n>> @@ -3,10 +3,13 @@ project('libcamera', 'c', 'cpp',\n>>      version : '0.0.0',\n>>      default_options : [\n>>          'werror=true',\n>>          'warning_level=2',\n>>          'cpp_std=c++11',\n>> +        'b_sanitize=address,undefined',\n>> +        #'b_sanitize=thread',\n>> +        #'b_sanitize=memory',\n>>      ],\n>>      license : 'LGPL 2.1+')\n>>\n>> The thread and memory sanitizers are mutually exclusive against the\n>> address,undefined-behaviour sanitiser - so it might be useful to use\n>> this in some automated test setup with three separate builds with each\n>> of them enabled.\n> \n> Agreed.\n> \n>> Anyway, all of the above is slightly unrelated to this patch, so is only\n>> discussion - as this patch reports more information on our own internal\n>> Fatal log-level which is a great addition IMO...\n>>\n>>>> This works nicely for me so far, so:\n>>>>\n>>>>   Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n>>>>\n>>>> It might be argued that the LogOutput changes required here could be\n>>>> split to clarify them, as it took a bit of parsing to understand what\n>>>> had to be moved and why. I don't see much requirement to split it, but\n>>>> it might be nice to add a description to the changelog to help\n>>>> understanding.\n>>>>\n>>>> From my parsing - something like:\n>>>>\n>>>> The LogOutput class is extended to support writing string messages\n>>>> directly to the output. Strings written directly will be considered as\n>>>> LogDebug messages when written to the Syslog.\n>>>>\n>>>> (Adapting that for correctness if you do decide to add it of course)\n>>>\n>>> Text added, thank you.\n>>>\n>>>>\n>>>>   Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>\n>>>>\n>>>>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>\n>>>>> ---\n>>>>>  meson.build           |  4 ++\n>>>>>  src/libcamera/log.cpp | 93 ++++++++++++++++++++++++++++++++++---------\n>>>>>  2 files changed, 79 insertions(+), 18 deletions(-)\n>>>>>\n>>>>> diff --git a/meson.build b/meson.build\n>>>>> index 0a222ba96dcb..634488589a46 100644\n>>>>> --- a/meson.build\n>>>>> +++ b/meson.build\n>>>>> @@ -26,6 +26,10 @@ libcamera_version = libcamera_git_version.split('+')[0]\n>>>>>  cc = meson.get_compiler('c')\n>>>>>  config_h = configuration_data()\n>>>>>  \n>>>>> +if cc.has_header_symbol('execinfo.h', 'backtrace')\n>>>>> +    config_h.set('HAVE_BACKTRACE', 1)\n>>>>> +endif\n>>>>> +\n>>>>>  if cc.has_header_symbol('stdlib.h', 'secure_getenv', prefix : '#define _GNU_SOURCE')\n>>>>>      config_h.set('HAVE_SECURE_GETENV', 1)\n>>>>>  endif\n>>>>> diff --git a/src/libcamera/log.cpp b/src/libcamera/log.cpp\n>>>>> index 50f345b98c74..f4eb8c11adc3 100644\n>>>>> --- a/src/libcamera/log.cpp\n>>>>> +++ b/src/libcamera/log.cpp\n>>>>> @@ -7,6 +7,9 @@\n>>>>>  \n>>>>>  #include \"log.h\"\n>>>>>  \n>>>>> +#if HAVE_BACKTRACE\n>>>>> +#include <execinfo.h>\n>>>>> +#endif\n>>>>>  #include <fstream>\n>>>>>  #include <iostream>\n>>>>>  #include <list>\n>>>>> @@ -108,10 +111,11 @@ public:\n>>>>>  \n>>>>>  \tbool isValid() const;\n>>>>>  \tvoid write(const LogMessage &msg);\n>>>>> +\tvoid write(const std::string &msg);\n>>>>>  \n>>>>>  private:\n>>>>> -\tvoid writeSyslog(const LogMessage &msg);\n>>>>> -\tvoid writeStream(const LogMessage &msg);\n>>>>> +\tvoid writeSyslog(LogSeverity severity, const std::string &msg);\n>>>>> +\tvoid writeStream(const std::string &msg);\n>>>>>  \n>>>>>  \tstd::ostream *stream_;\n>>>>>  \tLoggingTarget target_;\n>>>>> @@ -180,34 +184,55 @@ bool LogOutput::isValid() const\n>>>>>   * \\param[in] msg Message to write\n>>>>>   */\n>>>>>  void LogOutput::write(const LogMessage &msg)\n>>>>> +{\n>>>>> +\tstd::string str;\n>>>>> +\n>>>>> +\tswitch (target_) {\n>>>>> +\tcase LoggingTargetSyslog:\n>>>>> +\t\tstr = std::string(log_severity_name(msg.severity())) + \" \"\n>>>>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n>>>>> +\t\t    + msg.msg();\n>>>>> +\t\twriteSyslog(msg.severity(), str);\n>>>>> +\t\tbreak;\n>>>>> +\tcase LoggingTargetStream:\n>>>>> +\tcase LoggingTargetFile:\n>>>>> +\t\tstr = \"[\" + utils::time_point_to_string(msg.timestamp()) + \"]\"\n>>>>> +\t\t    + log_severity_name(msg.severity()) + \" \"\n>>>>> +\t\t    + msg.category().name() + \" \" + msg.fileInfo() + \" \"\n>>>>> +\t\t    + msg.msg();\n>>>>> +\t\twriteStream(str);\n>>>>> +\t\tbreak;\n>>>>> +\tdefault:\n>>>>> +\t\tbreak;\n>>>>> +\t}\n>>>>> +}\n>>>>> +\n>>>>> +/**\n>>>>> + * \\brief Write string to log output\n>>>>> + * \\param[in] str String to write\n>>>>> + */\n>>>>> +void LogOutput::write(const std::string &str)\n>>>>>  {\n>>>>>  \tswitch (target_) {\n>>>>>  \tcase LoggingTargetSyslog:\n>>>>> -\t\twriteSyslog(msg);\n>>>>> +\t\twriteSyslog(LogDebug, str);\n>>>>>  \t\tbreak;\n>>>>>  \tcase LoggingTargetStream:\n>>>>>  \tcase LoggingTargetFile:\n>>>>> -\t\twriteStream(msg);\n>>>>> +\t\twriteStream(str);\n>>>>>  \t\tbreak;\n>>>>>  \tdefault:\n>>>>>  \t\tbreak;\n>>>>>  \t}\n>>>>>  }\n>>>>>  \n>>>>> -void LogOutput::writeSyslog(const LogMessage &msg)\n>>>>> +void LogOutput::writeSyslog(LogSeverity severity, const std::string &str)\n>>>>>  {\n>>>>> -\tstd::string str = std::string(log_severity_name(msg.severity())) + \" \" +\n>>>>> -\t      \t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n>>>>> -\t\t\t  msg.msg();\n>>>>> -\tsyslog(log_severity_to_syslog(msg.severity()), \"%s\", str.c_str());\n>>>>> +\tsyslog(log_severity_to_syslog(severity), \"%s\", str.c_str());\n>>>>>  }\n>>>>>  \n>>>>> -void LogOutput::writeStream(const LogMessage &msg)\n>>>>> +void LogOutput::writeStream(const std::string &str)\n>>>>>  {\n>>>>> -\tstd::string str = \"[\" + utils::time_point_to_string(msg.timestamp()) +\n>>>>> -\t\t\t  \"]\" + log_severity_name(msg.severity()) + \" \" +\n>>>>> -\t\t\t  msg.category().name() + \" \" + msg.fileInfo() + \" \" +\n>>>>> -\t\t\t  msg.msg();\n>>>>>  \tstream_->write(str.c_str(), str.size());\n>>>>>  \tstream_->flush();\n>>>>>  }\n>>>>> @@ -223,6 +248,7 @@ public:\n>>>>>  \tstatic Logger *instance();\n>>>>>  \n>>>>>  \tvoid write(const LogMessage &msg);\n>>>>> +\tvoid backtrace();\n>>>>>  \n>>>>>  \tint logSetFile(const char *path);\n>>>>>  \tint logSetStream(std::ostream *stream);\n>>>>> @@ -240,9 +266,6 @@ private:\n>>>>>  \tvoid registerCategory(LogCategory *category);\n>>>>>  \tvoid unregisterCategory(LogCategory *category);\n>>>>>  \n>>>>> -\tvoid writeSyslog(const LogMessage &msg);\n>>>>> -\tvoid writeStream(const LogMessage &msg);\n>>>>> -\n>>>>>  \tstd::unordered_set<LogCategory *> categories_;\n>>>>>  \tstd::list<std::pair<std::string, LogSeverity>> levels_;\n>>>>>  \n>>>>> @@ -370,6 +393,38 @@ void Logger::write(const LogMessage &msg)\n>>>>>  \toutput->write(msg);\n>>>>>  }\n>>>>>  \n>>>>> +/**\n>>>>> + * \\brief Write a backtrace to the log\n>>>>> + */\n>>>>> +void Logger::backtrace()\n>>>>> +{\n>>>>> +#if HAVE_BACKTRACE\n>>>>\n>>>> Will this still be usable/desirable on release builds when\n>>>> function/symbol names are possibly unavailable?\n>>>>\n>>>> If not - then we might want to make this:\n>>>>\n>>>> \t#if HAVE_BACKTRACE && !defined(NDEBUG)\n>>>>\n>>>> But I'm not entirely averse to having the backtrace in regardless in\n>>>> fact, as it will likely backtrace all the way into the calling\n>>>> application (which should have symbols if it's being developed) and\n>>>> could add value in that scenario regardless.\n>>>\n>>> I think it's still desirable, as I'm using it with release builds :-)\n>>\n>>\n>> Excellent - I thought a release build would have stripped out the\n>> required symbol names.\n>>\n>> I certainly see some benefit to having this anyway.\n>>\n>> It will be even more beneficial if we later make Fatal 'non-fatal' in\n>> release builds, as there we really will want as much debug information\n>> to be reportable as possible, and Fatal will then give us a route to say\n>>\n>>  \"This is reallllly bad, we're trying to continue - but please report\n>> this issue\"\n>>\n>> (And in debug builds, I very much prefer the fail-fast abort on a fatal).\n>>\n>>>>> +\tstd::shared_ptr<LogOutput> output = std::atomic_load(&output_);\n>>>>> +\tif (!output)\n>>>>> +\t\treturn;\n>>>>> +\n>>>>> +\tvoid *buffer[32];\n>>>>> +\tint num_entries = ::backtrace(buffer, ARRAY_SIZE(buffer));\n>>>>> +\tchar **strings = backtrace_symbols(buffer, num_entries);\n>>>>> +\tif (!strings)\n>>>>> +\t\treturn;\n>>>>> +\n>>>>> +\tstd::ostringstream msg;\n>>>>> +\tmsg << \"Backtrace:\" << std::endl;\n>>>>> +\n>>>>> +\t/*\n>>>>> +\t * Skip the first two entries that correspond to this method and\n>>>>> +\t * ~LogMessage().\n>>>>> +\t */\n>>>>> +\tfor (int i = 2; i < num_entries; ++i)\n>>>>> +\t\tmsg << strings[i] << std::endl;\n>>>>> +\n>>>>> +\toutput->write(msg.str());\n>>>>> +\n>>>>> +\tfree(strings);\n>>>>> +#endif\n>>>>> +}\n>>>>> +\n>>>>>  /**\n>>>>>   * \\brief Set the log file\n>>>>>   * \\param[in] path Full path to the log file\n>>>>> @@ -783,8 +838,10 @@ LogMessage::~LogMessage()\n>>>>>  \tif (severity_ >= category_.severity())\n>>>>>  \t\tLogger::instance()->write(*this);\n>>>>>  \n>>>>> -\tif (severity_ == LogSeverity::LogFatal)\n>>>>> +\tif (severity_ == LogSeverity::LogFatal) {\n>>>>> +\t\tLogger::instance()->backtrace();\n>>>>>  \t\tstd::abort();\n>>>>> +\t}\n>>>>>  }\n>>>>>  \n>>>>>  /**\n>","headers":{"Return-Path":"<kieran.bingham@ideasonboard.com>","Received":["from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 91012607F3\n\tfor <libcamera-devel@lists.libcamera.org>;\n\tWed, 22 Jan 2020 16:17:30 +0100 (CET)","from [192.168.0.20]\n\t(cpc89242-aztw30-2-0-cust488.18-1.cable.virginm.net [86.31.129.233])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id DCB7C2E5;\n\tWed, 22 Jan 2020 16:17:29 +0100 (CET)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1579706250;\n\tbh=dgJDAfVgfq7BUZUavmteYChM2+ppJbcKgmcCAohhjx0=;\n\th=Reply-To:Subject:To:Cc:References:From:Date:In-Reply-To:From;\n\tb=HPIrdwmqT+3mjSS7hA7VR//0It/8qCpbz1GQIA89fE5sf1TBYfnhphXIifTDhsKKB\n\tsldDFCDt1PNmSUUZZgRuV3DQvfthVyGbdcJqMjmExANrVhW8885nLim10kd3InaX0g\n\t1JC7D56clfcQbpeR/hHs+vhXulVKlEwDUSrwiuLk=","Reply-To":"kieran.bingham@ideasonboard.com","To":"Laurent Pinchart <laurent.pinchart@ideasonboard.com>","Cc":"libcamera-devel@lists.libcamera.org","References":"<20191123124302.25863-1-laurent.pinchart@ideasonboard.com>\n\t<08297611-727b-7e93-cd87-067e8cce5d5b@ideasonboard.com>\n\t<20191126020148.GA4705@pendragon.ideasonboard.com>\n\t<1b79482c-e8ca-4a18-743f-0252e55a30de@ideasonboard.com>\n\t<20200111030724.GX4859@pendragon.ideasonboard.com>","From":"Kieran Bingham <kieran.bingham@ideasonboard.com>","Openpgp":"preference=signencrypt","Autocrypt":"addr=kieran.bingham@ideasonboard.com; keydata=\n\tmQINBFYE/WYBEACs1PwjMD9rgCu1hlIiUA1AXR4rv2v+BCLUq//vrX5S5bjzxKAryRf0uHat\n\tV/zwz6hiDrZuHUACDB7X8OaQcwhLaVlq6byfoBr25+hbZG7G3+5EUl9cQ7dQEdvNj6V6y/SC\n\trRanWfelwQThCHckbobWiQJfK9n7rYNcPMq9B8e9F020LFH7Kj6YmO95ewJGgLm+idg1Kb3C\n\tpotzWkXc1xmPzcQ1fvQMOfMwdS+4SNw4rY9f07Xb2K99rjMwZVDgESKIzhsDB5GY465sCsiQ\n\tcSAZRxqE49RTBq2+EQsbrQpIc8XiffAB8qexh5/QPzCmR4kJgCGeHIXBtgRj+nIkCJPZvZtf\n\tKr2EAbc6tgg6DkAEHJb+1okosV09+0+TXywYvtEop/WUOWQ+zo+Y/OBd+8Ptgt1pDRyOBzL8\n\tRXa8ZqRf0Mwg75D+dKntZeJHzPRJyrlfQokngAAs4PaFt6UfS+ypMAF37T6CeDArQC41V3ko\n\tlPn1yMsVD0p+6i3DPvA/GPIksDC4owjnzVX9kM8Zc5Cx+XoAN0w5Eqo4t6qEVbuettxx55gq\n\t8K8FieAjgjMSxngo/HST8TpFeqI5nVeq0/lqtBRQKumuIqDg+Bkr4L1V/PSB6XgQcOdhtd36\n\tOe9X9dXB8YSNt7VjOcO7BTmFn/Z8r92mSAfHXpb07YJWJosQOQARAQABtDBLaWVyYW4gQmlu\n\tZ2hhbSA8a2llcmFuLmJpbmdoYW1AaWRlYXNvbmJvYXJkLmNvbT6JAlcEEwEKAEECGwMFCwkI\n\tBwIGFQgJCgsCBBYCAwECHgECF4ACGQEWIQSQLdeYP70o/eNy1HqhHkZyEKRh/QUCXWTtygUJ\n\tCyJXZAAKCRChHkZyEKRh/f8dEACTDsbLN2nioNZMwyLuQRUAFcXNolDX48xcUXsWS2QjxaPm\n\tVsJx8Uy8aYkS85mdPBh0C83OovQR/OVbr8AxhGvYqBs3nQvbWuTl/+4od7DfK2VZOoKBAu5S\n\tQK2FYuUcikDqYcFWJ8DQnubxfE8dvzojHEkXw0sA4igINHDDFX3HJGZtLio+WpEFQtCbfTAG\n\tYZslasz1YZRbwEdSsmO3/kqy5eMnczlm8a21A3fKUo3g8oAZEFM+f4DUNzqIltg31OAB/kZS\n\tenKZQ/SWC8PmLg/ZXBrReYakxXtkP6w3FwMlzOlhGxqhIRNiAJfXJBaRhuUWzPOpEDE9q5YJ\n\tBmqQL2WJm1VSNNVxbXJHpaWMH1sA2R00vmvRrPXGwyIO0IPYeUYQa3gsy6k+En/aMQJd27dp\n\taScf9am9PFICPY5T4ppneeJLif2lyLojo0mcHOV+uyrds9XkLpp14GfTkeKPdPMrLLTsHRfH\n\tfA4I4OBpRrEPiGIZB/0im98MkGY/Mu6qxeZmYLCcgD6qz4idOvfgVOrNh+aA8HzIVR+RMW8H\n\tQGBN9f0E3kfwxuhl3omo6V7lDw8XOdmuWZNC9zPq1UfryVHANYbLGz9KJ4Aw6M+OgBC2JpkD\n\thXMdHUkC+d20dwXrwHTlrJi1YNp6rBc+xald3wsUPOZ5z8moTHUX/uPA/qhGsbkCDQRWBP1m\n\tARAAzijkb+Sau4hAncr1JjOY+KyFEdUNxRy+hqTJdJfaYihxyaj0Ee0P0zEi35CbE6lgU0Uz\n\ttih9fiUbSV3wfsWqg1Ut3/5rTKu7kLFp15kF7eqvV4uezXRD3Qu4yjv/rMmEJbbD4cTvGCYI\n\td6MDC417f7vK3hCbCVIZSp3GXxyC1LU+UQr3fFcOyCwmP9vDUR9JV0BSqHHxRDdpUXE26Dk6\n\tmhf0V1YkspE5St814ETXpEus2urZE5yJIUROlWPIL+hm3NEWfAP06vsQUyLvr/GtbOT79vXl\n\tEn1aulcYyu20dRRxhkQ6iILaURcxIAVJJKPi8dsoMnS8pB0QW12AHWuirPF0g6DiuUfPmrA5\n\tPKe56IGlpkjc8cO51lIxHkWTpCMWigRdPDexKX+Sb+W9QWK/0JjIc4t3KBaiG8O4yRX8ml2R\n\t+rxfAVKM6V769P/hWoRGdgUMgYHFpHGSgEt80OKK5HeUPy2cngDUXzwrqiM5Sz6Od0qw5pCk\n\tNlXqI0W/who0iSVM+8+RmyY0OEkxEcci7rRLsGnM15B5PjLJjh1f2ULYkv8s4SnDwMZ/kE04\n\t/UqCMK/KnX8pwXEMCjz0h6qWNpGwJ0/tYIgQJZh6bqkvBrDogAvuhf60Sogw+mH8b+PBlx1L\n\toeTK396wc+4c3BfiC6pNtUS5GpsPMMjYMk7kVvEAEQEAAYkCPAQYAQoAJgIbDBYhBJAt15g/\n\tvSj943LUeqEeRnIQpGH9BQJdizzIBQkLSKZiAAoJEKEeRnIQpGH9eYgQAJpjaWNgqNOnMTmD\n\tMJggbwjIotypzIXfhHNCeTkG7+qCDlSaBPclcPGYrTwCt0YWPU2TgGgJrVhYT20ierN8LUvj\n\t6qOPTd+Uk7NFzL65qkh80ZKNBFddx1AabQpSVQKbdcLb8OFs85kuSvFdgqZwgxA1vl4TFhNz\n\tPZ79NAmXLackAx3sOVFhk4WQaKRshCB7cSl+RIng5S/ThOBlwNlcKG7j7W2MC06BlTbdEkUp\n\tECzuuRBv8wX4OQl+hbWbB/VKIx5HKlLu1eypen/5lNVzSqMMIYkkZcjV2SWQyUGxSwq0O/sx\n\tS0A8/atCHUXOboUsn54qdxrVDaK+6jIAuo8JiRWctP16KjzUM7MO0/+4zllM8EY57rXrj48j\n\tsbEYX0YQnzaj+jO6kJtoZsIaYR7rMMq9aUAjyiaEZpmP1qF/2sYenDx0Fg2BSlLvLvXM0vU8\n\tpQk3kgDu7kb/7PRYrZvBsr21EIQoIjXbZxDz/o7z95frkP71EaICttZ6k9q5oxxA5WC6sTXc\n\tMW8zs8avFNuA9VpXt0YupJd2ijtZy2mpZNG02fFVXhIn4G807G7+9mhuC4XG5rKlBBUXTvPU\n\tAfYnB4JBDLmLzBFavQfvonSfbitgXwCG3vS+9HEwAjU30Bar1PEOmIbiAoMzuKeRm2LVpmq4\n\tWZw01QYHU/GUV/zHJSFk","Organization":"Ideas on Board","Message-ID":"<76ddf46c-95fb-b848-2985-d28f67af647e@ideasonboard.com>","Date":"Wed, 22 Jan 2020 15:17:27 +0000","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101\n\tThunderbird/60.9.0","MIME-Version":"1.0","In-Reply-To":"<20200111030724.GX4859@pendragon.ideasonboard.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-GB","Content-Transfer-Encoding":"8bit","Subject":"Re: [libcamera-devel] [PATCH] libcamera: Print backtrace on fatal\n\terrors","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"<libcamera-devel.lists.libcamera.org>","List-Unsubscribe":"<https://lists.libcamera.org/options/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=unsubscribe>","List-Archive":"<https://lists.libcamera.org/pipermail/libcamera-devel/>","List-Post":"<mailto:libcamera-devel@lists.libcamera.org>","List-Help":"<mailto:libcamera-devel-request@lists.libcamera.org?subject=help>","List-Subscribe":"<https://lists.libcamera.org/listinfo/libcamera-devel>,\n\t<mailto:libcamera-devel-request@lists.libcamera.org?subject=subscribe>","X-List-Received-Date":"Wed, 22 Jan 2020 15:17:30 -0000"}}]