From patchwork Tue Sep 15 14:20:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9611 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 2797BBF01C for ; Tue, 15 Sep 2020 14:21:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 1275F62E1B; Tue, 15 Sep 2020 16:21:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nln4C0e7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id A1F76603EE for ; Tue, 15 Sep 2020 16:21:01 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A12A7AEA; Tue, 15 Sep 2020 16:20:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179654; bh=hZ5z9Pn83/b9ODQDApDcjez+uT4fueazDjez05BOfEM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nln4C0e7U2NTC+07ZbVVju6fHevo+MWJB5vVkfBREKzxnGyqDM5rgbylVJLp2Ve54 Di+JKaQsVTyiLkK0myJjr/tL9KgcPM75T67ayJLUv7q9BuGOEUCN/+6SZAuB0VvVTu l+31JE2fd4ufF5f2I2U/BlkczwBNP+r5TXb8zFoM= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:16 +0900 Message-Id: <20200915142038.28757-2-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 01/23] libcamera: ProcessManager: make ProcessManager lifetime explicitly managed X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" If any Process instances are destroyed after the ProcessManager is destroyed, then a segfault will occur. Fix this by making the lifetime of the ProcessManager explicit, and make the CameraManager construct and deconstruct (automatically, via a unique pointer) the ProcessManager. Signed-off-by: Paul Elder Reviewed-by: Laurent Pinchart Reviewed-by: Niklas Söderlund --- include/libcamera/internal/process.h | 27 ++++++++++++++++ src/libcamera/camera_manager.cpp | 2 ++ src/libcamera/process.cpp | 46 ++++++++++++---------------- 3 files changed, 48 insertions(+), 27 deletions(-) diff --git a/include/libcamera/internal/process.h b/include/libcamera/internal/process.h index 36595106..df697c7f 100644 --- a/include/libcamera/internal/process.h +++ b/include/libcamera/internal/process.h @@ -7,6 +7,7 @@ #ifndef __LIBCAMERA_INTERNAL_PROCESS_H__ #define __LIBCAMERA_INTERNAL_PROCESS_H__ +#include #include #include @@ -50,6 +51,32 @@ private: friend class ProcessManager; }; +class ProcessManager +{ +public: + ProcessManager(); + ~ProcessManager(); + + void registerProcess(Process *proc); + + static ProcessManager *instance(); + + int writePipe() const; + + const struct sigaction &oldsa() const; + +private: + static ProcessManager *self_; + + void sighandler(EventNotifier *notifier); + + std::list processes_; + + struct sigaction oldsa_; + EventNotifier *sigEvent_; + int pipe_[2]; +}; + } /* namespace libcamera */ #endif /* __LIBCAMERA_INTERNAL_PROCESS_H__ */ diff --git a/src/libcamera/camera_manager.cpp b/src/libcamera/camera_manager.cpp index 47d56256..b8d3ccc8 100644 --- a/src/libcamera/camera_manager.cpp +++ b/src/libcamera/camera_manager.cpp @@ -18,6 +18,7 @@ #include "libcamera/internal/ipa_manager.h" #include "libcamera/internal/log.h" #include "libcamera/internal/pipeline_handler.h" +#include "libcamera/internal/process.h" #include "libcamera/internal/thread.h" #include "libcamera/internal/utils.h" @@ -67,6 +68,7 @@ private: std::unique_ptr enumerator_; IPAManager ipaManager_; + ProcessManager processManager_; }; CameraManager::Private::Private(CameraManager *cm) diff --git a/src/libcamera/process.cpp b/src/libcamera/process.cpp index 994190dc..72b5afe2 100644 --- a/src/libcamera/process.cpp +++ b/src/libcamera/process.cpp @@ -41,28 +41,6 @@ LOG_DEFINE_CATEGORY(Process) * The ProcessManager singleton keeps track of all created Process instances, * and manages the signal handling involved in terminating processes. */ -class ProcessManager -{ -public: - void registerProcess(Process *proc); - - static ProcessManager *instance(); - - int writePipe() const; - - const struct sigaction &oldsa() const; - -private: - void sighandler(EventNotifier *notifier); - ProcessManager(); - ~ProcessManager(); - - std::list processes_; - - struct sigaction oldsa_; - EventNotifier *sigEvent_; - int pipe_[2]; -}; namespace { @@ -127,8 +105,20 @@ void ProcessManager::registerProcess(Process *proc) processes_.push_back(proc); } +ProcessManager *ProcessManager::self_ = nullptr; + +/** + * \brief Construct a ProcessManager instance + * + * The ProcessManager class is meant to only be instantiated once, by the + * CameraManager. + */ ProcessManager::ProcessManager() { + if (self_) + LOG(Process, Fatal) + << "Multiple ProcessManager objects are not allowed"; + sigaction(SIGCHLD, NULL, &oldsa_); struct sigaction sa; @@ -145,6 +135,8 @@ ProcessManager::ProcessManager() << "Failed to initialize pipe for signal handling"; sigEvent_ = new EventNotifier(pipe_[0], EventNotifier::Read); sigEvent_->activated.connect(this, &ProcessManager::sighandler); + + self_ = this; } ProcessManager::~ProcessManager() @@ -153,21 +145,21 @@ ProcessManager::~ProcessManager() delete sigEvent_; close(pipe_[0]); close(pipe_[1]); + + self_ = nullptr; } /** * \brief Retrieve the Process manager instance * - * The ProcessManager is a singleton and can't be constructed manually. This - * method shall instead be used to retrieve the single global instance of the - * manager. + * The ProcessManager is constructed by the CameraManager. This function shall + * be used to retrieve the single instance of the manager. * * \return The Process manager instance */ ProcessManager *ProcessManager::instance() { - static ProcessManager processManager; - return &processManager; + return self_; } /** From patchwork Tue Sep 15 14:20:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9634 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 9F3C6BF01C for ; Tue, 15 Sep 2020 14:21:51 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 62EAC62E31; Tue, 15 Sep 2020 16:21:51 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="uUWIUCqZ"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E9F73603EE for ; Tue, 15 Sep 2020 16:21:01 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 03997FD8; Tue, 15 Sep 2020 16:20:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179658; bh=hJh9ZMbNHdXhItHEdwNfm79bFOiu6aKvOrR+LQrE4lQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uUWIUCqZU0VLfD4YcTeGmRHX/O+lLpx8ZSRdYjZfuE2s0ys0QrCgou7Md9gtMp0/A cbeEFLhNWjdC2x608kZSWxnM/GO5d97+me+fIUshua1RrqXUPSjRMGsSXBJmh5MsP6 HYxVbN3U1wX2NVUK/A427pEba1S5xLEd9LswlecY= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:17 +0900 Message-Id: <20200915142038.28757-3-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 15 Sep 2020 16:21:50 +0200 Subject: [libcamera-devel] [PATCH 02/23] utils: ipc: import mojo X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Import mojo from Chromium repository, so that we can use it for generating code for the IPC mechanism. This tree has been pruned somewhat; for example, the templates for code generation for unused languages are removed. Signed-off-by: Paul Elder --- utils/ipc/mojo/public/LICENSE | 27 + utils/ipc/mojo/public/tools/.style.yapf | 6 + utils/ipc/mojo/public/tools/BUILD.gn | 18 + utils/ipc/mojo/public/tools/bindings/BUILD.gn | 108 + .../ipc/mojo/public/tools/bindings/README.md | 816 +++++++ .../chromium_bindings_configuration.gni | 51 + .../tools/bindings/compile_typescript.py | 27 + .../tools/bindings/concatenate-files.py | 54 + ...concatenate_and_replace_closure_exports.py | 73 + .../bindings/format_typemap_generator_args.py | 36 + .../tools/bindings/gen_data_files_list.py | 52 + .../tools/bindings/generate_type_mappings.py | 187 ++ .../ipc/mojo/public/tools/bindings/mojom.gni | 1941 +++++++++++++++++ .../bindings/mojom_bindings_generator.py | 390 ++++ .../mojom_bindings_generator_unittest.py | 62 + .../tools/bindings/mojom_types_downgrader.py | 119 + .../tools/bindings/validate_typemap_config.py | 57 + utils/ipc/mojo/public/tools/mojom/README.md | 14 + .../mojom/check_stable_mojom_compatibility.py | 170 ++ ...eck_stable_mojom_compatibility_unittest.py | 260 +++ .../mojo/public/tools/mojom/const_unittest.py | 90 + .../mojo/public/tools/mojom/enum_unittest.py | 92 + .../mojo/public/tools/mojom/mojom/BUILD.gn | 43 + .../mojo/public/tools/mojom/mojom/__init__.py | 0 .../mojo/public/tools/mojom/mojom/error.py | 28 + .../mojo/public/tools/mojom/mojom/fileutil.py | 45 + .../tools/mojom/mojom/fileutil_unittest.py | 40 + .../tools/mojom/mojom/generate/__init__.py | 0 .../mojom/mojom/generate/constant_resolver.py | 93 + .../tools/mojom/mojom/generate/generator.py | 325 +++ .../mojom/generate/generator_unittest.py | 74 + .../tools/mojom/mojom/generate/module.py | 1635 ++++++++++++++ .../mojom/mojom/generate/module_unittest.py | 31 + .../public/tools/mojom/mojom/generate/pack.py | 258 +++ .../mojom/mojom/generate/pack_unittest.py | 225 ++ .../mojom/mojom/generate/template_expander.py | 83 + .../tools/mojom/mojom/generate/translate.py | 854 ++++++++ .../mojom/generate/translate_unittest.py | 73 + .../tools/mojom/mojom/parse/__init__.py | 0 .../public/tools/mojom/mojom/parse/ast.py | 427 ++++ .../tools/mojom/mojom/parse/ast_unittest.py | 121 + .../mojom/mojom/parse/conditional_features.py | 82 + .../parse/conditional_features_unittest.py | 233 ++ .../public/tools/mojom/mojom/parse/lexer.py | 251 +++ .../tools/mojom/mojom/parse/lexer_unittest.py | 198 ++ .../public/tools/mojom/mojom/parse/parser.py | 488 +++++ .../mojom/mojom/parse/parser_unittest.py | 1390 ++++++++++++ .../mojo/public/tools/mojom/mojom_parser.py | 361 +++ .../tools/mojom/mojom_parser_test_case.py | 73 + .../tools/mojom/mojom_parser_unittest.py | 171 ++ .../tools/mojom/stable_attribute_unittest.py | 127 ++ .../mojom/version_compatibility_unittest.py | 397 ++++ .../public/tools/run_all_python_unittests.py | 28 + utils/ipc/tools/diagnosis/crbug_1001171.py | 51 + 54 files changed, 12855 insertions(+) create mode 100644 utils/ipc/mojo/public/LICENSE create mode 100644 utils/ipc/mojo/public/tools/.style.yapf create mode 100644 utils/ipc/mojo/public/tools/BUILD.gn create mode 100644 utils/ipc/mojo/public/tools/bindings/BUILD.gn create mode 100644 utils/ipc/mojo/public/tools/bindings/README.md create mode 100644 utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni create mode 100644 utils/ipc/mojo/public/tools/bindings/compile_typescript.py create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate-files.py create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py create mode 100755 utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py create mode 100644 utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py create mode 100755 utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom.gni create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py create mode 100755 utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py create mode 100644 utils/ipc/mojo/public/tools/mojom/README.md create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/const_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/enum_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/__init__.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/error.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/__init__.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/__init__.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py create mode 100755 utils/ipc/mojo/public/tools/mojom/mojom_parser.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py create mode 100644 utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py create mode 100755 utils/ipc/mojo/public/tools/run_all_python_unittests.py create mode 100644 utils/ipc/tools/diagnosis/crbug_1001171.py diff --git a/utils/ipc/mojo/public/LICENSE b/utils/ipc/mojo/public/LICENSE new file mode 100644 index 00000000..972bb2ed --- /dev/null +++ b/utils/ipc/mojo/public/LICENSE @@ -0,0 +1,27 @@ +// Copyright 2014 The Chromium Authors. All rights reserved. +// +// Redistribution and use in source and binary forms, with or without +// modification, are permitted provided that the following conditions are +// met: +// +// * Redistributions of source code must retain the above copyright +// notice, this list of conditions and the following disclaimer. +// * Redistributions in binary form must reproduce the above +// copyright notice, this list of conditions and the following disclaimer +// in the documentation and/or other materials provided with the +// distribution. +// * Neither the name of Google Inc. nor the names of its +// contributors may be used to endorse or promote products derived from +// this software without specific prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/utils/ipc/mojo/public/tools/.style.yapf b/utils/ipc/mojo/public/tools/.style.yapf new file mode 100644 index 00000000..b4ebbe24 --- /dev/null +++ b/utils/ipc/mojo/public/tools/.style.yapf @@ -0,0 +1,6 @@ +[style] +based_on_style = pep8 + +# New directories should use a .style.yapf that does not include the following: +column_limit = 80 +indent_width = 2 diff --git a/utils/ipc/mojo/public/tools/BUILD.gn b/utils/ipc/mojo/public/tools/BUILD.gn new file mode 100644 index 00000000..4c68350b --- /dev/null +++ b/utils/ipc/mojo/public/tools/BUILD.gn @@ -0,0 +1,18 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +# The main target used to aggregate all unit tests for Python-based Mojo tools. +# This is used to generate a complete isolate which can be pushed to bots to run +# the tests. +group("mojo_python_unittests") { + data = [ + "run_all_python_unittests.py", + "//testing/scripts/common.py", + "//testing/scripts/run_isolated_script_test.py", + "//testing/test_env.py", + "//testing/xvfb.py", + ] + deps = [ "//mojo/public/tools/mojom/mojom:tests" ] + data_deps = [ "//third_party/catapult/third_party/typ/" ] +} diff --git a/utils/ipc/mojo/public/tools/bindings/BUILD.gn b/utils/ipc/mojo/public/tools/bindings/BUILD.gn new file mode 100644 index 00000000..8ba6e922 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/BUILD.gn @@ -0,0 +1,108 @@ +# Copyright 2016 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import("//mojo/public/tools/bindings/mojom.gni") +import("//third_party/jinja2/jinja2.gni") + +action("precompile_templates") { + sources = mojom_generator_sources + sources += [ + "$mojom_generator_root/generators/cpp_templates/enum_macros.tmpl", + "$mojom_generator_root/generators/cpp_templates/enum_serialization_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_macros.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_proxy_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_request_validator_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_response_validator_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/interface_stub_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-forward.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-import-headers.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-params-data.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-shared-internal.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-shared-message-ids.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-shared.cc.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-shared.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-test-utils.cc.tmpl", + "$mojom_generator_root/generators/cpp_templates/module-test-utils.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/module.cc.tmpl", + "$mojom_generator_root/generators/cpp_templates/module.h.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_data_view_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_data_view_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_macros.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_serialization_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_traits_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_traits_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/struct_unserialized_message_context.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_data_view_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_data_view_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_serialization_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_traits_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/union_traits_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/validation_macros.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_class_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_class_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_class_template_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_union_class_declaration.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_union_class_definition.tmpl", + "$mojom_generator_root/generators/cpp_templates/wrapper_union_class_template_definition.tmpl", + "$mojom_generator_root/generators/java_templates/constant_definition.tmpl", + "$mojom_generator_root/generators/java_templates/constants.java.tmpl", + "$mojom_generator_root/generators/java_templates/data_types_definition.tmpl", + "$mojom_generator_root/generators/java_templates/enum.java.tmpl", + "$mojom_generator_root/generators/java_templates/enum_definition.tmpl", + "$mojom_generator_root/generators/java_templates/header.java.tmpl", + "$mojom_generator_root/generators/java_templates/interface.java.tmpl", + "$mojom_generator_root/generators/java_templates/interface_definition.tmpl", + "$mojom_generator_root/generators/java_templates/interface_internal.java.tmpl", + "$mojom_generator_root/generators/java_templates/struct.java.tmpl", + "$mojom_generator_root/generators/java_templates/union.java.tmpl", + "$mojom_generator_root/generators/js_templates/enum_definition.tmpl", + "$mojom_generator_root/generators/js_templates/externs/interface_definition.tmpl", + "$mojom_generator_root/generators/js_templates/externs/module.externs.tmpl", + "$mojom_generator_root/generators/js_templates/externs/struct_definition.tmpl", + "$mojom_generator_root/generators/js_templates/fuzzing.tmpl", + "$mojom_generator_root/generators/js_templates/interface_definition.tmpl", + "$mojom_generator_root/generators/js_templates/lite/enum_definition.tmpl", + "$mojom_generator_root/generators/js_templates/lite/interface_definition.tmpl", + "$mojom_generator_root/generators/js_templates/lite/module_definition.tmpl", + "$mojom_generator_root/generators/js_templates/lite/mojom-lite.js.tmpl", + "$mojom_generator_root/generators/js_templates/lite/struct_definition.tmpl", + "$mojom_generator_root/generators/js_templates/lite/union_definition.tmpl", + "$mojom_generator_root/generators/js_templates/module.amd.tmpl", + "$mojom_generator_root/generators/js_templates/module_definition.tmpl", + "$mojom_generator_root/generators/js_templates/struct_definition.tmpl", + "$mojom_generator_root/generators/js_templates/union_definition.tmpl", + "$mojom_generator_root/generators/js_templates/validation_macros.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm.cc.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm.h.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm.proto.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm_from_proto_macros.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm_macros.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm_to_proto_macros.tmpl", + "$mojom_generator_root/generators/mojolpm_templates/mojolpm_traits_specialization_macros.tmpl", + "$mojom_generator_root/generators/ts_templates/module_definition.tmpl", + "$mojom_generator_root/generators/ts_templates/mojom.tmpl", + ] + script = mojom_generator_script + + inputs = jinja2_sources + outputs = [ + "$target_gen_dir/cpp_templates.zip", + "$target_gen_dir/java_templates.zip", + "$target_gen_dir/mojolpm_templates.zip", + "$target_gen_dir/js_templates.zip", + "$target_gen_dir/ts_templates.zip", + ] + args = [ + "-o", + rebase_path(target_gen_dir, root_build_dir), + "--use_bundled_pylibs", + "precompile", + ] +} diff --git a/utils/ipc/mojo/public/tools/bindings/README.md b/utils/ipc/mojo/public/tools/bindings/README.md new file mode 100644 index 00000000..1a3d5c58 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/README.md @@ -0,0 +1,816 @@ +# Mojom Interface Definition Language (IDL) +This document is a subset of the [Mojo documentation](/mojo/README.md). + +[TOC] + +## Overview + +Mojom is the IDL for Mojo interfaces. Given a `.mojom` file, the +[bindings +generator](https://cs.chromium.org/chromium/src/mojo/public/tools/bindings/) can +output bindings for any supported language: **C++**, **JavaScript**, or +**Java**. + +For a trivial example consider the following hypothetical Mojom file we write to +`//services/widget/public/mojom/frobinator.mojom`: + +``` +module widget.mojom; + +interface Frobinator { + Frobinate(); +}; +``` + +This defines a single [interface](#Interfaces) named `Frobinator` in a +[module](#Modules) named `widget.mojom` (and thus fully qualified in Mojom as +`widget.mojom.Frobinator`.) Note that many interfaces and/or other types of +definitions (structs, enums, *etc.*) may be included in a single Mojom file. + +If we add a corresponding GN target to +`//services/widget/public/mojom/BUILD.gn`: + +``` +import("mojo/public/tools/bindings/mojom.gni") + +mojom("mojom") { + sources = [ + "frobinator.mojom", + ] +} +``` + +and then build this target: + +``` +ninja -C out/r services/widget/public/mojom +``` + +we'll find several generated sources in our output directory: + +``` +out/r/gen/services/widget/public/mojom/frobinator.mojom.cc +out/r/gen/services/widget/public/mojom/frobinator.mojom.h +out/r/gen/services/widget/public/mojom/frobinator.mojom-shared.h +etc... +``` + +Each of these generated source modules includes a set of definitions +representing the Mojom contents in C++. You can also build or depend on suffixed +target names to get bindings for other languages. For example, + +``` +ninja -C out/r services/widget/public/mojom:mojom_js +ninja -C out/r services/widget/public/mojom:mojom_java +``` + +would generate JavaScript and Java bindings respectively, in the same generated +output directory. + +For more details regarding the generated +outputs please see +[documentation for individual target languages](#Generated-Code-For-Target-Languages). + +## Mojom Syntax + +Mojom IDL allows developers to define **structs**, **unions**, **interfaces**, +**constants**, and **enums**, all within the context of a **module**. These +definitions are used to generate code in the supported target languages at build +time. + +Mojom files may **import** other Mojom files in order to reference their +definitions. + +### Primitive Types +Mojom supports a few basic data types which may be composed into structs or used +for message parameters. + +| Type | Description +|-------------------------------|-------------------------------------------------------| +| `bool` | Boolean type (`true` or `false`.) +| `int8`, `uint8` | Signed or unsigned 8-bit integer. +| `int16`, `uint16` | Signed or unsigned 16-bit integer. +| `int32`, `uint32` | Signed or unsigned 32-bit integer. +| `int64`, `uint64` | Signed or unsigned 64-bit integer. +| `float`, `double` | 32- or 64-bit floating point number. +| `string` | UTF-8 encoded string. +| `array` | Array of any Mojom type *T*; for example, `array` or `array>`. +| `array` | Fixed-length array of any Mojom type *T*. The parameter *N* must be an integral constant. +| `map` | Associated array maping values of type *S* to values of type *T*. *S* may be a `string`, `enum`, or numeric type. +| `handle` | Generic Mojo handle. May be any type of handle, including a wrapped native platform handle. +| `handle` | Generic message pipe handle. +| `handle` | Shared buffer handle. +| `handle` | Data pipe producer handle. +| `handle` | Data pipe consumer handle. +| `handle` | A native platform/OS handle. +| *`pending_remote`* | Any user-defined Mojom interface type. This is sugar for a strongly-typed message pipe handle which should eventually be used to make outgoing calls on the interface. +| *`pending_receiver`* | A pending receiver for any user-defined Mojom interface type. This is sugar for a more strongly-typed message pipe handle which is expected to receive request messages and should therefore eventually be bound to an implementation of the interface. +| *`pending_associated_remote`* | An associated interface handle. See [Associated Interfaces](#Associated-Interfaces) +| *`pending_associated_receiver`* | A pending associated receiver. See [Associated Interfaces](#Associated-Interfaces) +| *T*? | An optional (nullable) value. Primitive numeric types (integers, floats, booleans, and enums) are not nullable. All other types are nullable. + +### Modules + +Every Mojom file may optionally specify a single **module** to which it belongs. + +This is used strictly for aggregaging all defined symbols therein within a +common Mojom namespace. The specific impact this has on generated binidngs code +varies for each target language. For example, if the following Mojom is used to +generate bindings: + +``` +module business.stuff; + +interface MoneyGenerator { + GenerateMoney(); +}; +``` + +Generated C++ bindings will define a class interface `MoneyGenerator` in the +`business::stuff` namespace, while Java bindings will define an interface +`MoneyGenerator` in the `org.chromium.business.stuff` package. JavaScript +bindings at this time are unaffected by module declarations. + +**NOTE:** By convention in the Chromium codebase, **all** Mojom files should +declare a module name with at least (and preferrably exactly) one top-level name +as well as an inner `mojom` module suffix. *e.g.*, `chrome.mojom`, +`business.mojom`, *etc.* + +This convention makes it easy to tell which symbols are generated by Mojom when +reading non-Mojom code, and it also avoids namespace collisions in the fairly +common scenario where you have a real C++ or Java `Foo` along with a +corresponding Mojom `Foo` for its serialized representation. + +### Imports + +If your Mojom references definitions from other Mojom files, you must **import** +those files. Import syntax is as follows: + +``` +import "services/widget/public/mojom/frobinator.mojom"; +``` + +Import paths are always relative to the top-level directory. + +Note that circular imports are **not** supported. + +### Structs + +Structs are defined using the **struct** keyword, and they provide a way to +group related fields together: + +``` cpp +struct StringPair { + string first; + string second; +}; +``` + +Struct fields may be comprised of any of the types listed above in the +[Primitive Types](#Primitive-Types) section. + +Default values may be specified as long as they are constant: + +``` cpp +struct Request { + int32 id = -1; + string details; +}; +``` + +What follows is a fairly +comprehensive example using the supported field types: + +``` cpp +struct StringPair { + string first; + string second; +}; + +enum AnEnum { + YES, + NO +}; + +interface SampleInterface { + DoStuff(); +}; + +struct AllTheThings { + // Note that these types can never be marked nullable! + bool boolean_value; + int8 signed_8bit_value = 42; + uint8 unsigned_8bit_value; + int16 signed_16bit_value; + uint16 unsigned_16bit_value; + int32 signed_32bit_value; + uint32 unsigned_32bit_value; + int64 signed_64bit_value; + uint64 unsigned_64bit_value; + float float_value_32bit; + double float_value_64bit; + AnEnum enum_value = AnEnum.YES; + + // Strings may be nullable. + string? maybe_a_string_maybe_not; + + // Structs may contain other structs. These may also be nullable. + StringPair some_strings; + StringPair? maybe_some_more_strings; + + // In fact structs can also be nested, though in practice you must always make + // such fields nullable -- otherwise messages would need to be infinitely long + // in order to pass validation! + AllTheThings? more_things; + + // Arrays may be templated over any Mojom type, and are always nullable: + array numbers; + array? maybe_more_numbers; + + // Arrays of arrays of arrays... are fine. + array>> this_works_but_really_plz_stop; + + // The element type may be nullable if it's a type which is allowed to be + // nullable. + array more_maybe_things; + + // Fixed-size arrays get some extra validation on the receiving end to ensure + // that the correct number of elements is always received. + array uuid; + + // Maps follow many of the same rules as arrays. Key types may be any + // non-handle, non-collection type, and value types may be any supported + // struct field type. Maps may also be nullable. + map one_map; + map? maybe_another_map; + map? maybe_a_pretty_weird_but_valid_map; + map?>?>?> ridiculous; + + // And finally, all handle types are valid as struct fields and may be + // nullable. Note that interfaces and interface requests (the "Foo" and + // "Foo&" type syntax respectively) are just strongly-typed message pipe + // handles. + handle generic_handle; + handle reader; + handle? maybe_writer; + handle dumping_ground; + handle raw_message_pipe; + pending_remote? maybe_a_sample_interface_client_pipe; + pending_receiver non_nullable_sample_pending_receiver; + pending_receiver? nullable_sample_pending_receiver; + pending_associated_remote associated_interface_client; + pending_associated_receiver associated_pending_receiver; + pending_associated_receiver? maybe_another_pending_receiver; +}; +``` + +For details on how all of these different types translate to usable generated +code, see +[documentation for individual target languages](#Generated-Code-For-Target-Languages). + +### Unions + +Mojom supports tagged unions using the **union** keyword. A union is a +collection of fields which may taken the value of any single one of those fields +at a time. Thus they provide a way to represent a variant value type while +minimizing storage requirements. + +Union fields may be of any type supported by [struct](#Structs) fields. For +example: + +```cpp +union ExampleUnion { + string str; + StringPair pair; + int64 id; + array guid; + SampleInterface iface; +}; +``` + +For details on how unions like this translate to generated bindings code, see +[documentation for individual target languages](#Generated-Code-For-Target-Languages). + +### Enumeration Types + +Enumeration types may be defined using the **enum** keyword either directly +within a module or nested within the namespace of some struct or interface: + +``` +module business.mojom; + +enum Department { + SALES = 0, + DEV, +}; + +struct Employee { + enum Type { + FULL_TIME, + PART_TIME, + }; + + Type type; + // ... +}; +``` + +Similar to C-style enums, individual values may be explicitly assigned within an +enum definition. By default, values are based at zero and increment by +1 sequentially. + +The effect of nested definitions on generated bindings varies depending on the +target language. See [documentation for individual target languages](#Generated-Code-For-Target-Languages) + +### Constants + +Constants may be defined using the **const** keyword either directly within a +module or nested within the namespace of some struct or interface: + +``` +module business.mojom; + +const string kServiceName = "business"; + +struct Employee { + const uint64 kInvalidId = 0; + + enum Type { + FULL_TIME, + PART_TIME, + }; + + uint64 id = kInvalidId; + Type type; +}; +``` + +The effect of nested definitions on generated bindings varies depending on the +target language. See [documentation for individual target languages](#Generated-Code-For-Target-Languages) + +### Interfaces + +An **interface** is a logical bundle of parameterized request messages. Each +request message may optionally define a parameterized response message. Here's +an example to define an interface `Foo` with various kinds of requests: + +``` +interface Foo { + // A request which takes no arguments and expects no response. + MyMessage(); + + // A request which has some arguments and expects no response. + MyOtherMessage(string name, array bytes); + + // A request which expects a single-argument response. + MyMessageWithResponse(string command) => (bool success); + + // A request which expects a response with multiple arguments. + MyMessageWithMoarResponse(string a, string b) => (int8 c, int8 d); +}; +``` + +Anything which is a valid struct field type (see [Structs](#Structs)) is also a +valid request or response argument type. The type notation is the same for both. + +### Attributes + +Mojom definitions may have their meaning altered by **attributes**, specified +with a syntax similar to Java or C# attributes. There are a handle of +interesting attributes supported today. + +**`[Sync]`** +: The `Sync` attribute may be specified for any interface method which expects + a response. This makes it so that callers of the method can wait + synchronously for a response. See + [Synchronous Calls](/mojo/public/cpp/bindings/README.md#Synchronous-Calls) + in the C++ bindings documentation. Note that sync methods are only actually + synchronous when called from C++. + +**`[Extensible]`** +: The `Extensible` attribute may be specified for any enum definition. This + essentially disables builtin range validation when receiving values of the + enum type in a message, allowing older bindings to tolerate unrecognized + values from newer versions of the enum. + +**`[Native]`** +: The `Native` attribute may be specified for an empty struct declaration to + provide a nominal bridge between Mojo IPC and legacy `IPC::ParamTraits` or + `IPC_STRUCT_TRAITS*` macros. + See + [Repurposing Legacy IPC Traits](/docs/mojo_ipc_conversion.md#repurposing-and-invocations) + for more details. Note support for this attribute is strictly limited to C++ + bindings generation. + +**`[MinVersion=N]`** +: The `MinVersion` attribute is used to specify the version at which a given + field, enum value, interface method, or method parameter was introduced. + See [Versioning](#Versioning) for more details. + +**`[Stable]`** +: The `Stable` attribute specifies that a given mojom type or interface + definition can be considered stable over time, meaning it is safe to use for + things like persistent storage or communication between independent + version-skewed binaries. Stable definitions may only depend on builtin mojom + types or other stable definitions, and changes to such definitions MUST + preserve backward-compatibility through appropriate use of versioning. + Backward-compatibility of changes is enforced in the Chromium tree using a + strict presubmit check. See [Versioning](#Versioning) for more details on + backward-compatibility constraints. + +**`[EnableIf=value]`** +: The `EnableIf` attribute is used to conditionally enable definitions when + the mojom is parsed. If the `mojom` target in the GN file does not include + the matching `value` in the list of `enabled_features`, the definition + will be disabled. This is useful for mojom definitions that only make + sense on one platform. Note that the `EnableIf` attribute can only be set + once per definition. + +## Generated Code For Target Languages + +When the bindings generator successfully processes an input Mojom file, it emits +corresponding code for each supported target language. For more details on how +Mojom concepts translate to a given target langauge, please refer to the +bindings API documentation for that language: + +* [C++ Bindings](/mojo/public/cpp/bindings/README.md) +* [JavaScript Bindings](/mojo/public/js/README.md) +* [Java Bindings](/mojo/public/java/bindings/README.md) + +## Message Validation + +Regardless of target language, all interface messages are validated during +deserialization before they are dispatched to a receiving implementation of the +interface. This helps to ensure consitent validation across interfaces without +leaving the burden to developers and security reviewers every time a new message +is added. + +If a message fails validation, it is never dispatched. Instead a **connection +error** is raised on the binding object (see +[C++ Connection Errors](/mojo/public/cpp/bindings/README.md#Connection-Errors), +[Java Connection Errors](/mojo/public/java/bindings/README.md#Connection-Errors), +or +[JavaScript Connection Errors](/mojo/public/js/README.md#Connection-Errors) for +details.) + +Some baseline level of validation is done automatically for primitive Mojom +types. + +### Non-Nullable Objects + +Mojom fields or parameter values (*e.g.*, structs, interfaces, arrays, *etc.*) +may be marked nullable in Mojom definitions (see +[Primitive Types](#Primitive-Types).) If a field or parameter is **not** marked +nullable but a message is received with a null value in its place, that message +will fail validation. + +### Enums + +Enums declared in Mojom are automatically validated against the range of legal +values. For example if a Mojom declares the enum: + +``` cpp +enum AdvancedBoolean { + TRUE = 0, + FALSE = 1, + FILE_NOT_FOUND = 2, +}; +``` + +and a message is received with the integral value 3 (or anything other than 0, +1, or 2) in place of some `AdvancedBoolean` field or parameter, the message will +fail validation. + +*** note +NOTE: It's possible to avoid this type of validation error by explicitly marking +an enum as [Extensible](#Attributes) if you anticipate your enum being exchanged +between two different versions of the binding interface. See +[Versioning](#Versioning). +*** + +### Other failures + +There are a host of internal validation errors that may occur when a malformed +message is received, but developers should not be concerned with these +specifically; in general they can only result from internal bindings bugs, +compromised processes, or some remote endpoint making a dubious effort to +manually encode their own bindings messages. + +### Custom Validation + +It's also possible for developers to define custom validation logic for specific +Mojom struct types by exploiting the +[type mapping](/mojo/public/cpp/bindings/README.md#Type-Mapping) system for C++ +bindings. Messages rejected by custom validation logic trigger the same +validation failure behavior as the built-in type validation routines. + +## Associated Interfaces + +As mentioned in the [Primitive Types](#Primitive-Types) section above, pending_remote +and pending_receiver fields and parameters may be marked as `associated`. This +essentially means that they are piggy-backed on some other interface's message +pipe. + +Because individual interface message pipes operate independently there can be no +relative ordering guarantees among them. Associated interfaces are useful when +one interface needs to guarantee strict FIFO ordering with respect to one or +more other interfaces, as they allow interfaces to share a single pipe. + +Currently associated interfaces are only supported in generated C++ bindings. +See the documentation for +[C++ Associated Interfaces](/mojo/public/cpp/bindings/README.md#Associated-Interfaces). + +## Versioning + +### Overview + +*** note +**NOTE:** You don't need to worry about versioning if you don't care about +backwards compatibility. Specifically, all parts of Chrome are updated +atomically today and there is not yet any possibility of any two Chrome +processes communicating with two different versions of any given Mojom +interface. +*** + +Services extend their interfaces to support new features over time, and clients +want to use those new features when they are available. If services and clients +are not updated at the same time, it's important for them to be able to +communicate with each other using different snapshots (versions) of their +interfaces. + +This document shows how to extend Mojom interfaces in a backwards-compatible +way. Changing interfaces in a non-backwards-compatible way is not discussed, +because in that case communication between different interface versions is +impossible anyway. + +### Versioned Structs + +You can use the `MinVersion` [attribute](#Attributes) to indicate from which +version a struct field is introduced. Assume you have the following struct: + +``` cpp +struct Employee { + uint64 employee_id; + string name; +}; +``` + +and you would like to add a birthday field. You can do: + +``` cpp +struct Employee { + uint64 employee_id; + string name; + [MinVersion=1] Date? birthday; +}; +``` + +By default, fields belong to version 0. New fields must be appended to the +struct definition (*i.e*., existing fields must not change **ordinal value**) +with the `MinVersion` attribute set to a number greater than any previous +existing versions. + +*** note +**NOTE:** do not change existing fields in versioned structs, as this is +not backwards-compatible. Instead, rename the old field to make its +deprecation clear and add a new field with the new version number. +*** + +**Ordinal value** refers to the relative positional layout of a struct's fields +(and an interface's methods) when encoded in a message. Implicitly, ordinal +numbers are assigned to fields according to lexical position. In the example +above, `employee_id` has an ordinal value of 0 and `name` has an ordinal value +of 1. + +Ordinal values can be specified explicitly using `**@**` notation, subject to +the following hard constraints: + +* For any given struct or interface, if any field or method explicitly specifies + an ordinal value, all fields or methods must explicitly specify an ordinal + value. +* For an *N*-field struct or *N*-method interface, the set of explicitly + assigned ordinal values must be limited to the range *[0, N-1]*. Interfaces + should include placeholder methods to fill the ordinal positions of removed + methods (for example "Unused_Message_7@7()" or "RemovedMessage@42()", etc). + +You may reorder fields, but you must ensure that the ordinal values of existing +fields remain unchanged. For example, the following struct remains +backwards-compatible: + +``` cpp +struct Employee { + uint64 employee_id@0; + [MinVersion=1] Date? birthday@2; + string name@1; +}; +``` + +*** note +**NOTE:** Newly added fields of Mojo object or handle types MUST be nullable. +See [Primitive Types](#Primitive-Types). +*** + +### Versioned Interfaces + +There are two dimensions on which an interface can be extended + +**Appending New Parameters To Existing Methods** +: Parameter lists are treated as structs internally, so all the rules of + versioned structs apply to method parameter lists. The only difference is + that the version number is scoped to the whole interface rather than to any + individual parameter list. + + Please note that adding a response to a message which did not previously + expect a response is a not a backwards-compatible change. + +**Appending New Methods** +: Similarly, you can reorder methods with explicit ordinal values as long as + the ordinal values of existing methods are unchanged. + +For example: + +``` cpp +// Old version: +interface HumanResourceDatabase { + AddEmployee(Employee employee) => (bool success); + QueryEmployee(uint64 id) => (Employee? employee); +}; + +// New version: +interface HumanResourceDatabase { + AddEmployee(Employee employee) => (bool success); + + QueryEmployee(uint64 id, [MinVersion=1] bool retrieve_finger_print) + => (Employee? employee, + [MinVersion=1] array? finger_print); + + [MinVersion=1] + AttachFingerPrint(uint64 id, array finger_print) + => (bool success); +}; +``` + +Similar to [versioned structs](#Versioned-Structs), when you pass the parameter +list of a request or response method to a destination using an older version of +an interface, unrecognized fields are silently discarded. However, if the method +call itself is not recognized, it is considered a validation error and the +receiver will close its end of the interface pipe. For example, if a client on +version 1 of the above interface sends an `AttachFingerPrint` request to an +implementation of version 0, the client will be disconnected. + +Bindings target languages that support versioning expose means to query or +assert the remote version from a client handle (*e.g.*, an +`mojo::Remote` in C++ bindings.) + +See +[C++ Versioning Considerations](/mojo/public/cpp/bindings/README.md#Versioning-Considerations) +and +[Java Versioning Considerations](/mojo/public/java/bindings/README.md#Versioning-Considerations) + +### Versioned Enums + +**By default, enums are non-extensible**, which means that generated message +validation code does not expect to see new values in the future. When an unknown +value is seen for a non-extensible enum field or parameter, a validation error +is raised. + +If you want an enum to be extensible in the future, you can apply the +`[Extensible]` [attribute](#Attributes): + +``` cpp +[Extensible] +enum Department { + SALES, + DEV, +}; +``` + +And later you can extend this enum without breaking backwards compatibility: + +``` cpp +[Extensible] +enum Department { + SALES, + DEV, + [MinVersion=1] RESEARCH, +}; +``` + +*** note +**NOTE:** For versioned enum definitions, the use of a `[MinVersion]` attribute +is strictly for documentation purposes. It has no impact on the generated code. +*** + +With extensible enums, bound interface implementations may receive unknown enum +values and will need to deal with them gracefully. See +[C++ Versioning Considerations](/mojo/public/cpp/bindings/README.md#Versioning-Considerations) +for details. + +## Grammar Reference + +Below is the (BNF-ish) context-free grammar of the Mojom language: + +``` +MojomFile = StatementList +StatementList = Statement StatementList | Statement +Statement = ModuleStatement | ImportStatement | Definition + +ModuleStatement = AttributeSection "module" Identifier ";" +ImportStatement = "import" StringLiteral ";" +Definition = Struct Union Interface Enum Const + +AttributeSection = | "[" AttributeList "]" +AttributeList = | NonEmptyAttributeList +NonEmptyAttributeList = Attribute + | Attribute "," NonEmptyAttributeList +Attribute = Name + | Name "=" Name + | Name "=" Literal + +Struct = AttributeSection "struct" Name "{" StructBody "}" ";" + | AttributeSection "struct" Name ";" +StructBody = + | StructBody Const + | StructBody Enum + | StructBody StructField +StructField = AttributeSection TypeSpec Name Ordinal Default ";" + +Union = AttributeSection "union" Name "{" UnionBody "}" ";" +UnionBody = | UnionBody UnionField +UnionField = AttributeSection TypeSpec Name Ordinal ";" + +Interface = AttributeSection "interface" Name "{" InterfaceBody "}" ";" +InterfaceBody = + | InterfaceBody Const + | InterfaceBody Enum + | InterfaceBody Method +Method = AttributeSection Name Ordinal "(" ParamterList ")" Response ";" +ParameterList = | NonEmptyParameterList +NonEmptyParameterList = Parameter + | Parameter "," NonEmptyParameterList +Parameter = AttributeSection TypeSpec Name Ordinal +Response = | "=>" "(" ParameterList ")" + +TypeSpec = TypeName "?" | TypeName +TypeName = BasicTypeName + | Array + | FixedArray + | Map + | InterfaceRequest +BasicTypeName = Identifier | "associated" Identifier | HandleType | NumericType +NumericType = "bool" | "int8" | "uint8" | "int16" | "uint16" | "int32" + | "uint32" | "int64" | "uint64" | "float" | "double" +HandleType = "handle" | "handle" "<" SpecificHandleType ">" +SpecificHandleType = "message_pipe" + | "shared_buffer" + | "data_pipe_consumer" + | "data_pipe_producer" + | "platform" +Array = "array" "<" TypeSpec ">" +FixedArray = "array" "<" TypeSpec "," IntConstDec ">" +Map = "map" "<" Identifier "," TypeSpec ">" +InterfaceRequest = Identifier "&" | "associated" Identifier "&" + +Ordinal = | OrdinalValue + +Default = | "=" Constant + +Enum = AttributeSection "enum" Name "{" NonEmptyEnumValueList "}" ";" + | AttributeSection "enum" Name "{" NonEmptyEnumValueList "," "}" ";" +NonEmptyEnumValueList = EnumValue | NonEmptyEnumValueList "," EnumValue +EnumValue = AttributeSection Name + | AttributeSection Name "=" Integer + | AttributeSection Name "=" Identifier + +Const = "const" TypeSpec Name "=" Constant ";" + +Constant = Literal | Identifier ";" + +Identifier = Name | Name "." Identifier + +Literal = Integer | Float | "true" | "false" | "default" | StringLiteral + +Integer = IntConst | "+" IntConst | "-" IntConst +IntConst = IntConstDec | IntConstHex + +Float = FloatConst | "+" FloatConst | "-" FloatConst + +; The rules below are for tokens matched strictly according to the given regexes + +Identifier = /[a-zA-Z_][0-9a-zA-Z_]*/ +IntConstDec = /0|(1-9[0-9]*)/ +IntConstHex = /0[xX][0-9a-fA-F]+/ +OrdinalValue = /@(0|(1-9[0-9]*))/ +FloatConst = ... # Imagine it's close enough to C-style float syntax. +StringLiteral = ... # Imagine it's close enough to C-style string literals, including escapes. +``` + +## Additional Documentation + +[Mojom Message Format](https://docs.google.com/document/d/13pv9cFh5YKuBggDBQ1-AL8VReF-IYpFOFpRfvWFrwio/edit) +: Describes the wire format used by Mojo bindings interfaces over message + pipes. + +[Input Format of Mojom Message Validation Tests](https://docs.google.com/document/d/1-y-2IYctyX2NPaLxJjpJfzVNWCC2SR2MJAD9MpIytHQ/edit) +: Describes a text format used to facilitate bindings message validation + tests. diff --git a/utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni b/utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni new file mode 100644 index 00000000..d8a13874 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni @@ -0,0 +1,51 @@ +# Copyright 2016 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +_typemap_imports = [ + "//chrome/chrome_cleaner/mojom/typemaps/typemaps.gni", + "//chrome/common/importer/typemaps.gni", + "//chrome/common/media_router/mojom/typemaps.gni", + "//chrome/typemaps.gni", + "//chromecast/typemaps.gni", + "//chromeos/typemaps.gni", + "//chromeos/components/multidevice/mojom/typemaps.gni", + "//chromeos/services/cros_healthd/public/mojom/typemaps.gni", + "//chromeos/services/device_sync/public/mojom/typemaps.gni", + "//chromeos/services/network_config/public/mojom/typemaps.gni", + "//chromeos/services/secure_channel/public/mojom/typemaps.gni", + "//components/arc/mojom/typemaps.gni", + "//components/chromeos_camera/common/typemaps.gni", + "//components/services/storage/public/cpp/filesystem/typemaps.gni", + "//components/sync/mojom/typemaps.gni", + "//components/typemaps.gni", + "//content/browser/typemaps.gni", + "//content/public/common/typemaps.gni", + "//sandbox/mac/mojom/typemaps.gni", + "//services/media_session/public/cpp/typemaps.gni", + "//services/proxy_resolver/public/cpp/typemaps.gni", + "//services/resource_coordinator/public/cpp/typemaps.gni", + "//services/service_manager/public/cpp/typemaps.gni", + "//services/tracing/public/mojom/typemaps.gni", +] + +_typemaps = [] +foreach(typemap_import, _typemap_imports) { + # Avoid reassignment error by assigning to empty scope first. + _imported = { + } + _imported = read_file(typemap_import, "scope") + _typemaps += _imported.typemaps +} + +typemaps = [] +foreach(typemap, _typemaps) { + typemaps += [ + { + filename = typemap + config = read_file(typemap, "scope") + }, + ] +} + +component_macro_suffix = "" diff --git a/utils/ipc/mojo/public/tools/bindings/compile_typescript.py b/utils/ipc/mojo/public/tools/bindings/compile_typescript.py new file mode 100644 index 00000000..a978901b --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/compile_typescript.py @@ -0,0 +1,27 @@ +# Copyright 2019 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import os +import sys +import argparse + +_HERE_PATH = os.path.dirname(__file__) +_SRC_PATH = os.path.normpath(os.path.join(_HERE_PATH, '..', '..', '..', '..')) + +sys.path.append(os.path.join(_SRC_PATH, 'third_party', 'node')) +import node +import node_modules + +def main(argv): + parser = argparse.ArgumentParser() + parser.add_argument('--tsconfig_path', required=True) + args = parser.parse_args(argv) + + result = node.RunNode([node_modules.PathToTypescript()] + + ['--project', args.tsconfig_path]) + if len(result) != 0: + raise RuntimeError('Failed to compile Typescript: \n%s' % result) + +if __name__ == '__main__': + main(sys.argv[1:]) diff --git a/utils/ipc/mojo/public/tools/bindings/concatenate-files.py b/utils/ipc/mojo/public/tools/bindings/concatenate-files.py new file mode 100755 index 00000000..48bc66fd --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/concatenate-files.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python +# Copyright 2019 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +# +# This utility concatenates several files into one. On Unix-like systems +# it is equivalent to: +# cat file1 file2 file3 ...files... > target +# +# The reason for writing a separate utility is that 'cat' is not available +# on all supported build platforms, but Python is, and hence this provides +# us with an easy and uniform way of doing this on all platforms. + +# for py2/py3 compatibility +from __future__ import print_function + +import optparse + + +def Concatenate(filenames): + """Concatenate files. + + Args: + files: Array of file names. + The last name is the target; all earlier ones are sources. + + Returns: + True, if the operation was successful. + """ + if len(filenames) < 2: + print("An error occurred generating %s:\nNothing to do." % filenames[-1]) + return False + + try: + with open(filenames[-1], "wb") as target: + for filename in filenames[:-1]: + with open(filename, "rb") as current: + target.write(current.read()) + return True + except IOError as e: + print("An error occurred when writing %s:\n%s" % (filenames[-1], e)) + return False + + +def main(): + parser = optparse.OptionParser() + parser.set_usage("""Concatenate several files into one. + Equivalent to: cat file1 ... > target.""") + (_options, args) = parser.parse_args() + exit(0 if Concatenate(args) else 1) + + +if __name__ == "__main__": + main() diff --git a/utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py b/utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py new file mode 100755 index 00000000..be8985ce --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python +# Copyright 2018 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +"""Simple utility which concatenates a set of files into a single output file +while also stripping any goog.provide or goog.require lines. This allows us to +provide a very primitive sort of "compilation" without any extra toolchain +support and without having to modify otherwise compilable sources in the tree +which use these directives. + +goog.provide lines are replaced with an equivalent invocation of +mojo.internal.exportModule, which accomplishes essentially the same thing in an +uncompiled context. A singular exception is made for the 'mojo.internal' export, +which is instead replaced with an inlined assignment to initialize the +namespace. +""" + +from __future__ import print_function + +import optparse +import re + + +_MOJO_INTERNAL_MODULE_NAME = "mojo.internal" +_MOJO_EXPORT_MODULE_SYMBOL = "mojo.internal.exportModule" + + +def FilterLine(filename, line, output): + if line.startswith("goog.require"): + return + + if line.startswith("goog.provide"): + match = re.match("goog.provide\('([^']+)'\);", line) + if not match: + print("Invalid goog.provide line in %s:\n%s" % (filename, line)) + exit(1) + + module_name = match.group(1) + if module_name == _MOJO_INTERNAL_MODULE_NAME: + output.write("self.mojo = { internal: {} };") + else: + output.write("%s('%s');\n" % (_MOJO_EXPORT_MODULE_SYMBOL, module_name)) + return + + output.write(line) + +def ConcatenateAndReplaceExports(filenames): + if (len(filenames) < 2): + print("At least two filenames (one input and the output) are required.") + return False + + try: + with open(filenames[-1], "w") as target: + for filename in filenames[:-1]: + with open(filename, "r") as current: + for line in current.readlines(): + FilterLine(filename, line, target) + return True + except IOError as e: + print("Error generating %s\n: %s" % (filenames[-1], e)) + return False + +def main(): + parser = optparse.OptionParser() + parser.set_usage("""file1 [file2...] outfile + Concatenate several files into one, stripping Closure provide and + require directives along the way.""") + (_, args) = parser.parse_args() + exit(0 if ConcatenateAndReplaceExports(args) else 1) + +if __name__ == "__main__": + main() diff --git a/utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py b/utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py new file mode 100755 index 00000000..7ac4af5f --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python +# Copyright 2016 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from __future__ import print_function + +import sys + +# This utility converts mojom dependencies into their corresponding typemap +# paths and formats them to be consumed by generate_type_mappings.py. + + +def FormatTypemap(typemap_filename): + # A simple typemap is valid Python with a minor alteration. + with open(typemap_filename) as f: + typemap_content = f.read().replace('=\n', '=') + typemap = {} + exec typemap_content in typemap + + for header in typemap.get('public_headers', []): + yield 'public_headers=%s' % header + for header in typemap.get('traits_headers', []): + yield 'traits_headers=%s' % header + for header in typemap.get('type_mappings', []): + yield 'type_mappings=%s' % header + + +def main(): + typemaps = sys.argv[1:] + print(' '.join('--start-typemap %s' % ' '.join(FormatTypemap(typemap)) + for typemap in typemaps)) + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py b/utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py new file mode 100644 index 00000000..79c9e50e --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py @@ -0,0 +1,52 @@ +# Copyright 2017 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Generates a list of all files in a directory. + +This script takes in a directory and an output file name as input. +It then reads the directory and creates a list of all file names +in that directory. The list is written to the output file. +There is also an option to pass in '-p' or '--pattern' +which will check each file name against a regular expression +pattern that is passed in. Only files which match the regex +will be written to the list. +""" + +from __future__ import print_function + +import os +import re +import sys + +from cStringIO import StringIO +from optparse import OptionParser + +sys.path.insert( + 0, + os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "mojom")) + +from mojom.generate.generator import WriteFile + + +def main(): + parser = OptionParser() + parser.add_option('-d', '--directory', help='Read files from DIRECTORY') + parser.add_option('-o', '--output', help='Write list to FILE') + parser.add_option('-p', + '--pattern', + help='Only reads files that name matches PATTERN', + default=".") + (options, _) = parser.parse_args() + pattern = re.compile(options.pattern) + files = [f for f in os.listdir(options.directory) if pattern.match(f)] + + stream = StringIO() + for f in files: + print(f, file=stream) + + WriteFile(stream.getvalue(), options.output) + stream.close() + +if __name__ == '__main__': + sys.exit(main()) diff --git a/utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py b/utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py new file mode 100755 index 00000000..64ca048f --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python +# Copyright 2016 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Generates a JSON typemap from its command-line arguments and dependencies. + +Each typemap should be specified in an command-line argument of the form +key=value, with an argument of "--start-typemap" preceding each typemap. + +For example, +generate_type_mappings.py --output=foo.typemap --start-typemap \\ + public_headers=foo.h traits_headers=foo_traits.h \\ + type_mappings=mojom.Foo=FooImpl + +generates a foo.typemap containing +{ + "c++": { + "mojom.Foo": { + "typename": "FooImpl", + "traits_headers": [ + "foo_traits.h" + ], + "public_headers": [ + "foo.h" + ] + } + } +} + +Then, +generate_type_mappings.py --dependency foo.typemap --output=bar.typemap \\ + --start-typemap public_headers=bar.h traits_headers=bar_traits.h \\ + type_mappings=mojom.Bar=BarImpl + +generates a bar.typemap containing +{ + "c++": { + "mojom.Bar": { + "typename": "BarImpl", + "traits_headers": [ + "bar_traits.h" + ], + "public_headers": [ + "bar.h" + ] + }, + "mojom.Foo": { + "typename": "FooImpl", + "traits_headers": [ + "foo_traits.h" + ], + "public_headers": [ + "foo.h" + ] + } + } +} +""" + +import argparse +import json +import os +import re +import sys + +sys.path.insert( + 0, + os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "mojom")) + +from mojom.generate.generator import WriteFile + +def ReadTypemap(path): + with open(path) as f: + return json.load(f)['c++'] + + +def ParseTypemapArgs(args): + typemaps = [s for s in '\n'.join(args).split('--start-typemap\n') if s] + result = {} + for typemap in typemaps: + result.update(ParseTypemap(typemap)) + return result + + +def LoadCppTypemapConfig(path): + configs = {} + with open(path) as f: + for config in json.load(f): + for entry in config['types']: + configs[entry['mojom']] = { + 'typename': entry['cpp'], + 'public_headers': config.get('traits_headers', []), + 'traits_headers': config.get('traits_private_headers', []), + 'copyable_pass_by_value': entry.get('copyable_pass_by_value', + False), + 'force_serialize': entry.get('force_serialize', False), + 'hashable': entry.get('hashable', False), + 'move_only': entry.get('move_only', False), + 'nullable_is_same_type': entry.get('nullable_is_same_type', False), + 'non_copyable_non_movable': False, + } + return configs + + +def ParseTypemap(typemap): + values = {'type_mappings': [], 'public_headers': [], 'traits_headers': []} + for line in typemap.split('\n'): + if not line: + continue + key, _, value = line.partition('=') + values[key].append(value.lstrip('/')) + result = {} + mapping_pattern = \ + re.compile(r"""^([^=]+) # mojom type + = + ([^[]+) # native type + (?:\[([^]]+)\])?$ # optional attribute in square brackets + """, re.X) + for typename in values['type_mappings']: + match_result = mapping_pattern.match(typename) + assert match_result, ( + "Cannot parse entry in the \"type_mappings\" section: %s" % typename) + + mojom_type = match_result.group(1) + native_type = match_result.group(2) + attributes = [] + if match_result.group(3): + attributes = match_result.group(3).split(',') + + assert mojom_type not in result, ( + "Cannot map multiple native types (%s, %s) to the same mojom type: %s" % + (result[mojom_type]['typename'], native_type, mojom_type)) + + result[mojom_type] = { + 'public_headers': values['public_headers'], + 'traits_headers': values['traits_headers'], + 'typename': native_type, + + # Attributes supported for individual mappings. + 'copyable_pass_by_value': 'copyable_pass_by_value' in attributes, + 'force_serialize': 'force_serialize' in attributes, + 'hashable': 'hashable' in attributes, + 'move_only': 'move_only' in attributes, + 'non_copyable_non_movable': 'non_copyable_non_movable' in attributes, + 'nullable_is_same_type': 'nullable_is_same_type' in attributes, + } + return result + + +def main(): + parser = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument( + '--dependency', + type=str, + action='append', + default=[], + help=('A path to another JSON typemap to merge into the output. ' + 'This may be repeated to merge multiple typemaps.')) + parser.add_argument( + '--cpp-typemap-config', + type=str, + action='store', + dest='cpp_config_path', + help=('A path to a single JSON-formatted typemap config as emitted by' + 'GN when processing a mojom_cpp_typemap build rule.')) + parser.add_argument('--output', + type=str, + required=True, + help='The path to which to write the generated JSON.') + params, typemap_params = parser.parse_known_args() + typemaps = ParseTypemapArgs(typemap_params) + if params.cpp_config_path: + typemaps.update(LoadCppTypemapConfig(params.cpp_config_path)) + missing = [path for path in params.dependency if not os.path.exists(path)] + if missing: + raise IOError('Missing dependencies: %s' % ', '.join(missing)) + for path in params.dependency: + typemaps.update(ReadTypemap(path)) + + WriteFile(json.dumps({'c++': typemaps}, indent=2), params.output) + + +if __name__ == '__main__': + main() diff --git a/utils/ipc/mojo/public/tools/bindings/mojom.gni b/utils/ipc/mojo/public/tools/bindings/mojom.gni new file mode 100644 index 00000000..a739fa6e --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/mojom.gni @@ -0,0 +1,1941 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import("//build/config/jumbo.gni") +import("//third_party/closure_compiler/closure_args.gni") +import("//third_party/closure_compiler/compile_js.gni") +import("//third_party/protobuf/proto_library.gni") +import("//ui/webui/webui_features.gni") + +# TODO(rockot): Maybe we can factor these dependencies out of //mojo. They're +# used to conditionally enable message ID scrambling in a way which is +# consistent across toolchains and which is affected by branded vs non-branded +# Chrome builds. Ideally we could create some generic knobs here that could be +# flipped elsewhere though. +import("//build/config/chrome_build.gni") +import("//build/config/chromecast_build.gni") +import("//build/config/chromeos/ui_mode.gni") +import("//build/config/nacl/config.gni") +import("//build/toolchain/kythe.gni") +import("//components/nacl/features.gni") +import("//third_party/jinja2/jinja2.gni") +import("//tools/ipc_fuzzer/ipc_fuzzer.gni") +declare_args() { + # Indicates whether typemapping should be supported in this build + # configuration. This may be disabled when building external projects which + # depend on //mojo but which do not need/want all of the Chromium tree + # dependencies that come with typemapping. + # + # Note that (perhaps obviously) a huge amount of Chromium code will not build + # with typemapping disabled, so it is never valid to set this to |false| in + # any Chromium build configuration. + enable_mojom_typemapping = true + + # Controls message ID scrambling behavior. If |true|, message IDs are + # scrambled (i.e. randomized based on the contents of //chrome/VERSION) on + # non-Chrome OS desktop platforms. Set to |false| to disable message ID + # scrambling on all platforms. + enable_mojom_message_id_scrambling = true + + # Enables Closure compilation of generated JS lite bindings. In environments + # where compilation is supported, any mojom target "foo" will also have a + # corresponding "foo_js_library_for_compile" target generated. + enable_mojom_closure_compile = enable_js_type_check && optimize_webui + + # Enables generating Typescript bindings and compiling them to JS bindings. + enable_typescript_bindings = false + + # Enables generating javascript fuzzing-related code and the bindings for the + # MojoLPM fuzzer targets. Off by default. + enable_mojom_fuzzer = false +} + +# NOTE: We would like to avoid scrambling message IDs where it doesn't add +# value, so we limit the behavior to desktop builds for now. There is some +# redundancy in the conditions here, but it is tolerated for clarity: +# We're explicit about Mac, Windows, and Linux desktop support, but it's +# also necessary to ensure that bindings in alternate toolchains (e.g. +# NaCl IRT) are always consistent with the default toolchain; for that +# reason we always enable scrambling within NaCl toolchains when possible, +# as well as within the default toolchain when NaCl is enabled. +# +# Finally, because we *cannot* enable scrambling on Chrome OS (it would break +# ARC) we have to explicitly opt out there even when NaCl is enabled (and +# consequently also when building for NaCl toolchains.) For this reason we +# check |target_os| explicitly, as it's consistent across all toolchains. +enable_scrambled_message_ids = + enable_mojom_message_id_scrambling && + (is_mac || is_win || (is_linux && !is_chromeos && !is_chromecast && + !chromeos_is_browser_only) || + ((enable_nacl || is_nacl || is_nacl_nonsfi) && + (target_os != "chromeos" && !chromeos_is_browser_only))) + +_mojom_tools_root = "//mojo/public/tools" +_mojom_library_root = "$_mojom_tools_root/mojom/mojom" +mojom_parser_script = "$_mojom_tools_root/mojom/mojom_parser.py" +mojom_parser_sources = [ + "$_mojom_library_root/__init__.py", + "$_mojom_library_root/error.py", + "$_mojom_library_root/generate/__init__.py", + "$_mojom_library_root/generate/constant_resolver.py", + "$_mojom_library_root/generate/generator.py", + "$_mojom_library_root/generate/module.py", + "$_mojom_library_root/generate/pack.py", + "$_mojom_library_root/generate/template_expander.py", + "$_mojom_library_root/generate/translate.py", + "$_mojom_library_root/parse/__init__.py", + "$_mojom_library_root/parse/ast.py", + "$_mojom_library_root/parse/lexer.py", + "$_mojom_library_root/parse/parser.py", +] + +mojom_generator_root = "$_mojom_tools_root/bindings" +mojom_generator_script = "$mojom_generator_root/mojom_bindings_generator.py" +mojom_generator_sources = + mojom_parser_sources + [ + "$mojom_generator_root/generators/mojom_cpp_generator.py", + "$mojom_generator_root/generators/mojom_java_generator.py", + "$mojom_generator_root/generators/mojom_mojolpm_generator.py", + "$mojom_generator_root/generators/mojom_js_generator.py", + "$mojom_generator_root/generators/mojom_ts_generator.py", + "$mojom_generator_script", + ] + +if (enable_scrambled_message_ids) { + declare_args() { + # The path to a file whose contents can be used as the basis for a message + # ID scrambling salt. + mojom_message_id_salt_path = "//chrome/VERSION" + + # The path to a file whose contents will be concatenated to the contents of + # the file at |mojom_message_id_salt_path| to form a complete salt for + # message ID scrambling. May be the empty string, in which case the contents + # of the above file alone are used as the complete salt. + if (is_chrome_branded) { + mojom_message_id_salt_suffix_path = + "//mojo/internal/chrome-message-id-salt-suffix" + } else { + mojom_message_id_salt_suffix_path = "" + } + } + + assert(mojom_message_id_salt_path != "") + message_scrambling_args = [ + "--scrambled_message_id_salt_path", + rebase_path(mojom_message_id_salt_path, root_build_dir), + ] + message_scrambling_inputs = [ mojom_message_id_salt_path ] + + if (mojom_message_id_salt_suffix_path != "") { + message_scrambling_args += [ + "--scrambled_message_id_salt_path", + rebase_path(mojom_message_id_salt_suffix_path, root_build_dir), + ] + message_scrambling_inputs += [ mojom_message_id_salt_suffix_path ] + } +} else { + message_scrambling_args = [] + message_scrambling_inputs = [] +} + +if (enable_mojom_typemapping) { + _bindings_configuration_files = + [ "//mojo/public/tools/bindings/chromium_bindings_configuration.gni" ] + _bindings_configurations = [] + foreach(config_file, _bindings_configuration_files) { + _bindings_configurations += [ read_file(config_file, "scope") ] + } + foreach(configuration, _bindings_configurations) { + # Check that the mojom field of each typemap refers to a mojom that exists. + foreach(typemap, configuration.typemaps) { + _typemap_config = { + } + _typemap_config = typemap.config + read_file(_typemap_config.mojom, "") + } + } +} else { + _bindings_configuration_files = [] + _bindings_configurations = [ + { + typemaps = [] + component_macro_suffix = "" + }, + ] +} + +if (!is_ios) { + _bindings_configurations += [ + { + variant = "blink" + component_macro_suffix = "_BLINK" + for_blink = true + typemaps = [] + }, + ] +} + +# Generates targets for building C++, JavaScript and Java bindings from mojom +# files. The output files will go under the generated file directory tree with +# the same path as each input file. +# +# Other targets should depend on one of these generated targets (where "foo" +# is the target name): +# +# foo +# C++ bindings. +# +# foo_blink +# C++ bindings using Blink standard types. +# +# foo_java +# Java bindings. +# +# foo_js +# JavaScript bindings; used as compile-time dependency. +# +# foo_js_data_deps +# JavaScript bindings; used as run-time dependency. +# +# Parameters: +# +# sources (optional if one of the deps sets listed below is present) +# List of source .mojom files to compile. +# +# deps (optional) +# Note: this can contain only other mojom targets. +# +# DEPRECATED: This is synonymous with public_deps because all mojom +# dependencies must be public by design. Please use public_deps. +# +# public_deps (optional) +# Note: this can contain only other mojom targets. +# +# parser_deps (optional) +# List of non-mojom targets required for the mojom sources to be parsed. +# +# import_dirs (optional) +# List of import directories that will get added when processing sources. +# +# testonly (optional) +# +# visibility (optional) +# +# visibility_blink (optional) +# The value to use for visibility for the blink variant. If unset, +# |visibility| is used. +# +# cpp_only (optional) +# If set to true, only the C++ bindings targets will be generated. +# +# NOTE: If the global |enable_mojom_fuzzer| build arg is true, JS bindings +# will still be generated even when |cpp_only| is set to |true|, unless +# you also set |enable_fuzzing| to |false| in your mojom target. +# +# cpp_typemaps (optional) +# A list of typemaps to be applied to the generated C++ bindings for this +# mojom target. Note that this only applies to the non-Blink variant of +# generated C++ bindings. +# +# Every typemap is a GN scope describing how one or more mojom types maps +# to a non-mojom C++ type, including necessary deps and headers required +# for the mapping to work. See the Typemaps section below. +# +# blink_cpp_typemaps (optional) +# Same as above, but for the Blink variant of generated C++ bindings. +# +# generate_java (optional) +# If set to true, Java bindings are generated for Android builds. If +# |cpp_only| is set to true, it overrides this to prevent generation of +# Java bindings. +# +# enable_fuzzing (optional) +# Enables generation of fuzzing sources for the target if the global build +# arg |enable_mojom_fuzzer| is also set to |true|. Defaults to |true|. If +# fuzzing generation is enabled for a target, the target will always +# generate JS bindings even if |cpp_only| is set to |true|. See note +# above. +# +# support_lazy_serialization (optional) +# If set to |true|, generated C++ bindings will effectively prefer to +# transmit messages in an unserialized form when going between endpoints +# in the same process. This avoids the runtime cost of serialization, +# deserialization, and validation logic at the expensive of increased +# code size. Defaults to |false|. +# +# disable_variants (optional) +# If |true|, no variant sources will be generated for the target. Defaults +# to |false|. +# +# disallow_native_types (optional) +# If set to |true|, mojoms in this target may not apply the [Native] +# attribute to struct or enum declarations. This avoids emitting code +# which depends on legacy IPC serialization. Default is |false|, meaning +# [Native] types are allowed. +# +# disallow_interfaces (optional) +# If set to |true|, mojoms in this target may not define interfaces. +# Generates bindings with a smaller set of dependencies. Defaults to +# |false|. +# +# scramble_message_ids (optional) +# If set to |true| (the default), generated mojom interfaces will use +# scrambled ordinal identifiers in encoded messages. +# +# component_output_prefix (optional) +# The prefix to use for the output_name of any component library emitted +# for generated C++ bindings. If this is omitted, C++ bindings targets are +# emitted as source_sets instead. Because this controls the name of the +# output shared library binary in the root output directory, it must be +# unique across the entire build configuration. +# +# This is required if |component_macro_prefix| is specified. +# +# component_macro_prefix (optional) +# This specifies a macro prefix to use for component export macros and +# should therefore be globally unique in the project. For example if this +# is "FOO_BAR", then the generated C++ sources will be built with +# IS_FOO_BAR_{suffix}_IMPL defined, and the generated public headers will +# annotate public symbol definitions with +# COMPONENT_EXPORT(FOO_BAR_{suffix}). "suffix" in this case depends on +# which internal subtarget is generating the code (e.g. "SHARED", or a +# variant name like "BLINK"). +# +# enabled_features (optional) +# Definitions in a mojom file can be guarded by an EnableIf attribute. If +# the value specified by the attribute does not match any items in the +# list of enabled_features, the definition will be disabled, with no code +# emitted for it. +# +# generate_closure_exports (optional) +# Generates JS lite bindings will use goog.provide and goog.require +# annotations to export its symbols and import core Mojo bindings support +# and other mojom dependency modules. Use this if you plan to compile your +# bindings into a larger JS binary. Defaults to |false|, instead +# generating JS lite bindings which assume they will be manually loaded in +# correct dependency order. Note that this only has an effect if +# the |enable_mojom_closure_compile| global arg is set to |true| as well. +# +# use_typescript_sources (optional) +# Uses the Typescript generator to generate JavaScript bindings. +# +# js_generate_struct_deserializers (optional) +# Generates JS deerialize methods for structs. +# +# extra_cpp_template_paths (optional) +# List of extra C++ templates that are used to generate additional source +# and/or header files. The templates should end with extension ".tmpl". +# +# The following parameters are used to support the component build. They are +# needed so that bindings which are linked with a component can use the same +# export settings for classes. The first three are for the chromium variant, and +# the last three are for the blink variant. These parameters can also override +# |component_macro_prefix| for a specific variant, allowing e.g. one variant +# to be linked into a larger non-mojom component target, while all other +# variants get their own unique component target. +# export_class_attribute (optional) +# The attribute to add to the class declaration. e.g. "CONTENT_EXPORT" +# export_define (optional) +# A define to be added to the source_set which is needed by the export +# header. e.g. "CONTENT_IMPLEMENTATION=1" +# export_header (optional) +# A header to be added to the generated bindings to support the component +# build. e.g. "content/common/content_export.h" +# export_class_attribute_blink (optional) +# export_define_blink (optional) +# export_header_blink (optional) +# These three parameters are the blink variants of the previous 3. +# +# The following parameters are used to correct component build dependencies. +# They are needed so mojom-mojom dependencies follow the rule that dependencies +# on a source set in another component are replaced by a dependency on the +# containing component. The first two are for the chromium variant; the other +# two are for the blink variant. +# overridden_deps (optional) +# The list of mojom deps to be overridden. +# component_deps (optional) +# The list of component deps to add to replace overridden_deps. +# overridden_deps_blink (optional) +# component_deps_blink (optional) +# These two parameters are the blink variants of the previous two. +# +# check_includes_blink (optional) +# Overrides the check_includes variable for the blink variant. +# If check_includes_blink is not defined, the check_includes variable +# retains its original value. +# +# Typemaps +# ======== +# The cpp_typemaps and blink_cpp_typemaps each specify an optional list of +# typemapping configurations. Each configuration is a GN scope with metadata +# describing what and how to map. +# +# Typemap scope parameters: +# types +# A list of type specifications for this typemap. Each type specification +# is a nested GN scope which can be expressed with the following syntax: +# +# { +# mojom = "foo.mojom.Bar" +# cpp = "::foo::LegitBar" +# move_only = true +# # etc... +# } +# +# Each type specification supports the following values: +# +# mojom (required) +# The fully qualified name of a mojom type to be mapped. This is a +# string like "foo.mojom.Bar". +# +# cpp (required) +# The fully qualified name of the C++ type to which the mojom type +# should be mapped in generated bindings. This is a string like +# "::base::Value" or "std::vector<::base::Value>". +# +# move_only (optional) +# A boolean value (default false) which indicates whether the C++ +# type is move-only. If true, generated bindings will pass the type +# by value and use std::move() at call sites. +# +# copyable_pass_by_value (optional) +# A boolean value (default false) which effectively indicates +# whether the C++ type is very cheap to copy. If so, generated +# bindings will pass by value but not use std::move() at call sites. +# +# nullable_is_same_type (optional) +# A boolean value (default false) which indicates that the C++ type +# has some baked-in semantic notion of a "null" state. If true, the +# traits for the type must define IsNull and SetToNull methods. +# +# When false, nullable fields are represented by wrapping the C++ +# type with base::Optional, and null values are simply +# base::nullopt. +# +# hashable (optional) +# A boolean value (default false) indicating whether the C++ type is +# hashable. Set to true if true AND needed (i.e. you need to use the +# type as the key of a mojom map). +# +# force_serialize (optional) +# A boolean value (default false) which disables lazy serialization +# of the typemapped type if lazy serialization is enabled for the +# mojom target applying this typemap. +# +# Additional typemap scope parameters: +# +# traits_headers (optional) +# Headers which must be included in the generated mojom in order for +# serialization to be possible. This generally means including at least +# the header for the corresponding mojom traits definitions. +# +# traits_private_headers (optional) +# Headers which must be included in generated C++ serialization code for +# a mojom using the typemap. This should be used only when including a +# header in |traits_headers| is problematic for compilation, as is +# sometimes the case with legacy IPC message headers. +# +# traits_sources (optional) +# The references to the source files (typically a single .cc and .h file) +# defining an appropriate set of EnumTraits or StructTraits, etc for the +# the type-mapping. Using this will cause the listed sources to be +# integrated directly into the dependent mojom's generated type-mapping +# targets. +# +# Prefer using |traits_public_deps| over inlined |traits_sources|, as this +# will generally lead to easier build maintenance over time. +# +# NOTE: If a typemap is shared by Blink and non-Blink bindings, you cannot +# use this and MUST use |traits_public_deps| to reference traits built +# within a separate target. +# +# traits_deps / traits_public_deps (optional) +# Any dependencies of sources in |traits_headers| or |traits_sources| must +# be listed here. +# +template("mojom") { + assert( + defined(invoker.sources) || defined(invoker.deps) || + defined(invoker.public_deps), + "\"sources\" or \"deps\" must be defined for the $target_name template.") + + if (defined(invoker.export_class_attribute) || + defined(invoker.export_define) || defined(invoker.export_header)) { + assert(defined(invoker.export_class_attribute)) + assert(defined(invoker.export_define)) + assert(defined(invoker.export_header)) + } + if (defined(invoker.export_class_attribute_blink) || + defined(invoker.export_define_blink) || + defined(invoker.export_header_blink)) { + assert(defined(invoker.export_class_attribute_blink)) + assert(defined(invoker.export_define_blink)) + assert(defined(invoker.export_header_blink)) + + # Not all platforms use the Blink variant, so make sure GN doesn't complain + # about these values being inconsequential. + not_needed(invoker, + [ + "export_class_attribute_blink", + "export_define_blink", + "export_header_blink", + ]) + } + if (defined(invoker.overridden_deps) || defined(invoker.component_deps)) { + assert(defined(invoker.overridden_deps)) + assert(defined(invoker.component_deps)) + } + + if (defined(invoker.overridden_deps_blink) || + defined(invoker.component_deps_blink)) { + assert(defined(invoker.overridden_deps_blink)) + assert(defined(invoker.component_deps_blink)) + } + + # Type-mapping may be disabled or we may not generate C++ bindings. + not_needed(invoker, + [ + "cpp_typemaps", + "blink_cpp_typemaps", + ]) + + require_full_cpp_deps = + !defined(invoker.disallow_native_types) || + !invoker.disallow_native_types || !defined(invoker.disallow_interfaces) || + !invoker.disallow_interfaces + + all_deps = [] + if (defined(invoker.deps)) { + all_deps += invoker.deps + } + if (defined(invoker.public_deps)) { + all_deps += invoker.public_deps + } + + if (defined(invoker.component_macro_prefix)) { + assert(defined(invoker.component_output_prefix)) + } + + group("${target_name}__is_mojom") { + } + + # Explicitly ensure that all dependencies (invoker.deps and + # invoker.public_deps) are mojom targets. + group("${target_name}__check_deps_are_all_mojom") { + deps = [] + foreach(d, all_deps) { + name = get_label_info(d, "label_no_toolchain") + toolchain = get_label_info(d, "toolchain") + deps += [ "${name}__is_mojom(${toolchain})" ] + } + } + + sources_list = [] + if (defined(invoker.sources)) { + sources_list = invoker.sources + } + + # Reset sources_assignment_filter for the BUILD.gn file to prevent + # regression during the migration of Chromium away from the feature. + # See docs/no_sources_assignment_filter.md for more information. + # TODO(crbug.com/1018739): remove this when migration is done. + set_sources_assignment_filter([]) + + # Listed sources may be relative to the current target dir, or they may be + # absolute paths, including paths to generated mojom files. While those are + # fine as-is for input references, deriving output paths can be more subtle. + # + # Here we rewrite all source paths to be relative to the root build dir and + # strip any root_gen_dir prefixes. + # + # So for a target in //foo/bar with: + # + # sources = [ + # "a.mojom", + # "b/c.mojom", + # "//baz/d.mojom", + # "$target_gen_dir/e.mojom", + # ] + # + # output_file_base_paths will be: + # + # [ + # "foo/bar/a.mojom", + # "foo/bar/b/c.mojom", + # "baz/d.mojom", + # "foo/bar/e.mojom", + # ] + # + # This result is essentially a list of base filename paths which are suitable + # for the naming of any generated output files derived from their + # corresponding input mojoms. These paths are always considered to be relative + # to root_gen_dir. + source_abspaths = rebase_path(sources_list, "//") + output_file_base_paths = [] + foreach(path, source_abspaths) { + output_file_base_paths += + [ string_replace(path, rebase_path(root_gen_dir, "//") + "/", "") ] + } + + # Sanity check that either all input files have a .mojom extension, or + # all input files have a .test-mojom extension AND |testonly| is |true|. + sources_list_filenames = + process_file_template(sources_list, "{{source_file_part}}") + sources_list_filenames_with_mojom_extension = + process_file_template(sources_list, "{{source_name_part}}.mojom") + if (sources_list_filenames != sources_list_filenames_with_mojom_extension) { + sources_list_filenames_with_test_mojom_extension = + process_file_template(sources_list, "{{source_name_part}}.test-mojom") + if (sources_list_filenames == + sources_list_filenames_with_test_mojom_extension) { + assert( + defined(invoker.testonly) && invoker.testonly, + "mojom targets for .test-mojom files must set |testonly| to |true|") + } else { + assert( + false, + "One or more mojom files has an invalid extension. The only " + + "allowed extensions are .mojom and .test-mojom, and any given " + + "mojom target must use one or the other exclusively.") + } + } + + build_metadata_filename = "$target_gen_dir/$target_name.build_metadata" + build_metadata = { + } + build_metadata.sources = rebase_path(sources_list) + build_metadata.deps = [] + foreach(dep, all_deps) { + dep_target_gen_dir = get_label_info(dep, "target_gen_dir") + dep_name = get_label_info(dep, "name") + build_metadata.deps += + [ rebase_path("$dep_target_gen_dir/$dep_name.build_metadata") ] + } + write_file(build_metadata_filename, build_metadata, "json") + + generate_fuzzing = + (!defined(invoker.enable_fuzzing) || invoker.enable_fuzzing) && + enable_mojom_fuzzer && (!defined(invoker.testonly) || !invoker.testonly) + + parser_target_name = "${target_name}__parser" + parser_deps = [] + foreach(dep, all_deps) { + _label = get_label_info(dep, "label_no_toolchain") + parser_deps += [ "${_label}__parser" ] + } + if (defined(invoker.parser_deps)) { + parser_deps += invoker.parser_deps + } + if (sources_list == []) { + # Even without sources we generate a parser target to at least forward + # other parser dependencies. + group(parser_target_name) { + public_deps = parser_deps + } + } else { + enabled_features = [] + if (defined(invoker.enabled_features)) { + enabled_features += invoker.enabled_features + } + if (is_posix) { + enabled_features += [ "is_posix" ] + } + if (is_android) { + enabled_features += [ "is_android" ] + } else if (is_chromeos) { + enabled_features += [ "is_chromeos" ] + } else if (is_fuchsia) { + enabled_features += [ "is_fuchsia" ] + } else if (is_ios) { + enabled_features += [ "is_ios" ] + } else if (is_linux) { + enabled_features += [ "is_linux" ] + } else if (is_mac) { + enabled_features += [ "is_mac" ] + } else if (is_win) { + enabled_features += [ "is_win" ] + } + + action(parser_target_name) { + script = mojom_parser_script + inputs = mojom_parser_sources + [ build_metadata_filename ] + sources = sources_list + deps = parser_deps + outputs = [] + foreach(base_path, output_file_base_paths) { + filename = get_path_info(base_path, "file") + dirname = get_path_info(base_path, "dir") + outputs += [ "$root_gen_dir/$dirname/${filename}-module" ] + } + + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path(source) ] + } + response_file_contents = filelist + + args = [ + # Resolve relative input mojom paths against both the root src dir and + # the root gen dir. + "--input-root", + rebase_path("//"), + "--input-root", + rebase_path(root_gen_dir), + + "--output-root", + rebase_path(root_gen_dir), + + "--mojom-file-list={{response_file_name}}", + + "--check-imports", + rebase_path(build_metadata_filename), + ] + + foreach(enabled_feature, enabled_features) { + args += [ + "--enable-feature", + enabled_feature, + ] + } + } + } + + generator_cpp_message_ids_target_name = "${target_name}__generate_message_ids" + + # Generate code that is shared by different variants. + if (sources_list != []) { + common_generator_args = [ + "--use_bundled_pylibs", + "-o", + rebase_path(root_gen_dir, root_build_dir), + "generate", + "-d", + rebase_path("//", root_build_dir), + "-I", + rebase_path("//", root_build_dir), + "--bytecode_path", + rebase_path("$root_gen_dir/mojo/public/tools/bindings", root_build_dir), + ] + + if (defined(invoker.disallow_native_types) && + invoker.disallow_native_types) { + common_generator_args += [ "--disallow_native_types" ] + } + + if (defined(invoker.disallow_interfaces) && invoker.disallow_interfaces) { + common_generator_args += [ "--disallow_interfaces" ] + } + + if (defined(invoker.import_dirs)) { + foreach(import_dir, invoker.import_dirs) { + common_generator_args += [ + "-I", + rebase_path(import_dir, root_build_dir), + ] + } + } + + if (defined(invoker.component_macro_prefix)) { + shared_component_export_macro = + "COMPONENT_EXPORT(${invoker.component_macro_prefix}_SHARED)" + shared_component_impl_macro = + "IS_${invoker.component_macro_prefix}_SHARED_IMPL" + shared_component_output_name = "${invoker.component_output_prefix}_shared" + } else if (defined(invoker.export_class_attribute_shared) || + defined(invoker.export_class_attribute)) { + if (defined(invoker.export_class_attribute_shared)) { + assert(defined(invoker.export_header_shared)) + shared_component_export_macro = invoker.export_class_attribute_shared + shared_component_impl_macro = invoker.export_define_shared + } else { + assert(!defined(invoker.export_header_shared)) + + # If no explicit shared attribute/define was provided by the invoker, + # we derive some reasonable settings frorm the default variant. + shared_component_export_macro = "COMPONENT_EXPORT(MOJOM_SHARED_" + + invoker.export_class_attribute + ")" + shared_component_impl_macro = + "IS_MOJOM_SHARED_" + invoker.export_class_attribute + "_IMPL" + } + + if (defined(invoker.component_output_prefix)) { + shared_component_output_name = + "${invoker.component_output_prefix}_shared" + } else { + shared_component_output_name = "${target_name}_shared" + } + } + + action(generator_cpp_message_ids_target_name) { + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + if (defined(invoker.parser_deps)) { + deps += invoker.parser_deps + } + outputs = [] + args = common_generator_args + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path("$source", root_build_dir) ] + } + foreach(base_path, output_file_base_paths) { + outputs += [ "$root_gen_dir/$base_path-shared-message-ids.h" ] + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "--generate_non_variant_code", + "--generate_message_ids", + "-g", + "c++", + ] + + if (!defined(invoker.scramble_message_ids) || + invoker.scramble_message_ids) { + inputs += message_scrambling_inputs + args += message_scrambling_args + } + } + + generator_shared_target_name = "${target_name}_shared__generator" + action(generator_shared_target_name) { + visibility = [ ":*" ] + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + if (defined(invoker.parser_deps)) { + deps += invoker.parser_deps + } + + outputs = [] + args = common_generator_args + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path("$source", root_build_dir) ] + } + foreach(base_path, output_file_base_paths) { + outputs += [ + "$root_gen_dir/$base_path-params-data.h", + "$root_gen_dir/$base_path-shared-internal.h", + "$root_gen_dir/$base_path-shared.cc", + "$root_gen_dir/$base_path-shared.h", + ] + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "--generate_non_variant_code", + "-g", + "c++", + ] + + if (defined(shared_component_export_macro)) { + args += [ + "--export_attribute", + shared_component_export_macro, + "--export_header", + "base/component_export.h", + ] + } + + # Enable adding annotations to generated C++ headers that are used for + # cross-references in CodeSearch. + if (enable_kythe_annotations) { + args += [ "--enable_kythe_annotations" ] + } + } + } else { + group(generator_cpp_message_ids_target_name) { + } + } + + shared_cpp_sources_target_name = "${target_name}_shared_cpp_sources" + jumbo_source_set(shared_cpp_sources_target_name) { + if (defined(invoker.testonly)) { + testonly = invoker.testonly + } + deps = [] + public_deps = [] + if (output_file_base_paths != []) { + sources = [] + foreach(base_path, output_file_base_paths) { + sources += [ + "$root_gen_dir/$base_path-params-data.h", + "$root_gen_dir/$base_path-shared-internal.h", + "$root_gen_dir/$base_path-shared.cc", + "$root_gen_dir/$base_path-shared.h", + ] + } + public_deps += [ ":$generator_shared_target_name" ] + } + if (require_full_cpp_deps) { + public_deps += [ "//mojo/public/cpp/bindings" ] + } else { + public_deps += [ "//mojo/public/cpp/bindings:bindings_base" ] + } + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append shared_cpp_sources_suffix + # to get the cpp dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps += [ "${full_name}_shared" ] + } + if (defined(shared_component_impl_macro)) { + defines = [ shared_component_impl_macro ] + } + } + + shared_cpp_library_target_name = "${target_name}_shared" + if (defined(shared_component_output_name)) { + component(shared_cpp_library_target_name) { + if (defined(invoker.testonly)) { + testonly = invoker.testonly + } + output_name = "$shared_component_output_name" + public_deps = [ ":$shared_cpp_sources_target_name" ] + } + } else { + group(shared_cpp_library_target_name) { + if (defined(invoker.testonly)) { + testonly = invoker.testonly + } + public_deps = [ ":$shared_cpp_sources_target_name" ] + } + } + + if (generate_fuzzing) { + # This block generates the proto files used for the MojoLPM fuzzer, + # and the corresponding proto targets that will be linked in the fuzzer + # targets. These are independent of the typemappings, and can be done + # separately here. + + generator_mojolpm_proto_target_name = + "${target_name}_mojolpm_proto_generator" + action(generator_mojolpm_proto_target_name) { + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = invoker.sources + deps = [ + ":$parser_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + + outputs = [] + args = common_generator_args + filelist = [] + foreach(source, invoker.sources) { + filelist += [ rebase_path("$source", root_build_dir) ] + outputs += [ "$target_gen_dir/$source.mojolpm.proto" ] + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "--generate_non_variant_code", + "-g", + "mojolpm", + ] + } + + mojolpm_proto_target_name = "${target_name}_mojolpm_proto" + if (defined(invoker.sources)) { + proto_library(mojolpm_proto_target_name) { + testonly = true + generate_python = false + sources = process_file_template( + invoker.sources, + [ "{{source_gen_dir}}/{{source_file_part}}.mojolpm.proto" ]) + import_dirs = [ "${root_gen_dir}" ] + proto_in_dir = "${root_gen_dir}" + proto_out_dir = "." + proto_deps = [ "//mojo/public/tools/fuzzers:mojolpm_proto_copy" ] + proto_deps += [ ":$generator_mojolpm_proto_target_name" ] + link_deps = [ "//mojo/public/tools/fuzzers:mojolpm_proto" ] + + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append mojolpm_proto_suffix + # to get the proto dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + proto_deps += [ "${full_name}_mojolpm_proto" ] + link_deps += [ "${full_name}_mojolpm_proto" ] + } + } + } else { + group(mojolpm_proto_target_name) { + testonly = true + public_deps = [ "//mojo/public/tools/fuzzers:mojolpm_proto" ] + if (defined(generator_shared_target_name)) { + public_deps += [ ":$generator_shared_target_name" ] + } + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append #mojolpm_proto_suffix + # to get the proto dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps += [ "${full_name}_mojolpm_proto" ] + } + } + } + } + + # Generate code for variants. + if (!defined(invoker.disable_variants) || !invoker.disable_variants) { + enabled_configurations = _bindings_configurations + } else { + first_config = _bindings_configurations[0] + assert(!defined(first_config.variant)) + enabled_configurations = [ first_config ] + } + foreach(bindings_configuration, enabled_configurations) { + cpp_only = false + if (defined(invoker.cpp_only)) { + cpp_only = invoker.cpp_only + } + variant_suffix = "" + if (defined(bindings_configuration.variant)) { + variant = bindings_configuration.variant + variant_suffix = "_${variant}" + cpp_only = true + } + + cpp_typemap_configs = [] + export_defines = [] + export_defines_overridden = false + force_source_set = false + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink) { + if (defined(invoker.blink_cpp_typemaps)) { + cpp_typemap_configs = invoker.blink_cpp_typemaps + } + if (defined(invoker.export_define_blink)) { + export_defines_overridden = true + export_defines = [ invoker.export_define_blink ] + force_source_set = true + } + } else { + if (defined(invoker.cpp_typemaps)) { + cpp_typemap_configs = invoker.cpp_typemaps + } + + if (defined(invoker.export_define)) { + export_defines_overridden = true + export_defines = [ invoker.export_define ] + force_source_set = true + } + } + not_needed([ "cpp_typemap_configs" ]) + + if (!export_defines_overridden && defined(invoker.component_macro_prefix)) { + output_name_override = + "${invoker.component_output_prefix}${variant_suffix}" + export_defines = + [ "IS_${invoker.component_macro_prefix}" + + "${bindings_configuration.component_macro_suffix}_IMPL" ] + } + + export_args = [] + export_args_overridden = false + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink) { + if (defined(invoker.export_class_attribute_blink)) { + export_args_overridden = true + export_args += [ + "--export_attribute", + invoker.export_class_attribute_blink, + "--export_header", + invoker.export_header_blink, + ] + } + } else if (defined(invoker.export_class_attribute)) { + export_args_overridden = true + export_args += [ + "--export_attribute", + invoker.export_class_attribute, + "--export_header", + invoker.export_header, + ] + } + + if (!export_args_overridden && defined(invoker.component_macro_prefix)) { + export_args += [ + "--export_attribute", + "COMPONENT_EXPORT(${invoker.component_macro_prefix}" + + "${bindings_configuration.component_macro_suffix})", + "--export_header", + "base/component_export.h", + ] + } + + generate_java = false + if (!cpp_only && defined(invoker.generate_java)) { + generate_java = invoker.generate_java + } + type_mappings_target_name = "${target_name}${variant_suffix}__type_mappings" + type_mappings_path = + "$target_gen_dir/${target_name}${variant_suffix}__type_mappings" + active_typemaps = [] + if (sources_list != []) { + generator_cpp_output_suffixes = [] + variant_dash_suffix = "" + if (defined(variant)) { + variant_dash_suffix = "-${variant}" + } + generator_cpp_output_suffixes += [ + "${variant_dash_suffix}-forward.h", + "${variant_dash_suffix}-import-headers.h", + "${variant_dash_suffix}-test-utils.cc", + "${variant_dash_suffix}-test-utils.h", + "${variant_dash_suffix}.cc", + "${variant_dash_suffix}.h", + ] + foreach(source, sources_list) { + # TODO(sammc): Use a map instead of a linear scan when GN supports maps. + foreach(typemap, bindings_configuration.typemaps) { + _typemap_config = { + } + _typemap_config = typemap.config + if (get_path_info(source, "abspath") == _typemap_config.mojom) { + active_typemaps += [ typemap ] + } + } + } + + generator_target_name = "${target_name}${variant_suffix}__generator" + action(generator_target_name) { + visibility = [ ":*" ] + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + ":$type_mappings_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + if (defined(invoker.parser_deps)) { + deps += invoker.parser_deps + } + outputs = [] + args = common_generator_args + export_args + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path("$source", root_build_dir) ] + } + foreach(base_path, output_file_base_paths) { + outputs += [ + "$root_gen_dir/${base_path}${variant_dash_suffix}-forward.h", + "$root_gen_dir/${base_path}${variant_dash_suffix}-import-headers.h", + "$root_gen_dir/${base_path}${variant_dash_suffix}-test-utils.cc", + "$root_gen_dir/${base_path}${variant_dash_suffix}-test-utils.h", + "$root_gen_dir/${base_path}${variant_dash_suffix}.cc", + "$root_gen_dir/${base_path}${variant_dash_suffix}.h", + ] + if (generate_fuzzing && !defined(bindings_configuration.variant)) { + outputs += [ + "$root_gen_dir/${base_path}${variant_dash_suffix}-mojolpm.cc", + "$root_gen_dir/${base_path}${variant_dash_suffix}-mojolpm.h", + ] + } + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "-g", + ] + + if (generate_fuzzing && !defined(bindings_configuration.variant)) { + args += [ "c++,mojolpm" ] + } else { + args += [ "c++" ] + } + + if (defined(bindings_configuration.variant)) { + args += [ + "--variant", + bindings_configuration.variant, + ] + } + + args += [ + "--typemap", + rebase_path(type_mappings_path, root_build_dir), + ] + + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink) { + args += [ "--for_blink" ] + } + + if (defined(invoker.support_lazy_serialization) && + invoker.support_lazy_serialization) { + args += [ "--support_lazy_serialization" ] + } + + if (enable_kythe_annotations) { + args += [ "--enable_kythe_annotations" ] + } + + if (!defined(invoker.scramble_message_ids) || + invoker.scramble_message_ids) { + inputs += message_scrambling_inputs + args += message_scrambling_args + } + + if (defined(invoker.extra_cpp_template_paths)) { + foreach(extra_cpp_template, invoker.extra_cpp_template_paths) { + args += [ + "--extra_cpp_template_paths", + rebase_path(extra_cpp_template, root_build_dir), + ] + assert( + get_path_info(extra_cpp_template, "extension") == "tmpl", + "--extra_cpp_template_paths only accepts template files ending in extension .tmpl") + foreach(base_path, output_file_base_paths) { + template_file_name = get_path_info("$extra_cpp_template", "name") + outputs += [ "$root_gen_dir/${base_path}${variant_dash_suffix}-${template_file_name}" ] + } + } + } + } + } + + if (generate_fuzzing && !defined(variant)) { + # This block contains the C++ targets for the MojoLPM fuzzer, we need to + # do this here so that we can use the typemap configuration for the + # empty-variant Mojo target. + + mojolpm_target_name = "${target_name}_mojolpm" + mojolpm_generator_target_name = "${target_name}__generator" + source_set(mojolpm_target_name) { + # There are still a few missing header dependencies between mojo targets + # with typemaps and the dependencies of their typemap headers. It would + # be good to enable include checking for these in the future though. + check_includes = false + testonly = true + if (defined(invoker.sources)) { + sources = process_file_template( + invoker.sources, + [ + "{{source_gen_dir}}/{{source_file_part}}-mojolpm.cc", + "{{source_gen_dir}}/{{source_file_part}}-mojolpm.h", + ]) + deps = [] + } else { + sources = [] + deps = [] + } + + public_deps = [ + ":$generator_shared_target_name", + + # NB: hardcoded dependency on the no-variant variant generator, since + # mojolpm only uses the no-variant type. + ":$mojolpm_generator_target_name", + ":$mojolpm_proto_target_name", + "//mojo/public/tools/fuzzers:mojolpm", + ] + + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append variant_suffix to + # get the cpp dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps += [ "${full_name}_mojolpm" ] + } + + foreach(typemap, active_typemaps) { + _typemap_config = { + } + _typemap_config = typemap.config + + if (defined(_typemap_config.deps)) { + deps += _typemap_config.deps + } + if (defined(_typemap_config.public_deps)) { + public_deps += _typemap_config.public_deps + } + } + foreach(config, cpp_typemap_configs) { + if (defined(config.traits_deps)) { + deps += config.traits_deps + } + if (defined(config.traits_public_deps)) { + public_deps += config.traits_public_deps + } + } + } + } + + # Write the typemapping configuration for this target out to a file to be + # validated by a Python script. This helps catch mistakes that can't + # be caught by logic in GN. + _typemap_config_filename = + "$target_gen_dir/${target_name}${variant_suffix}.typemap_config" + _typemap_stamp_filename = "${_typemap_config_filename}.validated" + _typemap_validator_target_name = "${type_mappings_target_name}__validator" + _rebased_typemap_configs = [] + foreach(config, cpp_typemap_configs) { + _rebased_config = { + } + _rebased_config = config + if (defined(config.traits_headers)) { + _rebased_config.traits_headers = [] + _rebased_config.traits_headers = + rebase_path(config.traits_headers, "//") + } + if (defined(config.traits_private_headers)) { + _rebased_config.traits_private_headers = [] + _rebased_config.traits_private_headers = + rebase_path(config.traits_private_headers, "//") + } + _rebased_typemap_configs += [ _rebased_config ] + } + write_file(_typemap_config_filename, _rebased_typemap_configs, "json") + _mojom_target_name = target_name + action(_typemap_validator_target_name) { + script = "$mojom_generator_root/validate_typemap_config.py" + inputs = [ _typemap_config_filename ] + outputs = [ _typemap_stamp_filename ] + args = [ + get_label_info(_mojom_target_name, "label_no_toolchain"), + rebase_path(_typemap_config_filename), + rebase_path(_typemap_stamp_filename), + ] + } + + action(type_mappings_target_name) { + inputs = _bindings_configuration_files + mojom_generator_sources + + jinja2_sources + [ _typemap_stamp_filename ] + outputs = [ type_mappings_path ] + script = "$mojom_generator_root/generate_type_mappings.py" + deps = [ ":$_typemap_validator_target_name" ] + args = [ + "--output", + rebase_path(type_mappings_path, root_build_dir), + ] + + foreach(d, all_deps) { + name = get_label_info(d, "label_no_toolchain") + toolchain = get_label_info(d, "toolchain") + dependency_output = "${name}${variant_suffix}__type_mappings" + dependency_target = "${dependency_output}(${toolchain})" + deps += [ dependency_target ] + dependency_output_dir = + get_label_info(dependency_output, "target_gen_dir") + dependency_name = get_label_info(dependency_output, "name") + dependency_path = + rebase_path("$dependency_output_dir/${dependency_name}", + root_build_dir) + args += [ + "--dependency", + dependency_path, + ] + } + + if (sources_list != []) { + # TODO(sammc): Pass the typemap description in a file to avoid command + # line length limitations. + typemap_description = [] + foreach(typemap, active_typemaps) { + _typemap_config = { + } + _typemap_config = typemap.config + + typemap_description += [ "--start-typemap" ] + if (defined(_typemap_config.public_headers)) { + foreach(value, _typemap_config.public_headers) { + typemap_description += [ "public_headers=$value" ] + } + } + if (defined(_typemap_config.traits_headers)) { + foreach(value, _typemap_config.traits_headers) { + typemap_description += [ "traits_headers=$value" ] + } + } + foreach(value, _typemap_config.type_mappings) { + typemap_description += [ "type_mappings=$value" ] + } + + # The typemap configuration files are not actually used as inputs here + # but this establishes a necessary build dependency to ensure that + # typemap changes force a rebuild of affected targets. + if (defined(typemap.filename)) { + inputs += [ typemap.filename ] + } + } + args += typemap_description + + # Newer GN-based typemaps are aggregated into a single config. + inputs += [ _typemap_config_filename ] + args += [ + "--cpp-typemap-config", + rebase_path(_typemap_config_filename, root_build_dir), + ] + } + } + + group("${target_name}${variant_suffix}_headers") { + public_deps = [] + if (sources_list != []) { + public_deps += [ + ":$generator_cpp_message_ids_target_name", + ":$generator_shared_target_name", + ":$generator_target_name", + ] + } + foreach(d, all_deps) { + full_name = get_label_info("$d", "label_no_toolchain") + public_deps += [ "${full_name}${variant_suffix}_headers" ] + } + } + + if (!force_source_set && defined(invoker.component_macro_prefix)) { + output_target_type = "component" + } else { + output_target_type = "source_set" + } + + js_data_deps_target_name = target_name + "_js_data_deps" + not_needed([ "js_data_deps_target_name" ]) + + target("jumbo_" + output_target_type, "${target_name}${variant_suffix}") { + if (defined(output_name_override)) { + output_name = output_name_override + } + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink && + defined(invoker.visibility_blink)) { + visibility = invoker.visibility_blink + } else if (defined(invoker.visibility)) { + visibility = invoker.visibility + } + if (defined(invoker.testonly)) { + testonly = invoker.testonly + } + defines = export_defines + if (output_file_base_paths != []) { + sources = [] + foreach(base_path, output_file_base_paths) { + foreach(suffix, generator_cpp_output_suffixes) { + sources += [ "$root_gen_dir/${base_path}$suffix" ] + } + } + } + deps = [ + ":$generator_cpp_message_ids_target_name", + "//mojo/public/cpp/bindings:struct_traits", + "//mojo/public/interfaces/bindings:bindings_headers", + ] + public_deps = [ + ":$shared_cpp_library_target_name", + "//base", + ] + if (require_full_cpp_deps) { + public_deps += [ "//mojo/public/cpp/bindings" ] + } else { + public_deps += [ "//mojo/public/cpp/bindings:bindings_base" ] + } + + if (sources_list != []) { + public_deps += [ ":$generator_target_name" ] + } + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append variant_suffix to + # get the cpp dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps += [ "${full_name}${variant_suffix}" ] + } + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink) { + if (defined(invoker.overridden_deps_blink)) { + foreach(d, invoker.overridden_deps_blink) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append variant_suffix + # to get the cpp dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps -= [ "${full_name}${variant_suffix}" ] + } + public_deps += invoker.component_deps_blink + } + if (defined(invoker.check_includes_blink)) { + check_includes = invoker.check_includes_blink + } + } else { + if (defined(invoker.check_includes_blink)) { + not_needed(invoker, [ "check_includes_blink" ]) + } + if (defined(invoker.overridden_deps)) { + foreach(d, invoker.overridden_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append variant_suffix + # to get the cpp dependency name. + full_name = get_label_info("$d", "label_no_toolchain") + public_deps -= [ "${full_name}${variant_suffix}" ] + } + public_deps += invoker.component_deps + } + } + foreach(typemap, active_typemaps) { + _typemap_config = { + } + _typemap_config = typemap.config + if (defined(_typemap_config.sources)) { + sources += _typemap_config.sources + } + if (defined(_typemap_config.public_deps)) { + public_deps += _typemap_config.public_deps + } + if (defined(_typemap_config.deps)) { + deps += _typemap_config.deps + } + } + foreach(config, cpp_typemap_configs) { + if (defined(config.traits_sources)) { + sources += config.traits_sources + } + if (defined(config.traits_deps)) { + deps += config.traits_deps + } + if (defined(config.traits_public_deps)) { + public_deps += config.traits_public_deps + } + } + if (defined(invoker.export_header)) { + sources += [ "//" + invoker.export_header ] + } + if (defined(bindings_configuration.for_blink) && + bindings_configuration.for_blink) { + public_deps += [ "//mojo/public/cpp/bindings:wtf_support" ] + } + + if (generate_fuzzing) { + # Generate JS bindings by default if IPC fuzzer is enabled. + public_deps += [ ":$js_data_deps_target_name" ] + } + } + + if (generate_java && is_android) { + import("//build/config/android/rules.gni") + + java_generator_target_name = target_name + "_java__generator" + if (sources_list != []) { + action(java_generator_target_name) { + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + ":$type_mappings_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + outputs = [] + args = common_generator_args + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path("$source", root_build_dir) ] + } + foreach(base_path, output_file_base_paths) { + outputs += [ "$root_gen_dir/$base_path.srcjar" ] + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "-g", + "java", + ] + + if (!defined(invoker.scramble_message_ids) || + invoker.scramble_message_ids) { + inputs += message_scrambling_inputs + args += message_scrambling_args + } + } + } else { + group(java_generator_target_name) { + } + } + + java_srcjar_target_name = target_name + "_java_sources" + action(java_srcjar_target_name) { + script = "//build/android/gyp/zip.py" + inputs = [] + if (output_file_base_paths != []) { + foreach(base_path, output_file_base_paths) { + inputs += [ "$root_gen_dir/${base_path}.srcjar" ] + } + } + output = "$target_gen_dir/$target_name.srcjar" + outputs = [ output ] + rebase_inputs = rebase_path(inputs, root_build_dir) + rebase_output = rebase_path(output, root_build_dir) + args = [ + "--input-zips=$rebase_inputs", + "--output=$rebase_output", + ] + deps = [] + if (sources_list != []) { + deps = [ ":$java_generator_target_name" ] + } + } + + java_target_name = target_name + "_java" + android_library(java_target_name) { + forward_variables_from(invoker, [ "enable_bytecode_checks" ]) + deps = [ + "//base:base_java", + "//mojo/public/java:bindings_java", + "//mojo/public/java:system_java", + ] + + # Disable warnings/checks on these generated files. + chromium_code = false + + foreach(d, all_deps) { + # Resolve the name, so that a target //mojo/something becomes + # //mojo/something:something and we can append "_java" to get the java + # dependency name. + full_name = get_label_info(d, "label_no_toolchain") + deps += [ "${full_name}_java" ] + } + + srcjar_deps = [ ":$java_srcjar_target_name" ] + } + } + } + + use_typescript_for_target = + enable_typescript_bindings && defined(invoker.use_typescript_sources) && + invoker.use_typescript_sources + + if (!use_typescript_for_target && defined(invoker.use_typescript_sources)) { + not_needed(invoker, [ "use_typescript_sources" ]) + } + + if ((generate_fuzzing || !defined(invoker.cpp_only) || !invoker.cpp_only) && + !use_typescript_for_target) { + if (sources_list != []) { + generator_js_target_name = "${target_name}_js__generator" + action(generator_js_target_name) { + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + if (defined(invoker.parser_deps)) { + deps += invoker.parser_deps + } + outputs = [] + args = common_generator_args + filelist = [] + foreach(source, sources_list) { + filelist += [ rebase_path("$source", root_build_dir) ] + } + foreach(base_path, output_file_base_paths) { + outputs += [ + "$root_gen_dir/$base_path.js", + "$root_gen_dir/$base_path.externs.js", + "$root_gen_dir/$base_path-lite.js", + "$root_gen_dir/$base_path.html", + "$root_gen_dir/$base_path-lite-for-compile.js", + ] + } + + response_file_contents = filelist + + args += [ + "--filelist={{response_file_name}}", + "-g", + "javascript", + "--js_bindings_mode=new", + ] + + if (defined(invoker.js_generate_struct_deserializers) && + invoker.js_generate_struct_deserializers) { + args += [ "--js_generate_struct_deserializers" ] + } + + if (!defined(invoker.scramble_message_ids) || + invoker.scramble_message_ids) { + inputs += message_scrambling_inputs + args += message_scrambling_args + } + + if (generate_fuzzing) { + args += [ "--generate_fuzzing" ] + } + } + } + + js_target_name = target_name + "_js" + group(js_target_name) { + public_deps = [] + if (sources_list != []) { + public_deps += [ ":$generator_js_target_name" ] + } + + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + public_deps += [ "${full_name}_js" ] + } + } + + group(js_data_deps_target_name) { + deps = [] + if (sources_list != []) { + data = [] + foreach(base_path, output_file_base_paths) { + data += [ + "$root_gen_dir/${base_path}.js", + "$root_gen_dir/${base_path}-lite.js", + ] + } + deps += [ ":$generator_js_target_name" ] + } + + data_deps = [] + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + data_deps += [ "${full_name}_js_data_deps" ] + } + } + + js_library_target_name = "${target_name}_js_library" + if (sources_list != []) { + js_library(js_library_target_name) { + extra_public_deps = [ ":$generator_js_target_name" ] + sources = [] + foreach(base_path, output_file_base_paths) { + sources += [ "$root_gen_dir/${base_path}-lite.js" ] + } + externs_list = [ + "${externs_path}/mojo_core.js", + "${externs_path}/pending.js", + ] + + deps = [] + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + deps += [ "${full_name}_js_library" ] + } + } + } else { + group(js_library_target_name) { + } + } + + js_library_for_compile_target_name = "${target_name}_js_library_for_compile" + if (sources_list != []) { + js_library(js_library_for_compile_target_name) { + extra_public_deps = [ ":$generator_js_target_name" ] + sources = [] + foreach(base_path, output_file_base_paths) { + sources += [ "$root_gen_dir/${base_path}-lite-for-compile.js" ] + } + externs_list = [ + "${externs_path}/mojo_core.js", + "${externs_path}/pending.js", + ] + deps = [] + if (!defined(invoker.disallow_native_types)) { + deps += [ "//mojo/public/js:bindings_lite_sources" ] + } + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + deps += [ "${full_name}_js_library_for_compile" ] + } + } + } else { + group(js_library_for_compile_target_name) { + } + } + } + if ((generate_fuzzing || !defined(invoker.cpp_only) || !invoker.cpp_only) && + use_typescript_for_target) { + generator_js_target_names = [] + source_filelist = [] + foreach(source, sources_list) { + source_filelist += [ rebase_path("$source", root_build_dir) ] + } + + dependency_types = [ + { + name = "regular" + ts_extension = ".ts" + js_extension = ".js" + }, + { + name = "es_modules" + ts_extension = ".m.ts" + js_extension = ".m.js" + }, + ] + + foreach(dependency_type, dependency_types) { + ts_outputs = [] + js_outputs = [] + + foreach(base_path, output_file_base_paths) { + ts_outputs += + [ "$root_gen_dir/$base_path-lite${dependency_type.ts_extension}" ] + js_outputs += + [ "$root_gen_dir/$base_path-lite${dependency_type.js_extension}" ] + } + + # Generate Typescript bindings. + generator_ts_target_name = + "${target_name}_${dependency_type.name}__ts__generator" + action(generator_ts_target_name) { + script = mojom_generator_script + inputs = mojom_generator_sources + jinja2_sources + sources = sources_list + deps = [ + ":$parser_target_name", + "//mojo/public/tools/bindings:precompile_templates", + ] + + outputs = ts_outputs + args = common_generator_args + response_file_contents = source_filelist + + args += [ + "--filelist={{response_file_name}}", + "-g", + "typescript", + ] + + if (dependency_type.name == "es_modules") { + args += [ "--ts_use_es_modules" ] + } + + # TODO(crbug.com/1007587): Support scramble_message_ids. + # TODO(crbug.com/1007591): Support generate_fuzzing. + } + + # Create tsconfig.json for the generated Typescript. + tsconfig_filename = + "$target_gen_dir/$target_name-${dependency_type.name}-tsconfig.json" + tsconfig = { + } + tsconfig.compilerOptions = { + composite = true + target = "es6" + module = "es6" + lib = [ + "es6", + "esnext.bigint", + ] + strict = true + } + tsconfig.files = [] + foreach(base_path, output_file_base_paths) { + tsconfig.files += [ rebase_path( + "$root_gen_dir/$base_path-lite${dependency_type.ts_extension}", + target_gen_dir, + root_gen_dir) ] + } + tsconfig.references = [] + + # Get tsconfigs for deps. + foreach(d, all_deps) { + dep_target_gen_dir = rebase_path(get_label_info(d, "target_gen_dir")) + dep_name = get_label_info(d, "name") + reference = { + } + reference.path = "$dep_target_gen_dir/$dep_name-${dependency_type.name}-tsconfig.json" + tsconfig.references += [ reference ] + } + write_file(tsconfig_filename, tsconfig, "json") + + # Compile previously generated Typescript to Javascript. + generator_js_target_name = + "${target_name}_${dependency_type.name}__js__generator" + generator_js_target_names += [ generator_js_target_name ] + + action(generator_js_target_name) { + script = "$mojom_generator_root/compile_typescript.py" + sources = ts_outputs + outputs = js_outputs + public_deps = [ ":$generator_ts_target_name" ] + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + public_deps += + [ "${full_name}_${dependency_type.name}__js__generator" ] + } + + absolute_tsconfig_path = + rebase_path(tsconfig_filename, "", target_gen_dir) + args = [ "--tsconfig_path=$absolute_tsconfig_path" ] + } + } + + js_target_name = target_name + "_js" + group(js_target_name) { + public_deps = [] + if (sources_list != []) { + foreach(generator_js_target_name, generator_js_target_names) { + public_deps += [ ":$generator_js_target_name" ] + } + } + + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + public_deps += [ "${full_name}_js" ] + } + } + + group(js_data_deps_target_name) { + data = js_outputs + deps = [] + foreach(generator_js_target_name, generator_js_target_names) { + deps += [ ":$generator_js_target_name" ] + } + data_deps = [] + foreach(d, all_deps) { + full_name = get_label_info(d, "label_no_toolchain") + data_deps += [ "${full_name}_js_data_deps" ] + } + } + } +} + +# A helper for the mojom() template above when component libraries are desired +# for generated C++ bindings units. Supports all the same arguments as mojom() +# except for the optional |component_output_prefix| and |component_macro_prefix| +# arguments. These are instead shortened to |output_prefix| and |macro_prefix| +# and are *required*. +template("mojom_component") { + assert(defined(invoker.output_prefix) && defined(invoker.macro_prefix)) + + mojom(target_name) { + forward_variables_from(invoker, + "*", + [ + "output_prefix", + "macro_prefix", + ]) + component_output_prefix = invoker.output_prefix + component_macro_prefix = invoker.macro_prefix + } +} diff --git a/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py b/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py new file mode 100755 index 00000000..da9efc71 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py @@ -0,0 +1,390 @@ +#!/usr/bin/env python +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +"""The frontend for the Mojo bindings system.""" + +from __future__ import print_function + +import argparse + +import hashlib +import importlib +import json +import os +import pprint +import re +import struct +import sys + +# Disable lint check for finding modules: +# pylint: disable=F0401 + +def _GetDirAbove(dirname): + """Returns the directory "above" this file containing |dirname| (which must + also be "above" this file).""" + path = os.path.abspath(__file__) + while True: + path, tail = os.path.split(path) + assert tail + if tail == dirname: + return path + + +sys.path.insert( + 0, + os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "mojom")) + +from mojom.error import Error +import mojom.fileutil as fileutil +from mojom.generate.module import Module +from mojom.generate import template_expander +from mojom.generate import translate +from mojom.generate.generator import WriteFile + +sys.path.append( + os.path.join(_GetDirAbove("mojo"), "tools", "diagnosis")) +import crbug_1001171 + + +_BUILTIN_GENERATORS = { + "c++": "mojom_cpp_generator", + "javascript": "mojom_js_generator", + "java": "mojom_java_generator", + "mojolpm": "mojom_mojolpm_generator", + "typescript": "mojom_ts_generator", +} + + +def LoadGenerators(generators_string): + if not generators_string: + return [] # No generators. + + generators = {} + for generator_name in [s.strip() for s in generators_string.split(",")]: + language = generator_name.lower() + if language not in _BUILTIN_GENERATORS: + print("Unknown generator name %s" % generator_name) + sys.exit(1) + generator_module = importlib.import_module( + "generators.%s" % _BUILTIN_GENERATORS[language]) + generators[language] = generator_module + return generators + + +def MakeImportStackMessage(imported_filename_stack): + """Make a (human-readable) message listing a chain of imports. (Returned + string begins with a newline (if nonempty) and does not end with one.)""" + return ''.join( + reversed(["\n %s was imported by %s" % (a, b) for (a, b) in \ + zip(imported_filename_stack[1:], imported_filename_stack)])) + + +class RelativePath(object): + """Represents a path relative to the source tree or generated output dir.""" + + def __init__(self, path, source_root, output_dir): + self.path = path + if path.startswith(source_root): + self.root = source_root + elif path.startswith(output_dir): + self.root = output_dir + else: + raise Exception("Invalid input path %s" % path) + + def relative_path(self): + return os.path.relpath( + os.path.abspath(self.path), os.path.abspath(self.root)) + + +def _GetModulePath(path, output_dir): + return os.path.join(output_dir, path.relative_path() + '-module') + + +def ScrambleMethodOrdinals(interfaces, salt): + already_generated = set() + for interface in interfaces: + i = 0 + already_generated.clear() + for method in interface.methods: + if method.explicit_ordinal is not None: + continue + while True: + i = i + 1 + if i == 1000000: + raise Exception("Could not generate %d method ordinals for %s" % + (len(interface.methods), interface.mojom_name)) + # Generate a scrambled method.ordinal value. The algorithm doesn't have + # to be very strong, cryptographically. It just needs to be non-trivial + # to guess the results without the secret salt, in order to make it + # harder for a compromised process to send fake Mojo messages. + sha256 = hashlib.sha256(salt) + sha256.update(interface.mojom_name.encode('utf-8')) + sha256.update(str(i).encode('utf-8')) + # Take the first 4 bytes as a little-endian uint32. + ordinal = struct.unpack('= 2: + args.import_directories[idx] = RelativePath(tokens[0], tokens[1], + args.output_dir) + else: + args.import_directories[idx] = RelativePath(tokens[0], args.depth, + args.output_dir) + generator_modules = LoadGenerators(args.generators_string) + + fileutil.EnsureDirectoryExists(args.output_dir) + + processor = MojomProcessor(lambda filename: filename in args.filename) + processor.LoadTypemaps(set(args.typemaps)) + + if args.filelist: + with open(args.filelist) as f: + args.filename.extend(f.read().split()) + + for filename in args.filename: + processor._GenerateModule( + args, remaining_args, generator_modules, + RelativePath(filename, args.depth, args.output_dir), []) + + return 0 + + +def _Precompile(args, _): + generator_modules = LoadGenerators(",".join(_BUILTIN_GENERATORS.keys())) + + template_expander.PrecompileTemplates(generator_modules, args.output_dir) + return 0 + + +def main(): + parser = argparse.ArgumentParser( + description="Generate bindings from mojom files.") + parser.add_argument("--use_bundled_pylibs", action="store_true", + help="use Python modules bundled in the SDK") + parser.add_argument( + "-o", + "--output_dir", + dest="output_dir", + default=".", + help="output directory for generated files") + + subparsers = parser.add_subparsers() + + generate_parser = subparsers.add_parser( + "generate", description="Generate bindings from mojom files.") + generate_parser.add_argument("filename", nargs="*", + help="mojom input file") + generate_parser.add_argument("--filelist", help="mojom input file list") + generate_parser.add_argument("-d", "--depth", dest="depth", default=".", + help="depth from source root") + generate_parser.add_argument("-g", + "--generators", + dest="generators_string", + metavar="GENERATORS", + default="c++,javascript,java,mojolpm", + help="comma-separated list of generators") + generate_parser.add_argument( + "--gen_dir", dest="gen_directories", action="append", metavar="directory", + default=[], help="add a directory to be searched for the syntax trees.") + generate_parser.add_argument( + "-I", dest="import_directories", action="append", metavar="directory", + default=[], + help="add a directory to be searched for import files. The depth from " + "source root can be specified for each import by appending it after " + "a colon") + generate_parser.add_argument("--typemap", action="append", metavar="TYPEMAP", + default=[], dest="typemaps", + help="apply TYPEMAP to generated output") + generate_parser.add_argument("--variant", dest="variant", default=None, + help="output a named variant of the bindings") + generate_parser.add_argument( + "--bytecode_path", required=True, help=( + "the path from which to load template bytecode; to generate template " + "bytecode, run %s precompile BYTECODE_PATH" % os.path.basename( + sys.argv[0]))) + generate_parser.add_argument("--for_blink", action="store_true", + help="Use WTF types as generated types for mojo " + "string/array/map.") + generate_parser.add_argument( + "--js_bindings_mode", choices=["new", "old"], default="old", + help="This option only affects the JavaScript bindings. The value could " + "be \"new\" to generate new-style lite JS bindings in addition to the " + "old, or \"old\" to only generate old bindings.") + generate_parser.add_argument( + "--js_generate_struct_deserializers", action="store_true", + help="Generate javascript deserialize methods for structs in " + "mojom-lite.js file") + generate_parser.add_argument( + "--export_attribute", default="", + help="Optional attribute to specify on class declaration to export it " + "for the component build.") + generate_parser.add_argument( + "--export_header", default="", + help="Optional header to include in the generated headers to support the " + "component build.") + generate_parser.add_argument( + "--generate_non_variant_code", action="store_true", + help="Generate code that is shared by different variants.") + generate_parser.add_argument( + "--scrambled_message_id_salt_path", + dest="scrambled_message_id_salt_paths", + help="If non-empty, the path to a file whose contents should be used as" + "a salt for generating scrambled message IDs. If this switch is specified" + "more than once, the contents of all salt files are concatenated to form" + "the salt value.", default=[], action="append") + generate_parser.add_argument( + "--support_lazy_serialization", + help="If set, generated bindings will serialize lazily when possible.", + action="store_true") + generate_parser.add_argument( + "--extra_cpp_template_paths", + dest="extra_cpp_template_paths", + action="append", + metavar="path_to_template", + default=[], + help="Provide a path to a new template (.tmpl) that is used to generate " + "additional C++ source/header files ") + generate_parser.add_argument( + "--generate_extra_cpp_only", + help="If set and extra_cpp_template_paths provided, will only generate" + "extra_cpp_template related C++ bindings", + action="store_true") + generate_parser.add_argument( + "--disallow_native_types", + help="Disallows the [Native] attribute to be specified on structs or " + "enums within the mojom file.", action="store_true") + generate_parser.add_argument( + "--disallow_interfaces", + help="Disallows interface definitions within the mojom file. It is an " + "error to specify this flag when processing a mojom file which defines " + "any interface.", action="store_true") + generate_parser.add_argument( + "--generate_message_ids", + help="Generates only the message IDs header for C++ bindings. Note that " + "this flag only matters if --generate_non_variant_code is also " + "specified.", action="store_true") + generate_parser.add_argument( + "--generate_fuzzing", + action="store_true", + help="Generates additional bindings for fuzzing in JS.") + generate_parser.add_argument( + "--enable_kythe_annotations", + action="store_true", + help="Adds annotations for kythe metadata generation.") + + generate_parser.set_defaults(func=_Generate) + + precompile_parser = subparsers.add_parser("precompile", + description="Precompile templates for the mojom bindings generator.") + precompile_parser.set_defaults(func=_Precompile) + + args, remaining_args = parser.parse_known_args() + return args.func(args, remaining_args) + + +if __name__ == "__main__": + with crbug_1001171.DumpStateOnLookupError(): + sys.exit(main()) diff --git a/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py b/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py new file mode 100644 index 00000000..bddbe3f4 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py @@ -0,0 +1,62 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import unittest + +from mojom_bindings_generator import MakeImportStackMessage +from mojom_bindings_generator import ScrambleMethodOrdinals + + +class FakeIface(object): + def __init__(self): + self.mojom_name = None + self.methods = None + + +class FakeMethod(object): + def __init__(self, explicit_ordinal=None): + self.explicit_ordinal = explicit_ordinal + self.ordinal = explicit_ordinal + self.ordinal_comment = None + + +class MojoBindingsGeneratorTest(unittest.TestCase): + """Tests mojo_bindings_generator.""" + + def testMakeImportStackMessage(self): + """Tests MakeImportStackMessage().""" + self.assertEqual(MakeImportStackMessage(["x"]), "") + self.assertEqual(MakeImportStackMessage(["x", "y"]), + "\n y was imported by x") + self.assertEqual(MakeImportStackMessage(["x", "y", "z"]), + "\n z was imported by y\n y was imported by x") + + def testScrambleMethodOrdinals(self): + """Tests ScrambleMethodOrdinals().""" + interface = FakeIface() + interface.mojom_name = 'RendererConfiguration' + interface.methods = [ + FakeMethod(), + FakeMethod(), + FakeMethod(), + FakeMethod(explicit_ordinal=42) + ] + ScrambleMethodOrdinals([interface], "foo".encode('utf-8')) + # These next three values are hard-coded. If the generation algorithm + # changes from being based on sha256(seed + interface.name + str(i)) then + # these numbers will obviously need to change too. + # + # Note that hashlib.sha256('fooRendererConfiguration1').digest()[:4] is + # '\xa5\xbc\xf9\xca' and that hex(1257880741) = '0x4af9bca5'. The + # difference in 0x4a vs 0xca is because we only take 31 bits. + self.assertEqual(interface.methods[0].ordinal, 1257880741) + self.assertEqual(interface.methods[1].ordinal, 631133653) + self.assertEqual(interface.methods[2].ordinal, 549336076) + + # Explicit method ordinals should not be scrambled. + self.assertEqual(interface.methods[3].ordinal, 42) + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py b/utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py new file mode 100755 index 00000000..15f0e3ba --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Downgrades *.mojom files to the old mojo types for remotes and receivers.""" + +import argparse +import fnmatch +import os +import re +import shutil +import sys +import tempfile + +# List of patterns and replacements to match and use against the contents of a +# mojo file. Each replacement string will be used with Python string's format() +# function, so the '{}' substring is used to mark where the mojo type should go. +_MOJO_REPLACEMENTS = { + r'pending_remote': r'{}', + r'pending_receiver': r'{}&', + r'pending_associated_remote': r'associated {}', + r'pending_associated_receiver': r'associated {}&', +} + +# Pre-compiled regular expression that matches against any of the replacements. +_REGEXP_PATTERN = re.compile( + r'|'.join( + ['{}\s*<\s*(.*?)\s*>'.format(k) for k in _MOJO_REPLACEMENTS.keys()]), + flags=re.DOTALL) + + +def ReplaceFunction(match_object): + """Returns the right replacement for the string matched against the regexp.""" + for index, (match, repl) in enumerate(_MOJO_REPLACEMENTS.items(), 1): + if match_object.group(0).startswith(match): + return repl.format(match_object.group(index)) + + +def DowngradeFile(path, output_dir=None): + """Downgrades the mojom file specified by |path| to the old mojo types. + + Optionally pass |output_dir| to place the result under a separate output + directory, preserving the relative path to the file included in |path|. + """ + # Use a temporary file to dump the new contents after replacing the patterns. + with open(path) as src_mojo_file: + with tempfile.NamedTemporaryFile(mode='w', delete=False) as tmp_mojo_file: + tmp_contents = _REGEXP_PATTERN.sub(ReplaceFunction, src_mojo_file.read()) + tmp_mojo_file.write(tmp_contents) + + # Files should be placed in the desired output directory + if output_dir: + output_filepath = os.path.join(output_dir, os.path.basename(path)) + if not os.path.exists(output_dir): + os.makedirs(output_dir) + else: + output_filepath = path + + # Write the new contents preserving the original file's attributes. + shutil.copystat(path, tmp_mojo_file.name) + shutil.move(tmp_mojo_file.name, output_filepath) + + # Make sure to "touch" the new file so that access, modify and change times + # are always newer than the source file's, otherwise Modify time will be kept + # as per the call to shutil.copystat(), causing unnecessary generations of the + # output file in subsequent builds due to ninja considering it dirty. + os.utime(output_filepath, None) + + +def DowngradeDirectory(path, output_dir=None): + """Downgrades mojom files inside directory |path| to the old mojo types. + + Optionally pass |output_dir| to place the result under a separate output + directory, preserving the relative path to the file included in |path|. + """ + # We don't have recursive glob.glob() nor pathlib.Path.rglob() in Python 2.7 + mojom_filepaths = [] + for dir_path, _, filenames in os.walk(path): + for filename in fnmatch.filter(filenames, "*mojom"): + mojom_filepaths.append(os.path.join(dir_path, filename)) + + for path in mojom_filepaths: + absolute_dirpath = os.path.dirname(os.path.abspath(path)) + if output_dir: + dest_dirpath = output_dir + absolute_dirpath + else: + dest_dirpath = absolute_dirpath + DowngradeFile(path, dest_dirpath) + + +def DowngradePath(src_path, output_dir=None): + """Downgrades the mojom files pointed by |src_path| to the old mojo types. + + Optionally pass |output_dir| to place the result under a separate output + directory, preserving the relative path to the file included in |path|. + """ + if os.path.isdir(src_path): + DowngradeDirectory(src_path, output_dir) + elif os.path.isfile(src_path): + DowngradeFile(src_path, output_dir) + else: + print(">>> {} not pointing to a valid file or directory".format(src_path)) + sys.exit(1) + + +def main(): + parser = argparse.ArgumentParser( + description="Downgrade *.mojom files to use the old mojo types.") + parser.add_argument( + "srcpath", help="path to the file or directory to apply the conversion") + parser.add_argument( + "--outdir", help="the directory to place the converted file(s) under") + args = parser.parse_args() + + DowngradePath(args.srcpath, args.outdir) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py b/utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py new file mode 100755 index 00000000..f1783d59 --- /dev/null +++ b/utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import argparse +import json +import os +import re +import sys + + +def CheckCppTypemapConfigs(target_name, config_filename, out_filename): + _SUPPORTED_CONFIG_KEYS = set([ + 'types', 'traits_headers', 'traits_private_headers', 'traits_sources', + 'traits_deps', 'traits_public_deps' + ]) + _SUPPORTED_TYPE_KEYS = set([ + 'mojom', 'cpp', 'copyable_pass_by_value', 'force_serialize', 'hashable', + 'move_only', 'nullable_is_same_type' + ]) + with open(config_filename, 'r') as f: + for config in json.load(f): + for key in config.keys(): + if key not in _SUPPORTED_CONFIG_KEYS: + raise ValueError('Invalid typemap property "%s" when processing %s' % + (key, target_name)) + + types = config.get('types') + if not types: + raise ValueError('Typemap for %s must specify at least one type to map' + % target_name) + + for entry in types: + for key in entry.keys(): + if key not in _SUPPORTED_TYPE_KEYS: + raise IOError( + 'Invalid type property "%s" in typemap for "%s" on target %s' % + (key, entry.get('mojom', '(unknown)'), target_name)) + + with open(out_filename, 'w') as f: + f.truncate(0) + + +def main(): + parser = argparse.ArgumentParser() + _, args = parser.parse_known_args() + if len(args) != 3: + print('Usage: validate_typemap_config.py target_name config_filename ' + 'stamp_filename') + sys.exit(1) + + CheckCppTypemapConfigs(args[0], args[1], args[2]) + + +if __name__ == '__main__': + main() diff --git a/utils/ipc/mojo/public/tools/mojom/README.md b/utils/ipc/mojo/public/tools/mojom/README.md new file mode 100644 index 00000000..6a4ff78a --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/README.md @@ -0,0 +1,14 @@ +# The Mojom Parser + +The Mojom format is an interface definition language (IDL) for describing +interprocess communication (IPC) messages and data types for use with the +low-level cross-platform +[Mojo IPC library](https://chromium.googlesource.com/chromium/src/+/master/mojo/public/c/system/README.md). + +This directory consists of a `mojom` Python module, its tests, and supporting +command-line tools. The Python module implements the parser used by the +command-line tools and exposes an API to help external bindings generators emit +useful code from the parser's outputs. + +TODO(https://crbug.com/1060464): Fill out this documentation once the library +and tools have stabilized. diff --git a/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py b/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py new file mode 100755 index 00000000..7e746112 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Verifies backward-compatibility of mojom type changes. + +Given a set of pre- and post-diff mojom file contents, and a root directory +for a project, this tool verifies that any changes to [Stable] mojom types are +backward-compatible with the previous version. + +This can be used e.g. by a presubmit check to prevent developers from making +breaking changes to stable mojoms.""" + +import argparse +import errno +import io +import json +import os +import os.path +import shutil +import six +import sys +import tempfile + +from mojom.generate import module +from mojom.generate import translate +from mojom.parse import parser + + +class ParseError(Exception): + pass + + +def _ValidateDelta(root, delta): + """Parses all modified mojoms (including all transitive mojom dependencies, + even if unmodified) to perform backward-compatibility checks on any types + marked with the [Stable] attribute. + + Note that unlike the normal build-time parser in mojom_parser.py, this does + not produce or rely on cached module translations, but instead parses the full + transitive closure of a mojom's input dependencies all at once. + """ + + # First build a map of all files covered by the delta + affected_files = set() + old_files = {} + new_files = {} + for change in delta: + # TODO(crbug.com/953884): Use pathlib once we're migrated fully to Python 3. + filename = change['filename'].replace('\\', '/') + affected_files.add(filename) + if change['old']: + old_files[filename] = change['old'] + if change['new']: + new_files[filename] = change['new'] + + # Parse and translate all mojoms relevant to the delta, including transitive + # imports that weren't modified. + unmodified_modules = {} + + def parseMojom(mojom, file_overrides, override_modules): + if mojom in unmodified_modules or mojom in override_modules: + return + + contents = file_overrides.get(mojom) + if contents: + modules = override_modules + else: + modules = unmodified_modules + with io.open(os.path.join(root, mojom), encoding='utf-8') as f: + contents = f.read() + + try: + ast = parser.Parse(contents, mojom) + except Exception as e: + six.reraise( + ParseError, + 'encountered exception {0} while parsing {1}'.format(e, mojom), + sys.exc_info()[2]) + for imp in ast.import_list: + parseMojom(imp.import_filename, file_overrides, override_modules) + + # Now that the transitive set of dependencies has been imported and parsed + # above, translate each mojom AST into a Module so that all types are fully + # defined and can be inspected. + all_modules = {} + all_modules.update(unmodified_modules) + all_modules.update(override_modules) + modules[mojom] = translate.OrderedModule(ast, mojom, all_modules) + + old_modules = {} + for mojom in old_files.keys(): + parseMojom(mojom, old_files, old_modules) + new_modules = {} + for mojom in new_files.keys(): + parseMojom(mojom, new_files, new_modules) + + # At this point we have a complete set of translated Modules from both the + # pre- and post-diff mojom contents. Now we can analyze backward-compatibility + # of the deltas. + # + # Note that for backward-compatibility checks we only care about types which + # were marked [Stable] before the diff. Types newly marked as [Stable] are not + # checked. + def collectTypes(modules): + types = {} + for m in modules.values(): + for kinds in (m.enums, m.structs, m.unions, m.interfaces): + for kind in kinds: + types[kind.qualified_name] = kind + return types + + old_types = collectTypes(old_modules) + new_types = collectTypes(new_modules) + + # Collect any renamed types so they can be compared accordingly. + renamed_types = {} + for name, kind in new_types.items(): + old_name = kind.attributes and kind.attributes.get('RenamedFrom') + if old_name: + renamed_types[old_name] = name + + for qualified_name, kind in old_types.items(): + if not kind.stable: + continue + + new_name = renamed_types.get(qualified_name, qualified_name) + if new_name not in new_types: + raise Exception( + 'Stable type %s appears to be deleted by this change. If it was ' + 'renamed, please add a [RenamedFrom] attribute to the new type. This ' + 'can be deleted by a subsequent change.' % qualified_name) + + if not new_types[new_name].IsBackwardCompatible(kind): + raise Exception('Stable type %s appears to have changed in a way which ' + 'breaks backward-compatibility. Please fix!\n\nIf you ' + 'believe this assessment to be incorrect, please file a ' + 'Chromium bug against the "Internals>Mojo>Bindings" ' + 'component.' % qualified_name) + + +def Run(command_line, delta=None): + """Runs the tool with the given command_line. Normally this will read the + change description from stdin as a JSON-encoded list, but tests may pass a + delta directly for convenience.""" + arg_parser = argparse.ArgumentParser( + description='Verifies backward-compatibility of mojom type changes.', + epilog=""" +This tool reads a change description from stdin and verifies that all modified +[Stable] mojom types will retain backward-compatibility. The change description +must be a JSON-encoded list of objects, each with a "filename" key (path to a +changed mojom file, relative to ROOT); an "old" key whose value is a string of +the full file contents before the change, or null if the file is being added; +and a "new" key whose value is a string of the full file contents after the +change, or null if the file is being deleted.""") + arg_parser.add_argument( + '--src-root', + required=True, + action='store', + metavar='ROOT', + help='The root of the source tree in which the checked mojoms live.') + + args, _ = arg_parser.parse_known_args(command_line) + if not delta: + delta = json.load(sys.stdin) + _ValidateDelta(args.src_root, delta) + + +if __name__ == '__main__': + Run(sys.argv[1:]) diff --git a/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py b/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py new file mode 100755 index 00000000..9f51ea77 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import json +import os +import os.path +import shutil +import tempfile +import unittest + +import check_stable_mojom_compatibility + +from mojom.generate import module + + +class Change(object): + """Helper to clearly define a mojom file delta to be analyzed.""" + + def __init__(self, filename, old=None, new=None): + """If old is None, this is a file addition. If new is None, this is a file + deletion. Otherwise it's a file change.""" + self.filename = filename + self.old = old + self.new = new + + +class UnchangedFile(Change): + def __init__(self, filename, contents): + super(UnchangedFile, self).__init__(filename, old=contents, new=contents) + + +class CheckStableMojomCompatibilityTest(unittest.TestCase): + """Tests covering the behavior of the compatibility checking tool. Note that + details of different compatibility checks and relevant failure modes are NOT + covered by these tests. Those are instead covered by unittests in + version_compatibility_unittest.py. Additionally, the tests which ensure a + given set of [Stable] mojom definitions are indeed plausibly stable (i.e. they + have no unstable dependencies) are covered by stable_attribute_unittest.py. + + These tests cover higher-level concerns of the compatibility checking tool, + like file or symbol, renames, changes spread over multiple files, etc.""" + + def verifyBackwardCompatibility(self, changes): + """Helper for implementing assertBackwardCompatible and + assertNotBackwardCompatible""" + + temp_dir = tempfile.mkdtemp() + for change in changes: + if change.old: + # Populate the old file on disk in our temporary fake source root + file_path = os.path.join(temp_dir, change.filename) + dir_path = os.path.dirname(file_path) + if not os.path.exists(dir_path): + os.makedirs(dir_path) + with open(file_path, 'w') as f: + f.write(change.old) + + delta = [] + for change in changes: + if change.old != change.new: + delta.append({ + 'filename': change.filename, + 'old': change.old, + 'new': change.new + }) + + try: + check_stable_mojom_compatibility.Run(['--src-root', temp_dir], + delta=delta) + finally: + shutil.rmtree(temp_dir) + + def assertBackwardCompatible(self, changes): + self.verifyBackwardCompatibility(changes) + + def assertNotBackwardCompatible(self, changes): + try: + self.verifyBackwardCompatibility(changes) + except Exception: + return + + raise Exception('Change unexpectedly passed a backward-compatibility check') + + def testBasicCompatibility(self): + """Minimal smoke test to verify acceptance of a simple valid change.""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old='[Stable] struct S {};', + new='[Stable] struct S { [MinVersion=1] int32 x; };') + ]) + + def testBasicIncompatibility(self): + """Minimal smoke test to verify rejection of a simple invalid change.""" + self.assertNotBackwardCompatible([ + Change('foo/foo.mojom', + old='[Stable] struct S {};', + new='[Stable] struct S { int32 x; };') + ]) + + def testIgnoreIfNotStable(self): + """We don't care about types not marked [Stable]""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old='struct S {};', + new='struct S { int32 x; };') + ]) + + def testRename(self): + """We can do checks for renamed types.""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old='[Stable] struct S {};', + new='[Stable, RenamedFrom="S"] struct T {};') + ]) + self.assertNotBackwardCompatible([ + Change('foo/foo.mojom', + old='[Stable] struct S {};', + new='[Stable, RenamedFrom="S"] struct T { int32 x; };') + ]) + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old='[Stable] struct S {};', + new="""\ + [Stable, RenamedFrom="S"] + struct T { [MinVersion=1] int32 x; }; + """) + ]) + + def testNewlyStable(self): + """We don't care about types newly marked as [Stable].""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old='struct S {};', + new='[Stable] struct S { int32 x; };') + ]) + + def testFileRename(self): + """Make sure we can still do compatibility checks after a file rename.""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', old='[Stable] struct S {};', new=None), + Change('bar/bar.mojom', + old=None, + new='[Stable] struct S { [MinVersion=1] int32 x; };') + ]) + self.assertNotBackwardCompatible([ + Change('foo/foo.mojom', old='[Stable] struct S {};', new=None), + Change('bar/bar.mojom', old=None, new='[Stable] struct S { int32 x; };') + ]) + + def testWithImport(self): + """Ensure that cross-module dependencies do not break the compatibility + checking tool.""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old="""\ + module foo; + [Stable] struct S {}; + """, + new="""\ + module foo; + [Stable] struct S { [MinVersion=2] int32 x; }; + """), + Change('bar/bar.mojom', + old="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; }; + """, + new="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; [MinVersion=1] int32 y; }; + """) + ]) + + def testWithMovedDefinition(self): + """If a definition moves from one file to another, we should still be able + to check compatibility accurately.""" + self.assertBackwardCompatible([ + Change('foo/foo.mojom', + old="""\ + module foo; + [Stable] struct S {}; + """, + new="""\ + module foo; + """), + Change('bar/bar.mojom', + old="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; }; + """, + new="""\ + module bar; + import "foo/foo.mojom"; + [Stable, RenamedFrom="foo.S"] struct S { + [MinVersion=2] int32 x; + }; + [Stable] struct T { S s; [MinVersion=1] int32 y; }; + """) + ]) + + self.assertNotBackwardCompatible([ + Change('foo/foo.mojom', + old="""\ + module foo; + [Stable] struct S {}; + """, + new="""\ + module foo; + """), + Change('bar/bar.mojom', + old="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; }; + """, + new="""\ + module bar; + import "foo/foo.mojom"; + [Stable, RenamedFrom="foo.S"] struct S { int32 x; }; + [Stable] struct T { S s; [MinVersion=1] int32 y; }; + """) + ]) + + def testWithUnmodifiedImport(self): + """Unchanged files in the filesystem are still parsed by the compatibility + checking tool if they're imported by a changed file.""" + self.assertBackwardCompatible([ + UnchangedFile('foo/foo.mojom', 'module foo; [Stable] struct S {};'), + Change('bar/bar.mojom', + old="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; }; + """, + new="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; [MinVersion=1] int32 x; }; + """) + ]) + + self.assertNotBackwardCompatible([ + UnchangedFile('foo/foo.mojom', 'module foo; [Stable] struct S {};'), + Change('bar/bar.mojom', + old="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; }; + """, + new="""\ + module bar; + import "foo/foo.mojom"; + [Stable] struct T { foo.S s; int32 x; }; + """) + ]) diff --git a/utils/ipc/mojo/public/tools/mojom/const_unittest.py b/utils/ipc/mojo/public/tools/mojom/const_unittest.py new file mode 100644 index 00000000..cb42dfac --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/const_unittest.py @@ -0,0 +1,90 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom_parser_test_case import MojomParserTestCase +from mojom.generate import module as mojom + + +class ConstTest(MojomParserTestCase): + """Tests constant parsing behavior.""" + + def testLiteralInt(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'const int32 k = 42;') + self.ParseMojoms([a_mojom]) + a = self.LoadModule(a_mojom) + self.assertEqual(1, len(a.constants)) + self.assertEqual('k', a.constants[0].mojom_name) + self.assertEqual('42', a.constants[0].value) + + def testLiteralFloat(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'const float k = 42.5;') + self.ParseMojoms([a_mojom]) + a = self.LoadModule(a_mojom) + self.assertEqual(1, len(a.constants)) + self.assertEqual('k', a.constants[0].mojom_name) + self.assertEqual('42.5', a.constants[0].value) + + def testLiteralString(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'const string k = "woot";') + self.ParseMojoms([a_mojom]) + a = self.LoadModule(a_mojom) + self.assertEqual(1, len(a.constants)) + self.assertEqual('k', a.constants[0].mojom_name) + self.assertEqual('"woot"', a.constants[0].value) + + def testEnumConstant(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'module a; enum E { kA = 41, kB };') + b_mojom = 'b.mojom' + self.WriteFile( + b_mojom, """\ + import "a.mojom"; + const a.E kE1 = a.E.kB; + + // We also allow value names to be unqualified, implying scope from the + // constant's type. + const a.E kE2 = kB; + """) + self.ParseMojoms([a_mojom, b_mojom]) + a = self.LoadModule(a_mojom) + b = self.LoadModule(b_mojom) + self.assertEqual(1, len(a.enums)) + self.assertEqual('E', a.enums[0].mojom_name) + self.assertEqual(2, len(b.constants)) + self.assertEqual('kE1', b.constants[0].mojom_name) + self.assertEqual(a.enums[0], b.constants[0].kind) + self.assertEqual(a.enums[0].fields[1], b.constants[0].value.field) + self.assertEqual(42, b.constants[0].value.field.numeric_value) + self.assertEqual('kE2', b.constants[1].mojom_name) + self.assertEqual(a.enums[0].fields[1], b.constants[1].value.field) + self.assertEqual(42, b.constants[1].value.field.numeric_value) + + def testConstantReference(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'const int32 kA = 42; const int32 kB = kA;') + self.ParseMojoms([a_mojom]) + a = self.LoadModule(a_mojom) + self.assertEqual(2, len(a.constants)) + self.assertEqual('kA', a.constants[0].mojom_name) + self.assertEqual('42', a.constants[0].value) + self.assertEqual('kB', a.constants[1].mojom_name) + self.assertEqual('42', a.constants[1].value) + + def testImportedConstantReference(self): + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'const int32 kA = 42;') + b_mojom = 'b.mojom' + self.WriteFile(b_mojom, 'import "a.mojom"; const int32 kB = kA;') + self.ParseMojoms([a_mojom, b_mojom]) + a = self.LoadModule(a_mojom) + b = self.LoadModule(b_mojom) + self.assertEqual(1, len(a.constants)) + self.assertEqual(1, len(b.constants)) + self.assertEqual('kA', a.constants[0].mojom_name) + self.assertEqual('42', a.constants[0].value) + self.assertEqual('kB', b.constants[0].mojom_name) + self.assertEqual('42', b.constants[0].value) diff --git a/utils/ipc/mojo/public/tools/mojom/enum_unittest.py b/utils/ipc/mojo/public/tools/mojom/enum_unittest.py new file mode 100644 index 00000000..d9005078 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/enum_unittest.py @@ -0,0 +1,92 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom_parser_test_case import MojomParserTestCase + + +class EnumTest(MojomParserTestCase): + """Tests enum parsing behavior.""" + + def testExplicitValues(self): + """Verifies basic parsing of assigned integral values.""" + types = self.ExtractTypes('enum E { kFoo=0, kBar=2, kBaz };') + self.assertEqual('kFoo', types['E'].fields[0].mojom_name) + self.assertEqual(0, types['E'].fields[0].numeric_value) + self.assertEqual('kBar', types['E'].fields[1].mojom_name) + self.assertEqual(2, types['E'].fields[1].numeric_value) + self.assertEqual('kBaz', types['E'].fields[2].mojom_name) + self.assertEqual(3, types['E'].fields[2].numeric_value) + + def testImplicitValues(self): + """Verifies basic automatic assignment of integral values at parse time.""" + types = self.ExtractTypes('enum E { kFoo, kBar, kBaz };') + self.assertEqual('kFoo', types['E'].fields[0].mojom_name) + self.assertEqual(0, types['E'].fields[0].numeric_value) + self.assertEqual('kBar', types['E'].fields[1].mojom_name) + self.assertEqual(1, types['E'].fields[1].numeric_value) + self.assertEqual('kBaz', types['E'].fields[2].mojom_name) + self.assertEqual(2, types['E'].fields[2].numeric_value) + + def testSameEnumReference(self): + """Verifies that an enum value can be assigned from the value of another + field within the same enum.""" + types = self.ExtractTypes('enum E { kA, kB, kFirst=kA };') + self.assertEqual('kA', types['E'].fields[0].mojom_name) + self.assertEqual(0, types['E'].fields[0].numeric_value) + self.assertEqual('kB', types['E'].fields[1].mojom_name) + self.assertEqual(1, types['E'].fields[1].numeric_value) + self.assertEqual('kFirst', types['E'].fields[2].mojom_name) + self.assertEqual(0, types['E'].fields[2].numeric_value) + + def testSameModuleOtherEnumReference(self): + """Verifies that an enum value can be assigned from the value of a field + in another enum within the same module.""" + types = self.ExtractTypes('enum E { kA, kB }; enum F { kA = E.kB };') + self.assertEqual(1, types['F'].fields[0].numeric_value) + + def testImportedEnumReference(self): + """Verifies that an enum value can be assigned from the value of a field + in another enum within a different module.""" + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'module a; enum E { kFoo=42, kBar };') + b_mojom = 'b.mojom' + self.WriteFile(b_mojom, + 'module b; import "a.mojom"; enum F { kFoo = a.E.kBar };') + self.ParseMojoms([a_mojom, b_mojom]) + b = self.LoadModule(b_mojom) + + self.assertEqual('F', b.enums[0].mojom_name) + self.assertEqual('kFoo', b.enums[0].fields[0].mojom_name) + self.assertEqual(43, b.enums[0].fields[0].numeric_value) + + def testConstantReference(self): + """Verifies that an enum value can be assigned from the value of an + integral constant within the same module.""" + types = self.ExtractTypes('const int32 kFoo = 42; enum E { kA = kFoo };') + self.assertEqual(42, types['E'].fields[0].numeric_value) + + def testInvalidConstantReference(self): + """Verifies that enum values cannot be assigned from the value of + non-integral constants.""" + with self.assertRaisesRegexp(ValueError, 'not an integer'): + self.ExtractTypes('const float kFoo = 1.0; enum E { kA = kFoo };') + with self.assertRaisesRegexp(ValueError, 'not an integer'): + self.ExtractTypes('const double kFoo = 1.0; enum E { kA = kFoo };') + with self.assertRaisesRegexp(ValueError, 'not an integer'): + self.ExtractTypes('const string kFoo = "lol"; enum E { kA = kFoo };') + + def testImportedConstantReference(self): + """Verifies that an enum value can be assigned from the value of an integral + constant within an imported module.""" + a_mojom = 'a.mojom' + self.WriteFile(a_mojom, 'module a; const int32 kFoo = 37;') + b_mojom = 'b.mojom' + self.WriteFile(b_mojom, + 'module b; import "a.mojom"; enum F { kFoo = a.kFoo };') + self.ParseMojoms([a_mojom, b_mojom]) + b = self.LoadModule(b_mojom) + + self.assertEqual('F', b.enums[0].mojom_name) + self.assertEqual('kFoo', b.enums[0].fields[0].mojom_name) + self.assertEqual(37, b.enums[0].fields[0].numeric_value) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn b/utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn new file mode 100644 index 00000000..7416ef19 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn @@ -0,0 +1,43 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +group("mojom") { + data = [ + "__init__.py", + "error.py", + "fileutil.py", + "generate/__init__.py", + "generate/constant_resolver.py", + "generate/generator.py", + "generate/module.py", + "generate/pack.py", + "generate/template_expander.py", + "generate/translate.py", + "parse/__init__.py", + "parse/ast.py", + "parse/conditional_features.py", + "parse/lexer.py", + "parse/parser.py", + + # Third-party module dependencies + "//third_party/jinja2/", + "//third_party/ply/", + ] +} + +group("tests") { + data = [ + "fileutil_unittest.py", + "generate/generator_unittest.py", + "generate/module_unittest.py", + "generate/pack_unittest.py", + "generate/translate_unittest.py", + "parse/ast_unittest.py", + "parse/conditional_features_unittest.py", + "parse/lexer_unittest.py", + "parse/parser_unittest.py", + ] + + public_deps = [ ":mojom" ] +} diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/__init__.py b/utils/ipc/mojo/public/tools/mojom/mojom/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/error.py b/utils/ipc/mojo/public/tools/mojom/mojom/error.py new file mode 100644 index 00000000..8a1e03da --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/error.py @@ -0,0 +1,28 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + + +class Error(Exception): + """Base class for Mojo IDL bindings parser/generator errors.""" + + def __init__(self, filename, message, lineno=None, addenda=None, **kwargs): + """|filename| is the (primary) file which caused the error, |message| is the + error message, |lineno| is the 1-based line number (or |None| if not + applicable/available), and |addenda| is a list of additional lines to append + to the final error message.""" + Exception.__init__(self, **kwargs) + self.filename = filename + self.message = message + self.lineno = lineno + self.addenda = addenda + + def __str__(self): + if self.lineno: + s = "%s:%d: Error: %s" % (self.filename, self.lineno, self.message) + else: + s = "%s: Error: %s" % (self.filename, self.message) + return "\n".join([s] + self.addenda) if self.addenda else s + + def __repr__(self): + return str(self) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py b/utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py new file mode 100644 index 00000000..bf626f54 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py @@ -0,0 +1,45 @@ +# Copyright 2015 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import errno +import imp +import os.path +import sys + + +def _GetDirAbove(dirname): + """Returns the directory "above" this file containing |dirname| (which must + also be "above" this file).""" + path = os.path.abspath(__file__) + while True: + path, tail = os.path.split(path) + if not tail: + return None + if tail == dirname: + return path + + +def EnsureDirectoryExists(path, always_try_to_create=False): + """A wrapper for os.makedirs that does not error if the directory already + exists. A different process could be racing to create this directory.""" + + if not os.path.exists(path) or always_try_to_create: + try: + os.makedirs(path) + except OSError as e: + # There may have been a race to create this directory. + if e.errno != errno.EEXIST: + raise + + +def AddLocalRepoThirdPartyDirToModulePath(): + """Helper function to find the top-level directory of this script's repository + assuming the script falls somewhere within a 'mojo' directory, and insert the + top-level 'third_party' directory early in the module search path. Used to + ensure that third-party dependencies provided within the repository itself + (e.g. Chromium sources include snapshots of jinja2 and ply) are preferred over + locally installed system library packages.""" + toplevel_dir = _GetDirAbove('mojo') + if toplevel_dir: + sys.path.insert(1, os.path.join(toplevel_dir, 'third_party')) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py new file mode 100644 index 00000000..ff5753a2 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py @@ -0,0 +1,40 @@ +# Copyright 2015 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import shutil +import sys +import tempfile +import unittest + +from mojom import fileutil + + +class FileUtilTest(unittest.TestCase): + def testEnsureDirectoryExists(self): + """Test that EnsureDirectoryExists fuctions correctly.""" + + temp_dir = tempfile.mkdtemp() + try: + self.assertTrue(os.path.exists(temp_dir)) + + # Directory does not exist, yet. + full = os.path.join(temp_dir, "foo", "bar") + self.assertFalse(os.path.exists(full)) + + # Create the directory. + fileutil.EnsureDirectoryExists(full) + self.assertTrue(os.path.exists(full)) + + # Trying to create it again does not cause an error. + fileutil.EnsureDirectoryExists(full) + self.assertTrue(os.path.exists(full)) + + # Bypass check for directory existence to tickle error handling that + # occurs in response to a race. + fileutil.EnsureDirectoryExists(full, always_try_to_create=True) + self.assertTrue(os.path.exists(full)) + finally: + shutil.rmtree(temp_dir) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/__init__.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py new file mode 100644 index 00000000..0dfd996e --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py @@ -0,0 +1,93 @@ +# Copyright 2015 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Resolves the values used for constants and enums.""" + +from itertools import ifilter + +from mojom.generate import module as mojom + + +def ResolveConstants(module, expression_to_text): + in_progress = set() + computed = set() + + def GetResolvedValue(named_value): + assert isinstance(named_value, (mojom.EnumValue, mojom.ConstantValue)) + if isinstance(named_value, mojom.EnumValue): + field = next( + ifilter(lambda field: field.name == named_value.name, + named_value.enum.fields), None) + if not field: + raise RuntimeError( + 'Unable to get computed value for field %s of enum %s' % + (named_value.name, named_value.enum.name)) + if field not in computed: + ResolveEnum(named_value.enum) + return field.resolved_value + else: + ResolveConstant(named_value.constant) + named_value.resolved_value = named_value.constant.resolved_value + return named_value.resolved_value + + def ResolveConstant(constant): + if constant in computed: + return + if constant in in_progress: + raise RuntimeError('Circular dependency for constant: %s' % constant.name) + in_progress.add(constant) + if isinstance(constant.value, (mojom.EnumValue, mojom.ConstantValue)): + resolved_value = GetResolvedValue(constant.value) + else: + resolved_value = expression_to_text(constant.value) + constant.resolved_value = resolved_value + in_progress.remove(constant) + computed.add(constant) + + def ResolveEnum(enum): + def ResolveEnumField(enum, field, default_value): + if field in computed: + return + if field in in_progress: + raise RuntimeError('Circular dependency for enum: %s' % enum.name) + in_progress.add(field) + if field.value: + if isinstance(field.value, mojom.EnumValue): + resolved_value = GetResolvedValue(field.value) + elif isinstance(field.value, str): + resolved_value = int(field.value, 0) + else: + raise RuntimeError('Unexpected value: %s' % field.value) + else: + resolved_value = default_value + field.resolved_value = resolved_value + in_progress.remove(field) + computed.add(field) + + current_value = 0 + for field in enum.fields: + ResolveEnumField(enum, field, current_value) + current_value = field.resolved_value + 1 + + for constant in module.constants: + ResolveConstant(constant) + + for enum in module.enums: + ResolveEnum(enum) + + for struct in module.structs: + for constant in struct.constants: + ResolveConstant(constant) + for enum in struct.enums: + ResolveEnum(enum) + for field in struct.fields: + if isinstance(field.default, (mojom.ConstantValue, mojom.EnumValue)): + field.default.resolved_value = GetResolvedValue(field.default) + + for interface in module.interfaces: + for constant in interface.constants: + ResolveConstant(constant) + for enum in interface.enums: + ResolveEnum(enum) + + return module diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py new file mode 100644 index 00000000..de62260a --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py @@ -0,0 +1,325 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Code shared by the various language-specific code generators.""" + +from __future__ import print_function + +from functools import partial +import os.path +import re + +from mojom import fileutil +from mojom.generate import module as mojom +from mojom.generate import pack + + +def ExpectedArraySize(kind): + if mojom.IsArrayKind(kind): + return kind.length + return None + + +def SplitCamelCase(identifier): + """Splits a camel-cased |identifier| and returns a list of lower-cased + strings. + """ + # Add underscores after uppercase letters when appropriate. An uppercase + # letter is considered the end of a word if it is followed by an upper and a + # lower. E.g. URLLoaderFactory -> URL_LoaderFactory + identifier = re.sub('([A-Z][0-9]*)(?=[A-Z][0-9]*[a-z])', r'\1_', identifier) + # Add underscores after lowercase letters when appropriate. A lowercase letter + # is considered the end of a word if it is followed by an upper. + # E.g. URLLoaderFactory -> URLLoader_Factory + identifier = re.sub('([a-z][0-9]*)(?=[A-Z])', r'\1_', identifier) + return [x.lower() for x in identifier.split('_')] + + +def ToCamel(identifier, lower_initial=False, digits_split=False, delimiter='_'): + """Splits |identifier| using |delimiter|, makes the first character of each + word uppercased (but makes the first character of the first word lowercased + if |lower_initial| is set to True), and joins the words. Please note that for + each word, all the characters except the first one are untouched. + """ + result = '' + capitalize_next = True + for i in range(len(identifier)): + if identifier[i] == delimiter: + capitalize_next = True + elif digits_split and identifier[i].isdigit(): + capitalize_next = True + result += identifier[i] + elif capitalize_next: + capitalize_next = False + result += identifier[i].upper() + else: + result += identifier[i] + + if lower_initial and result: + result = result[0].lower() + result[1:] + + return result + + +def _ToSnakeCase(identifier, upper=False): + """Splits camel-cased |identifier| into lower case words, removes the first + word if it's "k" and joins them using "_" e.g. for "URLLoaderFactory", returns + "URL_LOADER_FACTORY" if upper, otherwise "url_loader_factory". + """ + words = SplitCamelCase(identifier) + if words[0] == 'k' and len(words) > 1: + words = words[1:] + + # Variables cannot start with a digit + if (words[0][0].isdigit()): + words[0] = '_' + words[0] + + + if upper: + words = map(lambda x: x.upper(), words) + + return '_'.join(words) + + +def ToUpperSnakeCase(identifier): + """Splits camel-cased |identifier| into lower case words, removes the first + word if it's "k" and joins them using "_" e.g. for "URLLoaderFactory", returns + "URL_LOADER_FACTORY". + """ + return _ToSnakeCase(identifier, upper=True) + + +def ToLowerSnakeCase(identifier): + """Splits camel-cased |identifier| into lower case words, removes the first + word if it's "k" and joins them using "_" e.g. for "URLLoaderFactory", returns + "url_loader_factory". + """ + return _ToSnakeCase(identifier, upper=False) + + +class Stylizer(object): + """Stylizers specify naming rules to map mojom names to names in generated + code. For example, if you would like method_name in mojom to be mapped to + MethodName in the generated code, you need to define a subclass of Stylizer + and override StylizeMethod to do the conversion.""" + + def StylizeConstant(self, mojom_name): + return mojom_name + + def StylizeField(self, mojom_name): + return mojom_name + + def StylizeStruct(self, mojom_name): + return mojom_name + + def StylizeUnion(self, mojom_name): + return mojom_name + + def StylizeParameter(self, mojom_name): + return mojom_name + + def StylizeMethod(self, mojom_name): + return mojom_name + + def StylizeInterface(self, mojom_name): + return mojom_name + + def StylizeEnumField(self, mojom_name): + return mojom_name + + def StylizeEnum(self, mojom_name): + return mojom_name + + def StylizeModule(self, mojom_namespace): + return mojom_namespace + + +def WriteFile(contents, full_path): + # If |contents| is same with the file content, we skip updating. + if os.path.isfile(full_path): + with open(full_path, 'rb') as destination_file: + if destination_file.read() == contents: + return + + # Make sure the containing directory exists. + full_dir = os.path.dirname(full_path) + fileutil.EnsureDirectoryExists(full_dir) + + # Dump the data to disk. + with open(full_path, "wb") as f: + if not isinstance(contents, bytes): + f.write(contents.encode('utf-8')) + else: + f.write(contents) + + +def AddComputedData(module): + """Adds computed data to the given module. The data is computed once and + used repeatedly in the generation process.""" + + def _AddStructComputedData(exported, struct): + struct.packed = pack.PackedStruct(struct) + struct.bytes = pack.GetByteLayout(struct.packed) + struct.versions = pack.GetVersionInfo(struct.packed) + struct.exported = exported + + def _AddInterfaceComputedData(interface): + interface.version = 0 + for method in interface.methods: + # this field is never scrambled + method.sequential_ordinal = method.ordinal + + if method.min_version is not None: + interface.version = max(interface.version, method.min_version) + + method.param_struct = _GetStructFromMethod(method) + if interface.stable: + method.param_struct.attributes[mojom.ATTRIBUTE_STABLE] = True + if method.explicit_ordinal is None: + raise Exception( + 'Stable interfaces must declare explicit method ordinals. The ' + 'method %s on stable interface %s does not declare an explicit ' + 'ordinal.' % (method.mojom_name, interface.qualified_name)) + interface.version = max(interface.version, + method.param_struct.versions[-1].version) + + if method.response_parameters is not None: + method.response_param_struct = _GetResponseStructFromMethod(method) + if interface.stable: + method.response_param_struct.attributes[mojom.ATTRIBUTE_STABLE] = True + interface.version = max( + interface.version, + method.response_param_struct.versions[-1].version) + else: + method.response_param_struct = None + + def _GetStructFromMethod(method): + """Converts a method's parameters into the fields of a struct.""" + params_class = "%s_%s_Params" % (method.interface.mojom_name, + method.mojom_name) + struct = mojom.Struct(params_class, + module=method.interface.module, + attributes={}) + for param in method.parameters: + struct.AddField( + param.mojom_name, + param.kind, + param.ordinal, + attributes=param.attributes) + _AddStructComputedData(False, struct) + return struct + + def _GetResponseStructFromMethod(method): + """Converts a method's response_parameters into the fields of a struct.""" + params_class = "%s_%s_ResponseParams" % (method.interface.mojom_name, + method.mojom_name) + struct = mojom.Struct(params_class, + module=method.interface.module, + attributes={}) + for param in method.response_parameters: + struct.AddField( + param.mojom_name, + param.kind, + param.ordinal, + attributes=param.attributes) + _AddStructComputedData(False, struct) + return struct + + for struct in module.structs: + _AddStructComputedData(True, struct) + for interface in module.interfaces: + _AddInterfaceComputedData(interface) + + +class Generator(object): + # Pass |output_dir| to emit files to disk. Omit |output_dir| to echo all + # files to stdout. + def __init__(self, + module, + output_dir=None, + typemap=None, + variant=None, + bytecode_path=None, + for_blink=False, + js_bindings_mode="new", + js_generate_struct_deserializers=False, + export_attribute=None, + export_header=None, + generate_non_variant_code=False, + support_lazy_serialization=False, + disallow_native_types=False, + disallow_interfaces=False, + generate_message_ids=False, + generate_fuzzing=False, + enable_kythe_annotations=False, + extra_cpp_template_paths=None, + generate_extra_cpp_only=False): + self.module = module + self.output_dir = output_dir + self.typemap = typemap or {} + self.variant = variant + self.bytecode_path = bytecode_path + self.for_blink = for_blink + self.js_bindings_mode = js_bindings_mode + self.js_generate_struct_deserializers = js_generate_struct_deserializers + self.export_attribute = export_attribute + self.export_header = export_header + self.generate_non_variant_code = generate_non_variant_code + self.support_lazy_serialization = support_lazy_serialization + self.disallow_native_types = disallow_native_types + self.disallow_interfaces = disallow_interfaces + self.generate_message_ids = generate_message_ids + self.generate_fuzzing = generate_fuzzing + self.enable_kythe_annotations = enable_kythe_annotations + self.extra_cpp_template_paths = extra_cpp_template_paths + self.generate_extra_cpp_only = generate_extra_cpp_only + + def Write(self, contents, filename): + if self.output_dir is None: + print(contents) + return + full_path = os.path.join(self.output_dir, filename) + WriteFile(contents, full_path) + + def OptimizeEmpty(self, contents): + # Look for .cc files that contain no actual code. There are many of these + # and they collectively take a while to compile. + lines = contents.splitlines() + + for line in lines: + if line.startswith('#') or line.startswith('//'): + continue + if re.match(r'namespace .* {', line) or re.match(r'}.*//.*namespace', + line): + continue + if line.strip(): + # There is some actual code - return the unmodified contents. + return contents + + # If we reach here then we have a .cc file with no actual code. The + # includes are therefore unneeded and can be removed. + new_lines = [line for line in lines if not line.startswith('#include')] + if len(new_lines) < len(lines): + new_lines.append('') + new_lines.append('// Includes removed due to no code being generated.') + return '\n'.join(new_lines) + + def WriteWithComment(self, contents, filename): + generator_name = "mojom_bindings_generator.py" + comment = r"// %s is auto generated by %s, do not edit" % (filename, + generator_name) + contents = comment + '\n' + '\n' + contents; + if filename.endswith('.cc'): + contents = self.OptimizeEmpty(contents) + self.Write(contents, filename) + + def GenerateFiles(self, args): + raise NotImplementedError("Subclasses must override/implement this method") + + def GetJinjaParameters(self): + """Returns default constructor parameters for the jinja environment.""" + return {} + + def GetGlobals(self): + """Returns global mappings for the template generation.""" + return {} diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py new file mode 100644 index 00000000..32c884a8 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py @@ -0,0 +1,74 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys +import unittest + + +def _GetDirAbove(dirname): + """Returns the directory "above" this file containing |dirname| (which must + also be "above" this file).""" + path = os.path.abspath(__file__) + while True: + path, tail = os.path.split(path) + assert tail + if tail == dirname: + return path + + +try: + imp.find_module("mojom") +except ImportError: + sys.path.append(os.path.join(_GetDirAbove("pylib"), "pylib")) +from mojom.generate import generator + + +class StringManipulationTest(unittest.TestCase): + """generator contains some string utilities, this tests only those.""" + + def testSplitCamelCase(self): + self.assertEquals(["camel", "case"], generator.SplitCamelCase("CamelCase")) + self.assertEquals(["url", "loader", "factory"], + generator.SplitCamelCase('URLLoaderFactory')) + self.assertEquals(["get99", "entries"], + generator.SplitCamelCase('Get99Entries')) + self.assertEquals(["get99entries"], + generator.SplitCamelCase('Get99entries')) + + def testToCamel(self): + self.assertEquals("CamelCase", generator.ToCamel("camel_case")) + self.assertEquals("CAMELCASE", generator.ToCamel("CAMEL_CASE")) + self.assertEquals("camelCase", + generator.ToCamel("camel_case", lower_initial=True)) + self.assertEquals("CamelCase", generator.ToCamel( + "camel case", delimiter=' ')) + self.assertEquals("CaMelCaSe", generator.ToCamel("caMel_caSe")) + self.assertEquals("L2Tp", generator.ToCamel("l2tp", digits_split=True)) + self.assertEquals("l2tp", generator.ToCamel("l2tp", lower_initial=True)) + + def testToSnakeCase(self): + self.assertEquals("snake_case", generator.ToLowerSnakeCase("SnakeCase")) + self.assertEquals("snake_case", generator.ToLowerSnakeCase("snakeCase")) + self.assertEquals("snake_case", generator.ToLowerSnakeCase("SnakeCASE")) + self.assertEquals("snake_d3d11_case", + generator.ToLowerSnakeCase("SnakeD3D11Case")) + self.assertEquals("snake_d3d11_case", + generator.ToLowerSnakeCase("SnakeD3d11Case")) + self.assertEquals("snake_d3d11_case", + generator.ToLowerSnakeCase("snakeD3d11Case")) + self.assertEquals("SNAKE_CASE", generator.ToUpperSnakeCase("SnakeCase")) + self.assertEquals("SNAKE_CASE", generator.ToUpperSnakeCase("snakeCase")) + self.assertEquals("SNAKE_CASE", generator.ToUpperSnakeCase("SnakeCASE")) + self.assertEquals("SNAKE_D3D11_CASE", + generator.ToUpperSnakeCase("SnakeD3D11Case")) + self.assertEquals("SNAKE_D3D11_CASE", + generator.ToUpperSnakeCase("SnakeD3d11Case")) + self.assertEquals("SNAKE_D3D11_CASE", + generator.ToUpperSnakeCase("snakeD3d11Case")) + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py new file mode 100644 index 00000000..8547ff64 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py @@ -0,0 +1,1635 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +# This module's classes provide an interface to mojo modules. Modules are +# collections of interfaces and structs to be used by mojo ipc clients and +# servers. +# +# A simple interface would be created this way: +# module = mojom.generate.module.Module('Foo') +# interface = module.AddInterface('Bar') +# method = interface.AddMethod('Tat', 0) +# method.AddParameter('baz', 0, mojom.INT32) + +import pickle + +# We use our own version of __repr__ when displaying the AST, as the +# AST currently doesn't capture which nodes are reference (e.g. to +# types) and which nodes are definitions. This allows us to e.g. print +# the definition of a struct when it's defined inside a module, but +# only print its name when it's referenced in e.g. a method parameter. +def Repr(obj, as_ref=True): + """A version of __repr__ that can distinguish references. + + Sometimes we like to print an object's full representation + (e.g. with its fields) and sometimes we just want to reference an + object that was printed in full elsewhere. This function allows us + to make that distinction. + + Args: + obj: The object whose string representation we compute. + as_ref: If True, use the short reference representation. + + Returns: + A str representation of |obj|. + """ + if hasattr(obj, 'Repr'): + return obj.Repr(as_ref=as_ref) + # Since we cannot implement Repr for existing container types, we + # handle them here. + elif isinstance(obj, list): + if not obj: + return '[]' + else: + return ('[\n%s\n]' % (',\n'.join( + ' %s' % Repr(elem, as_ref).replace('\n', '\n ') + for elem in obj))) + elif isinstance(obj, dict): + if not obj: + return '{}' + else: + return ('{\n%s\n}' % (',\n'.join( + ' %s: %s' % (Repr(key, as_ref).replace('\n', '\n '), + Repr(val, as_ref).replace('\n', '\n ')) + for key, val in obj.items()))) + else: + return repr(obj) + + +def GenericRepr(obj, names): + """Compute generic Repr for |obj| based on the attributes in |names|. + + Args: + obj: The object to compute a Repr for. + names: A dict from attribute names to include, to booleans + specifying whether those attributes should be shown as + references or not. + + Returns: + A str representation of |obj|. + """ + + def ReprIndent(name, as_ref): + return ' %s=%s' % (name, Repr(getattr(obj, name), as_ref).replace( + '\n', '\n ')) + + return '%s(\n%s\n)' % (obj.__class__.__name__, ',\n'.join( + ReprIndent(name, as_ref) for (name, as_ref) in names.items())) + + +class Kind(object): + """Kind represents a type (e.g. int8, string). + + Attributes: + spec: A string uniquely identifying the type. May be None. + module: {Module} The defining module. Set to None for built-in types. + parent_kind: The enclosing type. For example, an enum defined + inside an interface has that interface as its parent. May be None. + """ + + def __init__(self, spec=None, module=None): + self.spec = spec + self.module = module + self.parent_kind = None + + def Repr(self, as_ref=True): + # pylint: disable=unused-argument + return '<%s spec=%r>' % (self.__class__.__name__, self.spec) + + def __repr__(self): + # Gives us a decent __repr__ for all kinds. + return self.Repr() + + def __eq__(self, rhs): + # pylint: disable=unidiomatic-typecheck + return (type(self) == type(rhs) + and (self.spec, self.parent_kind) == (rhs.spec, rhs.parent_kind)) + + def __hash__(self): + # TODO(crbug.com/1060471): Remove this and other __hash__ methods on Kind + # and its subclasses. This is to support existing generator code which uses + # some primitive Kinds as dict keys. The default hash (object identity) + # breaks these dicts when a pickled Module instance is unpickled and used + # during a subsequent run of the parser. + return hash((self.spec, self.parent_kind)) + + +class ReferenceKind(Kind): + """ReferenceKind represents pointer and handle types. + + A type is nullable if null (for pointer types) or invalid handle (for handle + types) is a legal value for the type. + + Attributes: + is_nullable: True if the type is nullable. + """ + + def __init__(self, spec=None, is_nullable=False, module=None): + assert spec is None or is_nullable == spec.startswith('?') + Kind.__init__(self, spec, module) + self.is_nullable = is_nullable + self.shared_definition = {} + + def Repr(self, as_ref=True): + return '<%s spec=%r is_nullable=%r>' % (self.__class__.__name__, self.spec, + self.is_nullable) + + def MakeNullableKind(self): + assert not self.is_nullable + + if self == STRING: + return NULLABLE_STRING + if self == HANDLE: + return NULLABLE_HANDLE + if self == DCPIPE: + return NULLABLE_DCPIPE + if self == DPPIPE: + return NULLABLE_DPPIPE + if self == MSGPIPE: + return NULLABLE_MSGPIPE + if self == SHAREDBUFFER: + return NULLABLE_SHAREDBUFFER + if self == PLATFORMHANDLE: + return NULLABLE_PLATFORMHANDLE + + nullable_kind = type(self)() + nullable_kind.shared_definition = self.shared_definition + if self.spec is not None: + nullable_kind.spec = '?' + self.spec + nullable_kind.is_nullable = True + nullable_kind.parent_kind = self.parent_kind + nullable_kind.module = self.module + + return nullable_kind + + @classmethod + def AddSharedProperty(cls, name): + """Adds a property |name| to |cls|, which accesses the corresponding item in + |shared_definition|. + + The reason of adding such indirection is to enable sharing definition + between a reference kind and its nullable variation. For example: + a = Struct('test_struct_1') + b = a.MakeNullableKind() + a.name = 'test_struct_2' + print(b.name) # Outputs 'test_struct_2'. + """ + + def Get(self): + try: + return self.shared_definition[name] + except KeyError: # Must raise AttributeError if property doesn't exist. + raise AttributeError + + def Set(self, value): + self.shared_definition[name] = value + + setattr(cls, name, property(Get, Set)) + + def __eq__(self, rhs): + return (isinstance(rhs, ReferenceKind) + and super(ReferenceKind, self).__eq__(rhs) + and self.is_nullable == rhs.is_nullable) + + def __hash__(self): + return hash((super(ReferenceKind, self).__hash__(), self.is_nullable)) + + +# Initialize the set of primitive types. These can be accessed by clients. +BOOL = Kind('b') +INT8 = Kind('i8') +INT16 = Kind('i16') +INT32 = Kind('i32') +INT64 = Kind('i64') +UINT8 = Kind('u8') +UINT16 = Kind('u16') +UINT32 = Kind('u32') +UINT64 = Kind('u64') +FLOAT = Kind('f') +DOUBLE = Kind('d') +STRING = ReferenceKind('s') +HANDLE = ReferenceKind('h') +DCPIPE = ReferenceKind('h:d:c') +DPPIPE = ReferenceKind('h:d:p') +MSGPIPE = ReferenceKind('h:m') +SHAREDBUFFER = ReferenceKind('h:s') +PLATFORMHANDLE = ReferenceKind('h:p') +NULLABLE_STRING = ReferenceKind('?s', True) +NULLABLE_HANDLE = ReferenceKind('?h', True) +NULLABLE_DCPIPE = ReferenceKind('?h:d:c', True) +NULLABLE_DPPIPE = ReferenceKind('?h:d:p', True) +NULLABLE_MSGPIPE = ReferenceKind('?h:m', True) +NULLABLE_SHAREDBUFFER = ReferenceKind('?h:s', True) +NULLABLE_PLATFORMHANDLE = ReferenceKind('?h:p', True) + +# Collection of all Primitive types +PRIMITIVES = ( + BOOL, + INT8, + INT16, + INT32, + INT64, + UINT8, + UINT16, + UINT32, + UINT64, + FLOAT, + DOUBLE, + STRING, + HANDLE, + DCPIPE, + DPPIPE, + MSGPIPE, + SHAREDBUFFER, + PLATFORMHANDLE, + NULLABLE_STRING, + NULLABLE_HANDLE, + NULLABLE_DCPIPE, + NULLABLE_DPPIPE, + NULLABLE_MSGPIPE, + NULLABLE_SHAREDBUFFER, + NULLABLE_PLATFORMHANDLE, +) + +ATTRIBUTE_MIN_VERSION = 'MinVersion' +ATTRIBUTE_EXTENSIBLE = 'Extensible' +ATTRIBUTE_STABLE = 'Stable' +ATTRIBUTE_SYNC = 'Sync' + + +class NamedValue(object): + def __init__(self, module, parent_kind, mojom_name): + self.module = module + self.parent_kind = parent_kind + self.mojom_name = mojom_name + + def GetSpec(self): + return (self.module.GetNamespacePrefix() + + (self.parent_kind and + (self.parent_kind.mojom_name + '.') or "") + self.mojom_name) + + def __eq__(self, rhs): + return (isinstance(rhs, NamedValue) + and (self.parent_kind, self.mojom_name) == (rhs.parent_kind, + rhs.mojom_name)) + + +class BuiltinValue(object): + def __init__(self, value): + self.value = value + + def __eq__(self, rhs): + return isinstance(rhs, BuiltinValue) and self.value == rhs.value + + +class ConstantValue(NamedValue): + def __init__(self, module, parent_kind, constant): + NamedValue.__init__(self, module, parent_kind, constant.mojom_name) + self.constant = constant + + @property + def name(self): + return self.constant.name + + +class EnumValue(NamedValue): + def __init__(self, module, enum, field): + NamedValue.__init__(self, module, enum.parent_kind, field.mojom_name) + self.field = field + self.enum = enum + + def GetSpec(self): + return (self.module.GetNamespacePrefix() + + (self.parent_kind and (self.parent_kind.mojom_name + '.') or "") + + self.enum.mojom_name + '.' + self.mojom_name) + + @property + def name(self): + return self.field.name + + +class Constant(object): + def __init__(self, mojom_name=None, kind=None, value=None, parent_kind=None): + self.mojom_name = mojom_name + self.name = None + self.kind = kind + self.value = value + self.parent_kind = parent_kind + + def Stylize(self, stylizer): + self.name = stylizer.StylizeConstant(self.mojom_name) + + def __eq__(self, rhs): + return (isinstance(rhs, Constant) + and (self.mojom_name, self.kind, self.value, + self.parent_kind) == (rhs.mojom_name, rhs.kind, rhs.value, + rhs.parent_kind)) + + +class Field(object): + def __init__(self, + mojom_name=None, + kind=None, + ordinal=None, + default=None, + attributes=None): + if self.__class__.__name__ == 'Field': + raise Exception() + self.mojom_name = mojom_name + self.name = None + self.kind = kind + self.ordinal = ordinal + self.default = default + self.attributes = attributes + + def Repr(self, as_ref=True): + # pylint: disable=unused-argument + # Fields are only referenced by objects which define them and thus + # they are always displayed as non-references. + return GenericRepr(self, {'mojom_name': False, 'kind': True}) + + def Stylize(self, stylizer): + self.name = stylizer.StylizeField(self.mojom_name) + + @property + def min_version(self): + return self.attributes.get(ATTRIBUTE_MIN_VERSION) \ + if self.attributes else None + + def __eq__(self, rhs): + return (isinstance(rhs, Field) + and (self.mojom_name, self.kind, self.ordinal, self.default, + self.attributes) == (rhs.mojom_name, rhs.kind, rhs.ordinal, + rhs.default, rhs.attributes)) + + def __hash__(self): + return hash((self.mojom_name, self.kind, self.ordinal, self.default)) + + +class StructField(Field): + pass + + +class UnionField(Field): + pass + + +def _IsFieldBackwardCompatible(new_field, old_field): + if (new_field.min_version or 0) != (old_field.min_version or 0): + return False + + if isinstance(new_field.kind, (Enum, Struct, Union)): + return new_field.kind.IsBackwardCompatible(old_field.kind) + + return new_field.kind == old_field.kind + + +class Struct(ReferenceKind): + """A struct with typed fields. + + Attributes: + mojom_name: {str} The name of the struct type as defined in mojom. + name: {str} The stylized name. + native_only: {bool} Does the struct have a body (i.e. any fields) or is it + purely a native struct. + custom_serializer: {bool} Should we generate a serializer for the struct or + will one be provided by non-generated code. + fields: {List[StructField]} The members of the struct. + enums: {List[Enum]} The enums defined in the struct scope. + constants: {List[Constant]} The constants defined in the struct scope. + attributes: {dict} Additional information about the struct, such as + if it's a native struct. + """ + + ReferenceKind.AddSharedProperty('mojom_name') + ReferenceKind.AddSharedProperty('name') + ReferenceKind.AddSharedProperty('native_only') + ReferenceKind.AddSharedProperty('custom_serializer') + ReferenceKind.AddSharedProperty('fields') + ReferenceKind.AddSharedProperty('enums') + ReferenceKind.AddSharedProperty('constants') + ReferenceKind.AddSharedProperty('attributes') + + def __init__(self, mojom_name=None, module=None, attributes=None): + if mojom_name is not None: + spec = 'x:' + mojom_name + else: + spec = None + ReferenceKind.__init__(self, spec, False, module) + self.mojom_name = mojom_name + self.name = None + self.native_only = False + self.custom_serializer = False + self.fields = [] + self.enums = [] + self.constants = [] + self.attributes = attributes + + def Repr(self, as_ref=True): + if as_ref: + return '<%s mojom_name=%r module=%s>' % (self.__class__.__name__, + self.mojom_name, + Repr(self.module, as_ref=True)) + else: + return GenericRepr(self, { + 'mojom_name': False, + 'fields': False, + 'module': True + }) + + def AddField(self, + mojom_name, + kind, + ordinal=None, + default=None, + attributes=None): + field = StructField(mojom_name, kind, ordinal, default, attributes) + self.fields.append(field) + return field + + def Stylize(self, stylizer): + self.name = stylizer.StylizeStruct(self.mojom_name) + for field in self.fields: + field.Stylize(stylizer) + for enum in self.enums: + enum.Stylize(stylizer) + for constant in self.constants: + constant.Stylize(stylizer) + + def IsBackwardCompatible(self, older_struct): + """This struct is backward-compatible with older_struct if and only if all + of the following conditions hold: + - Any newly added field is tagged with a [MinVersion] attribute specifying + a version number greater than all previously used [MinVersion] + attributes within the struct. + - All fields present in older_struct remain present in the new struct, + with the same ordinal position, same optional or non-optional status, + same (or backward-compatible) type and where applicable, the same + [MinVersion] attribute value. + - All [MinVersion] attributes must be non-decreasing in ordinal order. + - All reference-typed (string, array, map, struct, or union) fields tagged + with a [MinVersion] greater than zero must be optional. + """ + + def buildOrdinalFieldMap(struct): + fields_by_ordinal = {} + for field in struct.fields: + if field.ordinal in fields_by_ordinal: + raise Exception('Multiple fields with ordinal %s in struct %s.' % + (field.ordinal, struct.mojom_name)) + fields_by_ordinal[field.ordinal] = field + return fields_by_ordinal + + new_fields = buildOrdinalFieldMap(self) + old_fields = buildOrdinalFieldMap(older_struct) + if len(new_fields) < len(old_fields): + # At least one field was removed, which is not OK. + return False + + # If there are N fields, existing ordinal values must exactly cover the + # range from 0 to N-1. + num_old_ordinals = len(old_fields) + max_old_min_version = 0 + for ordinal in range(num_old_ordinals): + new_field = new_fields[ordinal] + old_field = old_fields[ordinal] + if (old_field.min_version or 0) > max_old_min_version: + max_old_min_version = old_field.min_version + if not _IsFieldBackwardCompatible(new_field, old_field): + # Type or min-version mismatch between old and new versions of the same + # ordinal field. + return False + + # At this point we know all old fields are intact in the new struct + # definition. Now verify that all new fields have a high enough min version + # and are appropriately optional where required. + num_new_ordinals = len(new_fields) + last_min_version = max_old_min_version + for ordinal in range(num_old_ordinals, num_new_ordinals): + new_field = new_fields[ordinal] + min_version = new_field.min_version or 0 + if min_version <= max_old_min_version: + # A new field is being added to an existing version, which is not OK. + return False + if min_version < last_min_version: + # The [MinVersion] of a field cannot be lower than the [MinVersion] of + # a field with lower ordinal value. + return False + if IsReferenceKind(new_field.kind) and not IsNullableKind(new_field.kind): + # New fields whose type can be nullable MUST be nullable. + return False + + return True + + @property + def stable(self): + return self.attributes.get(ATTRIBUTE_STABLE, False) \ + if self.attributes else False + + @property + def qualified_name(self): + if self.parent_kind: + prefix = self.parent_kind.qualified_name + '.' + else: + prefix = self.module.GetNamespacePrefix() + return '%s%s' % (prefix, self.mojom_name) + + def __eq__(self, rhs): + return (isinstance(rhs, Struct) and + (self.mojom_name, self.native_only, self.fields, self.constants, + self.attributes) == (rhs.mojom_name, rhs.native_only, rhs.fields, + rhs.constants, rhs.attributes)) + + def __hash__(self): + return id(self) + + +class Union(ReferenceKind): + """A union of several kinds. + + Attributes: + mojom_name: {str} The name of the union type as defined in mojom. + name: {str} The stylized name. + fields: {List[UnionField]} The members of the union. + attributes: {dict} Additional information about the union, such as + which Java class name to use to represent it in the generated + bindings. + """ + ReferenceKind.AddSharedProperty('mojom_name') + ReferenceKind.AddSharedProperty('name') + ReferenceKind.AddSharedProperty('fields') + ReferenceKind.AddSharedProperty('attributes') + + def __init__(self, mojom_name=None, module=None, attributes=None): + if mojom_name is not None: + spec = 'x:' + mojom_name + else: + spec = None + ReferenceKind.__init__(self, spec, False, module) + self.mojom_name = mojom_name + self.name = None + self.fields = [] + self.attributes = attributes + + def Repr(self, as_ref=True): + if as_ref: + return '<%s spec=%r is_nullable=%r fields=%s>' % ( + self.__class__.__name__, self.spec, self.is_nullable, Repr( + self.fields)) + else: + return GenericRepr(self, {'fields': True, 'is_nullable': False}) + + def AddField(self, mojom_name, kind, ordinal=None, attributes=None): + field = UnionField(mojom_name, kind, ordinal, None, attributes) + self.fields.append(field) + return field + + def Stylize(self, stylizer): + self.name = stylizer.StylizeUnion(self.mojom_name) + for field in self.fields: + field.Stylize(stylizer) + + def IsBackwardCompatible(self, older_union): + """This union is backward-compatible with older_union if and only if all + of the following conditions hold: + - Any newly added field is tagged with a [MinVersion] attribute specifying + a version number greater than all previously used [MinVersion] + attributes within the union. + - All fields present in older_union remain present in the new union, + with the same ordinal value, same optional or non-optional status, + same (or backward-compatible) type, and where applicable, the same + [MinVersion] attribute value. + """ + + def buildOrdinalFieldMap(union): + fields_by_ordinal = {} + for field in union.fields: + if field.ordinal in fields_by_ordinal: + raise Exception('Multiple fields with ordinal %s in union %s.' % + (field.ordinal, union.mojom_name)) + fields_by_ordinal[field.ordinal] = field + return fields_by_ordinal + + new_fields = buildOrdinalFieldMap(self) + old_fields = buildOrdinalFieldMap(older_union) + if len(new_fields) < len(old_fields): + # At least one field was removed, which is not OK. + return False + + max_old_min_version = 0 + for ordinal, old_field in old_fields.items(): + new_field = new_fields.get(ordinal) + if not new_field: + # A field was removed, which is not OK. + return False + if not _IsFieldBackwardCompatible(new_field, old_field): + # An field changed its type or MinVersion, which is not OK. + return False + old_min_version = old_field.min_version or 0 + if old_min_version > max_old_min_version: + max_old_min_version = old_min_version + + new_ordinals = set(new_fields.keys()) - set(old_fields.keys()) + for ordinal in new_ordinals: + if (new_fields[ordinal].min_version or 0) <= max_old_min_version: + # New fields must use a MinVersion greater than any old fields. + return False + + return True + + @property + def stable(self): + return self.attributes.get(ATTRIBUTE_STABLE, False) \ + if self.attributes else False + + @property + def qualified_name(self): + if self.parent_kind: + prefix = self.parent_kind.qualified_name + '.' + else: + prefix = self.module.GetNamespacePrefix() + return '%s%s' % (prefix, self.mojom_name) + + def __eq__(self, rhs): + return (isinstance(rhs, Union) and + (self.mojom_name, self.fields, + self.attributes) == (rhs.mojom_name, rhs.fields, rhs.attributes)) + + def __hash__(self): + return id(self) + + +class Array(ReferenceKind): + """An array. + + Attributes: + kind: {Kind} The type of the elements. May be None. + length: The number of elements. None if unknown. + """ + + ReferenceKind.AddSharedProperty('kind') + ReferenceKind.AddSharedProperty('length') + + def __init__(self, kind=None, length=None): + if kind is not None: + if length is not None: + spec = 'a%d:%s' % (length, kind.spec) + else: + spec = 'a:%s' % kind.spec + + ReferenceKind.__init__(self, spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + self.length = length + + def Repr(self, as_ref=True): + if as_ref: + return '<%s spec=%r is_nullable=%r kind=%s length=%r>' % ( + self.__class__.__name__, self.spec, self.is_nullable, Repr( + self.kind), self.length) + else: + return GenericRepr(self, { + 'kind': True, + 'length': False, + 'is_nullable': False + }) + + def __eq__(self, rhs): + return (isinstance(rhs, Array) + and (self.kind, self.length) == (rhs.kind, rhs.length)) + + def __hash__(self): + return id(self) + + +class Map(ReferenceKind): + """A map. + + Attributes: + key_kind: {Kind} The type of the keys. May be None. + value_kind: {Kind} The type of the elements. May be None. + """ + ReferenceKind.AddSharedProperty('key_kind') + ReferenceKind.AddSharedProperty('value_kind') + + def __init__(self, key_kind=None, value_kind=None): + if (key_kind is not None and value_kind is not None): + ReferenceKind.__init__( + self, 'm[' + key_kind.spec + '][' + value_kind.spec + ']') + if IsNullableKind(key_kind): + raise Exception("Nullable kinds cannot be keys in maps.") + if IsAnyHandleKind(key_kind): + raise Exception("Handles cannot be keys in maps.") + if IsAnyInterfaceKind(key_kind): + raise Exception("Interfaces cannot be keys in maps.") + if IsArrayKind(key_kind): + raise Exception("Arrays cannot be keys in maps.") + else: + ReferenceKind.__init__(self) + + self.key_kind = key_kind + self.value_kind = value_kind + + def Repr(self, as_ref=True): + if as_ref: + return '<%s spec=%r is_nullable=%r key_kind=%s value_kind=%s>' % ( + self.__class__.__name__, self.spec, self.is_nullable, + Repr(self.key_kind), Repr(self.value_kind)) + else: + return GenericRepr(self, {'key_kind': True, 'value_kind': True}) + + def __eq__(self, rhs): + return (isinstance(rhs, Map) and + (self.key_kind, self.value_kind) == (rhs.key_kind, rhs.value_kind)) + + def __hash__(self): + return id(self) + + +class PendingRemote(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + 'pending_remote requires T to be an interface type. Got %r' % + kind.spec) + ReferenceKind.__init__(self, 'rmt:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, PendingRemote) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class PendingReceiver(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + 'pending_receiver requires T to be an interface type. Got %r' % + kind.spec) + ReferenceKind.__init__(self, 'rcv:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, PendingReceiver) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class PendingAssociatedRemote(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + 'pending_associated_remote requires T to be an interface ' + + 'type. Got %r' % kind.spec) + ReferenceKind.__init__(self, 'rma:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, PendingAssociatedRemote) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class PendingAssociatedReceiver(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + 'pending_associated_receiver requires T to be an interface' + + 'type. Got %r' % kind.spec) + ReferenceKind.__init__(self, 'rca:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, PendingAssociatedReceiver) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class InterfaceRequest(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + "Interface request requires %r to be an interface." % kind.spec) + ReferenceKind.__init__(self, 'r:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, InterfaceRequest) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class AssociatedInterfaceRequest(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, InterfaceRequest): + raise Exception( + "Associated interface request requires %r to be an interface " + "request." % kind.spec) + assert not kind.is_nullable + ReferenceKind.__init__(self, 'asso:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind.kind if kind is not None else None + + def __eq__(self, rhs): + return isinstance(rhs, AssociatedInterfaceRequest) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class Parameter(object): + def __init__(self, + mojom_name=None, + kind=None, + ordinal=None, + default=None, + attributes=None): + self.mojom_name = mojom_name + self.name = None + self.ordinal = ordinal + self.kind = kind + self.default = default + self.attributes = attributes + + def Repr(self, as_ref=True): + # pylint: disable=unused-argument + return '<%s mojom_name=%r kind=%s>' % ( + self.__class__.__name__, self.mojom_name, self.kind.Repr(as_ref=True)) + + def Stylize(self, stylizer): + self.name = stylizer.StylizeParameter(self.mojom_name) + + @property + def min_version(self): + return self.attributes.get(ATTRIBUTE_MIN_VERSION) \ + if self.attributes else None + + def __eq__(self, rhs): + return (isinstance(rhs, Parameter) + and (self.mojom_name, self.ordinal, self.kind, self.default, + self.attributes) == (rhs.mojom_name, rhs.ordinal, rhs.kind, + rhs.default, rhs.attributes)) + + +class Method(object): + def __init__(self, interface, mojom_name, ordinal=None, attributes=None): + self.interface = interface + self.mojom_name = mojom_name + self.name = None + self.explicit_ordinal = ordinal + self.ordinal = ordinal + self.parameters = [] + self.param_struct = None + self.response_parameters = None + self.response_param_struct = None + self.attributes = attributes + + def Repr(self, as_ref=True): + if as_ref: + return '<%s mojom_name=%r>' % (self.__class__.__name__, self.mojom_name) + else: + return GenericRepr(self, { + 'mojom_name': False, + 'parameters': True, + 'response_parameters': True + }) + + def AddParameter(self, + mojom_name, + kind, + ordinal=None, + default=None, + attributes=None): + parameter = Parameter(mojom_name, kind, ordinal, default, attributes) + self.parameters.append(parameter) + return parameter + + def AddResponseParameter(self, + mojom_name, + kind, + ordinal=None, + default=None, + attributes=None): + if self.response_parameters == None: + self.response_parameters = [] + parameter = Parameter(mojom_name, kind, ordinal, default, attributes) + self.response_parameters.append(parameter) + return parameter + + def Stylize(self, stylizer): + self.name = stylizer.StylizeMethod(self.mojom_name) + for param in self.parameters: + param.Stylize(stylizer) + if self.response_parameters is not None: + for param in self.response_parameters: + param.Stylize(stylizer) + + if self.param_struct: + self.param_struct.Stylize(stylizer) + if self.response_param_struct: + self.response_param_struct.Stylize(stylizer) + + @property + def min_version(self): + return self.attributes.get(ATTRIBUTE_MIN_VERSION) \ + if self.attributes else None + + @property + def sync(self): + return self.attributes.get(ATTRIBUTE_SYNC) \ + if self.attributes else None + + def __eq__(self, rhs): + return (isinstance(rhs, Method) and + (self.mojom_name, self.ordinal, self.parameters, + self.response_parameters, + self.attributes) == (rhs.mojom_name, rhs.ordinal, rhs.parameters, + rhs.response_parameters, rhs.attributes)) + + +class Interface(ReferenceKind): + ReferenceKind.AddSharedProperty('mojom_name') + ReferenceKind.AddSharedProperty('name') + ReferenceKind.AddSharedProperty('methods') + ReferenceKind.AddSharedProperty('enums') + ReferenceKind.AddSharedProperty('constants') + ReferenceKind.AddSharedProperty('attributes') + + def __init__(self, mojom_name=None, module=None, attributes=None): + if mojom_name is not None: + spec = 'x:' + mojom_name + else: + spec = None + ReferenceKind.__init__(self, spec, False, module) + self.mojom_name = mojom_name + self.name = None + self.methods = [] + self.enums = [] + self.constants = [] + self.attributes = attributes + + def Repr(self, as_ref=True): + if as_ref: + return '<%s mojom_name=%r>' % (self.__class__.__name__, self.mojom_name) + else: + return GenericRepr(self, { + 'mojom_name': False, + 'attributes': False, + 'methods': False + }) + + def AddMethod(self, mojom_name, ordinal=None, attributes=None): + method = Method(self, mojom_name, ordinal, attributes) + self.methods.append(method) + return method + + def Stylize(self, stylizer): + self.name = stylizer.StylizeInterface(self.mojom_name) + for method in self.methods: + method.Stylize(stylizer) + for enum in self.enums: + enum.Stylize(stylizer) + for constant in self.constants: + constant.Stylize(stylizer) + + def IsBackwardCompatible(self, older_interface): + """This interface is backward-compatible with older_interface if and only + if all of the following conditions hold: + - All defined methods in older_interface (when identified by ordinal) have + backward-compatible definitions in this interface. For each method this + means: + - The parameter list is backward-compatible, according to backward- + compatibility rules for structs, where each parameter is essentially + a struct field. + - If the old method definition does not specify a reply message, the + new method definition must not specify a reply message. + - If the old method definition specifies a reply message, the new + method definition must also specify a reply message with a parameter + list that is backward-compatible according to backward-compatibility + rules for structs. + - All newly introduced methods in this interface have a [MinVersion] + attribute specifying a version greater than any method in + older_interface. + """ + + def buildOrdinalMethodMap(interface): + methods_by_ordinal = {} + for method in interface.methods: + if method.ordinal in methods_by_ordinal: + raise Exception('Multiple methods with ordinal %s in interface %s.' % + (method.ordinal, interface.mojom_name)) + methods_by_ordinal[method.ordinal] = method + return methods_by_ordinal + + new_methods = buildOrdinalMethodMap(self) + old_methods = buildOrdinalMethodMap(older_interface) + max_old_min_version = 0 + for ordinal, old_method in old_methods.items(): + new_method = new_methods.get(ordinal) + if not new_method: + # A method was removed, which is not OK. + return False + + if not new_method.param_struct.IsBackwardCompatible( + old_method.param_struct): + # The parameter list is not backward-compatible, which is not OK. + return False + + if old_method.response_param_struct is None: + if new_method.response_param_struct is not None: + # A reply was added to a message which didn't have one before, and + # this is not OK. + return False + else: + if new_method.response_param_struct is None: + # A reply was removed from a message, which is not OK. + return False + if not new_method.response_param_struct.IsBackwardCompatible( + old_method.response_param_struct): + # The new message's reply is not backward-compatible with the old + # message's reply, which is not OK. + return False + + if (old_method.min_version or 0) > max_old_min_version: + max_old_min_version = old_method.min_version + + # All the old methods are compatible with their new counterparts. Now verify + # that newly added methods are properly versioned. + new_ordinals = set(new_methods.keys()) - set(old_methods.keys()) + for ordinal in new_ordinals: + new_method = new_methods[ordinal] + if (new_method.min_version or 0) <= max_old_min_version: + # A method was added to an existing version, which is not OK. + return False + + return True + + @property + def stable(self): + return self.attributes.get(ATTRIBUTE_STABLE, False) \ + if self.attributes else False + + @property + def qualified_name(self): + if self.parent_kind: + prefix = self.parent_kind.qualified_name + '.' + else: + prefix = self.module.GetNamespacePrefix() + return '%s%s' % (prefix, self.mojom_name) + + def __eq__(self, rhs): + return (isinstance(rhs, Interface) + and (self.mojom_name, self.methods, self.enums, self.constants, + self.attributes) == (rhs.mojom_name, rhs.methods, rhs.enums, + rhs.constants, rhs.attributes)) + + def __hash__(self): + return id(self) + + +class AssociatedInterface(ReferenceKind): + ReferenceKind.AddSharedProperty('kind') + + def __init__(self, kind=None): + if kind is not None: + if not isinstance(kind, Interface): + raise Exception( + "Associated interface requires %r to be an interface." % kind.spec) + assert not kind.is_nullable + ReferenceKind.__init__(self, 'asso:' + kind.spec) + else: + ReferenceKind.__init__(self) + self.kind = kind + + def __eq__(self, rhs): + return isinstance(rhs, AssociatedInterface) and self.kind == rhs.kind + + def __hash__(self): + return id(self) + + +class EnumField(object): + def __init__(self, + mojom_name=None, + value=None, + attributes=None, + numeric_value=None): + self.mojom_name = mojom_name + self.name = None + self.value = value + self.attributes = attributes + self.numeric_value = numeric_value + + def Stylize(self, stylizer): + self.name = stylizer.StylizeEnumField(self.mojom_name) + + @property + def min_version(self): + return self.attributes.get(ATTRIBUTE_MIN_VERSION) \ + if self.attributes else None + + def __eq__(self, rhs): + return (isinstance(rhs, EnumField) + and (self.mojom_name, self.value, self.attributes, + self.numeric_value) == (rhs.mojom_name, rhs.value, + rhs.attributes, rhs.numeric_value)) + + +class Enum(Kind): + def __init__(self, mojom_name=None, module=None, attributes=None): + self.mojom_name = mojom_name + self.name = None + self.native_only = False + if mojom_name is not None: + spec = 'x:' + mojom_name + else: + spec = None + Kind.__init__(self, spec, module) + self.fields = [] + self.attributes = attributes + self.min_value = None + self.max_value = None + + def Repr(self, as_ref=True): + if as_ref: + return '<%s mojom_name=%r>' % (self.__class__.__name__, self.mojom_name) + else: + return GenericRepr(self, {'mojom_name': False, 'fields': False}) + + def Stylize(self, stylizer): + self.name = stylizer.StylizeEnum(self.mojom_name) + for field in self.fields: + field.Stylize(stylizer) + + @property + def extensible(self): + return self.attributes.get(ATTRIBUTE_EXTENSIBLE, False) \ + if self.attributes else False + + @property + def stable(self): + return self.attributes.get(ATTRIBUTE_STABLE, False) \ + if self.attributes else False + + @property + def qualified_name(self): + if self.parent_kind: + prefix = self.parent_kind.qualified_name + '.' + else: + prefix = self.module.GetNamespacePrefix() + return '%s%s' % (prefix, self.mojom_name) + + def IsBackwardCompatible(self, older_enum): + """This enum is backward-compatible with older_enum if and only if one of + the following conditions holds: + - Neither enum is [Extensible] and both have the exact same set of valid + numeric values. Field names and aliases for the same numeric value do + not affect compatibility. + - older_enum is [Extensible], and for every version defined by + older_enum, this enum has the exact same set of valid numeric values. + """ + + def buildVersionFieldMap(enum): + fields_by_min_version = {} + for field in enum.fields: + if field.min_version not in fields_by_min_version: + fields_by_min_version[field.min_version] = set() + fields_by_min_version[field.min_version].add(field.numeric_value) + return fields_by_min_version + + old_fields = buildVersionFieldMap(older_enum) + new_fields = buildVersionFieldMap(self) + + if new_fields.keys() != old_fields.keys() and not older_enum.extensible: + return False + + for min_version, valid_values in old_fields.items(): + if (min_version not in new_fields + or new_fields[min_version] != valid_values): + return False + + return True + + def __eq__(self, rhs): + return (isinstance(rhs, Enum) and + (self.mojom_name, self.native_only, self.fields, self.attributes, + self.min_value, + self.max_value) == (rhs.mojom_name, rhs.native_only, rhs.fields, + rhs.attributes, rhs.min_value, rhs.max_value)) + + def __hash__(self): + return id(self) + + +class Module(object): + def __init__(self, path=None, mojom_namespace=None, attributes=None): + self.path = path + self.mojom_namespace = mojom_namespace + self.namespace = None + self.structs = [] + self.unions = [] + self.interfaces = [] + self.enums = [] + self.constants = [] + self.kinds = {} + self.attributes = attributes + self.imports = [] + self.imported_kinds = {} + + def __repr__(self): + # Gives us a decent __repr__ for modules. + return self.Repr() + + def __eq__(self, rhs): + return (isinstance(rhs, Module) and + (self.path, self.attributes, self.mojom_namespace, self.imports, + self.constants, self.enums, self.structs, self.unions, + self.interfaces) == (rhs.path, rhs.attributes, rhs.mojom_namespace, + rhs.imports, rhs.constants, rhs.enums, + rhs.structs, rhs.unions, rhs.interfaces)) + + def Repr(self, as_ref=True): + if as_ref: + return '<%s path=%r mojom_namespace=%r>' % ( + self.__class__.__name__, self.path, self.mojom_namespace) + else: + return GenericRepr( + self, { + 'path': False, + 'mojom_namespace': False, + 'attributes': False, + 'structs': False, + 'interfaces': False, + 'unions': False + }) + + def GetNamespacePrefix(self): + return '%s.' % self.mojom_namespace if self.mojom_namespace else '' + + def AddInterface(self, mojom_name, attributes=None): + interface = Interface(mojom_name, self, attributes) + self.interfaces.append(interface) + return interface + + def AddStruct(self, mojom_name, attributes=None): + struct = Struct(mojom_name, self, attributes) + self.structs.append(struct) + return struct + + def AddUnion(self, mojom_name, attributes=None): + union = Union(mojom_name, self, attributes) + self.unions.append(union) + return union + + def Stylize(self, stylizer): + self.namespace = stylizer.StylizeModule(self.mojom_namespace) + for struct in self.structs: + struct.Stylize(stylizer) + for union in self.unions: + union.Stylize(stylizer) + for interface in self.interfaces: + interface.Stylize(stylizer) + for enum in self.enums: + enum.Stylize(stylizer) + for constant in self.constants: + constant.Stylize(stylizer) + + for imported_module in self.imports: + imported_module.Stylize(stylizer) + + def Dump(self, f): + pickle.dump(self, f, 2) + + @classmethod + def Load(cls, f): + result = pickle.load(f) + assert isinstance(result, Module) + return result + + +def IsBoolKind(kind): + return kind.spec == BOOL.spec + + +def IsFloatKind(kind): + return kind.spec == FLOAT.spec + + +def IsDoubleKind(kind): + return kind.spec == DOUBLE.spec + + +def IsIntegralKind(kind): + return (kind.spec == BOOL.spec or kind.spec == INT8.spec + or kind.spec == INT16.spec or kind.spec == INT32.spec + or kind.spec == INT64.spec or kind.spec == UINT8.spec + or kind.spec == UINT16.spec or kind.spec == UINT32.spec + or kind.spec == UINT64.spec) + + +def IsStringKind(kind): + return kind.spec == STRING.spec or kind.spec == NULLABLE_STRING.spec + + +def IsGenericHandleKind(kind): + return kind.spec == HANDLE.spec or kind.spec == NULLABLE_HANDLE.spec + + +def IsDataPipeConsumerKind(kind): + return kind.spec == DCPIPE.spec or kind.spec == NULLABLE_DCPIPE.spec + + +def IsDataPipeProducerKind(kind): + return kind.spec == DPPIPE.spec or kind.spec == NULLABLE_DPPIPE.spec + + +def IsMessagePipeKind(kind): + return kind.spec == MSGPIPE.spec or kind.spec == NULLABLE_MSGPIPE.spec + + +def IsSharedBufferKind(kind): + return (kind.spec == SHAREDBUFFER.spec + or kind.spec == NULLABLE_SHAREDBUFFER.spec) + + +def IsPlatformHandleKind(kind): + return (kind.spec == PLATFORMHANDLE.spec + or kind.spec == NULLABLE_PLATFORMHANDLE.spec) + + +def IsStructKind(kind): + return isinstance(kind, Struct) + + +def IsUnionKind(kind): + return isinstance(kind, Union) + + +def IsArrayKind(kind): + return isinstance(kind, Array) + + +def IsInterfaceKind(kind): + return isinstance(kind, Interface) + + +def IsAssociatedInterfaceKind(kind): + return isinstance(kind, AssociatedInterface) + + +def IsInterfaceRequestKind(kind): + return isinstance(kind, InterfaceRequest) + + +def IsAssociatedInterfaceRequestKind(kind): + return isinstance(kind, AssociatedInterfaceRequest) + + +def IsPendingRemoteKind(kind): + return isinstance(kind, PendingRemote) + + +def IsPendingReceiverKind(kind): + return isinstance(kind, PendingReceiver) + + +def IsPendingAssociatedRemoteKind(kind): + return isinstance(kind, PendingAssociatedRemote) + + +def IsPendingAssociatedReceiverKind(kind): + return isinstance(kind, PendingAssociatedReceiver) + + +def IsEnumKind(kind): + return isinstance(kind, Enum) + + +def IsReferenceKind(kind): + return isinstance(kind, ReferenceKind) + + +def IsNullableKind(kind): + return IsReferenceKind(kind) and kind.is_nullable + + +def IsMapKind(kind): + return isinstance(kind, Map) + + +def IsObjectKind(kind): + return IsPointerKind(kind) or IsUnionKind(kind) + + +def IsPointerKind(kind): + return (IsStructKind(kind) or IsArrayKind(kind) or IsStringKind(kind) + or IsMapKind(kind)) + + +# Please note that it doesn't include any interface kind. +def IsAnyHandleKind(kind): + return (IsGenericHandleKind(kind) or IsDataPipeConsumerKind(kind) + or IsDataPipeProducerKind(kind) or IsMessagePipeKind(kind) + or IsSharedBufferKind(kind) or IsPlatformHandleKind(kind)) + + +def IsAnyInterfaceKind(kind): + return (IsInterfaceKind(kind) or IsInterfaceRequestKind(kind) + or IsAssociatedKind(kind) or IsPendingRemoteKind(kind) + or IsPendingReceiverKind(kind)) + + +def IsAnyHandleOrInterfaceKind(kind): + return IsAnyHandleKind(kind) or IsAnyInterfaceKind(kind) + + +def IsAssociatedKind(kind): + return (IsAssociatedInterfaceKind(kind) + or IsAssociatedInterfaceRequestKind(kind) + or IsPendingAssociatedRemoteKind(kind) + or IsPendingAssociatedReceiverKind(kind)) + + +def HasCallbacks(interface): + for method in interface.methods: + if method.response_parameters != None: + return True + return False + + +# Finds out whether an interface passes associated interfaces and associated +# interface requests. +def PassesAssociatedKinds(interface): + visited_kinds = set() + for method in interface.methods: + if MethodPassesAssociatedKinds(method, visited_kinds): + return True + return False + + +def _AnyMethodParameterRecursive(method, predicate, visited_kinds=None): + def _HasProperty(kind): + if kind in visited_kinds: + # No need to examine the kind again. + return False + visited_kinds.add(kind) + if predicate(kind): + return True + if IsArrayKind(kind): + return _HasProperty(kind.kind) + if IsStructKind(kind) or IsUnionKind(kind): + for field in kind.fields: + if _HasProperty(field.kind): + return True + if IsMapKind(kind): + if _HasProperty(kind.key_kind) or _HasProperty(kind.value_kind): + return True + return False + + if visited_kinds is None: + visited_kinds = set() + + for param in method.parameters: + if _HasProperty(param.kind): + return True + if method.response_parameters != None: + for param in method.response_parameters: + if _HasProperty(param.kind): + return True + return False + + +# Finds out whether a method passes associated interfaces and associated +# interface requests. +def MethodPassesAssociatedKinds(method, visited_kinds=None): + return _AnyMethodParameterRecursive( + method, IsAssociatedKind, visited_kinds=visited_kinds) + + +# Determines whether a method passes interfaces. +def MethodPassesInterfaces(method): + return _AnyMethodParameterRecursive(method, IsInterfaceKind) + + +def HasSyncMethods(interface): + for method in interface.methods: + if method.sync: + return True + return False + + +def ContainsHandlesOrInterfaces(kind): + """Check if the kind contains any handles. + + This check is recursive so it checks all struct fields, containers elements, + etc. + + Args: + struct: {Kind} The kind to check. + + Returns: + {bool}: True if the kind contains handles. + """ + # We remember the types we already checked to avoid infinite recursion when + # checking recursive (or mutually recursive) types: + checked = set() + + def Check(kind): + if kind.spec in checked: + return False + checked.add(kind.spec) + if IsStructKind(kind): + return any(Check(field.kind) for field in kind.fields) + elif IsUnionKind(kind): + return any(Check(field.kind) for field in kind.fields) + elif IsAnyHandleKind(kind): + return True + elif IsAnyInterfaceKind(kind): + return True + elif IsArrayKind(kind): + return Check(kind.kind) + elif IsMapKind(kind): + return Check(kind.key_kind) or Check(kind.value_kind) + else: + return False + + return Check(kind) + + +def ContainsNativeTypes(kind): + """Check if the kind contains any native type (struct or enum). + + This check is recursive so it checks all struct fields, scoped interface + enums, etc. + + Args: + struct: {Kind} The kind to check. + + Returns: + {bool}: True if the kind contains native types. + """ + # We remember the types we already checked to avoid infinite recursion when + # checking recursive (or mutually recursive) types: + checked = set() + + def Check(kind): + if kind.spec in checked: + return False + checked.add(kind.spec) + if IsEnumKind(kind): + return kind.native_only + elif IsStructKind(kind): + if kind.native_only: + return True + if any(enum.native_only for enum in kind.enums): + return True + return any(Check(field.kind) for field in kind.fields) + elif IsUnionKind(kind): + return any(Check(field.kind) for field in kind.fields) + elif IsInterfaceKind(kind): + return any(enum.native_only for enum in kind.enums) + elif IsArrayKind(kind): + return Check(kind.kind) + elif IsMapKind(kind): + return Check(kind.key_kind) or Check(kind.value_kind) + else: + return False + + return Check(kind) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py new file mode 100644 index 00000000..e8fd4936 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py @@ -0,0 +1,31 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import sys +import unittest + +from mojom.generate import module as mojom + + +class ModuleTest(unittest.TestCase): + def testNonInterfaceAsInterfaceRequest(self): + """Tests that a non-interface cannot be used for interface requests.""" + module = mojom.Module('test_module', 'test_namespace') + struct = mojom.Struct('TestStruct', module=module) + with self.assertRaises(Exception) as e: + mojom.InterfaceRequest(struct) + self.assertEquals( + e.exception.__str__(), + 'Interface request requires \'x:TestStruct\' to be an interface.') + + def testNonInterfaceAsAssociatedInterface(self): + """Tests that a non-interface type cannot be used for associated interfaces. + """ + module = mojom.Module('test_module', 'test_namespace') + struct = mojom.Struct('TestStruct', module=module) + with self.assertRaises(Exception) as e: + mojom.AssociatedInterface(struct) + self.assertEquals( + e.exception.__str__(), + 'Associated interface requires \'x:TestStruct\' to be an interface.') diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py new file mode 100644 index 00000000..88b77c98 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py @@ -0,0 +1,258 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom.generate import module as mojom + +# This module provides a mechanism for determining the packed order and offsets +# of a mojom.Struct. +# +# ps = pack.PackedStruct(struct) +# ps.packed_fields will access a list of PackedField objects, each of which +# will have an offset, a size and a bit (for mojom.BOOLs). + +# Size of struct header in bytes: num_bytes [4B] + version [4B]. +HEADER_SIZE = 8 + + +class PackedField(object): + kind_to_size = { + mojom.BOOL: 1, + mojom.INT8: 1, + mojom.UINT8: 1, + mojom.INT16: 2, + mojom.UINT16: 2, + mojom.INT32: 4, + mojom.UINT32: 4, + mojom.FLOAT: 4, + mojom.HANDLE: 4, + mojom.MSGPIPE: 4, + mojom.SHAREDBUFFER: 4, + mojom.PLATFORMHANDLE: 4, + mojom.DCPIPE: 4, + mojom.DPPIPE: 4, + mojom.NULLABLE_HANDLE: 4, + mojom.NULLABLE_MSGPIPE: 4, + mojom.NULLABLE_SHAREDBUFFER: 4, + mojom.NULLABLE_PLATFORMHANDLE: 4, + mojom.NULLABLE_DCPIPE: 4, + mojom.NULLABLE_DPPIPE: 4, + mojom.INT64: 8, + mojom.UINT64: 8, + mojom.DOUBLE: 8, + mojom.STRING: 8, + mojom.NULLABLE_STRING: 8 + } + + @classmethod + def GetSizeForKind(cls, kind): + if isinstance(kind, (mojom.Array, mojom.Map, mojom.Struct, mojom.Interface, + mojom.AssociatedInterface, mojom.PendingRemote, + mojom.PendingAssociatedRemote)): + return 8 + if isinstance(kind, mojom.Union): + return 16 + if isinstance(kind, (mojom.InterfaceRequest, mojom.PendingReceiver)): + kind = mojom.MSGPIPE + if isinstance( + kind, + (mojom.AssociatedInterfaceRequest, mojom.PendingAssociatedReceiver)): + return 4 + if isinstance(kind, mojom.Enum): + # TODO(mpcomplete): what about big enums? + return cls.kind_to_size[mojom.INT32] + if not kind in cls.kind_to_size: + raise Exception("Undefined type: %s. Did you forget to import the file " + "containing the definition?" % kind.spec) + return cls.kind_to_size[kind] + + @classmethod + def GetAlignmentForKind(cls, kind): + if isinstance(kind, (mojom.Interface, mojom.AssociatedInterface, + mojom.PendingRemote, mojom.PendingAssociatedRemote)): + return 4 + if isinstance(kind, mojom.Union): + return 8 + return cls.GetSizeForKind(kind) + + def __init__(self, field, index, ordinal): + """ + Args: + field: the original field. + index: the position of the original field in the struct. + ordinal: the ordinal of the field for serialization. + """ + self.field = field + self.index = index + self.ordinal = ordinal + self.size = self.GetSizeForKind(field.kind) + self.alignment = self.GetAlignmentForKind(field.kind) + self.offset = None + self.bit = None + self.min_version = None + + +def GetPad(offset, alignment): + """Returns the pad necessary to reserve space so that |offset + pad| equals to + some multiple of |alignment|.""" + return (alignment - (offset % alignment)) % alignment + + +def GetFieldOffset(field, last_field): + """Returns a 2-tuple of the field offset and bit (for BOOLs).""" + if (field.field.kind == mojom.BOOL and last_field.field.kind == mojom.BOOL + and last_field.bit < 7): + return (last_field.offset, last_field.bit + 1) + + offset = last_field.offset + last_field.size + pad = GetPad(offset, field.alignment) + return (offset + pad, 0) + + +def GetPayloadSizeUpToField(field): + """Returns the payload size (not including struct header) if |field| is the + last field. + """ + if not field: + return 0 + offset = field.offset + field.size + pad = GetPad(offset, 8) + return offset + pad + + +class PackedStruct(object): + def __init__(self, struct): + self.struct = struct + # |packed_fields| contains all the fields, in increasing offset order. + self.packed_fields = [] + # |packed_fields_in_ordinal_order| refers to the same fields as + # |packed_fields|, but in ordinal order. + self.packed_fields_in_ordinal_order = [] + + # No fields. + if (len(struct.fields) == 0): + return + + # Start by sorting by ordinal. + src_fields = self.packed_fields_in_ordinal_order + ordinal = 0 + for index, field in enumerate(struct.fields): + if field.ordinal is not None: + ordinal = field.ordinal + src_fields.append(PackedField(field, index, ordinal)) + ordinal += 1 + src_fields.sort(key=lambda field: field.ordinal) + + # Set |min_version| for each field. + next_min_version = 0 + for packed_field in src_fields: + if packed_field.field.min_version is None: + assert next_min_version == 0 + else: + assert packed_field.field.min_version >= next_min_version + next_min_version = packed_field.field.min_version + packed_field.min_version = next_min_version + + if (packed_field.min_version != 0 + and mojom.IsReferenceKind(packed_field.field.kind) + and not packed_field.field.kind.is_nullable): + raise Exception("Non-nullable fields are only allowed in version 0 of " + "a struct. %s.%s is defined with [MinVersion=%d]." % + (self.struct.name, packed_field.field.name, + packed_field.min_version)) + + src_field = src_fields[0] + src_field.offset = 0 + src_field.bit = 0 + dst_fields = self.packed_fields + dst_fields.append(src_field) + + # Then find first slot that each field will fit. + for src_field in src_fields[1:]: + last_field = dst_fields[0] + for i in range(1, len(dst_fields)): + next_field = dst_fields[i] + offset, bit = GetFieldOffset(src_field, last_field) + if offset + src_field.size <= next_field.offset: + # Found hole. + src_field.offset = offset + src_field.bit = bit + dst_fields.insert(i, src_field) + break + last_field = next_field + if src_field.offset is None: + # Add to end + src_field.offset, src_field.bit = GetFieldOffset(src_field, last_field) + dst_fields.append(src_field) + + +class ByteInfo(object): + def __init__(self): + self.is_padding = False + self.packed_fields = [] + + +def GetByteLayout(packed_struct): + total_payload_size = GetPayloadSizeUpToField( + packed_struct.packed_fields[-1] if packed_struct.packed_fields else None) + byte_info = [ByteInfo() for i in range(total_payload_size)] + + limit_of_previous_field = 0 + for packed_field in packed_struct.packed_fields: + for i in range(limit_of_previous_field, packed_field.offset): + byte_info[i].is_padding = True + byte_info[packed_field.offset].packed_fields.append(packed_field) + limit_of_previous_field = packed_field.offset + packed_field.size + + for i in range(limit_of_previous_field, len(byte_info)): + byte_info[i].is_padding = True + + for byte in byte_info: + # A given byte cannot both be padding and have a fields packed into it. + assert not (byte.is_padding and byte.packed_fields) + + return byte_info + + +class VersionInfo(object): + def __init__(self, version, num_fields, num_bytes): + self.version = version + self.num_fields = num_fields + self.num_bytes = num_bytes + + +def GetVersionInfo(packed_struct): + """Get version information for a struct. + + Args: + packed_struct: A PackedStruct instance. + + Returns: + A non-empty list of VersionInfo instances, sorted by version in increasing + order. + Note: The version numbers may not be consecutive. + """ + versions = [] + last_version = 0 + last_num_fields = 0 + last_payload_size = 0 + + for packed_field in packed_struct.packed_fields_in_ordinal_order: + if packed_field.min_version != last_version: + versions.append( + VersionInfo(last_version, last_num_fields, + last_payload_size + HEADER_SIZE)) + last_version = packed_field.min_version + + last_num_fields += 1 + # The fields are iterated in ordinal order here. However, the size of a + # version is determined by the last field of that version in pack order, + # instead of ordinal order. Therefore, we need to calculate the max value. + last_payload_size = max( + GetPayloadSizeUpToField(packed_field), last_payload_size) + + assert len(versions) == 0 or last_num_fields != versions[-1].num_fields + versions.append( + VersionInfo(last_version, last_num_fields, + last_payload_size + HEADER_SIZE)) + return versions diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py new file mode 100644 index 00000000..98c705ad --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py @@ -0,0 +1,225 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import sys +import unittest + +from mojom.generate import module as mojom +from mojom.generate import pack + + +class PackTest(unittest.TestCase): + def testOrdinalOrder(self): + struct = mojom.Struct('test') + struct.AddField('testfield1', mojom.INT32, 2) + struct.AddField('testfield2', mojom.INT32, 1) + ps = pack.PackedStruct(struct) + + self.assertEqual(2, len(ps.packed_fields)) + self.assertEqual('testfield2', ps.packed_fields[0].field.mojom_name) + self.assertEqual('testfield1', ps.packed_fields[1].field.mojom_name) + + def testZeroFields(self): + struct = mojom.Struct('test') + ps = pack.PackedStruct(struct) + self.assertEqual(0, len(ps.packed_fields)) + + def testOneField(self): + struct = mojom.Struct('test') + struct.AddField('testfield1', mojom.INT8) + ps = pack.PackedStruct(struct) + self.assertEqual(1, len(ps.packed_fields)) + + def _CheckPackSequence(self, kinds, fields, offsets): + """Checks the pack order and offsets of a sequence of mojom.Kinds. + + Args: + kinds: A sequence of mojom.Kinds that specify the fields that are to be + created. + fields: The expected order of the resulting fields, with the integer "1" + first. + offsets: The expected order of offsets, with the integer "0" first. + """ + struct = mojom.Struct('test') + index = 1 + for kind in kinds: + struct.AddField('%d' % index, kind) + index += 1 + ps = pack.PackedStruct(struct) + num_fields = len(ps.packed_fields) + self.assertEqual(len(kinds), num_fields) + for i in range(num_fields): + self.assertEqual('%d' % fields[i], ps.packed_fields[i].field.mojom_name) + self.assertEqual(offsets[i], ps.packed_fields[i].offset) + + def testPaddingPackedInOrder(self): + return self._CheckPackSequence((mojom.INT8, mojom.UINT8, mojom.INT32), + (1, 2, 3), (0, 1, 4)) + + def testPaddingPackedOutOfOrder(self): + return self._CheckPackSequence((mojom.INT8, mojom.INT32, mojom.UINT8), + (1, 3, 2), (0, 1, 4)) + + def testPaddingPackedOverflow(self): + kinds = (mojom.INT8, mojom.INT32, mojom.INT16, mojom.INT8, mojom.INT8) + # 2 bytes should be packed together first, followed by short, then by int. + fields = (1, 4, 3, 2, 5) + offsets = (0, 1, 2, 4, 8) + return self._CheckPackSequence(kinds, fields, offsets) + + def testNullableTypes(self): + kinds = (mojom.STRING.MakeNullableKind(), mojom.HANDLE.MakeNullableKind(), + mojom.Struct('test_struct').MakeNullableKind(), + mojom.DCPIPE.MakeNullableKind(), mojom.Array().MakeNullableKind(), + mojom.DPPIPE.MakeNullableKind(), + mojom.Array(length=5).MakeNullableKind(), + mojom.MSGPIPE.MakeNullableKind(), + mojom.Interface('test_interface').MakeNullableKind(), + mojom.SHAREDBUFFER.MakeNullableKind(), + mojom.InterfaceRequest().MakeNullableKind()) + fields = (1, 2, 4, 3, 5, 6, 8, 7, 9, 10, 11) + offsets = (0, 8, 12, 16, 24, 32, 36, 40, 48, 56, 60) + return self._CheckPackSequence(kinds, fields, offsets) + + def testAllTypes(self): + return self._CheckPackSequence( + (mojom.BOOL, mojom.INT8, mojom.STRING, mojom.UINT8, mojom.INT16, + mojom.DOUBLE, mojom.UINT16, mojom.INT32, mojom.UINT32, mojom.INT64, + mojom.FLOAT, mojom.STRING, mojom.HANDLE, mojom.UINT64, + mojom.Struct('test'), mojom.Array(), mojom.STRING.MakeNullableKind()), + (1, 2, 4, 5, 7, 3, 6, 8, 9, 10, 11, 13, 12, 14, 15, 16, 17, 18), + (0, 1, 2, 4, 6, 8, 16, 24, 28, 32, 40, 44, 48, 56, 64, 72, 80, 88)) + + def testPaddingPackedOutOfOrderByOrdinal(self): + struct = mojom.Struct('test') + struct.AddField('testfield1', mojom.INT8) + struct.AddField('testfield3', mojom.UINT8, 3) + struct.AddField('testfield2', mojom.INT32, 2) + ps = pack.PackedStruct(struct) + self.assertEqual(3, len(ps.packed_fields)) + + # Second byte should be packed in behind first, altering order. + self.assertEqual('testfield1', ps.packed_fields[0].field.mojom_name) + self.assertEqual('testfield3', ps.packed_fields[1].field.mojom_name) + self.assertEqual('testfield2', ps.packed_fields[2].field.mojom_name) + + # Second byte should be packed with first. + self.assertEqual(0, ps.packed_fields[0].offset) + self.assertEqual(1, ps.packed_fields[1].offset) + self.assertEqual(4, ps.packed_fields[2].offset) + + def testBools(self): + struct = mojom.Struct('test') + struct.AddField('bit0', mojom.BOOL) + struct.AddField('bit1', mojom.BOOL) + struct.AddField('int', mojom.INT32) + struct.AddField('bit2', mojom.BOOL) + struct.AddField('bit3', mojom.BOOL) + struct.AddField('bit4', mojom.BOOL) + struct.AddField('bit5', mojom.BOOL) + struct.AddField('bit6', mojom.BOOL) + struct.AddField('bit7', mojom.BOOL) + struct.AddField('bit8', mojom.BOOL) + ps = pack.PackedStruct(struct) + self.assertEqual(10, len(ps.packed_fields)) + + # First 8 bits packed together. + for i in range(8): + pf = ps.packed_fields[i] + self.assertEqual(0, pf.offset) + self.assertEqual("bit%d" % i, pf.field.mojom_name) + self.assertEqual(i, pf.bit) + + # Ninth bit goes into second byte. + self.assertEqual("bit8", ps.packed_fields[8].field.mojom_name) + self.assertEqual(1, ps.packed_fields[8].offset) + self.assertEqual(0, ps.packed_fields[8].bit) + + # int comes last. + self.assertEqual("int", ps.packed_fields[9].field.mojom_name) + self.assertEqual(4, ps.packed_fields[9].offset) + + def testMinVersion(self): + """Tests that |min_version| is properly set for packed fields.""" + struct = mojom.Struct('test') + struct.AddField('field_2', mojom.BOOL, 2) + struct.AddField('field_0', mojom.INT32, 0) + struct.AddField('field_1', mojom.INT64, 1) + ps = pack.PackedStruct(struct) + + self.assertEqual('field_0', ps.packed_fields[0].field.mojom_name) + self.assertEqual('field_2', ps.packed_fields[1].field.mojom_name) + self.assertEqual('field_1', ps.packed_fields[2].field.mojom_name) + + self.assertEqual(0, ps.packed_fields[0].min_version) + self.assertEqual(0, ps.packed_fields[1].min_version) + self.assertEqual(0, ps.packed_fields[2].min_version) + + struct.fields[0].attributes = {'MinVersion': 1} + ps = pack.PackedStruct(struct) + + self.assertEqual(0, ps.packed_fields[0].min_version) + self.assertEqual(1, ps.packed_fields[1].min_version) + self.assertEqual(0, ps.packed_fields[2].min_version) + + def testGetVersionInfoEmptyStruct(self): + """Tests that pack.GetVersionInfo() never returns an empty list, even for + empty structs. + """ + struct = mojom.Struct('test') + ps = pack.PackedStruct(struct) + + versions = pack.GetVersionInfo(ps) + self.assertEqual(1, len(versions)) + self.assertEqual(0, versions[0].version) + self.assertEqual(0, versions[0].num_fields) + self.assertEqual(8, versions[0].num_bytes) + + def testGetVersionInfoComplexOrder(self): + """Tests pack.GetVersionInfo() using a struct whose definition order, + ordinal order and pack order for fields are all different. + """ + struct = mojom.Struct('test') + struct.AddField( + 'field_3', mojom.BOOL, ordinal=3, attributes={'MinVersion': 3}) + struct.AddField('field_0', mojom.INT32, ordinal=0) + struct.AddField( + 'field_1', mojom.INT64, ordinal=1, attributes={'MinVersion': 2}) + struct.AddField( + 'field_2', mojom.INT64, ordinal=2, attributes={'MinVersion': 3}) + ps = pack.PackedStruct(struct) + + versions = pack.GetVersionInfo(ps) + self.assertEqual(3, len(versions)) + + self.assertEqual(0, versions[0].version) + self.assertEqual(1, versions[0].num_fields) + self.assertEqual(16, versions[0].num_bytes) + + self.assertEqual(2, versions[1].version) + self.assertEqual(2, versions[1].num_fields) + self.assertEqual(24, versions[1].num_bytes) + + self.assertEqual(3, versions[2].version) + self.assertEqual(4, versions[2].num_fields) + self.assertEqual(32, versions[2].num_bytes) + + def testInterfaceAlignment(self): + """Tests that interfaces are aligned on 4-byte boundaries, although the size + of an interface is 8 bytes. + """ + kinds = (mojom.INT32, mojom.Interface('test_interface')) + fields = (1, 2) + offsets = (0, 4) + self._CheckPackSequence(kinds, fields, offsets) + + def testAssociatedInterfaceAlignment(self): + """Tests that associated interfaces are aligned on 4-byte boundaries, + although the size of an associated interface is 8 bytes. + """ + kinds = (mojom.INT32, + mojom.AssociatedInterface(mojom.Interface('test_interface'))) + fields = (1, 2) + offsets = (0, 4) + self._CheckPackSequence(kinds, fields, offsets) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py new file mode 100644 index 00000000..7a300560 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py @@ -0,0 +1,83 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +# Based on third_party/WebKit/Source/build/scripts/template_expander.py. + +import os.path +import sys + +from mojom import fileutil + +fileutil.AddLocalRepoThirdPartyDirToModulePath() +import jinja2 + + +def ApplyTemplate(mojo_generator, path_to_template, params, **kwargs): + loader = jinja2.ModuleLoader( + os.path.join(mojo_generator.bytecode_path, + "%s.zip" % mojo_generator.GetTemplatePrefix())) + final_kwargs = dict(mojo_generator.GetJinjaParameters()) + final_kwargs.update(kwargs) + + jinja_env = jinja2.Environment( + loader=loader, keep_trailing_newline=True, **final_kwargs) + jinja_env.globals.update(mojo_generator.GetGlobals()) + jinja_env.filters.update(mojo_generator.GetFilters()) + template = jinja_env.get_template(path_to_template) + return template.render(params) + + +def UseJinja(path_to_template, **kwargs): + def RealDecorator(generator): + def GeneratorInternal(*args, **kwargs2): + parameters = generator(*args, **kwargs2) + return ApplyTemplate(args[0], path_to_template, parameters, **kwargs) + + GeneratorInternal.__name__ = generator.__name__ + return GeneratorInternal + + return RealDecorator + + +def ApplyImportedTemplate(mojo_generator, path_to_template, filename, params, + **kwargs): + loader = jinja2.FileSystemLoader(searchpath=path_to_template) + final_kwargs = dict(mojo_generator.GetJinjaParameters()) + final_kwargs.update(kwargs) + + jinja_env = jinja2.Environment( + loader=loader, keep_trailing_newline=True, **final_kwargs) + jinja_env.globals.update(mojo_generator.GetGlobals()) + jinja_env.filters.update(mojo_generator.GetFilters()) + template = jinja_env.get_template(filename) + return template.render(params) + + +def UseJinjaForImportedTemplate(func): + def wrapper(*args, **kwargs): + parameters = func(*args, **kwargs) + path_to_template = args[1] + filename = args[2] + return ApplyImportedTemplate(args[0], path_to_template, filename, + parameters) + + wrapper.__name__ = func.__name__ + return wrapper + + +def PrecompileTemplates(generator_modules, output_dir): + for module in generator_modules.values(): + generator = module.Generator(None) + jinja_env = jinja2.Environment( + loader=jinja2.FileSystemLoader([ + os.path.join( + os.path.dirname(module.__file__), generator.GetTemplatePrefix()) + ])) + jinja_env.filters.update(generator.GetFilters()) + jinja_env.compile_templates( + os.path.join(output_dir, "%s.zip" % generator.GetTemplatePrefix()), + extensions=["tmpl"], + zip="stored", + py_compile=True, + ignore_errors=False) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py new file mode 100644 index 00000000..d6df3ca6 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py @@ -0,0 +1,854 @@ +# Copyright 2013 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Convert parse tree to AST. + +This module converts the parse tree to the AST we use for code generation. The +main entry point is OrderedModule, which gets passed the parser +representation of a mojom file. When called it's assumed that all imports have +already been parsed and converted to ASTs before. +""" + +import itertools +import os +import re +import sys + +from mojom.generate import generator +from mojom.generate import module as mojom +from mojom.parse import ast + + +def _IsStrOrUnicode(x): + if sys.version_info[0] < 3: + return isinstance(x, (unicode, str)) + return isinstance(x, str) + + +def _DuplicateName(values): + """Returns the 'mojom_name' of the first entry in |values| whose 'mojom_name' + has already been encountered. If there are no duplicates, returns None.""" + names = set() + for value in values: + if value.mojom_name in names: + return value.mojom_name + names.add(value.mojom_name) + return None + + +def _ElemsOfType(elems, elem_type, scope): + """Find all elements of the given type. + + Args: + elems: {Sequence[Any]} Sequence of elems. + elem_type: {Type[C]} Extract all elems of this type. + scope: {str} The name of the surrounding scope (e.g. struct + definition). Used in error messages. + + Returns: + {List[C]} All elems of matching type. + """ + assert isinstance(elem_type, type) + result = [elem for elem in elems if isinstance(elem, elem_type)] + duplicate_name = _DuplicateName(result) + if duplicate_name: + raise Exception('Names in mojom must be unique within a scope. The name ' + '"%s" is used more than once within the scope "%s".' % + (duplicate_name, scope)) + return result + + +def _ProcessElements(scope, elements, operations_by_type): + """Iterates over the given elements, running a function from + operations_by_type for any element that matches a key in that dict. The scope + is the name of the surrounding scope, such as a filename or struct name, used + only in error messages.""" + names_in_this_scope = set() + for element in elements: + # pylint: disable=unidiomatic-typecheck + element_type = type(element) + if element_type in operations_by_type: + if element.mojom_name in names_in_this_scope: + raise Exception('Names must be unique within a scope. The name "%s" is ' + 'used more than once within the scope "%s".' % + (duplicate_name, scope)) + operations_by_type[element_type](element) + + +def _MapKind(kind): + map_to_kind = { + 'bool': 'b', + 'int8': 'i8', + 'int16': 'i16', + 'int32': 'i32', + 'int64': 'i64', + 'uint8': 'u8', + 'uint16': 'u16', + 'uint32': 'u32', + 'uint64': 'u64', + 'float': 'f', + 'double': 'd', + 'string': 's', + 'handle': 'h', + 'handle': 'h:d:c', + 'handle': 'h:d:p', + 'handle': 'h:m', + 'handle': 'h:s', + 'handle': 'h:p' + } + if kind.endswith('?'): + base_kind = _MapKind(kind[0:-1]) + # NOTE: This doesn't rule out enum types. Those will be detected later, when + # cross-reference is established. + reference_kinds = ('m', 's', 'h', 'a', 'r', 'x', 'asso', 'rmt', 'rcv', + 'rma', 'rca') + if re.split('[^a-z]', base_kind, 1)[0] not in reference_kinds: + raise Exception('A type (spec "%s") cannot be made nullable' % base_kind) + return '?' + base_kind + if kind.endswith('}'): + lbracket = kind.rfind('{') + value = kind[0:lbracket] + return 'm[' + _MapKind(kind[lbracket + 1:-1]) + '][' + _MapKind(value) + ']' + if kind.endswith(']'): + lbracket = kind.rfind('[') + typename = kind[0:lbracket] + return 'a' + kind[lbracket + 1:-1] + ':' + _MapKind(typename) + if kind.endswith('&'): + return 'r:' + _MapKind(kind[0:-1]) + if kind.startswith('asso<'): + assert kind.endswith('>') + return 'asso:' + _MapKind(kind[5:-1]) + if kind.startswith('rmt<'): + assert kind.endswith('>') + return 'rmt:' + _MapKind(kind[4:-1]) + if kind.startswith('rcv<'): + assert kind.endswith('>') + return 'rcv:' + _MapKind(kind[4:-1]) + if kind.startswith('rma<'): + assert kind.endswith('>') + return 'rma:' + _MapKind(kind[4:-1]) + if kind.startswith('rca<'): + assert kind.endswith('>') + return 'rca:' + _MapKind(kind[4:-1]) + if kind in map_to_kind: + return map_to_kind[kind] + return 'x:' + kind + + +def _AttributeListToDict(attribute_list): + if attribute_list is None: + return None + assert isinstance(attribute_list, ast.AttributeList) + # TODO(vtl): Check for duplicate keys here. + return dict( + [(attribute.key, attribute.value) for attribute in attribute_list]) + + +builtin_values = frozenset([ + "double.INFINITY", "double.NEGATIVE_INFINITY", "double.NAN", + "float.INFINITY", "float.NEGATIVE_INFINITY", "float.NAN" +]) + + +def _IsBuiltinValue(value): + return value in builtin_values + + +def _LookupKind(kinds, spec, scope): + """Tries to find which Kind a spec refers to, given the scope in which its + referenced. Starts checking from the narrowest scope to most general. For + example, given a struct field like + Foo.Bar x; + Foo.Bar could refer to the type 'Bar' in the 'Foo' namespace, or an inner + type 'Bar' in the struct 'Foo' in the current namespace. + + |scope| is a tuple that looks like (namespace, struct/interface), referring + to the location where the type is referenced.""" + if spec.startswith('x:'): + mojom_name = spec[2:] + for i in range(len(scope), -1, -1): + test_spec = 'x:' + if i > 0: + test_spec += '.'.join(scope[:i]) + '.' + test_spec += mojom_name + kind = kinds.get(test_spec) + if kind: + return kind + + return kinds.get(spec) + + +def _GetScopeForKind(module, kind): + """For a given kind, returns a tuple of progressively more specific names + used to qualify the kind. For example if kind is an enum named Bar nested in a + struct Foo within module 'foo', this would return ('foo', 'Foo', 'Bar')""" + if isinstance(kind, mojom.Enum) and kind.parent_kind: + # Enums may be nested in other kinds. + return _GetScopeForKind(module, kind.parent_kind) + (kind.mojom_name, ) + + module_fragment = (module.mojom_namespace, ) if module.mojom_namespace else () + kind_fragment = (kind.mojom_name, ) if kind else () + return module_fragment + kind_fragment + + +def _LookupValueInScope(module, kind, identifier): + """Given a kind and an identifier, this attempts to resolve the given + identifier to a concrete NamedValue within the scope of the given kind.""" + scope = _GetScopeForKind(module, kind) + for i in reversed(range(len(scope) + 1)): + qualified_name = '.'.join(scope[:i] + (identifier, )) + value = module.values.get(qualified_name) + if value: + return value + return None + + +def _LookupValue(module, parent_kind, implied_kind, ast_leaf_node): + """Resolves a leaf node in the form ('IDENTIFIER', 'x') to a constant value + identified by 'x' in some mojom definition. parent_kind is used as context + when resolving the identifier. If the given leaf node is not an IDENTIFIER + (e.g. already a constant value), it is returned as-is. + + If implied_kind is provided, the parsed identifier may also be resolved within + its scope as fallback. This can be useful for more concise value references + when assigning enum-typed constants or field values.""" + if not isinstance(ast_leaf_node, tuple) or ast_leaf_node[0] != 'IDENTIFIER': + return ast_leaf_node + + # First look for a known user-defined identifier to resolve this within the + # enclosing scope. + identifier = ast_leaf_node[1] + + value = _LookupValueInScope(module, parent_kind, identifier) + if value: + return value + + # Next look in the scope of implied_kind, if provided. + value = (implied_kind and implied_kind.module and _LookupValueInScope( + implied_kind.module, implied_kind, identifier)) + if value: + return value + + # Fall back on defined builtin symbols + if _IsBuiltinValue(identifier): + return mojom.BuiltinValue(identifier) + + raise ValueError('Unknown identifier %s' % identifier) + + +def _Kind(kinds, spec, scope): + """Convert a type name into a mojom.Kind object. + + As a side-effect this function adds the result to 'kinds'. + + Args: + kinds: {Dict[str, mojom.Kind]} All known kinds up to this point, indexed by + their names. + spec: {str} A name uniquely identifying a type. + scope: {Tuple[str, str]} A tuple that looks like (namespace, + struct/interface), referring to the location where the type is + referenced. + + Returns: + {mojom.Kind} The type corresponding to 'spec'. + """ + kind = _LookupKind(kinds, spec, scope) + if kind: + return kind + + if spec.startswith('?'): + kind = _Kind(kinds, spec[1:], scope).MakeNullableKind() + elif spec.startswith('a:'): + kind = mojom.Array(_Kind(kinds, spec[2:], scope)) + elif spec.startswith('asso:'): + inner_kind = _Kind(kinds, spec[5:], scope) + if isinstance(inner_kind, mojom.InterfaceRequest): + kind = mojom.AssociatedInterfaceRequest(inner_kind) + else: + kind = mojom.AssociatedInterface(inner_kind) + elif spec.startswith('a'): + colon = spec.find(':') + length = int(spec[1:colon]) + kind = mojom.Array(_Kind(kinds, spec[colon + 1:], scope), length) + elif spec.startswith('r:'): + kind = mojom.InterfaceRequest(_Kind(kinds, spec[2:], scope)) + elif spec.startswith('rmt:'): + kind = mojom.PendingRemote(_Kind(kinds, spec[4:], scope)) + elif spec.startswith('rcv:'): + kind = mojom.PendingReceiver(_Kind(kinds, spec[4:], scope)) + elif spec.startswith('rma:'): + kind = mojom.PendingAssociatedRemote(_Kind(kinds, spec[4:], scope)) + elif spec.startswith('rca:'): + kind = mojom.PendingAssociatedReceiver(_Kind(kinds, spec[4:], scope)) + elif spec.startswith('m['): + # Isolate the two types from their brackets. + + # It is not allowed to use map as key, so there shouldn't be nested ']'s + # inside the key type spec. + key_end = spec.find(']') + assert key_end != -1 and key_end < len(spec) - 1 + assert spec[key_end + 1] == '[' and spec[-1] == ']' + + first_kind = spec[2:key_end] + second_kind = spec[key_end + 2:-1] + + kind = mojom.Map( + _Kind(kinds, first_kind, scope), _Kind(kinds, second_kind, scope)) + else: + kind = mojom.Kind(spec) + + kinds[spec] = kind + return kind + + +def _Import(module, import_module): + # Copy the struct kinds from our imports into the current module. + importable_kinds = (mojom.Struct, mojom.Union, mojom.Enum, mojom.Interface) + for kind in import_module.kinds.values(): + if (isinstance(kind, importable_kinds) + and kind.module.path == import_module.path): + module.kinds[kind.spec] = kind + # Ditto for values. + for value in import_module.values.values(): + if value.module.path == import_module.path: + module.values[value.GetSpec()] = value + + return import_module + + +def _Struct(module, parsed_struct): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_struct: {ast.Struct} Parsed struct. + + Returns: + {mojom.Struct} AST struct. + """ + struct = mojom.Struct(module=module) + struct.mojom_name = parsed_struct.mojom_name + struct.native_only = parsed_struct.body is None + struct.spec = 'x:' + module.GetNamespacePrefix() + struct.mojom_name + module.kinds[struct.spec] = struct + struct.enums = [] + struct.constants = [] + struct.fields_data = [] + if not struct.native_only: + _ProcessElements( + parsed_struct.mojom_name, parsed_struct.body, { + ast.Enum: + lambda enum: struct.enums.append(_Enum(module, enum, struct)), + ast.Const: + lambda const: struct.constants.append( + _Constant(module, const, struct)), + ast.StructField: + struct.fields_data.append, + }) + + struct.attributes = _AttributeListToDict(parsed_struct.attribute_list) + + # Enforce that a [Native] attribute is set to make native-only struct + # declarations more explicit. + if struct.native_only: + if not struct.attributes or not struct.attributes.get('Native', False): + raise Exception("Native-only struct declarations must include a " + + "Native attribute.") + + if struct.attributes and struct.attributes.get('CustomSerializer', False): + struct.custom_serializer = True + + return struct + + +def _Union(module, parsed_union): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_union: {ast.Union} Parsed union. + + Returns: + {mojom.Union} AST union. + """ + union = mojom.Union(module=module) + union.mojom_name = parsed_union.mojom_name + union.spec = 'x:' + module.GetNamespacePrefix() + union.mojom_name + module.kinds[union.spec] = union + # Stash fields parsed_union here temporarily. + union.fields_data = [] + _ProcessElements(parsed_union.mojom_name, parsed_union.body, + {ast.UnionField: union.fields_data.append}) + union.attributes = _AttributeListToDict(parsed_union.attribute_list) + return union + + +def _StructField(module, parsed_field, struct): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_field: {ast.StructField} Parsed struct field. + struct: {mojom.Struct} Struct this field belongs to. + + Returns: + {mojom.StructField} AST struct field. + """ + field = mojom.StructField() + field.mojom_name = parsed_field.mojom_name + field.kind = _Kind(module.kinds, _MapKind(parsed_field.typename), + (module.mojom_namespace, struct.mojom_name)) + field.ordinal = parsed_field.ordinal.value if parsed_field.ordinal else None + field.default = _LookupValue(module, struct, field.kind, + parsed_field.default_value) + field.attributes = _AttributeListToDict(parsed_field.attribute_list) + return field + + +def _UnionField(module, parsed_field, union): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_field: {ast.UnionField} Parsed union field. + union: {mojom.Union} Union this fields belong to. + + Returns: + {mojom.UnionField} AST union. + """ + field = mojom.UnionField() + field.mojom_name = parsed_field.mojom_name + field.kind = _Kind(module.kinds, _MapKind(parsed_field.typename), + (module.mojom_namespace, union.mojom_name)) + field.ordinal = parsed_field.ordinal.value if parsed_field.ordinal else None + field.default = None + field.attributes = _AttributeListToDict(parsed_field.attribute_list) + return field + + +def _Parameter(module, parsed_param, interface): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_param: {ast.Parameter} Parsed parameter. + union: {mojom.Interface} Interface this parameter belongs to. + + Returns: + {mojom.Parameter} AST parameter. + """ + parameter = mojom.Parameter() + parameter.mojom_name = parsed_param.mojom_name + parameter.kind = _Kind(module.kinds, _MapKind(parsed_param.typename), + (module.mojom_namespace, interface.mojom_name)) + parameter.ordinal = (parsed_param.ordinal.value + if parsed_param.ordinal else None) + parameter.default = None # TODO(tibell): We never have these. Remove field? + parameter.attributes = _AttributeListToDict(parsed_param.attribute_list) + return parameter + + +def _Method(module, parsed_method, interface): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_method: {ast.Method} Parsed method. + interface: {mojom.Interface} Interface this method belongs to. + + Returns: + {mojom.Method} AST method. + """ + method = mojom.Method( + interface, + parsed_method.mojom_name, + ordinal=parsed_method.ordinal.value if parsed_method.ordinal else None) + method.parameters = list( + map(lambda parameter: _Parameter(module, parameter, interface), + parsed_method.parameter_list)) + if parsed_method.response_parameter_list is not None: + method.response_parameters = list( + map(lambda parameter: _Parameter(module, parameter, interface), + parsed_method.response_parameter_list)) + method.attributes = _AttributeListToDict(parsed_method.attribute_list) + + # Enforce that only methods with response can have a [Sync] attribute. + if method.sync and method.response_parameters is None: + raise Exception("Only methods with response can include a [Sync] " + "attribute. If no response parameters are needed, you " + "could use an empty response parameter list, i.e., " + "\"=> ()\".") + + return method + + +def _Interface(module, parsed_iface): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_iface: {ast.Interface} Parsed interface. + + Returns: + {mojom.Interface} AST interface. + """ + interface = mojom.Interface(module=module) + interface.mojom_name = parsed_iface.mojom_name + interface.spec = 'x:' + module.GetNamespacePrefix() + interface.mojom_name + module.kinds[interface.spec] = interface + interface.attributes = _AttributeListToDict(parsed_iface.attribute_list) + interface.enums = [] + interface.constants = [] + interface.methods_data = [] + _ProcessElements( + parsed_iface.mojom_name, parsed_iface.body, { + ast.Enum: + lambda enum: interface.enums.append(_Enum(module, enum, interface)), + ast.Const: + lambda const: interface.constants.append( + _Constant(module, const, interface)), + ast.Method: + interface.methods_data.append, + }) + return interface + + +def _EnumField(module, enum, parsed_field): + """ + Args: + module: {mojom.Module} Module currently being constructed. + enum: {mojom.Enum} Enum this field belongs to. + parsed_field: {ast.EnumValue} Parsed enum value. + + Returns: + {mojom.EnumField} AST enum field. + """ + field = mojom.EnumField() + field.mojom_name = parsed_field.mojom_name + field.value = _LookupValue(module, enum, None, parsed_field.value) + field.attributes = _AttributeListToDict(parsed_field.attribute_list) + value = mojom.EnumValue(module, enum, field) + module.values[value.GetSpec()] = value + return field + + +def _ResolveNumericEnumValues(enum): + """ + Given a reference to a mojom.Enum, resolves and assigns the numeric value of + each field, and also computes the min_value and max_value of the enum. + """ + + # map of -> integral value + prev_value = -1 + min_value = None + max_value = None + for field in enum.fields: + # This enum value is +1 the previous enum value (e.g: BEGIN). + if field.value is None: + prev_value += 1 + + # Integral value (e.g: BEGIN = -0x1). + elif _IsStrOrUnicode(field.value): + prev_value = int(field.value, 0) + + # Reference to a previous enum value (e.g: INIT = BEGIN). + elif isinstance(field.value, mojom.EnumValue): + prev_value = field.value.field.numeric_value + elif isinstance(field.value, mojom.ConstantValue): + constant = field.value.constant + kind = constant.kind + if not mojom.IsIntegralKind(kind) or mojom.IsBoolKind(kind): + raise ValueError('Enum values must be integers. %s is not an integer.' % + constant.mojom_name) + prev_value = int(constant.value, 0) + else: + raise Exception('Unresolved enum value for %s' % field.value.GetSpec()) + + #resolved_enum_values[field.mojom_name] = prev_value + field.numeric_value = prev_value + if min_value is None or prev_value < min_value: + min_value = prev_value + if max_value is None or prev_value > max_value: + max_value = prev_value + + enum.min_value = min_value + enum.max_value = max_value + + +def _Enum(module, parsed_enum, parent_kind): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_enum: {ast.Enum} Parsed enum. + + Returns: + {mojom.Enum} AST enum. + """ + enum = mojom.Enum(module=module) + enum.mojom_name = parsed_enum.mojom_name + enum.native_only = parsed_enum.enum_value_list is None + mojom_name = enum.mojom_name + if parent_kind: + mojom_name = parent_kind.mojom_name + '.' + mojom_name + enum.spec = 'x:%s.%s' % (module.mojom_namespace, mojom_name) + enum.parent_kind = parent_kind + enum.attributes = _AttributeListToDict(parsed_enum.attribute_list) + + if not enum.native_only: + enum.fields = list( + map(lambda field: _EnumField(module, enum, field), + parsed_enum.enum_value_list)) + _ResolveNumericEnumValues(enum) + + module.kinds[enum.spec] = enum + + # Enforce that a [Native] attribute is set to make native-only enum + # declarations more explicit. + if enum.native_only: + if not enum.attributes or not enum.attributes.get('Native', False): + raise Exception("Native-only enum declarations must include a " + + "Native attribute.") + + return enum + + +def _Constant(module, parsed_const, parent_kind): + """ + Args: + module: {mojom.Module} Module currently being constructed. + parsed_const: {ast.Const} Parsed constant. + + Returns: + {mojom.Constant} AST constant. + """ + constant = mojom.Constant() + constant.mojom_name = parsed_const.mojom_name + if parent_kind: + scope = (module.mojom_namespace, parent_kind.mojom_name) + else: + scope = (module.mojom_namespace, ) + # TODO(mpcomplete): maybe we should only support POD kinds. + constant.kind = _Kind(module.kinds, _MapKind(parsed_const.typename), scope) + constant.parent_kind = parent_kind + constant.value = _LookupValue(module, parent_kind, constant.kind, + parsed_const.value) + + # Iteratively resolve this constant reference to a concrete value + while isinstance(constant.value, mojom.ConstantValue): + constant.value = constant.value.constant.value + + value = mojom.ConstantValue(module, parent_kind, constant) + module.values[value.GetSpec()] = value + return constant + + +def _CollectReferencedKinds(module, all_defined_kinds): + """ + Takes a {mojom.Module} object and a list of all defined kinds within that + module, and enumerates the complete dict of user-defined mojom types + (as {mojom.Kind} objects) referenced by the module's own defined kinds (i.e. + as types of struct or union or interface parameters. The returned dict is + keyed by kind spec. + """ + + def extract_referenced_user_kinds(kind): + if mojom.IsArrayKind(kind): + return extract_referenced_user_kinds(kind.kind) + if mojom.IsMapKind(kind): + return (extract_referenced_user_kinds(kind.key_kind) + + extract_referenced_user_kinds(kind.value_kind)) + if mojom.IsInterfaceRequestKind(kind) or mojom.IsAssociatedKind(kind): + return [kind.kind] + if mojom.IsStructKind(kind): + return [kind] + if (mojom.IsInterfaceKind(kind) or mojom.IsEnumKind(kind) + or mojom.IsUnionKind(kind)): + return [kind] + return [] + + def sanitize_kind(kind): + """Removes nullability from a kind""" + if kind.spec.startswith('?'): + return _Kind(module.kinds, kind.spec[1:], (module.mojom_namespace, '')) + return kind + + referenced_user_kinds = {} + for defined_kind in all_defined_kinds: + if mojom.IsStructKind(defined_kind) or mojom.IsUnionKind(defined_kind): + for field in defined_kind.fields: + for referenced_kind in extract_referenced_user_kinds(field.kind): + sanitized_kind = sanitize_kind(referenced_kind) + referenced_user_kinds[sanitized_kind.spec] = sanitized_kind + + # Also scan for references in parameter lists + for interface in module.interfaces: + for method in interface.methods: + for param in itertools.chain(method.parameters or [], + method.response_parameters or []): + if (mojom.IsStructKind(param.kind) or mojom.IsUnionKind(param.kind) + or mojom.IsEnumKind(param.kind) + or mojom.IsAnyInterfaceKind(param.kind)): + for referenced_kind in extract_referenced_user_kinds(param.kind): + sanitized_kind = sanitize_kind(referenced_kind) + referenced_user_kinds[sanitized_kind.spec] = sanitized_kind + + return referenced_user_kinds + + +def _AssignDefaultOrdinals(items): + """Assigns default ordinal values to a sequence of items if necessary.""" + next_ordinal = 0 + for item in items: + if item.ordinal is not None: + next_ordinal = item.ordinal + 1 + else: + item.ordinal = next_ordinal + next_ordinal += 1 + + +def _AssertTypeIsStable(kind): + """Raises an error if a type is not stable, meaning it is composed of at least + one type that is not marked [Stable].""" + + def assertDependencyIsStable(dependency): + if (mojom.IsEnumKind(dependency) or mojom.IsStructKind(dependency) + or mojom.IsUnionKind(dependency) or mojom.IsInterfaceKind(dependency)): + if not dependency.stable: + raise Exception( + '%s is marked [Stable] but cannot be stable because it depends on ' + '%s, which is not marked [Stable].' % + (kind.mojom_name, dependency.mojom_name)) + elif mojom.IsArrayKind(dependency) or mojom.IsAnyInterfaceKind(dependency): + assertDependencyIsStable(dependency.kind) + elif mojom.IsMapKind(dependency): + assertDependencyIsStable(dependency.key_kind) + assertDependencyIsStable(dependency.value_kind) + + if mojom.IsStructKind(kind) or mojom.IsUnionKind(kind): + for field in kind.fields: + assertDependencyIsStable(field.kind) + elif mojom.IsInterfaceKind(kind): + for method in kind.methods: + for param in method.param_struct.fields: + assertDependencyIsStable(param.kind) + if method.response_param_struct: + for response_param in method.response_param_struct.fields: + assertDependencyIsStable(response_param.kind) + + +def _Module(tree, path, imports): + """ + Args: + tree: {ast.Mojom} The parse tree. + path: {str} The path to the mojom file. + imports: {Dict[str, mojom.Module]} Mapping from filenames, as they appear in + the import list, to already processed modules. Used to process imports. + + Returns: + {mojom.Module} An AST for the mojom. + """ + module = mojom.Module(path=path) + module.kinds = {} + for kind in mojom.PRIMITIVES: + module.kinds[kind.spec] = kind + + module.values = {} + + module.mojom_namespace = tree.module.mojom_namespace[1] if tree.module else '' + # Imports must come first, because they add to module.kinds which is used + # by by the others. + module.imports = [ + _Import(module, imports[imp.import_filename]) for imp in tree.import_list + ] + if tree.module and tree.module.attribute_list: + assert isinstance(tree.module.attribute_list, ast.AttributeList) + # TODO(vtl): Check for duplicate keys here. + module.attributes = dict((attribute.key, attribute.value) + for attribute in tree.module.attribute_list) + + filename = os.path.basename(path) + # First pass collects kinds. + module.constants = [] + module.enums = [] + module.structs = [] + module.unions = [] + module.interfaces = [] + _ProcessElements( + filename, tree.definition_list, { + ast.Const: + lambda const: module.constants.append(_Constant(module, const, None)), + ast.Enum: + lambda enum: module.enums.append(_Enum(module, enum, None)), + ast.Struct: + lambda struct: module.structs.append(_Struct(module, struct)), + ast.Union: + lambda union: module.unions.append(_Union(module, union)), + ast.Interface: + lambda interface: module.interfaces.append( + _Interface(module, interface)), + }) + + # Second pass expands fields and methods. This allows fields and parameters + # to refer to kinds defined anywhere in the mojom. + all_defined_kinds = {} + for struct in module.structs: + struct.fields = list( + map(lambda field: _StructField(module, field, struct), + struct.fields_data)) + _AssignDefaultOrdinals(struct.fields) + del struct.fields_data + all_defined_kinds[struct.spec] = struct + for enum in struct.enums: + all_defined_kinds[enum.spec] = enum + + for union in module.unions: + union.fields = list( + map(lambda field: _UnionField(module, field, union), union.fields_data)) + _AssignDefaultOrdinals(union.fields) + del union.fields_data + all_defined_kinds[union.spec] = union + + for interface in module.interfaces: + interface.methods = list( + map(lambda method: _Method(module, method, interface), + interface.methods_data)) + _AssignDefaultOrdinals(interface.methods) + del interface.methods_data + all_defined_kinds[interface.spec] = interface + for enum in interface.enums: + all_defined_kinds[enum.spec] = enum + for enum in module.enums: + all_defined_kinds[enum.spec] = enum + + all_referenced_kinds = _CollectReferencedKinds(module, + all_defined_kinds.values()) + imported_kind_specs = set(all_referenced_kinds.keys()).difference( + set(all_defined_kinds.keys())) + module.imported_kinds = dict( + (spec, all_referenced_kinds[spec]) for spec in imported_kind_specs) + + generator.AddComputedData(module) + for iface in module.interfaces: + for method in iface.methods: + if method.param_struct: + _AssignDefaultOrdinals(method.param_struct.fields) + if method.response_param_struct: + _AssignDefaultOrdinals(method.response_param_struct.fields) + + # Ensure that all types marked [Stable] are actually stable. Enums are + # automatically OK since they don't depend on other definitions. + for kinds in (module.structs, module.unions, module.interfaces): + for kind in kinds: + if kind.stable: + _AssertTypeIsStable(kind) + + return module + + +def OrderedModule(tree, path, imports): + """Convert parse tree to AST module. + + Args: + tree: {ast.Mojom} The parse tree. + path: {str} The path to the mojom file. + imports: {Dict[str, mojom.Module]} Mapping from filenames, as they appear in + the import list, to already processed modules. Used to process imports. + + Returns: + {mojom.Module} An AST for the mojom. + """ + module = _Module(tree, path, imports) + return module diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py new file mode 100644 index 00000000..19905c8a --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py @@ -0,0 +1,73 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys +import unittest + +from mojom.generate import module as mojom +from mojom.generate import translate +from mojom.parse import ast + + +class TranslateTest(unittest.TestCase): + """Tests |parser.Parse()|.""" + + def testSimpleArray(self): + """Tests a simple int32[].""" + # pylint: disable=W0212 + self.assertEquals(translate._MapKind("int32[]"), "a:i32") + + def testAssociativeArray(self): + """Tests a simple uint8{string}.""" + # pylint: disable=W0212 + self.assertEquals(translate._MapKind("uint8{string}"), "m[s][u8]") + + def testLeftToRightAssociativeArray(self): + """Makes sure that parsing is done from right to left on the internal kinds + in the presence of an associative array.""" + # pylint: disable=W0212 + self.assertEquals(translate._MapKind("uint8[]{string}"), "m[s][a:u8]") + + def testTranslateSimpleUnions(self): + """Makes sure that a simple union is translated correctly.""" + tree = ast.Mojom(None, ast.ImportList(), [ + ast.Union( + "SomeUnion", None, + ast.UnionBody([ + ast.UnionField("a", None, None, "int32"), + ast.UnionField("b", None, None, "string") + ])) + ]) + + translation = translate.OrderedModule(tree, "mojom_tree", []) + self.assertEqual(1, len(translation.unions)) + + union = translation.unions[0] + self.assertTrue(isinstance(union, mojom.Union)) + self.assertEqual("SomeUnion", union.mojom_name) + self.assertEqual(2, len(union.fields)) + self.assertEqual("a", union.fields[0].mojom_name) + self.assertEqual(mojom.INT32.spec, union.fields[0].kind.spec) + self.assertEqual("b", union.fields[1].mojom_name) + self.assertEqual(mojom.STRING.spec, union.fields[1].kind.spec) + + def testMapKindRaisesWithDuplicate(self): + """Verifies _MapTreeForType() raises when passed two values with the same + name.""" + methods = [ + ast.Method('dup', None, None, ast.ParameterList(), None), + ast.Method('dup', None, None, ast.ParameterList(), None) + ] + with self.assertRaises(Exception): + translate._ElemsOfType(methods, ast.Method, 'scope') + + def testAssociatedKinds(self): + """Tests type spec translation of associated interfaces and requests.""" + # pylint: disable=W0212 + self.assertEquals( + translate._MapKind("asso?"), "?asso:x:SomeInterface") + self.assertEquals( + translate._MapKind("asso?"), "?asso:r:x:SomeInterface") diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/__init__.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py new file mode 100644 index 00000000..1f0db200 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py @@ -0,0 +1,427 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Node classes for the AST for a Mojo IDL file.""" + +# Note: For convenience of testing, you probably want to define __eq__() methods +# for all node types; it's okay to be slightly lax (e.g., not compare filename +# and lineno). You may also define __repr__() to help with analyzing test +# failures, especially for more complex types. + + +import sys + + +def _IsStrOrUnicode(x): + if sys.version_info[0] < 3: + return isinstance(x, (unicode, str)) + return isinstance(x, str) + + +class NodeBase(object): + """Base class for nodes in the AST.""" + + def __init__(self, filename=None, lineno=None): + self.filename = filename + self.lineno = lineno + + def __eq__(self, other): + # We want strict comparison of the two object's types. Disable pylint's + # insistence upon recommending isinstance(). + # pylint: disable=unidiomatic-typecheck + return type(self) == type(other) + + # Make != the inverse of ==. (Subclasses shouldn't have to override this.) + def __ne__(self, other): + return not self == other + + +# TODO(vtl): Some of this is complicated enough that it should be tested. +class NodeListBase(NodeBase): + """Represents a list of other nodes, all having the same type. (This is meant + to be subclassed, with subclasses defining _list_item_type to be the class (or + classes, in a tuple) of the members of the list.)""" + + def __init__(self, item_or_items=None, **kwargs): + super(NodeListBase, self).__init__(**kwargs) + self.items = [] + if item_or_items is None: + pass + elif isinstance(item_or_items, list): + for item in item_or_items: + assert isinstance(item, self._list_item_type) + self.Append(item) + else: + assert isinstance(item_or_items, self._list_item_type) + self.Append(item_or_items) + + # Support iteration. For everything else, users should just access |items| + # directly. (We intentionally do NOT supply |__len__()| or |__nonzero__()|, so + # |bool(NodeListBase())| is true.) + def __iter__(self): + return self.items.__iter__() + + def __eq__(self, other): + return super(NodeListBase, self).__eq__(other) and \ + self.items == other.items + + # Implement this so that on failure, we get slightly more sensible output. + def __repr__(self): + return self.__class__.__name__ + "([" + \ + ", ".join([repr(elem) for elem in self.items]) + "])" + + def Insert(self, item): + """Inserts item at the front of the list.""" + + assert isinstance(item, self._list_item_type) + self.items.insert(0, item) + self._UpdateFilenameAndLineno() + + def Append(self, item): + """Appends item to the end of the list.""" + + assert isinstance(item, self._list_item_type) + self.items.append(item) + self._UpdateFilenameAndLineno() + + def _UpdateFilenameAndLineno(self): + if self.items: + self.filename = self.items[0].filename + self.lineno = self.items[0].lineno + + +class Definition(NodeBase): + """Represents a definition of anything that has a global name (e.g., enums, + enum values, consts, structs, struct fields, interfaces). (This does not + include parameter definitions.) This class is meant to be subclassed.""" + + def __init__(self, mojom_name, **kwargs): + assert _IsStrOrUnicode(mojom_name) + NodeBase.__init__(self, **kwargs) + self.mojom_name = mojom_name + + +################################################################################ + + +class Attribute(NodeBase): + """Represents an attribute.""" + + def __init__(self, key, value, **kwargs): + assert _IsStrOrUnicode(key) + super(Attribute, self).__init__(**kwargs) + self.key = key + self.value = value + + def __eq__(self, other): + return super(Attribute, self).__eq__(other) and \ + self.key == other.key and \ + self.value == other.value + + +class AttributeList(NodeListBase): + """Represents a list attributes.""" + + _list_item_type = Attribute + + +class Const(Definition): + """Represents a const definition.""" + + def __init__(self, mojom_name, attribute_list, typename, value, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + # The typename is currently passed through as a string. + assert _IsStrOrUnicode(typename) + # The value is either a literal (currently passed through as a string) or a + # "wrapped identifier". + assert _IsStrOrUnicode or isinstance(value, tuple) + super(Const, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.typename = typename + self.value = value + + def __eq__(self, other): + return super(Const, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.typename == other.typename and \ + self.value == other.value + + +class Enum(Definition): + """Represents an enum definition.""" + + def __init__(self, mojom_name, attribute_list, enum_value_list, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert enum_value_list is None or isinstance(enum_value_list, EnumValueList) + super(Enum, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.enum_value_list = enum_value_list + + def __eq__(self, other): + return super(Enum, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.enum_value_list == other.enum_value_list + + +class EnumValue(Definition): + """Represents a definition of an enum value.""" + + def __init__(self, mojom_name, attribute_list, value, **kwargs): + # The optional value is either an int (which is current a string) or a + # "wrapped identifier". + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert value is None or _IsStrOrUnicode(value) or isinstance(value, tuple) + super(EnumValue, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.value = value + + def __eq__(self, other): + return super(EnumValue, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.value == other.value + + +class EnumValueList(NodeListBase): + """Represents a list of enum value definitions (i.e., the "body" of an enum + definition).""" + + _list_item_type = EnumValue + + +class Import(NodeBase): + """Represents an import statement.""" + + def __init__(self, attribute_list, import_filename, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert _IsStrOrUnicode(import_filename) + super(Import, self).__init__(**kwargs) + self.attribute_list = attribute_list + self.import_filename = import_filename + + def __eq__(self, other): + return super(Import, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.import_filename == other.import_filename + + +class ImportList(NodeListBase): + """Represents a list (i.e., sequence) of import statements.""" + + _list_item_type = Import + + +class Interface(Definition): + """Represents an interface definition.""" + + def __init__(self, mojom_name, attribute_list, body, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert isinstance(body, InterfaceBody) + super(Interface, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.body = body + + def __eq__(self, other): + return super(Interface, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.body == other.body + + +class Method(Definition): + """Represents a method definition.""" + + def __init__(self, mojom_name, attribute_list, ordinal, parameter_list, + response_parameter_list, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert ordinal is None or isinstance(ordinal, Ordinal) + assert isinstance(parameter_list, ParameterList) + assert response_parameter_list is None or \ + isinstance(response_parameter_list, ParameterList) + super(Method, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.ordinal = ordinal + self.parameter_list = parameter_list + self.response_parameter_list = response_parameter_list + + def __eq__(self, other): + return super(Method, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.ordinal == other.ordinal and \ + self.parameter_list == other.parameter_list and \ + self.response_parameter_list == other.response_parameter_list + + +# This needs to be declared after |Method|. +class InterfaceBody(NodeListBase): + """Represents the body of (i.e., list of definitions inside) an interface.""" + + _list_item_type = (Const, Enum, Method) + + +class Module(NodeBase): + """Represents a module statement.""" + + def __init__(self, mojom_namespace, attribute_list, **kwargs): + # |mojom_namespace| is either none or a "wrapped identifier". + assert mojom_namespace is None or isinstance(mojom_namespace, tuple) + assert attribute_list is None or isinstance(attribute_list, AttributeList) + super(Module, self).__init__(**kwargs) + self.mojom_namespace = mojom_namespace + self.attribute_list = attribute_list + + def __eq__(self, other): + return super(Module, self).__eq__(other) and \ + self.mojom_namespace == other.mojom_namespace and \ + self.attribute_list == other.attribute_list + + +class Mojom(NodeBase): + """Represents an entire .mojom file. (This is the root node.)""" + + def __init__(self, module, import_list, definition_list, **kwargs): + assert module is None or isinstance(module, Module) + assert isinstance(import_list, ImportList) + assert isinstance(definition_list, list) + super(Mojom, self).__init__(**kwargs) + self.module = module + self.import_list = import_list + self.definition_list = definition_list + + def __eq__(self, other): + return super(Mojom, self).__eq__(other) and \ + self.module == other.module and \ + self.import_list == other.import_list and \ + self.definition_list == other.definition_list + + def __repr__(self): + return "%s(%r, %r, %r)" % (self.__class__.__name__, self.module, + self.import_list, self.definition_list) + + +class Ordinal(NodeBase): + """Represents an ordinal value labeling, e.g., a struct field.""" + + def __init__(self, value, **kwargs): + assert isinstance(value, int) + super(Ordinal, self).__init__(**kwargs) + self.value = value + + def __eq__(self, other): + return super(Ordinal, self).__eq__(other) and \ + self.value == other.value + + +class Parameter(NodeBase): + """Represents a method request or response parameter.""" + + def __init__(self, mojom_name, attribute_list, ordinal, typename, **kwargs): + assert _IsStrOrUnicode(mojom_name) + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert ordinal is None or isinstance(ordinal, Ordinal) + assert _IsStrOrUnicode(typename) + super(Parameter, self).__init__(**kwargs) + self.mojom_name = mojom_name + self.attribute_list = attribute_list + self.ordinal = ordinal + self.typename = typename + + def __eq__(self, other): + return super(Parameter, self).__eq__(other) and \ + self.mojom_name == other.mojom_name and \ + self.attribute_list == other.attribute_list and \ + self.ordinal == other.ordinal and \ + self.typename == other.typename + + +class ParameterList(NodeListBase): + """Represents a list of (method request or response) parameters.""" + + _list_item_type = Parameter + + +class Struct(Definition): + """Represents a struct definition.""" + + def __init__(self, mojom_name, attribute_list, body, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert isinstance(body, StructBody) or body is None + super(Struct, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.body = body + + def __eq__(self, other): + return super(Struct, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.body == other.body + + +class StructField(Definition): + """Represents a struct field definition.""" + + def __init__(self, mojom_name, attribute_list, ordinal, typename, + default_value, **kwargs): + assert _IsStrOrUnicode(mojom_name) + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert ordinal is None or isinstance(ordinal, Ordinal) + assert _IsStrOrUnicode(typename) + # The optional default value is currently either a value as a string or a + # "wrapped identifier". + assert default_value is None or _IsStrOrUnicode(default_value) or \ + isinstance(default_value, tuple) + super(StructField, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.ordinal = ordinal + self.typename = typename + self.default_value = default_value + + def __eq__(self, other): + return super(StructField, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.ordinal == other.ordinal and \ + self.typename == other.typename and \ + self.default_value == other.default_value + + +# This needs to be declared after |StructField|. +class StructBody(NodeListBase): + """Represents the body of (i.e., list of definitions inside) a struct.""" + + _list_item_type = (Const, Enum, StructField) + + +class Union(Definition): + """Represents a union definition.""" + + def __init__(self, mojom_name, attribute_list, body, **kwargs): + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert isinstance(body, UnionBody) + super(Union, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.body = body + + def __eq__(self, other): + return super(Union, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.body == other.body + + +class UnionField(Definition): + def __init__(self, mojom_name, attribute_list, ordinal, typename, **kwargs): + assert _IsStrOrUnicode(mojom_name) + assert attribute_list is None or isinstance(attribute_list, AttributeList) + assert ordinal is None or isinstance(ordinal, Ordinal) + assert _IsStrOrUnicode(typename) + super(UnionField, self).__init__(mojom_name, **kwargs) + self.attribute_list = attribute_list + self.ordinal = ordinal + self.typename = typename + + def __eq__(self, other): + return super(UnionField, self).__eq__(other) and \ + self.attribute_list == other.attribute_list and \ + self.ordinal == other.ordinal and \ + self.typename == other.typename + + +class UnionBody(NodeListBase): + + _list_item_type = UnionField diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py new file mode 100644 index 00000000..62798631 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py @@ -0,0 +1,121 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys +import unittest + +from mojom.parse import ast + + +class _TestNode(ast.NodeBase): + """Node type for tests.""" + + def __init__(self, value, **kwargs): + super(_TestNode, self).__init__(**kwargs) + self.value = value + + def __eq__(self, other): + return super(_TestNode, self).__eq__(other) and self.value == other.value + + +class _TestNodeList(ast.NodeListBase): + """Node list type for tests.""" + + _list_item_type = _TestNode + + +class ASTTest(unittest.TestCase): + """Tests various AST classes.""" + + def testNodeBase(self): + # Test |__eq__()|; this is only used for testing, where we want to do + # comparison by value and ignore filenames/line numbers (for convenience). + node1 = ast.NodeBase(filename="hello.mojom", lineno=123) + node2 = ast.NodeBase() + self.assertEquals(node1, node2) + self.assertEquals(node2, node1) + + # Check that |__ne__()| just defers to |__eq__()| properly. + self.assertFalse(node1 != node2) + self.assertFalse(node2 != node1) + + # Check that |filename| and |lineno| are set properly (and are None by + # default). + self.assertEquals(node1.filename, "hello.mojom") + self.assertEquals(node1.lineno, 123) + self.assertIsNone(node2.filename) + self.assertIsNone(node2.lineno) + + # |NodeBase|'s |__eq__()| should compare types (and a subclass's |__eq__()| + # should first defer to its superclass's). + node3 = _TestNode(123) + self.assertNotEqual(node1, node3) + self.assertNotEqual(node3, node1) + # Also test |__eq__()| directly. + self.assertFalse(node1 == node3) + self.assertFalse(node3 == node1) + + node4 = _TestNode(123, filename="world.mojom", lineno=123) + self.assertEquals(node4, node3) + node5 = _TestNode(456) + self.assertNotEquals(node5, node4) + + def testNodeListBase(self): + node1 = _TestNode(1, filename="foo.mojom", lineno=1) + # Equal to, but not the same as, |node1|: + node1b = _TestNode(1, filename="foo.mojom", lineno=1) + node2 = _TestNode(2, filename="foo.mojom", lineno=2) + + nodelist1 = _TestNodeList() # Contains: (empty). + self.assertEquals(nodelist1, nodelist1) + self.assertEquals(nodelist1.items, []) + self.assertIsNone(nodelist1.filename) + self.assertIsNone(nodelist1.lineno) + + nodelist2 = _TestNodeList(node1) # Contains: 1. + self.assertEquals(nodelist2, nodelist2) + self.assertEquals(nodelist2.items, [node1]) + self.assertNotEqual(nodelist2, nodelist1) + self.assertEquals(nodelist2.filename, "foo.mojom") + self.assertEquals(nodelist2.lineno, 1) + + nodelist3 = _TestNodeList([node2]) # Contains: 2. + self.assertEquals(nodelist3.items, [node2]) + self.assertNotEqual(nodelist3, nodelist1) + self.assertNotEqual(nodelist3, nodelist2) + self.assertEquals(nodelist3.filename, "foo.mojom") + self.assertEquals(nodelist3.lineno, 2) + + nodelist1.Append(node1b) # Contains: 1. + self.assertEquals(nodelist1.items, [node1]) + self.assertEquals(nodelist1, nodelist2) + self.assertNotEqual(nodelist1, nodelist3) + self.assertEquals(nodelist1.filename, "foo.mojom") + self.assertEquals(nodelist1.lineno, 1) + + nodelist1.Append(node2) # Contains: 1, 2. + self.assertEquals(nodelist1.items, [node1, node2]) + self.assertNotEqual(nodelist1, nodelist2) + self.assertNotEqual(nodelist1, nodelist3) + self.assertEquals(nodelist1.lineno, 1) + + nodelist2.Append(node2) # Contains: 1, 2. + self.assertEquals(nodelist2.items, [node1, node2]) + self.assertEquals(nodelist2, nodelist1) + self.assertNotEqual(nodelist2, nodelist3) + self.assertEquals(nodelist2.lineno, 1) + + nodelist3.Insert(node1) # Contains: 1, 2. + self.assertEquals(nodelist3.items, [node1, node2]) + self.assertEquals(nodelist3, nodelist1) + self.assertEquals(nodelist3, nodelist2) + self.assertEquals(nodelist3.lineno, 1) + + # Test iteration: + i = 1 + for item in nodelist1: + self.assertEquals(item.value, i) + i += 1 diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py new file mode 100644 index 00000000..3cb73c5d --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py @@ -0,0 +1,82 @@ +# Copyright 2018 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Helpers for processing conditionally enabled features in a mojom.""" + +from mojom.error import Error +from mojom.parse import ast + + +class EnableIfError(Error): + """ Class for errors from .""" + + def __init__(self, filename, message, lineno=None): + Error.__init__(self, filename, message, lineno=lineno, addenda=None) + + +def _IsEnabled(definition, enabled_features): + """Returns true if a definition is enabled. + + A definition is enabled if it has no EnableIf attribute, or if the value of + the EnableIf attribute is in enabled_features. + """ + if not hasattr(definition, "attribute_list"): + return True + if not definition.attribute_list: + return True + + already_defined = False + for a in definition.attribute_list: + if a.key == 'EnableIf': + if already_defined: + raise EnableIfError( + definition.filename, + "EnableIf attribute may only be defined once per field.", + definition.lineno) + already_defined = True + + for attribute in definition.attribute_list: + if attribute.key == 'EnableIf' and attribute.value not in enabled_features: + return False + return True + + +def _FilterDisabledFromNodeList(node_list, enabled_features): + if not node_list: + return + assert isinstance(node_list, ast.NodeListBase) + node_list.items = [ + item for item in node_list.items if _IsEnabled(item, enabled_features) + ] + for item in node_list.items: + _FilterDefinition(item, enabled_features) + + +def _FilterDefinition(definition, enabled_features): + """Filters definitions with a body.""" + if isinstance(definition, ast.Enum): + _FilterDisabledFromNodeList(definition.enum_value_list, enabled_features) + elif isinstance(definition, ast.Interface): + _FilterDisabledFromNodeList(definition.body, enabled_features) + elif isinstance(definition, ast.Method): + _FilterDisabledFromNodeList(definition.parameter_list, enabled_features) + _FilterDisabledFromNodeList(definition.response_parameter_list, + enabled_features) + elif isinstance(definition, ast.Struct): + _FilterDisabledFromNodeList(definition.body, enabled_features) + elif isinstance(definition, ast.Union): + _FilterDisabledFromNodeList(definition.body, enabled_features) + + +def RemoveDisabledDefinitions(mojom, enabled_features): + """Removes conditionally disabled definitions from a Mojom node.""" + mojom.import_list = ast.ImportList([ + imported_file for imported_file in mojom.import_list + if _IsEnabled(imported_file, enabled_features) + ]) + mojom.definition_list = [ + definition for definition in mojom.definition_list + if _IsEnabled(definition, enabled_features) + ] + for definition in mojom.definition_list: + _FilterDefinition(definition, enabled_features) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py new file mode 100644 index 00000000..aa609be7 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py @@ -0,0 +1,233 @@ +# Copyright 2018 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os +import sys +import unittest + + +def _GetDirAbove(dirname): + """Returns the directory "above" this file containing |dirname| (which must + also be "above" this file).""" + path = os.path.abspath(__file__) + while True: + path, tail = os.path.split(path) + assert tail + if tail == dirname: + return path + + +try: + imp.find_module('mojom') +except ImportError: + sys.path.append(os.path.join(_GetDirAbove('pylib'), 'pylib')) +import mojom.parse.ast as ast +import mojom.parse.conditional_features as conditional_features +import mojom.parse.parser as parser + +ENABLED_FEATURES = frozenset({'red', 'green', 'blue'}) + + +class ConditionalFeaturesTest(unittest.TestCase): + """Tests |mojom.parse.conditional_features|.""" + + def parseAndAssertEqual(self, source, expected_source): + definition = parser.Parse(source, "my_file.mojom") + conditional_features.RemoveDisabledDefinitions(definition, ENABLED_FEATURES) + expected = parser.Parse(expected_source, "my_file.mojom") + self.assertEquals(definition, expected) + + def testFilterConst(self): + """Test that Consts are correctly filtered.""" + const_source = """ + [EnableIf=blue] + const int kMyConst1 = 1; + [EnableIf=orange] + const double kMyConst2 = 2; + const int kMyConst3 = 3; + """ + expected_source = """ + [EnableIf=blue] + const int kMyConst1 = 1; + const int kMyConst3 = 3; + """ + self.parseAndAssertEqual(const_source, expected_source) + + def testFilterEnum(self): + """Test that EnumValues are correctly filtered from an Enum.""" + enum_source = """ + enum MyEnum { + [EnableIf=purple] + VALUE1, + [EnableIf=blue] + VALUE2, + VALUE3, + }; + """ + expected_source = """ + enum MyEnum { + [EnableIf=blue] + VALUE2, + VALUE3 + }; + """ + self.parseAndAssertEqual(enum_source, expected_source) + + def testFilterImport(self): + """Test that imports are correctly filtered from a Mojom.""" + import_source = """ + [EnableIf=blue] + import "foo.mojom"; + import "bar.mojom"; + [EnableIf=purple] + import "baz.mojom"; + """ + expected_source = """ + [EnableIf=blue] + import "foo.mojom"; + import "bar.mojom"; + """ + self.parseAndAssertEqual(import_source, expected_source) + + def testFilterInterface(self): + """Test that definitions are correctly filtered from an Interface.""" + interface_source = """ + interface MyInterface { + [EnableIf=blue] + enum MyEnum { + [EnableIf=purple] + VALUE1, + VALUE2, + }; + [EnableIf=blue] + const int32 kMyConst = 123; + [EnableIf=purple] + MyMethod(); + }; + """ + expected_source = """ + interface MyInterface { + [EnableIf=blue] + enum MyEnum { + VALUE2, + }; + [EnableIf=blue] + const int32 kMyConst = 123; + }; + """ + self.parseAndAssertEqual(interface_source, expected_source) + + def testFilterMethod(self): + """Test that Parameters are correctly filtered from a Method.""" + method_source = """ + interface MyInterface { + [EnableIf=blue] + MyMethod([EnableIf=purple] int32 a) => ([EnableIf=red] int32 b); + }; + """ + expected_source = """ + interface MyInterface { + [EnableIf=blue] + MyMethod() => ([EnableIf=red] int32 b); + }; + """ + self.parseAndAssertEqual(method_source, expected_source) + + def testFilterStruct(self): + """Test that definitions are correctly filtered from a Struct.""" + struct_source = """ + struct MyStruct { + [EnableIf=blue] + enum MyEnum { + VALUE1, + [EnableIf=purple] + VALUE2, + }; + [EnableIf=yellow] + const double kMyConst = 1.23; + [EnableIf=green] + int32 a; + double b; + [EnableIf=purple] + int32 c; + [EnableIf=blue] + double d; + int32 e; + [EnableIf=orange] + double f; + }; + """ + expected_source = """ + struct MyStruct { + [EnableIf=blue] + enum MyEnum { + VALUE1, + }; + [EnableIf=green] + int32 a; + double b; + [EnableIf=blue] + double d; + int32 e; + }; + """ + self.parseAndAssertEqual(struct_source, expected_source) + + def testFilterUnion(self): + """Test that UnionFields are correctly filtered from a Union.""" + union_source = """ + union MyUnion { + [EnableIf=yellow] + int32 a; + [EnableIf=red] + bool b; + }; + """ + expected_source = """ + union MyUnion { + [EnableIf=red] + bool b; + }; + """ + self.parseAndAssertEqual(union_source, expected_source) + + def testSameNameFields(self): + mojom_source = """ + enum Foo { + [EnableIf=red] + VALUE1 = 5, + [EnableIf=yellow] + VALUE1 = 6, + }; + [EnableIf=red] + const double kMyConst = 1.23; + [EnableIf=yellow] + const double kMyConst = 4.56; + """ + expected_source = """ + enum Foo { + [EnableIf=red] + VALUE1 = 5, + }; + [EnableIf=red] + const double kMyConst = 1.23; + """ + self.parseAndAssertEqual(mojom_source, expected_source) + + def testMultipleEnableIfs(self): + source = """ + enum Foo { + [EnableIf=red,EnableIf=yellow] + kBarValue = 5, + }; + """ + definition = parser.Parse(source, "my_file.mojom") + self.assertRaises(conditional_features.EnableIfError, + conditional_features.RemoveDisabledDefinitions, + definition, ENABLED_FEATURES) + + +if __name__ == '__main__': + unittest.main() diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py new file mode 100644 index 00000000..3e084bbf --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py @@ -0,0 +1,251 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys + +from mojom import fileutil +from mojom.error import Error + +fileutil.AddLocalRepoThirdPartyDirToModulePath() +from ply.lex import TOKEN + + +class LexError(Error): + """Class for errors from the lexer.""" + + def __init__(self, filename, message, lineno): + Error.__init__(self, filename, message, lineno=lineno) + + +# We have methods which look like they could be functions: +# pylint: disable=R0201 +class Lexer(object): + def __init__(self, filename): + self.filename = filename + + ######################-- PRIVATE --###################### + + ## + ## Internal auxiliary methods + ## + def _error(self, msg, token): + raise LexError(self.filename, msg, token.lineno) + + ## + ## Reserved keywords + ## + keywords = ( + 'HANDLE', + 'IMPORT', + 'MODULE', + 'STRUCT', + 'UNION', + 'INTERFACE', + 'ENUM', + 'CONST', + 'TRUE', + 'FALSE', + 'DEFAULT', + 'ARRAY', + 'MAP', + 'ASSOCIATED', + 'PENDING_REMOTE', + 'PENDING_RECEIVER', + 'PENDING_ASSOCIATED_REMOTE', + 'PENDING_ASSOCIATED_RECEIVER', + ) + + keyword_map = {} + for keyword in keywords: + keyword_map[keyword.lower()] = keyword + + ## + ## All the tokens recognized by the lexer + ## + tokens = keywords + ( + # Identifiers + 'NAME', + + # Constants + 'ORDINAL', + 'INT_CONST_DEC', + 'INT_CONST_HEX', + 'FLOAT_CONST', + + # String literals + 'STRING_LITERAL', + + # Operators + 'MINUS', + 'PLUS', + 'AMP', + 'QSTN', + + # Assignment + 'EQUALS', + + # Request / response + 'RESPONSE', + + # Delimiters + 'LPAREN', + 'RPAREN', # ( ) + 'LBRACKET', + 'RBRACKET', # [ ] + 'LBRACE', + 'RBRACE', # { } + 'LANGLE', + 'RANGLE', # < > + 'SEMI', # ; + 'COMMA', + 'DOT' # , . + ) + + ## + ## Regexes for use in tokens + ## + + # valid C identifiers (K&R2: A.2.3) + identifier = r'[a-zA-Z_][0-9a-zA-Z_]*' + + hex_prefix = '0[xX]' + hex_digits = '[0-9a-fA-F]+' + + # integer constants (K&R2: A.2.5.1) + decimal_constant = '0|([1-9][0-9]*)' + hex_constant = hex_prefix + hex_digits + # Don't allow octal constants (even invalid octal). + octal_constant_disallowed = '0[0-9]+' + + # character constants (K&R2: A.2.5.2) + # Note: a-zA-Z and '.-~^_!=&;,' are allowed as escape chars to support #line + # directives with Windows paths as filenames (..\..\dir\file) + # For the same reason, decimal_escape allows all digit sequences. We want to + # parse all correct code, even if it means to sometimes parse incorrect + # code. + # + simple_escape = r"""([a-zA-Z._~!=&\^\-\\?'"])""" + decimal_escape = r"""(\d+)""" + hex_escape = r"""(x[0-9a-fA-F]+)""" + bad_escape = r"""([\\][^a-zA-Z._~^!=&\^\-\\?'"x0-7])""" + + escape_sequence = \ + r"""(\\("""+simple_escape+'|'+decimal_escape+'|'+hex_escape+'))' + + # string literals (K&R2: A.2.6) + string_char = r"""([^"\\\n]|""" + escape_sequence + ')' + string_literal = '"' + string_char + '*"' + bad_string_literal = '"' + string_char + '*' + bad_escape + string_char + '*"' + + # floating constants (K&R2: A.2.5.3) + exponent_part = r"""([eE][-+]?[0-9]+)""" + fractional_constant = r"""([0-9]*\.[0-9]+)|([0-9]+\.)""" + floating_constant = \ + '(((('+fractional_constant+')'+ \ + exponent_part+'?)|([0-9]+'+exponent_part+')))' + + # Ordinals + ordinal = r'@[0-9]+' + missing_ordinal_value = r'@' + # Don't allow ordinal values in octal (even invalid octal, like 09) or + # hexadecimal. + octal_or_hex_ordinal_disallowed = ( + r'@((0[0-9]+)|(' + hex_prefix + hex_digits + '))') + + ## + ## Rules for the normal state + ## + t_ignore = ' \t\r' + + # Newlines + def t_NEWLINE(self, t): + r'\n+' + t.lexer.lineno += len(t.value) + + # Operators + t_MINUS = r'-' + t_PLUS = r'\+' + t_AMP = r'&' + t_QSTN = r'\?' + + # = + t_EQUALS = r'=' + + # => + t_RESPONSE = r'=>' + + # Delimiters + t_LPAREN = r'\(' + t_RPAREN = r'\)' + t_LBRACKET = r'\[' + t_RBRACKET = r'\]' + t_LBRACE = r'\{' + t_RBRACE = r'\}' + t_LANGLE = r'<' + t_RANGLE = r'>' + t_COMMA = r',' + t_DOT = r'\.' + t_SEMI = r';' + + t_STRING_LITERAL = string_literal + + # The following floating and integer constants are defined as + # functions to impose a strict order (otherwise, decimal + # is placed before the others because its regex is longer, + # and this is bad) + # + @TOKEN(floating_constant) + def t_FLOAT_CONST(self, t): + return t + + @TOKEN(hex_constant) + def t_INT_CONST_HEX(self, t): + return t + + @TOKEN(octal_constant_disallowed) + def t_OCTAL_CONSTANT_DISALLOWED(self, t): + msg = "Octal values not allowed" + self._error(msg, t) + + @TOKEN(decimal_constant) + def t_INT_CONST_DEC(self, t): + return t + + # unmatched string literals are caught by the preprocessor + + @TOKEN(bad_string_literal) + def t_BAD_STRING_LITERAL(self, t): + msg = "String contains invalid escape code" + self._error(msg, t) + + # Handle ordinal-related tokens in the right order: + @TOKEN(octal_or_hex_ordinal_disallowed) + def t_OCTAL_OR_HEX_ORDINAL_DISALLOWED(self, t): + msg = "Octal and hexadecimal ordinal values not allowed" + self._error(msg, t) + + @TOKEN(ordinal) + def t_ORDINAL(self, t): + return t + + @TOKEN(missing_ordinal_value) + def t_BAD_ORDINAL(self, t): + msg = "Missing ordinal value" + self._error(msg, t) + + @TOKEN(identifier) + def t_NAME(self, t): + t.type = self.keyword_map.get(t.value, "NAME") + return t + + # Ignore C and C++ style comments + def t_COMMENT(self, t): + r'(/\*(.|\n)*?\*/)|(//.*(\n[ \t]*//.*)*)' + t.lexer.lineno += t.value.count("\n") + + def t_error(self, t): + msg = "Illegal character %s" % repr(t.value[0]) + self._error(msg, t) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py new file mode 100644 index 00000000..eadc6587 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py @@ -0,0 +1,198 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys +import unittest + + +def _GetDirAbove(dirname): + """Returns the directory "above" this file containing |dirname| (which must + also be "above" this file).""" + path = os.path.abspath(__file__) + while True: + path, tail = os.path.split(path) + assert tail + if tail == dirname: + return path + + +sys.path.insert(1, os.path.join(_GetDirAbove("mojo"), "third_party")) +from ply import lex + +try: + imp.find_module("mojom") +except ImportError: + sys.path.append(os.path.join(_GetDirAbove("pylib"), "pylib")) +import mojom.parse.lexer + + +# This (monkey-patching LexToken to make comparison value-based) is evil, but +# we'll do it anyway. (I'm pretty sure ply's lexer never cares about comparing +# for object identity.) +def _LexTokenEq(self, other): + return self.type == other.type and self.value == other.value and \ + self.lineno == other.lineno and self.lexpos == other.lexpos + + +setattr(lex.LexToken, '__eq__', _LexTokenEq) + + +def _MakeLexToken(token_type, value, lineno=1, lexpos=0): + """Makes a LexToken with the given parameters. (Note that lineno is 1-based, + but lexpos is 0-based.)""" + rv = lex.LexToken() + rv.type, rv.value, rv.lineno, rv.lexpos = token_type, value, lineno, lexpos + return rv + + +def _MakeLexTokenForKeyword(keyword, **kwargs): + """Makes a LexToken for the given keyword.""" + return _MakeLexToken(keyword.upper(), keyword.lower(), **kwargs) + + +class LexerTest(unittest.TestCase): + """Tests |mojom.parse.lexer.Lexer|.""" + + def __init__(self, *args, **kwargs): + unittest.TestCase.__init__(self, *args, **kwargs) + # Clone all lexer instances from this one, since making a lexer is slow. + self._zygote_lexer = lex.lex(mojom.parse.lexer.Lexer("my_file.mojom")) + + def testValidKeywords(self): + """Tests valid keywords.""" + self.assertEquals( + self._SingleTokenForInput("handle"), _MakeLexTokenForKeyword("handle")) + self.assertEquals( + self._SingleTokenForInput("import"), _MakeLexTokenForKeyword("import")) + self.assertEquals( + self._SingleTokenForInput("module"), _MakeLexTokenForKeyword("module")) + self.assertEquals( + self._SingleTokenForInput("struct"), _MakeLexTokenForKeyword("struct")) + self.assertEquals( + self._SingleTokenForInput("union"), _MakeLexTokenForKeyword("union")) + self.assertEquals( + self._SingleTokenForInput("interface"), + _MakeLexTokenForKeyword("interface")) + self.assertEquals( + self._SingleTokenForInput("enum"), _MakeLexTokenForKeyword("enum")) + self.assertEquals( + self._SingleTokenForInput("const"), _MakeLexTokenForKeyword("const")) + self.assertEquals( + self._SingleTokenForInput("true"), _MakeLexTokenForKeyword("true")) + self.assertEquals( + self._SingleTokenForInput("false"), _MakeLexTokenForKeyword("false")) + self.assertEquals( + self._SingleTokenForInput("default"), + _MakeLexTokenForKeyword("default")) + self.assertEquals( + self._SingleTokenForInput("array"), _MakeLexTokenForKeyword("array")) + self.assertEquals( + self._SingleTokenForInput("map"), _MakeLexTokenForKeyword("map")) + self.assertEquals( + self._SingleTokenForInput("associated"), + _MakeLexTokenForKeyword("associated")) + + def testValidIdentifiers(self): + """Tests identifiers.""" + self.assertEquals( + self._SingleTokenForInput("abcd"), _MakeLexToken("NAME", "abcd")) + self.assertEquals( + self._SingleTokenForInput("AbC_d012_"), + _MakeLexToken("NAME", "AbC_d012_")) + self.assertEquals( + self._SingleTokenForInput("_0123"), _MakeLexToken("NAME", "_0123")) + + def testInvalidIdentifiers(self): + with self.assertRaisesRegexp( + mojom.parse.lexer.LexError, + r"^my_file\.mojom:1: Error: Illegal character '\$'$"): + self._TokensForInput("$abc") + with self.assertRaisesRegexp( + mojom.parse.lexer.LexError, + r"^my_file\.mojom:1: Error: Illegal character '\$'$"): + self._TokensForInput("a$bc") + + def testDecimalIntegerConstants(self): + self.assertEquals( + self._SingleTokenForInput("0"), _MakeLexToken("INT_CONST_DEC", "0")) + self.assertEquals( + self._SingleTokenForInput("1"), _MakeLexToken("INT_CONST_DEC", "1")) + self.assertEquals( + self._SingleTokenForInput("123"), _MakeLexToken("INT_CONST_DEC", "123")) + self.assertEquals( + self._SingleTokenForInput("10"), _MakeLexToken("INT_CONST_DEC", "10")) + + def testValidTokens(self): + """Tests valid tokens (which aren't tested elsewhere).""" + # Keywords tested in |testValidKeywords|. + # NAME tested in |testValidIdentifiers|. + self.assertEquals( + self._SingleTokenForInput("@123"), _MakeLexToken("ORDINAL", "@123")) + self.assertEquals( + self._SingleTokenForInput("456"), _MakeLexToken("INT_CONST_DEC", "456")) + self.assertEquals( + self._SingleTokenForInput("0x01aB2eF3"), + _MakeLexToken("INT_CONST_HEX", "0x01aB2eF3")) + self.assertEquals( + self._SingleTokenForInput("123.456"), + _MakeLexToken("FLOAT_CONST", "123.456")) + self.assertEquals( + self._SingleTokenForInput("\"hello\""), + _MakeLexToken("STRING_LITERAL", "\"hello\"")) + self.assertEquals( + self._SingleTokenForInput("+"), _MakeLexToken("PLUS", "+")) + self.assertEquals( + self._SingleTokenForInput("-"), _MakeLexToken("MINUS", "-")) + self.assertEquals(self._SingleTokenForInput("&"), _MakeLexToken("AMP", "&")) + self.assertEquals( + self._SingleTokenForInput("?"), _MakeLexToken("QSTN", "?")) + self.assertEquals( + self._SingleTokenForInput("="), _MakeLexToken("EQUALS", "=")) + self.assertEquals( + self._SingleTokenForInput("=>"), _MakeLexToken("RESPONSE", "=>")) + self.assertEquals( + self._SingleTokenForInput("("), _MakeLexToken("LPAREN", "(")) + self.assertEquals( + self._SingleTokenForInput(")"), _MakeLexToken("RPAREN", ")")) + self.assertEquals( + self._SingleTokenForInput("["), _MakeLexToken("LBRACKET", "[")) + self.assertEquals( + self._SingleTokenForInput("]"), _MakeLexToken("RBRACKET", "]")) + self.assertEquals( + self._SingleTokenForInput("{"), _MakeLexToken("LBRACE", "{")) + self.assertEquals( + self._SingleTokenForInput("}"), _MakeLexToken("RBRACE", "}")) + self.assertEquals( + self._SingleTokenForInput("<"), _MakeLexToken("LANGLE", "<")) + self.assertEquals( + self._SingleTokenForInput(">"), _MakeLexToken("RANGLE", ">")) + self.assertEquals( + self._SingleTokenForInput(";"), _MakeLexToken("SEMI", ";")) + self.assertEquals( + self._SingleTokenForInput(","), _MakeLexToken("COMMA", ",")) + self.assertEquals(self._SingleTokenForInput("."), _MakeLexToken("DOT", ".")) + + def _TokensForInput(self, input_string): + """Gets a list of tokens for the given input string.""" + lexer = self._zygote_lexer.clone() + lexer.input(input_string) + rv = [] + while True: + tok = lexer.token() + if not tok: + return rv + rv.append(tok) + + def _SingleTokenForInput(self, input_string): + """Gets the single token for the given input string. (Raises an exception if + the input string does not result in exactly one token.)""" + toks = self._TokensForInput(input_string) + assert len(toks) == 1 + return toks[0] + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py new file mode 100644 index 00000000..b3b803d6 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py @@ -0,0 +1,488 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Generates a syntax tree from a Mojo IDL file.""" + +import os.path +import sys + +from mojom import fileutil +from mojom.error import Error +from mojom.parse import ast +from mojom.parse.lexer import Lexer + +fileutil.AddLocalRepoThirdPartyDirToModulePath() +from ply import lex +from ply import yacc + +_MAX_ORDINAL_VALUE = 0xffffffff +_MAX_ARRAY_SIZE = 0xffffffff + + +class ParseError(Error): + """Class for errors from the parser.""" + + def __init__(self, filename, message, lineno=None, snippet=None): + Error.__init__( + self, + filename, + message, + lineno=lineno, + addenda=([snippet] if snippet else None)) + + +# We have methods which look like they could be functions: +# pylint: disable=R0201 +class Parser(object): + def __init__(self, lexer, source, filename): + self.tokens = lexer.tokens + self.source = source + self.filename = filename + + # Names of functions + # + # In general, we name functions after the left-hand-side of the rule(s) that + # they handle. E.g., |p_foo_bar| for a rule |foo_bar : ...|. + # + # There may be multiple functions handling rules for the same left-hand-side; + # then we name the functions |p_foo_bar_N| (for left-hand-side |foo_bar|), + # where N is a number (numbered starting from 1). Note that using multiple + # functions is actually more efficient than having single functions handle + # multiple rules (and, e.g., distinguishing them by examining |len(p)|). + # + # It's also possible to have a function handling multiple rules with different + # left-hand-sides. We do not do this. + # + # See http://www.dabeaz.com/ply/ply.html#ply_nn25 for more details. + + # TODO(vtl): Get rid of the braces in the module "statement". (Consider + # renaming "module" -> "package".) Then we'll be able to have a single rule + # for root (by making module "optional"). + def p_root_1(self, p): + """root : """ + p[0] = ast.Mojom(None, ast.ImportList(), []) + + def p_root_2(self, p): + """root : root module""" + if p[1].module is not None: + raise ParseError( + self.filename, + "Multiple \"module\" statements not allowed:", + p[2].lineno, + snippet=self._GetSnippet(p[2].lineno)) + if p[1].import_list.items or p[1].definition_list: + raise ParseError( + self.filename, + "\"module\" statements must precede imports and definitions:", + p[2].lineno, + snippet=self._GetSnippet(p[2].lineno)) + p[0] = p[1] + p[0].module = p[2] + + def p_root_3(self, p): + """root : root import""" + if p[1].definition_list: + raise ParseError( + self.filename, + "\"import\" statements must precede definitions:", + p[2].lineno, + snippet=self._GetSnippet(p[2].lineno)) + p[0] = p[1] + p[0].import_list.Append(p[2]) + + def p_root_4(self, p): + """root : root definition""" + p[0] = p[1] + p[0].definition_list.append(p[2]) + + def p_import(self, p): + """import : attribute_section IMPORT STRING_LITERAL SEMI""" + # 'eval' the literal to strip the quotes. + # TODO(vtl): This eval is dubious. We should unquote/unescape ourselves. + p[0] = ast.Import( + p[1], eval(p[3]), filename=self.filename, lineno=p.lineno(2)) + + def p_module(self, p): + """module : attribute_section MODULE identifier_wrapped SEMI""" + p[0] = ast.Module(p[3], p[1], filename=self.filename, lineno=p.lineno(2)) + + def p_definition(self, p): + """definition : struct + | union + | interface + | enum + | const""" + p[0] = p[1] + + def p_attribute_section_1(self, p): + """attribute_section : """ + p[0] = None + + def p_attribute_section_2(self, p): + """attribute_section : LBRACKET attribute_list RBRACKET""" + p[0] = p[2] + + def p_attribute_list_1(self, p): + """attribute_list : """ + p[0] = ast.AttributeList() + + def p_attribute_list_2(self, p): + """attribute_list : nonempty_attribute_list""" + p[0] = p[1] + + def p_nonempty_attribute_list_1(self, p): + """nonempty_attribute_list : attribute""" + p[0] = ast.AttributeList(p[1]) + + def p_nonempty_attribute_list_2(self, p): + """nonempty_attribute_list : nonempty_attribute_list COMMA attribute""" + p[0] = p[1] + p[0].Append(p[3]) + + def p_attribute_1(self, p): + """attribute : NAME EQUALS evaled_literal + | NAME EQUALS NAME""" + p[0] = ast.Attribute(p[1], p[3], filename=self.filename, lineno=p.lineno(1)) + + def p_attribute_2(self, p): + """attribute : NAME""" + p[0] = ast.Attribute(p[1], True, filename=self.filename, lineno=p.lineno(1)) + + def p_evaled_literal(self, p): + """evaled_literal : literal""" + # 'eval' the literal to strip the quotes. Handle keywords "true" and "false" + # specially since they cannot directly be evaluated to python boolean + # values. + if p[1] == "true": + p[0] = True + elif p[1] == "false": + p[0] = False + else: + p[0] = eval(p[1]) + + def p_struct_1(self, p): + """struct : attribute_section STRUCT NAME LBRACE struct_body RBRACE SEMI""" + p[0] = ast.Struct(p[3], p[1], p[5]) + + def p_struct_2(self, p): + """struct : attribute_section STRUCT NAME SEMI""" + p[0] = ast.Struct(p[3], p[1], None) + + def p_struct_body_1(self, p): + """struct_body : """ + p[0] = ast.StructBody() + + def p_struct_body_2(self, p): + """struct_body : struct_body const + | struct_body enum + | struct_body struct_field""" + p[0] = p[1] + p[0].Append(p[2]) + + def p_struct_field(self, p): + """struct_field : attribute_section typename NAME ordinal default SEMI""" + p[0] = ast.StructField(p[3], p[1], p[4], p[2], p[5]) + + def p_union(self, p): + """union : attribute_section UNION NAME LBRACE union_body RBRACE SEMI""" + p[0] = ast.Union(p[3], p[1], p[5]) + + def p_union_body_1(self, p): + """union_body : """ + p[0] = ast.UnionBody() + + def p_union_body_2(self, p): + """union_body : union_body union_field""" + p[0] = p[1] + p[1].Append(p[2]) + + def p_union_field(self, p): + """union_field : attribute_section typename NAME ordinal SEMI""" + p[0] = ast.UnionField(p[3], p[1], p[4], p[2]) + + def p_default_1(self, p): + """default : """ + p[0] = None + + def p_default_2(self, p): + """default : EQUALS constant""" + p[0] = p[2] + + def p_interface(self, p): + """interface : attribute_section INTERFACE NAME LBRACE interface_body \ + RBRACE SEMI""" + p[0] = ast.Interface(p[3], p[1], p[5]) + + def p_interface_body_1(self, p): + """interface_body : """ + p[0] = ast.InterfaceBody() + + def p_interface_body_2(self, p): + """interface_body : interface_body const + | interface_body enum + | interface_body method""" + p[0] = p[1] + p[0].Append(p[2]) + + def p_response_1(self, p): + """response : """ + p[0] = None + + def p_response_2(self, p): + """response : RESPONSE LPAREN parameter_list RPAREN""" + p[0] = p[3] + + def p_method(self, p): + """method : attribute_section NAME ordinal LPAREN parameter_list RPAREN \ + response SEMI""" + p[0] = ast.Method(p[2], p[1], p[3], p[5], p[7]) + + def p_parameter_list_1(self, p): + """parameter_list : """ + p[0] = ast.ParameterList() + + def p_parameter_list_2(self, p): + """parameter_list : nonempty_parameter_list""" + p[0] = p[1] + + def p_nonempty_parameter_list_1(self, p): + """nonempty_parameter_list : parameter""" + p[0] = ast.ParameterList(p[1]) + + def p_nonempty_parameter_list_2(self, p): + """nonempty_parameter_list : nonempty_parameter_list COMMA parameter""" + p[0] = p[1] + p[0].Append(p[3]) + + def p_parameter(self, p): + """parameter : attribute_section typename NAME ordinal""" + p[0] = ast.Parameter( + p[3], p[1], p[4], p[2], filename=self.filename, lineno=p.lineno(3)) + + def p_typename(self, p): + """typename : nonnullable_typename QSTN + | nonnullable_typename""" + if len(p) == 2: + p[0] = p[1] + else: + p[0] = p[1] + "?" + + def p_nonnullable_typename(self, p): + """nonnullable_typename : basictypename + | array + | fixed_array + | associative_array + | interfacerequest""" + p[0] = p[1] + + def p_basictypename(self, p): + """basictypename : remotetype + | receivertype + | associatedremotetype + | associatedreceivertype + | identifier + | ASSOCIATED identifier + | handletype""" + if len(p) == 2: + p[0] = p[1] + else: + p[0] = "asso<" + p[2] + ">" + + def p_remotetype(self, p): + """remotetype : PENDING_REMOTE LANGLE identifier RANGLE""" + p[0] = "rmt<%s>" % p[3] + + def p_receivertype(self, p): + """receivertype : PENDING_RECEIVER LANGLE identifier RANGLE""" + p[0] = "rcv<%s>" % p[3] + + def p_associatedremotetype(self, p): + """associatedremotetype : PENDING_ASSOCIATED_REMOTE LANGLE identifier \ + RANGLE""" + p[0] = "rma<%s>" % p[3] + + def p_associatedreceivertype(self, p): + """associatedreceivertype : PENDING_ASSOCIATED_RECEIVER LANGLE identifier \ + RANGLE""" + p[0] = "rca<%s>" % p[3] + + def p_handletype(self, p): + """handletype : HANDLE + | HANDLE LANGLE NAME RANGLE""" + if len(p) == 2: + p[0] = p[1] + else: + if p[3] not in ('data_pipe_consumer', 'data_pipe_producer', + 'message_pipe', 'shared_buffer', 'platform'): + # Note: We don't enable tracking of line numbers for everything, so we + # can't use |p.lineno(3)|. + raise ParseError( + self.filename, + "Invalid handle type %r:" % p[3], + lineno=p.lineno(1), + snippet=self._GetSnippet(p.lineno(1))) + p[0] = "handle<" + p[3] + ">" + + def p_array(self, p): + """array : ARRAY LANGLE typename RANGLE""" + p[0] = p[3] + "[]" + + def p_fixed_array(self, p): + """fixed_array : ARRAY LANGLE typename COMMA INT_CONST_DEC RANGLE""" + value = int(p[5]) + if value == 0 or value > _MAX_ARRAY_SIZE: + raise ParseError( + self.filename, + "Fixed array size %d invalid:" % value, + lineno=p.lineno(5), + snippet=self._GetSnippet(p.lineno(5))) + p[0] = p[3] + "[" + p[5] + "]" + + def p_associative_array(self, p): + """associative_array : MAP LANGLE identifier COMMA typename RANGLE""" + p[0] = p[5] + "{" + p[3] + "}" + + def p_interfacerequest(self, p): + """interfacerequest : identifier AMP + | ASSOCIATED identifier AMP""" + if len(p) == 3: + p[0] = p[1] + "&" + else: + p[0] = "asso<" + p[2] + "&>" + + def p_ordinal_1(self, p): + """ordinal : """ + p[0] = None + + def p_ordinal_2(self, p): + """ordinal : ORDINAL""" + value = int(p[1][1:]) + if value > _MAX_ORDINAL_VALUE: + raise ParseError( + self.filename, + "Ordinal value %d too large:" % value, + lineno=p.lineno(1), + snippet=self._GetSnippet(p.lineno(1))) + p[0] = ast.Ordinal(value, filename=self.filename, lineno=p.lineno(1)) + + def p_enum_1(self, p): + """enum : attribute_section ENUM NAME LBRACE enum_value_list \ + RBRACE SEMI + | attribute_section ENUM NAME LBRACE nonempty_enum_value_list \ + COMMA RBRACE SEMI""" + p[0] = ast.Enum( + p[3], p[1], p[5], filename=self.filename, lineno=p.lineno(2)) + + def p_enum_2(self, p): + """enum : attribute_section ENUM NAME SEMI""" + p[0] = ast.Enum( + p[3], p[1], None, filename=self.filename, lineno=p.lineno(2)) + + def p_enum_value_list_1(self, p): + """enum_value_list : """ + p[0] = ast.EnumValueList() + + def p_enum_value_list_2(self, p): + """enum_value_list : nonempty_enum_value_list""" + p[0] = p[1] + + def p_nonempty_enum_value_list_1(self, p): + """nonempty_enum_value_list : enum_value""" + p[0] = ast.EnumValueList(p[1]) + + def p_nonempty_enum_value_list_2(self, p): + """nonempty_enum_value_list : nonempty_enum_value_list COMMA enum_value""" + p[0] = p[1] + p[0].Append(p[3]) + + def p_enum_value(self, p): + """enum_value : attribute_section NAME + | attribute_section NAME EQUALS int + | attribute_section NAME EQUALS identifier_wrapped""" + p[0] = ast.EnumValue( + p[2], + p[1], + p[4] if len(p) == 5 else None, + filename=self.filename, + lineno=p.lineno(2)) + + def p_const(self, p): + """const : attribute_section CONST typename NAME EQUALS constant SEMI""" + p[0] = ast.Const(p[4], p[1], p[3], p[6]) + + def p_constant(self, p): + """constant : literal + | identifier_wrapped""" + p[0] = p[1] + + def p_identifier_wrapped(self, p): + """identifier_wrapped : identifier""" + p[0] = ('IDENTIFIER', p[1]) + + # TODO(vtl): Make this produce a "wrapped" identifier (probably as an + # |ast.Identifier|, to be added) and get rid of identifier_wrapped. + def p_identifier(self, p): + """identifier : NAME + | NAME DOT identifier""" + p[0] = ''.join(p[1:]) + + def p_literal(self, p): + """literal : int + | float + | TRUE + | FALSE + | DEFAULT + | STRING_LITERAL""" + p[0] = p[1] + + def p_int(self, p): + """int : int_const + | PLUS int_const + | MINUS int_const""" + p[0] = ''.join(p[1:]) + + def p_int_const(self, p): + """int_const : INT_CONST_DEC + | INT_CONST_HEX""" + p[0] = p[1] + + def p_float(self, p): + """float : FLOAT_CONST + | PLUS FLOAT_CONST + | MINUS FLOAT_CONST""" + p[0] = ''.join(p[1:]) + + def p_error(self, e): + if e is None: + # Unexpected EOF. + # TODO(vtl): Can we figure out what's missing? + raise ParseError(self.filename, "Unexpected end of file") + + raise ParseError( + self.filename, + "Unexpected %r:" % e.value, + lineno=e.lineno, + snippet=self._GetSnippet(e.lineno)) + + def _GetSnippet(self, lineno): + return self.source.split('\n')[lineno - 1] + + +def Parse(source, filename): + """Parse source file to AST. + + Args: + source: The source text as a str (Python 2 or 3) or unicode (Python 2). + filename: The filename that |source| originates from. + + Returns: + The AST as a mojom.parse.ast.Mojom object. + """ + lexer = Lexer(filename) + parser = Parser(lexer, source, filename) + + lex.lex(object=lexer) + yacc.yacc(module=parser, debug=0, write_tables=0) + + tree = yacc.parse(source) + return tree diff --git a/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py new file mode 100644 index 00000000..6d6b7153 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py @@ -0,0 +1,1390 @@ +# Copyright 2014 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import imp +import os.path +import sys +import unittest + +from mojom.parse import ast +from mojom.parse import lexer +from mojom.parse import parser + + +class ParserTest(unittest.TestCase): + """Tests |parser.Parse()|.""" + + def testTrivialValidSource(self): + """Tests a trivial, but valid, .mojom source.""" + + source = """\ + // This is a comment. + + module my_module; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), []) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testSourceWithCrLfs(self): + """Tests a .mojom source with CR-LFs instead of LFs.""" + + source = "// This is a comment.\r\n\r\nmodule my_module;\r\n" + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), []) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testUnexpectedEOF(self): + """Tests a "truncated" .mojom source.""" + + source = """\ + // This is a comment. + + module my_module + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom: Error: Unexpected end of file$"): + parser.Parse(source, "my_file.mojom") + + def testCommentLineNumbers(self): + """Tests that line numbers are correctly tracked when comments are + present.""" + + source1 = """\ + // Isolated C++-style comments. + + // Foo. + asdf1 + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:4: Error: Unexpected 'asdf1':\n *asdf1$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + // Consecutive C++-style comments. + // Foo. + // Bar. + + struct Yada { // Baz. + // Quux. + int32 x; + }; + + asdf2 + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:10: Error: Unexpected 'asdf2':\n *asdf2$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + /* Single-line C-style comments. */ + /* Foobar. */ + + /* Baz. */ + asdf3 + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:5: Error: Unexpected 'asdf3':\n *asdf3$"): + parser.Parse(source3, "my_file.mojom") + + source4 = """\ + /* Multi-line C-style comments. + */ + /* + Foo. + Bar. + */ + + /* Baz + Quux. */ + asdf4 + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:10: Error: Unexpected 'asdf4':\n *asdf4$"): + parser.Parse(source4, "my_file.mojom") + + def testSimpleStruct(self): + """Tests a simple .mojom source that just defines a struct.""" + + source = """\ + module my_module; + + struct MyStruct { + int32 a; + double b; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a', None, None, 'int32', None), + ast.StructField('b', None, None, 'double', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testSimpleStructWithoutModule(self): + """Tests a simple struct without an explict module statement.""" + + source = """\ + struct MyStruct { + int32 a; + double b; + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a', None, None, 'int32', None), + ast.StructField('b', None, None, 'double', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testValidStructDefinitions(self): + """Tests all types of definitions that can occur in a struct.""" + + source = """\ + struct MyStruct { + enum MyEnum { VALUE }; + const double kMyConst = 1.23; + int32 a; + SomeOtherStruct b; // Invalidity detected at another stage. + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.Enum('MyEnum', None, + ast.EnumValueList(ast.EnumValue('VALUE', None, None))), + ast.Const('kMyConst', None, 'double', '1.23'), + ast.StructField('a', None, None, 'int32', None), + ast.StructField('b', None, None, 'SomeOtherStruct', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidStructDefinitions(self): + """Tests that definitions that aren't allowed in a struct are correctly + detected.""" + + source1 = """\ + struct MyStruct { + MyMethod(int32 a); + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '\(':\n" + r" *MyMethod\(int32 a\);$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + struct MyStruct { + struct MyInnerStruct { + int32 a; + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'struct':\n" + r" *struct MyInnerStruct {$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + struct MyStruct { + interface MyInterface { + MyMethod(int32 a); + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Unexpected 'interface':\n" + r" *interface MyInterface {$"): + parser.Parse(source3, "my_file.mojom") + + def testMissingModuleName(self): + """Tests an (invalid) .mojom with a missing module name.""" + + source1 = """\ + // Missing module name. + module ; + struct MyStruct { + int32 a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Unexpected ';':\n *module ;$"): + parser.Parse(source1, "my_file.mojom") + + # Another similar case, but make sure that line-number tracking/reporting + # is correct. + source2 = """\ + module + // This line intentionally left unblank. + + struct MyStruct { + int32 a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:4: Error: Unexpected 'struct':\n" + r" *struct MyStruct {$"): + parser.Parse(source2, "my_file.mojom") + + def testMultipleModuleStatements(self): + """Tests an (invalid) .mojom with multiple module statements.""" + + source = """\ + module foo; + module bar; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Multiple \"module\" statements not " + r"allowed:\n *module bar;$"): + parser.Parse(source, "my_file.mojom") + + def testModuleStatementAfterImport(self): + """Tests an (invalid) .mojom with a module statement after an import.""" + + source = """\ + import "foo.mojom"; + module foo; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: \"module\" statements must precede imports " + r"and definitions:\n *module foo;$"): + parser.Parse(source, "my_file.mojom") + + def testModuleStatementAfterDefinition(self): + """Tests an (invalid) .mojom with a module statement after a definition.""" + + source = """\ + struct MyStruct { + int32 a; + }; + module foo; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:4: Error: \"module\" statements must precede imports " + r"and definitions:\n *module foo;$"): + parser.Parse(source, "my_file.mojom") + + def testImportStatementAfterDefinition(self): + """Tests an (invalid) .mojom with an import statement after a definition.""" + + source = """\ + struct MyStruct { + int32 a; + }; + import "foo.mojom"; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:4: Error: \"import\" statements must precede " + r"definitions:\n *import \"foo.mojom\";$"): + parser.Parse(source, "my_file.mojom") + + def testEnums(self): + """Tests that enum statements are correctly parsed.""" + + source = """\ + module my_module; + enum MyEnum1 { VALUE1, VALUE2 }; // No trailing comma. + enum MyEnum2 { + VALUE1 = -1, + VALUE2 = 0, + VALUE3 = + 987, // Check that space is allowed. + VALUE4 = 0xAF12, + VALUE5 = -0x09bcd, + VALUE6 = VALUE5, + VALUE7, // Leave trailing comma. + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Enum( + 'MyEnum1', None, + ast.EnumValueList([ + ast.EnumValue('VALUE1', None, None), + ast.EnumValue('VALUE2', None, None) + ])), + ast.Enum( + 'MyEnum2', None, + ast.EnumValueList([ + ast.EnumValue('VALUE1', None, '-1'), + ast.EnumValue('VALUE2', None, '0'), + ast.EnumValue('VALUE3', None, '+987'), + ast.EnumValue('VALUE4', None, '0xAF12'), + ast.EnumValue('VALUE5', None, '-0x09bcd'), + ast.EnumValue('VALUE6', None, ('IDENTIFIER', 'VALUE5')), + ast.EnumValue('VALUE7', None, None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidEnumInitializers(self): + """Tests that invalid enum initializers are correctly detected.""" + + # Floating point value. + source2 = "enum MyEnum { VALUE = 0.123 };" + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:1: Error: Unexpected '0\.123':\n" + r"enum MyEnum { VALUE = 0\.123 };$"): + parser.Parse(source2, "my_file.mojom") + + # Boolean value. + source2 = "enum MyEnum { VALUE = true };" + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:1: Error: Unexpected 'true':\n" + r"enum MyEnum { VALUE = true };$"): + parser.Parse(source2, "my_file.mojom") + + def testConsts(self): + """Tests some constants and struct members initialized with them.""" + + source = """\ + module my_module; + + struct MyStruct { + const int8 kNumber = -1; + int8 number@0 = kNumber; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.Const('kNumber', None, 'int8', '-1'), + ast.StructField('number', None, ast.Ordinal(0), 'int8', + ('IDENTIFIER', 'kNumber')) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testNoConditionals(self): + """Tests that ?: is not allowed.""" + + source = """\ + module my_module; + + enum MyEnum { + MY_ENUM_1 = 1 ? 2 : 3 + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:4: Error: Unexpected '\?':\n" + r" *MY_ENUM_1 = 1 \? 2 : 3$"): + parser.Parse(source, "my_file.mojom") + + def testSimpleOrdinals(self): + """Tests that (valid) ordinal values are scanned correctly.""" + + source = """\ + module my_module; + + // This isn't actually valid .mojom, but the problem (missing ordinals) + // should be handled at a different level. + struct MyStruct { + int32 a0@0; + int32 a1@1; + int32 a2@2; + int32 a9@9; + int32 a10 @10; + int32 a11 @11; + int32 a29 @29; + int32 a1234567890 @1234567890; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a0', None, ast.Ordinal(0), 'int32', None), + ast.StructField('a1', None, ast.Ordinal(1), 'int32', None), + ast.StructField('a2', None, ast.Ordinal(2), 'int32', None), + ast.StructField('a9', None, ast.Ordinal(9), 'int32', None), + ast.StructField('a10', None, ast.Ordinal(10), 'int32', + None), + ast.StructField('a11', None, ast.Ordinal(11), 'int32', + None), + ast.StructField('a29', None, ast.Ordinal(29), 'int32', + None), + ast.StructField('a1234567890', None, + ast.Ordinal(1234567890), 'int32', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidOrdinals(self): + """Tests that (lexically) invalid ordinals are correctly detected.""" + + source1 = """\ + module my_module; + + struct MyStruct { + int32 a_missing@; + }; + """ + with self.assertRaisesRegexp( + lexer.LexError, r"^my_file\.mojom:4: Error: Missing ordinal value$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + module my_module; + + struct MyStruct { + int32 a_octal@01; + }; + """ + with self.assertRaisesRegexp( + lexer.LexError, r"^my_file\.mojom:4: Error: " + r"Octal and hexadecimal ordinal values not allowed$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + module my_module; struct MyStruct { int32 a_invalid_octal@08; }; + """ + with self.assertRaisesRegexp( + lexer.LexError, r"^my_file\.mojom:1: Error: " + r"Octal and hexadecimal ordinal values not allowed$"): + parser.Parse(source3, "my_file.mojom") + + source4 = "module my_module; struct MyStruct { int32 a_hex@0x1aB9; };" + with self.assertRaisesRegexp( + lexer.LexError, r"^my_file\.mojom:1: Error: " + r"Octal and hexadecimal ordinal values not allowed$"): + parser.Parse(source4, "my_file.mojom") + + source5 = "module my_module; struct MyStruct { int32 a_hex@0X0; };" + with self.assertRaisesRegexp( + lexer.LexError, r"^my_file\.mojom:1: Error: " + r"Octal and hexadecimal ordinal values not allowed$"): + parser.Parse(source5, "my_file.mojom") + + source6 = """\ + struct MyStruct { + int32 a_too_big@999999999999; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: " + r"Ordinal value 999999999999 too large:\n" + r" *int32 a_too_big@999999999999;$"): + parser.Parse(source6, "my_file.mojom") + + def testNestedNamespace(self): + """Tests that "nested" namespaces work.""" + + source = """\ + module my.mod; + + struct MyStruct { + int32 a; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my.mod'), None), ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody(ast.StructField('a', None, None, 'int32', None))) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testValidHandleTypes(self): + """Tests (valid) handle types.""" + + source = """\ + struct MyStruct { + handle a; + handle b; + handle c; + handle < message_pipe > d; + handle + < shared_buffer + > e; + handle + f; + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a', None, None, 'handle', None), + ast.StructField('b', None, None, 'handle', + None), + ast.StructField('c', None, None, 'handle', + None), + ast.StructField('d', None, None, 'handle', None), + ast.StructField('e', None, None, 'handle', None), + ast.StructField('f', None, None, 'handle', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidHandleType(self): + """Tests an invalid (unknown) handle type.""" + + source = """\ + struct MyStruct { + handle foo; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: " + r"Invalid handle type 'wtf_is_this':\n" + r" *handle foo;$"): + parser.Parse(source, "my_file.mojom") + + def testValidDefaultValues(self): + """Tests default values that are valid (to the parser).""" + + source = """\ + struct MyStruct { + int16 a0 = 0; + uint16 a1 = 0x0; + uint16 a2 = 0x00; + uint16 a3 = 0x01; + uint16 a4 = 0xcd; + int32 a5 = 12345; + int64 a6 = -12345; + int64 a7 = +12345; + uint32 a8 = 0x12cd3; + uint32 a9 = -0x12cD3; + uint32 a10 = +0x12CD3; + bool a11 = true; + bool a12 = false; + float a13 = 1.2345; + float a14 = -1.2345; + float a15 = +1.2345; + float a16 = 123.; + float a17 = .123; + double a18 = 1.23E10; + double a19 = 1.E-10; + double a20 = .5E+10; + double a21 = -1.23E10; + double a22 = +.123E10; + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a0', None, None, 'int16', '0'), + ast.StructField('a1', None, None, 'uint16', '0x0'), + ast.StructField('a2', None, None, 'uint16', '0x00'), + ast.StructField('a3', None, None, 'uint16', '0x01'), + ast.StructField('a4', None, None, 'uint16', '0xcd'), + ast.StructField('a5', None, None, 'int32', '12345'), + ast.StructField('a6', None, None, 'int64', '-12345'), + ast.StructField('a7', None, None, 'int64', '+12345'), + ast.StructField('a8', None, None, 'uint32', '0x12cd3'), + ast.StructField('a9', None, None, 'uint32', '-0x12cD3'), + ast.StructField('a10', None, None, 'uint32', '+0x12CD3'), + ast.StructField('a11', None, None, 'bool', 'true'), + ast.StructField('a12', None, None, 'bool', 'false'), + ast.StructField('a13', None, None, 'float', '1.2345'), + ast.StructField('a14', None, None, 'float', '-1.2345'), + ast.StructField('a15', None, None, 'float', '+1.2345'), + ast.StructField('a16', None, None, 'float', '123.'), + ast.StructField('a17', None, None, 'float', '.123'), + ast.StructField('a18', None, None, 'double', '1.23E10'), + ast.StructField('a19', None, None, 'double', '1.E-10'), + ast.StructField('a20', None, None, 'double', '.5E+10'), + ast.StructField('a21', None, None, 'double', '-1.23E10'), + ast.StructField('a22', None, None, 'double', '+.123E10') + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testValidFixedSizeArray(self): + """Tests parsing a fixed size array.""" + + source = """\ + struct MyStruct { + array normal_array; + array fixed_size_array_one_entry; + array fixed_size_array_ten_entries; + array>, 2> nested_arrays; + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('normal_array', None, None, 'int32[]', None), + ast.StructField('fixed_size_array_one_entry', None, None, + 'int32[1]', None), + ast.StructField('fixed_size_array_ten_entries', None, None, + 'int32[10]', None), + ast.StructField('nested_arrays', None, None, 'int32[1][][2]', + None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testValidNestedArray(self): + """Tests parsing a nested array.""" + + source = "struct MyStruct { array> nested_array; };" + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody( + ast.StructField('nested_array', None, None, 'int32[][]', None))) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidFixedArraySize(self): + """Tests that invalid fixed array bounds are correctly detected.""" + + source1 = """\ + struct MyStruct { + array zero_size_array; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Fixed array size 0 invalid:\n" + r" *array zero_size_array;$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + struct MyStruct { + array too_big_array; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Fixed array size 999999999999 invalid:\n" + r" *array too_big_array;$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + struct MyStruct { + array not_a_number; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'abcdefg':\n" + r" *array not_a_number;"): + parser.Parse(source3, "my_file.mojom") + + def testValidAssociativeArrays(self): + """Tests that we can parse valid associative array structures.""" + + source1 = "struct MyStruct { map data; };" + expected1 = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody( + [ast.StructField('data', None, None, 'uint8{string}', None)])) + ]) + self.assertEquals(parser.Parse(source1, "my_file.mojom"), expected1) + + source2 = "interface MyInterface { MyMethod(map a); };" + expected2 = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody( + ast.Method( + 'MyMethod', None, None, + ast.ParameterList( + ast.Parameter('a', None, None, 'uint8{string}')), + None))) + ]) + self.assertEquals(parser.Parse(source2, "my_file.mojom"), expected2) + + source3 = "struct MyStruct { map> data; };" + expected3 = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody( + [ast.StructField('data', None, None, 'uint8[]{string}', None)])) + ]) + self.assertEquals(parser.Parse(source3, "my_file.mojom"), expected3) + + def testValidMethod(self): + """Tests parsing method declarations.""" + + source1 = "interface MyInterface { MyMethod(int32 a); };" + expected1 = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody( + ast.Method( + 'MyMethod', None, None, + ast.ParameterList(ast.Parameter('a', None, None, 'int32')), + None))) + ]) + self.assertEquals(parser.Parse(source1, "my_file.mojom"), expected1) + + source2 = """\ + interface MyInterface { + MyMethod1@0(int32 a@0, int64 b@1); + MyMethod2@1() => (); + }; + """ + expected2 = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody([ + ast.Method( + 'MyMethod1', None, ast.Ordinal(0), + ast.ParameterList([ + ast.Parameter('a', None, ast.Ordinal(0), 'int32'), + ast.Parameter('b', None, ast.Ordinal(1), 'int64') + ]), None), + ast.Method('MyMethod2', None, ast.Ordinal(1), + ast.ParameterList(), ast.ParameterList()) + ])) + ]) + self.assertEquals(parser.Parse(source2, "my_file.mojom"), expected2) + + source3 = """\ + interface MyInterface { + MyMethod(string a) => (int32 a, bool b); + }; + """ + expected3 = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody( + ast.Method( + 'MyMethod', None, None, + ast.ParameterList(ast.Parameter('a', None, None, 'string')), + ast.ParameterList([ + ast.Parameter('a', None, None, 'int32'), + ast.Parameter('b', None, None, 'bool') + ])))) + ]) + self.assertEquals(parser.Parse(source3, "my_file.mojom"), expected3) + + def testInvalidMethods(self): + """Tests that invalid method declarations are correctly detected.""" + + # No trailing commas. + source1 = """\ + interface MyInterface { + MyMethod(string a,); + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '\)':\n" + r" *MyMethod\(string a,\);$"): + parser.Parse(source1, "my_file.mojom") + + # No leading commas. + source2 = """\ + interface MyInterface { + MyMethod(, string a); + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected ',':\n" + r" *MyMethod\(, string a\);$"): + parser.Parse(source2, "my_file.mojom") + + def testValidInterfaceDefinitions(self): + """Tests all types of definitions that can occur in an interface.""" + + source = """\ + interface MyInterface { + enum MyEnum { VALUE }; + const int32 kMyConst = 123; + MyMethod(int32 x) => (MyEnum y); + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody([ + ast.Enum('MyEnum', None, + ast.EnumValueList(ast.EnumValue('VALUE', None, None))), + ast.Const('kMyConst', None, 'int32', '123'), + ast.Method( + 'MyMethod', None, None, + ast.ParameterList(ast.Parameter('x', None, None, 'int32')), + ast.ParameterList(ast.Parameter('y', None, None, 'MyEnum'))) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidInterfaceDefinitions(self): + """Tests that definitions that aren't allowed in an interface are correctly + detected.""" + + source1 = """\ + interface MyInterface { + struct MyStruct { + int32 a; + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'struct':\n" + r" *struct MyStruct {$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + interface MyInterface { + interface MyInnerInterface { + MyMethod(int32 x); + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Unexpected 'interface':\n" + r" *interface MyInnerInterface {$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + interface MyInterface { + int32 my_field; + }; + """ + # The parser thinks that "int32" is a plausible name for a method, so it's + # "my_field" that gives it away. + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'my_field':\n" + r" *int32 my_field;$"): + parser.Parse(source3, "my_file.mojom") + + def testValidAttributes(self): + """Tests parsing attributes (and attribute lists).""" + + # Note: We use structs because they have (optional) attribute lists. + + # Empty attribute list. + source1 = "[] struct MyStruct {};" + expected1 = ast.Mojom( + None, ast.ImportList(), + [ast.Struct('MyStruct', ast.AttributeList(), ast.StructBody())]) + self.assertEquals(parser.Parse(source1, "my_file.mojom"), expected1) + + # One-element attribute list, with name value. + source2 = "[MyAttribute=MyName] struct MyStruct {};" + expected2 = ast.Mojom(None, ast.ImportList(), [ + ast.Struct('MyStruct', + ast.AttributeList(ast.Attribute("MyAttribute", "MyName")), + ast.StructBody()) + ]) + self.assertEquals(parser.Parse(source2, "my_file.mojom"), expected2) + + # Two-element attribute list, with one string value and one integer value. + source3 = "[MyAttribute1 = \"hello\", MyAttribute2 = 5] struct MyStruct {};" + expected3 = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', + ast.AttributeList([ + ast.Attribute("MyAttribute1", "hello"), + ast.Attribute("MyAttribute2", 5) + ]), ast.StructBody()) + ]) + self.assertEquals(parser.Parse(source3, "my_file.mojom"), expected3) + + # Various places that attribute list is allowed. + source4 = """\ + [Attr0=0] module my_module; + + [Attr1=1] import "my_import"; + + [Attr2=2] struct MyStruct { + [Attr3=3] int32 a; + }; + [Attr4=4] union MyUnion { + [Attr5=5] int32 a; + }; + [Attr6=6] enum MyEnum { + [Attr7=7] a + }; + [Attr8=8] interface MyInterface { + [Attr9=9] MyMethod([Attr10=10] int32 a) => ([Attr11=11] bool b); + }; + [Attr12=12] const double kMyConst = 1.23; + """ + expected4 = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), + ast.AttributeList([ast.Attribute("Attr0", 0)])), + ast.ImportList( + ast.Import( + ast.AttributeList([ast.Attribute("Attr1", 1)]), "my_import")), + [ + ast.Struct( + 'MyStruct', ast.AttributeList(ast.Attribute("Attr2", 2)), + ast.StructBody( + ast.StructField( + 'a', ast.AttributeList([ast.Attribute("Attr3", 3)]), + None, 'int32', None))), + ast.Union( + 'MyUnion', ast.AttributeList(ast.Attribute("Attr4", 4)), + ast.UnionBody( + ast.UnionField( + 'a', ast.AttributeList([ast.Attribute("Attr5", 5)]), + None, 'int32'))), + ast.Enum( + 'MyEnum', ast.AttributeList(ast.Attribute("Attr6", 6)), + ast.EnumValueList( + ast.EnumValue( + 'VALUE', ast.AttributeList([ast.Attribute("Attr7", 7)]), + None))), + ast.Interface( + 'MyInterface', ast.AttributeList(ast.Attribute("Attr8", 8)), + ast.InterfaceBody( + ast.Method( + 'MyMethod', ast.AttributeList( + ast.Attribute("Attr9", 9)), None, + ast.ParameterList( + ast.Parameter( + 'a', + ast.AttributeList([ast.Attribute("Attr10", 10) + ]), None, 'int32')), + ast.ParameterList( + ast.Parameter( + 'b', + ast.AttributeList([ast.Attribute("Attr11", 11) + ]), None, 'bool'))))), + ast.Const('kMyConst', ast.AttributeList( + ast.Attribute("Attr12", 12)), 'double', '1.23') + ]) + self.assertEquals(parser.Parse(source4, "my_file.mojom"), expected4) + + # TODO(vtl): Boolean attributes don't work yet. (In fact, we just |eval()| + # literal (non-name) values, which is extremely dubious.) + + def testInvalidAttributes(self): + """Tests that invalid attributes and attribute lists are correctly + detected.""" + + # Trailing commas not allowed. + source1 = "[MyAttribute=MyName,] struct MyStruct {};" + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:1: Error: Unexpected '\]':\n" + r"\[MyAttribute=MyName,\] struct MyStruct {};$"): + parser.Parse(source1, "my_file.mojom") + + # Missing value. + source2 = "[MyAttribute=] struct MyStruct {};" + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:1: Error: Unexpected '\]':\n" + r"\[MyAttribute=\] struct MyStruct {};$"): + parser.Parse(source2, "my_file.mojom") + + # Missing key. + source3 = "[=MyName] struct MyStruct {};" + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:1: Error: Unexpected '=':\n" + r"\[=MyName\] struct MyStruct {};$"): + parser.Parse(source3, "my_file.mojom") + + def testValidImports(self): + """Tests parsing import statements.""" + + # One import (no module statement). + source1 = "import \"somedir/my.mojom\";" + expected1 = ast.Mojom(None, + ast.ImportList(ast.Import(None, "somedir/my.mojom")), + []) + self.assertEquals(parser.Parse(source1, "my_file.mojom"), expected1) + + # Two imports (no module statement). + source2 = """\ + import "somedir/my1.mojom"; + import "somedir/my2.mojom"; + """ + expected2 = ast.Mojom( + None, + ast.ImportList([ + ast.Import(None, "somedir/my1.mojom"), + ast.Import(None, "somedir/my2.mojom") + ]), []) + self.assertEquals(parser.Parse(source2, "my_file.mojom"), expected2) + + # Imports with module statement. + source3 = """\ + module my_module; + import "somedir/my1.mojom"; + import "somedir/my2.mojom"; + """ + expected3 = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), + ast.ImportList([ + ast.Import(None, "somedir/my1.mojom"), + ast.Import(None, "somedir/my2.mojom") + ]), []) + self.assertEquals(parser.Parse(source3, "my_file.mojom"), expected3) + + def testInvalidImports(self): + """Tests that invalid import statements are correctly detected.""" + + source1 = """\ + // Make the error occur on line 2. + import invalid + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'invalid':\n" + r" *import invalid$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + import // Missing string. + struct MyStruct { + int32 a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'struct':\n" + r" *struct MyStruct {$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + import "foo.mojom" // Missing semicolon. + struct MyStruct { + int32 a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'struct':\n" + r" *struct MyStruct {$"): + parser.Parse(source3, "my_file.mojom") + + def testValidNullableTypes(self): + """Tests parsing nullable types.""" + + source = """\ + struct MyStruct { + int32? a; // This is actually invalid, but handled at a different + // level. + string? b; + array ? c; + array ? d; + array?>? e; + array? f; + array? g; + some_struct? h; + handle? i; + handle? j; + handle? k; + handle? l; + handle? m; + some_interface&? n; + handle? o; + }; + """ + expected = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a', None, None, 'int32?', None), + ast.StructField('b', None, None, 'string?', None), + ast.StructField('c', None, None, 'int32[]?', None), + ast.StructField('d', None, None, 'string?[]?', None), + ast.StructField('e', None, None, 'int32[]?[]?', None), + ast.StructField('f', None, None, 'int32[1]?', None), + ast.StructField('g', None, None, 'string?[1]?', None), + ast.StructField('h', None, None, 'some_struct?', None), + ast.StructField('i', None, None, 'handle?', None), + ast.StructField('j', None, None, 'handle?', + None), + ast.StructField('k', None, None, 'handle?', + None), + ast.StructField('l', None, None, 'handle?', None), + ast.StructField('m', None, None, 'handle?', + None), + ast.StructField('n', None, None, 'some_interface&?', None), + ast.StructField('o', None, None, 'handle?', None) + ])) + ]) + self.assertEquals(parser.Parse(source, "my_file.mojom"), expected) + + def testInvalidNullableTypes(self): + """Tests that invalid nullable types are correctly detected.""" + source1 = """\ + struct MyStruct { + string?? a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '\?':\n" + r" *string\?\? a;$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + struct MyStruct { + handle? a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '<':\n" + r" *handle\? a;$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + struct MyStruct { + some_interface?& a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '&':\n" + r" *some_interface\?& a;$"): + parser.Parse(source3, "my_file.mojom") + + def testSimpleUnion(self): + """Tests a simple .mojom source that just defines a union.""" + source = """\ + module my_module; + + union MyUnion { + int32 a; + double b; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Union( + 'MyUnion', None, + ast.UnionBody([ + ast.UnionField('a', None, None, 'int32'), + ast.UnionField('b', None, None, 'double') + ])) + ]) + actual = parser.Parse(source, "my_file.mojom") + self.assertEquals(actual, expected) + + def testUnionWithOrdinals(self): + """Test that ordinals are assigned to fields.""" + source = """\ + module my_module; + + union MyUnion { + int32 a @10; + double b @30; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Union( + 'MyUnion', None, + ast.UnionBody([ + ast.UnionField('a', None, ast.Ordinal(10), 'int32'), + ast.UnionField('b', None, ast.Ordinal(30), 'double') + ])) + ]) + actual = parser.Parse(source, "my_file.mojom") + self.assertEquals(actual, expected) + + def testUnionWithStructMembers(self): + """Test that struct members are accepted.""" + source = """\ + module my_module; + + union MyUnion { + SomeStruct s; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Union( + 'MyUnion', None, + ast.UnionBody([ast.UnionField('s', None, None, 'SomeStruct')])) + ]) + actual = parser.Parse(source, "my_file.mojom") + self.assertEquals(actual, expected) + + def testUnionWithArrayMember(self): + """Test that array members are accepted.""" + source = """\ + module my_module; + + union MyUnion { + array a; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Union( + 'MyUnion', None, + ast.UnionBody([ast.UnionField('a', None, None, 'int32[]')])) + ]) + actual = parser.Parse(source, "my_file.mojom") + self.assertEquals(actual, expected) + + def testUnionWithMapMember(self): + """Test that map members are accepted.""" + source = """\ + module my_module; + + union MyUnion { + map m; + }; + """ + expected = ast.Mojom( + ast.Module(('IDENTIFIER', 'my_module'), None), ast.ImportList(), [ + ast.Union( + 'MyUnion', None, + ast.UnionBody( + [ast.UnionField('m', None, None, 'string{int32}')])) + ]) + actual = parser.Parse(source, "my_file.mojom") + self.assertEquals(actual, expected) + + def testUnionDisallowNestedStruct(self): + """Tests that structs cannot be nested in unions.""" + source = """\ + module my_module; + + union MyUnion { + struct MyStruct { + int32 a; + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:4: Error: Unexpected 'struct':\n" + r" *struct MyStruct {$"): + parser.Parse(source, "my_file.mojom") + + def testUnionDisallowNestedInterfaces(self): + """Tests that interfaces cannot be nested in unions.""" + source = """\ + module my_module; + + union MyUnion { + interface MyInterface { + MyMethod(int32 a); + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:4: Error: Unexpected 'interface':\n" + r" *interface MyInterface {$"): + parser.Parse(source, "my_file.mojom") + + def testUnionDisallowNestedUnion(self): + """Tests that unions cannot be nested in unions.""" + source = """\ + module my_module; + + union MyUnion { + union MyOtherUnion { + int32 a; + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:4: Error: Unexpected 'union':\n" + r" *union MyOtherUnion {$"): + parser.Parse(source, "my_file.mojom") + + def testUnionDisallowNestedEnum(self): + """Tests that enums cannot be nested in unions.""" + source = """\ + module my_module; + + union MyUnion { + enum MyEnum { + A, + }; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:4: Error: Unexpected 'enum':\n" + r" *enum MyEnum {$"): + parser.Parse(source, "my_file.mojom") + + def testValidAssociatedKinds(self): + """Tests parsing associated interfaces and requests.""" + source1 = """\ + struct MyStruct { + associated MyInterface a; + associated MyInterface& b; + associated MyInterface? c; + associated MyInterface&? d; + }; + """ + expected1 = ast.Mojom(None, ast.ImportList(), [ + ast.Struct( + 'MyStruct', None, + ast.StructBody([ + ast.StructField('a', None, None, 'asso', None), + ast.StructField('b', None, None, 'asso', None), + ast.StructField('c', None, None, 'asso?', None), + ast.StructField('d', None, None, 'asso?', None) + ])) + ]) + self.assertEquals(parser.Parse(source1, "my_file.mojom"), expected1) + + source2 = """\ + interface MyInterface { + MyMethod(associated A a) =>(associated B& b); + };""" + expected2 = ast.Mojom(None, ast.ImportList(), [ + ast.Interface( + 'MyInterface', None, + ast.InterfaceBody( + ast.Method( + 'MyMethod', None, None, + ast.ParameterList( + ast.Parameter('a', None, None, 'asso')), + ast.ParameterList( + ast.Parameter('b', None, None, 'asso'))))) + ]) + self.assertEquals(parser.Parse(source2, "my_file.mojom"), expected2) + + def testInvalidAssociatedKinds(self): + """Tests that invalid associated interfaces and requests are correctly + detected.""" + source1 = """\ + struct MyStruct { + associated associated SomeInterface a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, + r"^my_file\.mojom:2: Error: Unexpected 'associated':\n" + r" *associated associated SomeInterface a;$"): + parser.Parse(source1, "my_file.mojom") + + source2 = """\ + struct MyStruct { + associated handle a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected 'handle':\n" + r" *associated handle a;$"): + parser.Parse(source2, "my_file.mojom") + + source3 = """\ + struct MyStruct { + associated? MyInterface& a; + }; + """ + with self.assertRaisesRegexp( + parser.ParseError, r"^my_file\.mojom:2: Error: Unexpected '\?':\n" + r" *associated\? MyInterface& a;$"): + parser.Parse(source3, "my_file.mojom") + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/ipc/mojo/public/tools/mojom/mojom_parser.py b/utils/ipc/mojo/public/tools/mojom/mojom_parser.py new file mode 100755 index 00000000..12adbfb9 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom_parser.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. +"""Parses mojom IDL files. + +This script parses one or more input mojom files and produces corresponding +module files fully describing the definitions contained within each mojom. The +module data is pickled and can be easily consumed by other tools to, e.g., +generate usable language bindings. +""" + +import argparse +import codecs +import errno +import json +import os +import os.path +import sys +from collections import defaultdict + +from mojom.generate import module +from mojom.generate import translate +from mojom.parse import parser +from mojom.parse import conditional_features + + +def _ResolveRelativeImportPath(path, roots): + """Attempts to resolve a relative import path against a set of possible roots. + + Args: + path: The relative import path to resolve. + roots: A list of absolute paths which will be checked in descending length + order for a match against path. + + Returns: + A normalized absolute path combining one of the roots with the input path if + and only if such a file exists. + + Raises: + ValueError: The path could not be resolved against any of the given roots. + """ + for root in reversed(sorted(roots, key=len)): + abs_path = os.path.join(root, path) + if os.path.isfile(abs_path): + return os.path.normcase(os.path.normpath(abs_path)) + + raise ValueError('"%s" does not exist in any of %s' % (path, roots)) + + +def _RebaseAbsolutePath(path, roots): + """Rewrites an absolute file path as relative to an absolute directory path in + roots. + + Args: + path: The absolute path of an existing file. + roots: A list of absolute directory paths. The given path argument must fall + within one of these directories. + + Returns: + A path equivalent to the input path, but relative to one of the provided + roots. If the input path falls within multiple roots, the longest root is + chosen (and thus the shortest relative path is returned). + + Paths returned by this method always use forward slashes as a separator to + mirror mojom import syntax. + + Raises: + ValueError if the given path does not fall within any of the listed roots. + """ + assert os.path.isabs(path) + assert os.path.isfile(path) + assert all(map(os.path.isabs, roots)) + + sorted_roots = list(reversed(sorted(roots, key=len))) + + def try_rebase_path(path, root): + head, rebased_path = os.path.split(path) + while head != root: + head, tail = os.path.split(head) + if not tail: + return None + rebased_path = os.path.join(tail, rebased_path) + return rebased_path + + for root in sorted_roots: + relative_path = try_rebase_path(path, root) + if relative_path: + # TODO(crbug.com/953884): Use pathlib for this kind of thing once we're + # fully migrated to Python 3. + return relative_path.replace('\\', '/') + + raise ValueError('%s does not fall within any of %s' % (path, sorted_roots)) + + +def _GetModuleFilename(mojom_filename): + return mojom_filename + '-module' + + +def _EnsureInputLoaded(mojom_abspath, module_path, abs_paths, asts, + dependencies, loaded_modules): + """Recursively ensures that a module and its dependencies are loaded. + + Args: + mojom_abspath: An absolute file path pointing to a mojom file to load. + module_path: The relative path used to identify mojom_abspath. + abs_paths: A mapping from module paths to absolute file paths for all + inputs given to this execution of the script. + asts: A map from each input mojom's absolute path to its parsed AST. + dependencies: A mapping of which input mojoms depend on each other, indexed + by absolute file path. + loaded_modules: A mapping of all modules loaded so far, including non-input + modules that were pulled in as transitive dependencies of the inputs. + import_set: The working set of mojom imports processed so far in this + call stack. Used to detect circular dependencies. + import_stack: An ordered list of imports processed so far in this call + stack. Used to report circular dependencies. + + Returns: + None + + On return, loaded_modules will be populated with the loaded input mojom's + Module as well as the Modules of all of its transitive dependencies.""" + + if mojom_abspath in loaded_modules: + # Already done. + return + + for dep_abspath, dep_path in dependencies[mojom_abspath]: + if dep_abspath not in loaded_modules: + _EnsureInputLoaded(dep_abspath, dep_path, abs_paths, asts, dependencies, + loaded_modules) + + imports = {} + for imp in asts[mojom_abspath].import_list: + path = imp.import_filename + imports[path] = loaded_modules[abs_paths[path]] + loaded_modules[mojom_abspath] = translate.OrderedModule( + asts[mojom_abspath], module_path, imports) + + +def _CollectAllowedImportsFromBuildMetadata(build_metadata_filename): + allowed_imports = set() + processed_deps = set() + + def collect(metadata_filename): + processed_deps.add(metadata_filename) + with open(metadata_filename) as f: + metadata = json.load(f) + allowed_imports.update( + map(os.path.normcase, map(os.path.normpath, metadata['sources']))) + for dep_metadata in metadata['deps']: + if dep_metadata not in processed_deps: + collect(dep_metadata) + + collect(build_metadata_filename) + return allowed_imports + + +def _ParseMojoms(mojom_files, + input_root_paths, + output_root_path, + enabled_features, + allowed_imports=None): + """Parses a set of mojom files and produces serialized module outputs. + + Args: + mojom_files: A list of mojom files to process. Paths must be absolute paths + which fall within one of the input or output root paths. + input_root_paths: A list of absolute filesystem paths which may be used to + resolve relative mojom file paths. + output_root_path: An absolute filesystem path which will service as the root + for all emitted artifacts. Artifacts produced from a given mojom file + are based on the mojom's relative path, rebased onto this path. + Additionally, the script expects this root to contain already-generated + modules for any transitive dependencies not listed in mojom_files. + enabled_features: A list of enabled feature names, controlling which AST + nodes are filtered by [EnableIf] attributes. + + Returns: + None. + + Upon completion, a mojom-module file will be saved for each input mojom. + """ + assert input_root_paths + assert output_root_path + + loaded_mojom_asts = {} + loaded_modules = {} + input_dependencies = defaultdict(set) + mojom_files_to_parse = dict((os.path.normcase(abs_path), + _RebaseAbsolutePath(abs_path, input_root_paths)) + for abs_path in mojom_files) + abs_paths = dict( + (path, abs_path) for abs_path, path in mojom_files_to_parse.items()) + for mojom_abspath, _ in mojom_files_to_parse.items(): + with codecs.open(mojom_abspath, encoding='utf-8') as f: + ast = parser.Parse(''.join(f.readlines()), mojom_abspath) + conditional_features.RemoveDisabledDefinitions(ast, enabled_features) + loaded_mojom_asts[mojom_abspath] = ast + invalid_imports = [] + for imp in ast.import_list: + import_abspath = _ResolveRelativeImportPath(imp.import_filename, + input_root_paths) + if allowed_imports and import_abspath not in allowed_imports: + invalid_imports.append(imp.import_filename) + + abs_paths[imp.import_filename] = import_abspath + if import_abspath in mojom_files_to_parse: + # This import is in the input list, so we're going to translate it + # into a module below; however it's also a dependency of another input + # module. We retain record of dependencies to help with input + # processing later. + input_dependencies[mojom_abspath].add((import_abspath, + imp.import_filename)) + else: + # We have an import that isn't being parsed right now. It must already + # be parsed and have a module file sitting in a corresponding output + # location. + module_path = _GetModuleFilename(imp.import_filename) + module_abspath = _ResolveRelativeImportPath(module_path, + [output_root_path]) + with open(module_abspath, 'rb') as module_file: + loaded_modules[import_abspath] = module.Module.Load(module_file) + + if invalid_imports: + raise ValueError( + '\nThe file %s imports the following files not allowed by build ' + 'dependencies:\n\n%s\n' % (mojom_abspath, + '\n'.join(invalid_imports))) + + + # At this point all transitive imports not listed as inputs have been loaded + # and we have a complete dependency tree of the unprocessed inputs. Now we can + # load all the inputs, resolving dependencies among them recursively as we go. + num_existing_modules_loaded = len(loaded_modules) + for mojom_abspath, mojom_path in mojom_files_to_parse.items(): + _EnsureInputLoaded(mojom_abspath, mojom_path, abs_paths, loaded_mojom_asts, + input_dependencies, loaded_modules) + assert (num_existing_modules_loaded + + len(mojom_files_to_parse) == len(loaded_modules)) + + # Now we have fully translated modules for every input and every transitive + # dependency. We can dump the modules to disk for other tools to use. + for mojom_abspath, mojom_path in mojom_files_to_parse.items(): + module_path = os.path.join(output_root_path, _GetModuleFilename(mojom_path)) + module_dir = os.path.dirname(module_path) + if not os.path.exists(module_dir): + try: + # Python 2 doesn't support exist_ok on makedirs(), so we just ignore + # that failure if it happens. It's possible during build due to races + # among build steps with module outputs in the same directory. + os.makedirs(module_dir) + except OSError as e: + if e.errno != errno.EEXIST: + raise + with open(module_path, 'wb') as f: + loaded_modules[mojom_abspath].Dump(f) + + +def Run(command_line): + arg_parser = argparse.ArgumentParser( + description=""" +Parses one or more mojom files and produces corresponding module outputs fully +describing the definitions therein. The output is exhaustive, stable, and +sufficient for another tool to consume and emit e.g. usable language +bindings based on the original mojoms.""", + epilog=""" +Note that each transitive import dependency reachable from the input mojoms must +either also be listed as an input or must have its corresponding compiled module +already present in the provided output root.""") + + arg_parser.add_argument( + '--input-root', + default=[], + action='append', + metavar='ROOT', + dest='input_root_paths', + help='Adds ROOT to the set of root paths against which relative input ' + 'paths should be resolved. Provided root paths are always searched ' + 'in order from longest absolute path to shortest.') + arg_parser.add_argument( + '--output-root', + action='store', + required=True, + dest='output_root_path', + metavar='ROOT', + help='Use ROOT as the root path in which the parser should emit compiled ' + 'modules for each processed input mojom. The path of emitted module is ' + 'based on the relative input path, rebased onto this root. Note that ' + 'ROOT is also searched for existing modules of any transitive imports ' + 'which were not included in the set of inputs.') + arg_parser.add_argument( + '--mojoms', + nargs='+', + dest='mojom_files', + default=[], + metavar='MOJOM_FILE', + help='Input mojom filename(s). Each filename must be either an absolute ' + 'path which falls within one of the given input or output roots, or a ' + 'relative path the parser will attempt to resolve using each of those ' + 'roots in unspecified order.') + arg_parser.add_argument( + '--mojom-file-list', + action='store', + metavar='LIST_FILENAME', + help='Input file whose contents are a list of mojoms to process. This ' + 'may be provided in lieu of --mojoms to avoid hitting command line ' + 'length limtations') + arg_parser.add_argument( + '--enable-feature', + dest='enabled_features', + default=[], + action='append', + metavar='FEATURE', + help='Enables a named feature when parsing the given mojoms. Features ' + 'are identified by arbitrary string values. Specifying this flag with a ' + 'given FEATURE name will cause the parser to process any syntax elements ' + 'tagged with an [EnableIf=FEATURE] attribute. If this flag is not ' + 'provided for a given FEATURE, such tagged elements are discarded by the ' + 'parser and will not be present in the compiled output.') + arg_parser.add_argument( + '--check-imports', + dest='build_metadata_filename', + action='store', + metavar='METADATA_FILENAME', + help='Instructs the parser to check imports against a set of allowed ' + 'imports. Allowed imports are based on build metadata within ' + 'METADATA_FILENAME. This is a JSON file with a `sources` key listing ' + 'paths to the set of input mojom files being processed by this parser ' + 'run, and a `deps` key listing paths to metadata files for any ' + 'dependencies of these inputs. This feature can be used to implement ' + 'build-time dependency checking for mojom imports, where each build ' + 'metadata file corresponds to a build target in the dependency graph of ' + 'a typical build system.') + + args, _ = arg_parser.parse_known_args(command_line) + if args.mojom_file_list: + with open(args.mojom_file_list) as f: + args.mojom_files.extend(f.read().split()) + + if not args.mojom_files: + raise ValueError( + 'Must list at least one mojom file via --mojoms or --mojom-file-list') + + mojom_files = list(map(os.path.abspath, args.mojom_files)) + input_roots = list(map(os.path.abspath, args.input_root_paths)) + output_root = os.path.abspath(args.output_root_path) + + if args.build_metadata_filename: + allowed_imports = _CollectAllowedImportsFromBuildMetadata( + args.build_metadata_filename) + else: + allowed_imports = None + + _ParseMojoms(mojom_files, input_roots, output_root, args.enabled_features, + allowed_imports) + + +if __name__ == '__main__': + Run(sys.argv[1:]) diff --git a/utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py b/utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py new file mode 100644 index 00000000..e213fbfa --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py @@ -0,0 +1,73 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import json +import os +import os.path +import shutil +import tempfile +import unittest + +import mojom_parser + +from mojom.generate import module + + +class MojomParserTestCase(unittest.TestCase): + """Tests covering the behavior defined by the main mojom_parser.py script. + This includes behavior around input and output path manipulation, dependency + resolution, and module serialization and deserialization.""" + + def __init__(self, method_name): + super(MojomParserTestCase, self).__init__(method_name) + self._temp_dir = None + + def setUp(self): + self._temp_dir = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self._temp_dir) + self._temp_dir = None + + def GetPath(self, path): + assert not os.path.isabs(path) + return os.path.join(self._temp_dir, path) + + def GetModulePath(self, path): + assert not os.path.isabs(path) + return os.path.join(self.GetPath('out'), path) + '-module' + + def WriteFile(self, path, contents): + full_path = self.GetPath(path) + dirname = os.path.dirname(full_path) + if not os.path.exists(dirname): + os.makedirs(dirname) + with open(full_path, 'w') as f: + f.write(contents) + + def LoadModule(self, mojom_path): + with open(self.GetModulePath(mojom_path), 'rb') as f: + return module.Module.Load(f) + + def ParseMojoms(self, mojoms, metadata=None): + """Parse all input mojoms relative the temp dir.""" + out_dir = self.GetPath('out') + args = [ + '--input-root', self._temp_dir, '--input-root', out_dir, + '--output-root', out_dir, '--mojoms' + ] + list(map(lambda mojom: os.path.join(self._temp_dir, mojom), mojoms)) + if metadata: + args.extend(['--check-imports', self.GetPath(metadata)]) + mojom_parser.Run(args) + + def ExtractTypes(self, mojom): + filename = 'test.mojom' + self.WriteFile(filename, mojom) + self.ParseMojoms([filename]) + m = self.LoadModule(filename) + definitions = {} + for kinds in (m.enums, m.structs, m.unions, m.interfaces): + for kind in kinds: + definitions[kind.mojom_name] = kind + return definitions diff --git a/utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py b/utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py new file mode 100644 index 00000000..a93f34ba --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py @@ -0,0 +1,171 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom_parser_test_case import MojomParserTestCase + + +class MojomParserTest(MojomParserTestCase): + """Tests covering the behavior defined by the main mojom_parser.py script. + This includes behavior around input and output path manipulation, dependency + resolution, and module serialization and deserialization.""" + + def testBasicParse(self): + """Basic test to verify that we can parse a mojom file and get a module.""" + mojom = 'foo/bar.mojom' + self.WriteFile( + mojom, """\ + module test; + enum TestEnum { kFoo }; + """) + self.ParseMojoms([mojom]) + + m = self.LoadModule(mojom) + self.assertEqual('foo/bar.mojom', m.path) + self.assertEqual('test', m.mojom_namespace) + self.assertEqual(1, len(m.enums)) + + def testBasicParseWithAbsolutePaths(self): + """Verifies that we can parse a mojom file given an absolute path input.""" + mojom = 'foo/bar.mojom' + self.WriteFile( + mojom, """\ + module test; + enum TestEnum { kFoo }; + """) + self.ParseMojoms([self.GetPath(mojom)]) + + m = self.LoadModule(mojom) + self.assertEqual('foo/bar.mojom', m.path) + self.assertEqual('test', m.mojom_namespace) + self.assertEqual(1, len(m.enums)) + + def testImport(self): + """Verify imports within the same set of mojom inputs.""" + a = 'a.mojom' + b = 'b.mojom' + self.WriteFile( + a, """\ + module a; + import "b.mojom"; + struct Foo { b.Bar bar; };""") + self.WriteFile(b, """\ + module b; + struct Bar {};""") + self.ParseMojoms([a, b]) + + ma = self.LoadModule(a) + mb = self.LoadModule(b) + self.assertEqual('a.mojom', ma.path) + self.assertEqual('b.mojom', mb.path) + self.assertEqual(1, len(ma.imports)) + self.assertEqual(mb, ma.imports[0]) + + def testPreProcessedImport(self): + """Verify imports processed by a previous parser execution can be loaded + properly when parsing a dependent mojom.""" + a = 'a.mojom' + self.WriteFile(a, """\ + module a; + struct Bar {};""") + self.ParseMojoms([a]) + + b = 'b.mojom' + self.WriteFile( + b, """\ + module b; + import "a.mojom"; + struct Foo { a.Bar bar; };""") + self.ParseMojoms([b]) + + def testMissingImport(self): + """Verify that an import fails if the imported mojom does not exist.""" + a = 'a.mojom' + self.WriteFile( + a, """\ + module a; + import "non-existent.mojom"; + struct Bar {};""") + with self.assertRaisesRegexp(ValueError, "does not exist"): + self.ParseMojoms([a]) + + def testUnparsedImport(self): + """Verify that an import fails if the imported mojom is not in the set of + mojoms provided to the parser on this execution AND there is no pre-existing + parsed output module already on disk for it.""" + a = 'a.mojom' + b = 'b.mojom' + self.WriteFile(a, """\ + module a; + struct Bar {};""") + self.WriteFile( + b, """\ + module b; + import "a.mojom"; + struct Foo { a.Bar bar; };""") + + # a.mojom has not been parsed yet, so its import will fail when processing + # b.mojom here. + with self.assertRaisesRegexp(ValueError, "does not exist"): + self.ParseMojoms([b]) + + def testCheckImportsBasic(self): + """Verify that the parser can handle --check-imports with a valid set of + inputs, including support for transitive dependency resolution.""" + a = 'a.mojom' + a_metadata = 'out/a.build_metadata' + b = 'b.mojom' + b_metadata = 'out/b.build_metadata' + c = 'c.mojom' + c_metadata = 'out/c.build_metadata' + self.WriteFile(a_metadata, + '{"sources": ["%s"], "deps": []}\n' % self.GetPath(a)) + self.WriteFile( + b_metadata, + '{"sources": ["%s"], "deps": ["%s"]}\n' % (self.GetPath(b), + self.GetPath(a_metadata))) + self.WriteFile( + c_metadata, + '{"sources": ["%s"], "deps": ["%s"]}\n' % (self.GetPath(c), + self.GetPath(b_metadata))) + self.WriteFile(a, """\ + module a; + struct Bar {};""") + self.WriteFile( + b, """\ + module b; + import "a.mojom"; + struct Foo { a.Bar bar; };""") + self.WriteFile( + c, """\ + module c; + import "a.mojom"; + import "b.mojom"; + struct Baz { b.Foo foo; };""") + self.ParseMojoms([a], metadata=a_metadata) + self.ParseMojoms([b], metadata=b_metadata) + self.ParseMojoms([c], metadata=c_metadata) + + def testCheckImportsMissing(self): + """Verify that the parser rejects valid input mojoms when imports don't + agree with build metadata given via --check-imports.""" + a = 'a.mojom' + a_metadata = 'out/a.build_metadata' + b = 'b.mojom' + b_metadata = 'out/b.build_metadata' + self.WriteFile(a_metadata, + '{"sources": ["%s"], "deps": []}\n' % self.GetPath(a)) + self.WriteFile(b_metadata, + '{"sources": ["%s"], "deps": []}\n' % self.GetPath(b)) + self.WriteFile(a, """\ + module a; + struct Bar {};""") + self.WriteFile( + b, """\ + module b; + import "a.mojom"; + struct Foo { a.Bar bar; };""") + + self.ParseMojoms([a], metadata=a_metadata) + with self.assertRaisesRegexp(ValueError, "not allowed by build"): + self.ParseMojoms([b], metadata=b_metadata) diff --git a/utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py b/utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py new file mode 100644 index 00000000..d45ec586 --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py @@ -0,0 +1,127 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom_parser_test_case import MojomParserTestCase + +from mojom.generate import module + + +class StableAttributeTest(MojomParserTestCase): + """Tests covering usage of the [Stable] attribute.""" + + def testStableAttributeTagging(self): + """Verify that we recognize the [Stable] attribute on relevant definitions + and the resulting parser outputs are tagged accordingly.""" + mojom = 'test.mojom' + self.WriteFile( + mojom, """\ + [Stable] enum TestEnum { kFoo }; + enum UnstableEnum { kBar }; + [Stable] struct TestStruct { TestEnum a; }; + struct UnstableStruct { UnstableEnum a; }; + [Stable] union TestUnion { TestEnum a; TestStruct b; }; + union UnstableUnion { UnstableEnum a; UnstableStruct b; }; + [Stable] interface TestInterface { Foo@0(TestUnion x) => (); }; + interface UnstableInterface { Foo(UnstableUnion x) => (); }; + """) + self.ParseMojoms([mojom]) + + m = self.LoadModule(mojom) + self.assertEqual(2, len(m.enums)) + self.assertTrue(m.enums[0].stable) + self.assertFalse(m.enums[1].stable) + self.assertEqual(2, len(m.structs)) + self.assertTrue(m.structs[0].stable) + self.assertFalse(m.structs[1].stable) + self.assertEqual(2, len(m.unions)) + self.assertTrue(m.unions[0].stable) + self.assertFalse(m.unions[1].stable) + self.assertEqual(2, len(m.interfaces)) + self.assertTrue(m.interfaces[0].stable) + self.assertFalse(m.interfaces[1].stable) + + def testStableStruct(self): + """A [Stable] struct is valid if all its fields are also stable.""" + self.ExtractTypes('[Stable] struct S {};') + self.ExtractTypes('[Stable] struct S { int32 x; bool b; };') + self.ExtractTypes('[Stable] enum E { A }; [Stable] struct S { E e; };') + self.ExtractTypes('[Stable] struct S {}; [Stable] struct T { S s; };') + self.ExtractTypes( + '[Stable] struct S {}; [Stable] struct T { array ss; };') + self.ExtractTypes( + '[Stable] interface F {}; [Stable] struct T { pending_remote f; };') + + with self.assertRaisesRegexp(Exception, 'because it depends on E'): + self.ExtractTypes('enum E { A }; [Stable] struct S { E e; };') + with self.assertRaisesRegexp(Exception, 'because it depends on X'): + self.ExtractTypes('struct X {}; [Stable] struct S { X x; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] struct S { array xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] struct S { map xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] struct S { map xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on F'): + self.ExtractTypes( + 'interface F {}; [Stable] struct S { pending_remote f; };') + with self.assertRaisesRegexp(Exception, 'because it depends on F'): + self.ExtractTypes( + 'interface F {}; [Stable] struct S { pending_receiver f; };') + + def testStableUnion(self): + """A [Stable] union is valid if all its fields' types are also stable.""" + self.ExtractTypes('[Stable] union U {};') + self.ExtractTypes('[Stable] union U { int32 x; bool b; };') + self.ExtractTypes('[Stable] enum E { A }; [Stable] union U { E e; };') + self.ExtractTypes('[Stable] struct S {}; [Stable] union U { S s; };') + self.ExtractTypes( + '[Stable] struct S {}; [Stable] union U { array ss; };') + self.ExtractTypes( + '[Stable] interface F {}; [Stable] union U { pending_remote f; };') + + with self.assertRaisesRegexp(Exception, 'because it depends on E'): + self.ExtractTypes('enum E { A }; [Stable] union U { E e; };') + with self.assertRaisesRegexp(Exception, 'because it depends on X'): + self.ExtractTypes('struct X {}; [Stable] union U { X x; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] union U { array xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] union U { map xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on T'): + self.ExtractTypes('struct T {}; [Stable] union U { map xs; };') + with self.assertRaisesRegexp(Exception, 'because it depends on F'): + self.ExtractTypes( + 'interface F {}; [Stable] union U { pending_remote f; };') + with self.assertRaisesRegexp(Exception, 'because it depends on F'): + self.ExtractTypes( + 'interface F {}; [Stable] union U { pending_receiver f; };') + + def testStableInterface(self): + """A [Stable] interface is valid if all its methods' parameter types are + stable, including response parameters where applicable.""" + self.ExtractTypes('[Stable] interface F {};') + self.ExtractTypes('[Stable] interface F { A@0(int32 x); };') + self.ExtractTypes('[Stable] interface F { A@0(int32 x) => (bool b); };') + self.ExtractTypes("""\ + [Stable] enum E { A, B, C }; + [Stable] struct S {}; + [Stable] interface F { A@0(E e, S s) => (bool b, array s); }; + """) + + with self.assertRaisesRegexp(Exception, 'because it depends on E'): + self.ExtractTypes( + 'enum E { A, B, C }; [Stable] interface F { A@0(E e); };') + with self.assertRaisesRegexp(Exception, 'because it depends on E'): + self.ExtractTypes( + 'enum E { A, B, C }; [Stable] interface F { A@0(int32 x) => (E e); };' + ) + with self.assertRaisesRegexp(Exception, 'because it depends on S'): + self.ExtractTypes( + 'struct S {}; [Stable] interface F { A@0(int32 x) => (S s); };') + with self.assertRaisesRegexp(Exception, 'because it depends on S'): + self.ExtractTypes( + 'struct S {}; [Stable] interface F { A@0(S s) => (bool b); };') + + with self.assertRaisesRegexp(Exception, 'explicit method ordinals'): + self.ExtractTypes('[Stable] interface F { A() => (); };') diff --git a/utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py b/utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py new file mode 100644 index 00000000..a0ee150e --- /dev/null +++ b/utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py @@ -0,0 +1,397 @@ +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +from mojom_parser_test_case import MojomParserTestCase + + +class VersionCompatibilityTest(MojomParserTestCase): + """Tests covering compatibility between two versions of the same mojom type + definition. This coverage ensures that we can reliably detect unsafe changes + to definitions that are expected to tolerate version skew in production + environments.""" + + def _GetTypeCompatibilityMap(self, old_mojom, new_mojom): + """Helper to support the implementation of assertBackwardCompatible and + assertNotBackwardCompatible.""" + + old = self.ExtractTypes(old_mojom) + new = self.ExtractTypes(new_mojom) + self.assertEqual(set(old.keys()), set(new.keys()), + 'Old and new test mojoms should use the same type names.') + + compatibility_map = {} + for name in old.keys(): + compatibility_map[name] = new[name].IsBackwardCompatible(old[name]) + return compatibility_map + + def assertBackwardCompatible(self, old_mojom, new_mojom): + compatibility_map = self._GetTypeCompatibilityMap(old_mojom, new_mojom) + for name, compatible in compatibility_map.items(): + if not compatible: + raise AssertionError( + 'Given the old mojom:\n\n %s\n\nand the new mojom:\n\n %s\n\n' + 'The new definition of %s should pass a backward-compatibiity ' + 'check, but it does not.' % (old_mojom, new_mojom, name)) + + def assertNotBackwardCompatible(self, old_mojom, new_mojom): + compatibility_map = self._GetTypeCompatibilityMap(old_mojom, new_mojom) + if all(compatibility_map.values()): + raise AssertionError( + 'Given the old mojom:\n\n %s\n\nand the new mojom:\n\n %s\n\n' + 'The new mojom should fail a backward-compatibility check, but it ' + 'does not.' % (old_mojom, new_mojom)) + + def testNewNonExtensibleEnumValue(self): + """Adding a value to a non-extensible enum breaks backward-compatibility.""" + self.assertNotBackwardCompatible('enum E { kFoo, kBar };', + 'enum E { kFoo, kBar, kBaz };') + + def testNewNonExtensibleEnumValueWithMinVersion(self): + """Adding a value to a non-extensible enum breaks backward-compatibility, + even with a new [MinVersion] specified for the value.""" + self.assertNotBackwardCompatible( + 'enum E { kFoo, kBar };', 'enum E { kFoo, kBar, [MinVersion=1] kBaz };') + + def testNewValueInExistingVersion(self): + """Adding a value to an existing version is not allowed, even if the old + enum was marked [Extensible]. Note that it is irrelevant whether or not the + new enum is marked [Extensible].""" + self.assertNotBackwardCompatible('[Extensible] enum E { kFoo, kBar };', + 'enum E { kFoo, kBar, kBaz };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kFoo, kBar };', + '[Extensible] enum E { kFoo, kBar, kBaz };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kFoo, [MinVersion=1] kBar };', + 'enum E { kFoo, [MinVersion=1] kBar, [MinVersion=1] kBaz };') + + def testEnumValueRemoval(self): + """Removal of an enum value is never valid even for [Extensible] enums.""" + self.assertNotBackwardCompatible('enum E { kFoo, kBar };', + 'enum E { kFoo };') + self.assertNotBackwardCompatible('[Extensible] enum E { kFoo, kBar };', + '[Extensible] enum E { kFoo };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kA, [MinVersion=1] kB };', + '[Extensible] enum E { kA, };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kA, [MinVersion=1] kB, [MinVersion=1] kZ };', + '[Extensible] enum E { kA, [MinVersion=1] kB };') + + def testNewExtensibleEnumValueWithMinVersion(self): + """Adding a new and properly [MinVersion]'d value to an [Extensible] enum + is a backward-compatible change. Note that it is irrelevant whether or not + the new enum is marked [Extensible].""" + self.assertBackwardCompatible('[Extensible] enum E { kA, kB };', + 'enum E { kA, kB, [MinVersion=1] kC };') + self.assertBackwardCompatible( + '[Extensible] enum E { kA, kB };', + '[Extensible] enum E { kA, kB, [MinVersion=1] kC };') + self.assertBackwardCompatible( + '[Extensible] enum E { kA, [MinVersion=1] kB };', + '[Extensible] enum E { kA, [MinVersion=1] kB, [MinVersion=2] kC };') + + def testRenameEnumValue(self): + """Renaming an enum value does not affect backward-compatibility. Only + numeric value is relevant.""" + self.assertBackwardCompatible('enum E { kA, kB };', 'enum E { kX, kY };') + + def testAddEnumValueAlias(self): + """Adding new enum fields does not affect backward-compatibility if it does + not introduce any new numeric values.""" + self.assertBackwardCompatible( + 'enum E { kA, kB };', 'enum E { kA, kB, kC = kA, kD = 1, kE = kD };') + + def testEnumIdentity(self): + """An unchanged enum is obviously backward-compatible.""" + self.assertBackwardCompatible('enum E { kA, kB, kC };', + 'enum E { kA, kB, kC };') + + def testNewStructFieldUnversioned(self): + """Adding a new field to a struct without a new (i.e. higher than any + existing version) [MinVersion] tag breaks backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string a; };', + 'struct S { string a; string b; };') + + def testStructFieldRemoval(self): + """Removing a field from a struct breaks backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string a; string b; };', + 'struct S { string a; };') + + def testStructFieldTypeChange(self): + """Changing the type of an existing field always breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string a; };', + 'struct S { array a; };') + + def testStructFieldBecomingOptional(self): + """Changing a field from non-optional to optional breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string a; };', + 'struct S { string? a; };') + + def testStructFieldBecomingNonOptional(self): + """Changing a field from optional to non-optional breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string? a; };', + 'struct S { string a; };') + + def testStructFieldOrderChange(self): + """Changing the order of fields breaks backward-compatibility.""" + self.assertNotBackwardCompatible('struct S { string a; bool b; };', + 'struct S { bool b; string a; };') + self.assertNotBackwardCompatible('struct S { string a@0; bool b@1; };', + 'struct S { string a@1; bool b@0; };') + + def testStructFieldMinVersionChange(self): + """Changing the MinVersion of a field breaks backward-compatibility.""" + self.assertNotBackwardCompatible( + 'struct S { string a; [MinVersion=1] string? b; };', + 'struct S { string a; [MinVersion=2] string? b; };') + + def testStructFieldTypeChange(self): + """If a struct field's own type definition changes, the containing struct + is backward-compatible if and only if the field type's change is + backward-compatible.""" + self.assertBackwardCompatible( + 'struct S {}; struct T { S s; };', + 'struct S { [MinVersion=1] int32 x; }; struct T { S s; };') + self.assertBackwardCompatible( + '[Extensible] enum E { kA }; struct S { E e; };', + '[Extensible] enum E { kA, [MinVersion=1] kB }; struct S { E e; };') + self.assertNotBackwardCompatible( + 'struct S {}; struct T { S s; };', + 'struct S { int32 x; }; struct T { S s; };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kA }; struct S { E e; };', + '[Extensible] enum E { kA, kB }; struct S { E e; };') + + def testNewStructFieldWithInvalidMinVersion(self): + """Adding a new field using an existing MinVersion breaks backward- + compatibility.""" + self.assertNotBackwardCompatible( + """\ + struct S { + string a; + [MinVersion=1] string? b; + }; + """, """\ + struct S { + string a; + [MinVersion=1] string? b; + [MinVersion=1] string? c; + };""") + + def testNewStructFieldWithValidMinVersion(self): + """Adding a new field is safe if tagged with a MinVersion greater than any + previously used MinVersion in the struct.""" + self.assertBackwardCompatible( + 'struct S { int32 a; };', + 'struct S { int32 a; [MinVersion=1] int32 b; };') + self.assertBackwardCompatible( + 'struct S { int32 a; [MinVersion=1] int32 b; };', + 'struct S { int32 a; [MinVersion=1] int32 b; [MinVersion=2] bool c; };') + + def testNewStructFieldNullableReference(self): + """Adding a new nullable reference-typed field is fine if versioned + properly.""" + self.assertBackwardCompatible( + 'struct S { int32 a; };', + 'struct S { int32 a; [MinVersion=1] string? b; };') + + def testStructFieldRename(self): + """Renaming a field has no effect on backward-compatibility.""" + self.assertBackwardCompatible('struct S { int32 x; bool b; };', + 'struct S { int32 a; bool b; };') + + def testStructFieldReorderWithExplicitOrdinals(self): + """Reordering fields has no effect on backward-compatibility when field + ordinals are explicitly labeled and remain unchanged.""" + self.assertBackwardCompatible('struct S { bool b@1; int32 a@0; };', + 'struct S { int32 a@0; bool b@1; };') + + def testNewUnionFieldUnversioned(self): + """Adding a new field to a union without a new (i.e. higher than any + existing version) [MinVersion] tag breaks backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string a; };', + 'union U { string a; string b; };') + + def testUnionFieldRemoval(self): + """Removing a field from a union breaks backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string a; string b; };', + 'union U { string a; };') + + def testUnionFieldTypeChange(self): + """Changing the type of an existing field always breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string a; };', + 'union U { array a; };') + + def testUnionFieldBecomingOptional(self): + """Changing a field from non-optional to optional breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string a; };', + 'union U { string? a; };') + + def testUnionFieldBecomingNonOptional(self): + """Changing a field from optional to non-optional breaks + backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string? a; };', + 'union U { string a; };') + + def testUnionFieldOrderChange(self): + """Changing the order of fields breaks backward-compatibility.""" + self.assertNotBackwardCompatible('union U { string a; bool b; };', + 'union U { bool b; string a; };') + self.assertNotBackwardCompatible('union U { string a@0; bool b@1; };', + 'union U { string a@1; bool b@0; };') + + def testUnionFieldMinVersionChange(self): + """Changing the MinVersion of a field breaks backward-compatibility.""" + self.assertNotBackwardCompatible( + 'union U { string a; [MinVersion=1] string b; };', + 'union U { string a; [MinVersion=2] string b; };') + + def testUnionFieldTypeChange(self): + """If a union field's own type definition changes, the containing union + is backward-compatible if and only if the field type's change is + backward-compatible.""" + self.assertBackwardCompatible( + 'struct S {}; union U { S s; };', + 'struct S { [MinVersion=1] int32 x; }; union U { S s; };') + self.assertBackwardCompatible( + '[Extensible] enum E { kA }; union U { E e; };', + '[Extensible] enum E { kA, [MinVersion=1] kB }; union U { E e; };') + self.assertNotBackwardCompatible( + 'struct S {}; union U { S s; };', + 'struct S { int32 x; }; union U { S s; };') + self.assertNotBackwardCompatible( + '[Extensible] enum E { kA }; union U { E e; };', + '[Extensible] enum E { kA, kB }; union U { E e; };') + + def testNewUnionFieldWithInvalidMinVersion(self): + """Adding a new field using an existing MinVersion breaks backward- + compatibility.""" + self.assertNotBackwardCompatible( + """\ + union U { + string a; + [MinVersion=1] string b; + }; + """, """\ + union U { + string a; + [MinVersion=1] string b; + [MinVersion=1] string c; + };""") + + def testNewUnionFieldWithValidMinVersion(self): + """Adding a new field is safe if tagged with a MinVersion greater than any + previously used MinVersion in the union.""" + self.assertBackwardCompatible( + 'union U { int32 a; };', + 'union U { int32 a; [MinVersion=1] int32 b; };') + self.assertBackwardCompatible( + 'union U { int32 a; [MinVersion=1] int32 b; };', + 'union U { int32 a; [MinVersion=1] int32 b; [MinVersion=2] bool c; };') + + def testUnionFieldRename(self): + """Renaming a field has no effect on backward-compatibility.""" + self.assertBackwardCompatible('union U { int32 x; bool b; };', + 'union U { int32 a; bool b; };') + + def testUnionFieldReorderWithExplicitOrdinals(self): + """Reordering fields has no effect on backward-compatibility when field + ordinals are explicitly labeled and remain unchanged.""" + self.assertBackwardCompatible('union U { bool b@1; int32 a@0; };', + 'union U { int32 a@0; bool b@1; };') + + def testNewInterfaceMethodUnversioned(self): + """Adding a new method to an interface without a new (i.e. higher than any + existing version) [MinVersion] tag breaks backward-compatibility.""" + self.assertNotBackwardCompatible('interface F { A(); };', + 'interface F { A(); B(); };') + + def testInterfaceMethodRemoval(self): + """Removing a method from an interface breaks backward-compatibility.""" + self.assertNotBackwardCompatible('interface F { A(); B(); };', + 'interface F { A(); };') + + def testInterfaceMethodParamsChanged(self): + """Changes to the parameter list are only backward-compatible if they meet + backward-compatibility requirements of an equivalent struct definition.""" + self.assertNotBackwardCompatible('interface F { A(); };', + 'interface F { A(int32 x); };') + self.assertNotBackwardCompatible('interface F { A(int32 x); };', + 'interface F { A(bool x); };') + self.assertNotBackwardCompatible( + 'interface F { A(int32 x, [MinVersion=1] string? s); };', """\ + interface F { + A(int32 x, [MinVersion=1] string? s, [MinVersion=1] int32 y); + };""") + + self.assertBackwardCompatible('interface F { A(int32 x); };', + 'interface F { A(int32 a); };') + self.assertBackwardCompatible( + 'interface F { A(int32 x); };', + 'interface F { A(int32 x, [MinVersion=1] string? s); };') + + self.assertBackwardCompatible( + 'struct S {}; interface F { A(S s); };', + 'struct S { [MinVersion=1] int32 x; }; interface F { A(S s); };') + self.assertBackwardCompatible( + 'struct S {}; struct T {}; interface F { A(S s); };', + 'struct S {}; struct T {}; interface F { A(T s); };') + self.assertNotBackwardCompatible( + 'struct S {}; struct T { int32 x; }; interface F { A(S s); };', + 'struct S {}; struct T { int32 x; }; interface F { A(T t); };') + + def testInterfaceMethodReplyAdded(self): + """Adding a reply to a message breaks backward-compatibilty.""" + self.assertNotBackwardCompatible('interface F { A(); };', + 'interface F { A() => (); };') + + def testInterfaceMethodReplyRemoved(self): + """Removing a reply from a message breaks backward-compatibility.""" + self.assertNotBackwardCompatible('interface F { A() => (); };', + 'interface F { A(); };') + + def testInterfaceMethodReplyParamsChanged(self): + """Similar to request parameters, a change to reply parameters is considered + backward-compatible if it meets the same backward-compatibility + requirements imposed on equivalent struct changes.""" + self.assertNotBackwardCompatible('interface F { A() => (); };', + 'interface F { A() => (int32 x); };') + self.assertNotBackwardCompatible('interface F { A() => (int32 x); };', + 'interface F { A() => (); };') + self.assertNotBackwardCompatible('interface F { A() => (bool x); };', + 'interface F { A() => (int32 x); };') + + self.assertBackwardCompatible('interface F { A() => (int32 a); };', + 'interface F { A() => (int32 x); };') + self.assertBackwardCompatible( + 'interface F { A() => (int32 x); };', + 'interface F { A() => (int32 x, [MinVersion] string? s); };') + + def testNewInterfaceMethodWithInvalidMinVersion(self): + """Adding a new method to an existing version is not backward-compatible.""" + self.assertNotBackwardCompatible( + """\ + interface F { + A(); + [MinVersion=1] B(); + }; + """, """\ + interface F { + A(); + [MinVersion=1] B(); + [MinVersion=1] C(); + }; + """) + + def testNewInterfaceMethodWithValidMinVersion(self): + """Adding a new method is fine as long as its MinVersion exceeds that of any + method on the old interface definition.""" + self.assertBackwardCompatible('interface F { A(); };', + 'interface F { A(); [MinVersion=1] B(); };') diff --git a/utils/ipc/mojo/public/tools/run_all_python_unittests.py b/utils/ipc/mojo/public/tools/run_all_python_unittests.py new file mode 100755 index 00000000..b2010958 --- /dev/null +++ b/utils/ipc/mojo/public/tools/run_all_python_unittests.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python +# Copyright 2020 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +import os.path +import sys + +_TOOLS_DIR = os.path.dirname(__file__) +_MOJOM_DIR = os.path.join(_TOOLS_DIR, 'mojom') +_SRC_DIR = os.path.join(_TOOLS_DIR, os.path.pardir, os.path.pardir, + os.path.pardir) + +# Ensure that the mojom library is discoverable. +sys.path.append(_MOJOM_DIR) + +# Help Python find typ in //third_party/catapult/third_party/typ/ +sys.path.append( + os.path.join(_SRC_DIR, 'third_party', 'catapult', 'third_party', 'typ')) +import typ + + +def Main(): + return typ.main(top_level_dir=_MOJOM_DIR) + + +if __name__ == '__main__': + sys.exit(Main()) diff --git a/utils/ipc/tools/diagnosis/crbug_1001171.py b/utils/ipc/tools/diagnosis/crbug_1001171.py new file mode 100644 index 00000000..478fb8c1 --- /dev/null +++ b/utils/ipc/tools/diagnosis/crbug_1001171.py @@ -0,0 +1,51 @@ +# Copyright 2019 The Chromium Authors. All rights reserved. +# Use of this source code is governed by a BSD-style license that can be +# found in the LICENSE file. + +"""Helper context wrapper for diagnosing crbug.com/1001171. + +This module and all uses thereof can and should be removed once +crbug.com/1001171 has been resolved. +""" + +from __future__ import print_function + +import contextlib +import os +import sys + + +@contextlib.contextmanager +def DumpStateOnLookupError(): + """Prints potentially useful state info in the event of a LookupError.""" + try: + yield + except LookupError: + print('LookupError diagnosis for crbug.com/1001171:') + for path_index, path_entry in enumerate(sys.path): + desc = 'unknown' + if not os.path.exists(path_entry): + desc = 'missing' + elif os.path.islink(path_entry): + desc = 'link -> %s' % os.path.realpath(path_entry) + elif os.path.isfile(path_entry): + desc = 'file' + elif os.path.isdir(path_entry): + desc = 'dir' + print(' sys.path[%d]: %s (%s)' % (path_index, path_entry, desc)) + + real_path_entry = os.path.realpath(path_entry) + if (path_entry.endswith(os.path.join('lib', 'python2.7')) + and os.path.isdir(real_path_entry)): + encodings_dir = os.path.realpath( + os.path.join(real_path_entry, 'encodings')) + if os.path.exists(encodings_dir): + if os.path.isdir(encodings_dir): + print(' %s contents: %s' % (encodings_dir, + str(os.listdir(encodings_dir)))) + else: + print(' %s exists but is not a directory' % encodings_dir) + else: + print(' %s missing' % encodings_dir) + + raise From patchwork Tue Sep 15 14:20:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9612 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 58269C3B5D for ; Tue, 15 Sep 2020 14:21:04 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 7E98562E32; Tue, 15 Sep 2020 16:21:03 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="hcagVfXM"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DEA262E0F for ; Tue, 15 Sep 2020 16:21:02 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id DD40D276; Tue, 15 Sep 2020 16:20:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179660; bh=WmmqVrXHBzIcvoKzMxXwCvSu1OjhoxDOgXFOGXqsnbI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hcagVfXMkiwWOKsAZB1IceXVG3q4ihr0N/5XJg8wdBfnoqJtOc193UBnT7ECkKn/Q H75DqAmR0HhupWSj12f2ujj6xtiJB7JxXLD0pToxjIa9uDVle82hB07iX4EVYxMDss 5MmkJ1hiKHpY6qjeo3M3nG1Qoc3rUW1dVBXEPpJk= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:18 +0900 Message-Id: <20200915142038.28757-4-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 03/23] utils: ipc: add templates for code generation for IPC mechanism X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add templates to mojo to generate code for the IPC mechanism. These templates generate: - module header - module serializer - IPA proxy cpp, header, and worker Given an input data definition mojom file for a pipeline. Signed-off-by: Paul Elder --- .../module_generated.h.tmpl | 95 +++++ .../module_ipa_proxy.cpp.tmpl | 219 ++++++++++ .../module_ipa_proxy.h.tmpl | 108 +++++ .../module_ipa_proxy_worker.cpp.tmpl | 167 ++++++++ .../module_serializer.h.tmpl | 43 ++ .../libcamera_templates/proxy_functions.tmpl | 180 ++++++++ .../libcamera_templates/serializer.tmpl | 262 ++++++++++++ .../generators/mojom_libcamera_generator.py | 403 ++++++++++++++++++ 8 files changed, 1477 insertions(+) create mode 100644 utils/ipc/generators/libcamera_templates/module_generated.h.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py diff --git a/utils/ipc/generators/libcamera_templates/module_generated.h.tmpl b/utils/ipc/generators/libcamera_templates/module_generated.h.tmpl new file mode 100644 index 00000000..ea810406 --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/module_generated.h.tmpl @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) {{ year }}, Google Inc. + * + * {{ module_name }}_generated.h - Image Processing Algorithm interface for {{ module_name }} + */ + +// automatically generated by custom compiler + +#ifndef __LIBCAMERA_IPA_INTERFACE_{{ module_name|upper }}_GENERATED_H__ +#define __LIBCAMERA_IPA_INTERFACE_{{ module_name|upper }}_GENERATED_H__ + +#include + +#include + +{% if has_map %}#include {% endif %} +{% if has_array %}#include {% endif %} + +namespace libcamera { + +enum {{ cmd_enum_name }} { + CMD_EXIT = 0, +{%- for method in interface_main.methods %} + CMD_{{ method.mojom_name|upper }} = {{loop.index}}, +{%- endfor %} +{%- set count = interface_main.methods|length -%} +{%- for method in interface_cb.methods %} + CMD_{{ method.mojom_name|upper }} = {{loop.index + count}}, +{%- endfor %} +}; + +{% for enum in enums %} +enum {{ enum.mojom_name }} { +{%- for field in enum.fields %} + {{ field.mojom_name }} = {{ field.numeric_value + 1 }}, +{%- endfor %} +}; +{% endfor %} + +{%- for struct in structs_nonempty %} +struct {{ struct.mojom_name }} +{ +public: + {{ struct.mojom_name }}() {%- if struct|has_default_fields %} :{% endif %} +{%- for field in struct.fields|with_default_values -%} +{{" " if loop.first}}{{ field.mojom_name}}_({{ field|default_value }}){{ ", " if not loop.last }} +{%- endfor %} {} + ~{{ struct.mojom_name }}() {} + + {{ struct.mojom_name }}( +{%- for field in struct.fields -%} +{{ field|name }} {{ field.mojom_name }}{{ ", " if not loop.last }} +{%- endfor -%} +) : +{%- for field in struct.fields -%} +{{" " if loop.first}}{{ field.mojom_name}}_({{ field.mojom_name }}){{ ", " if not loop.last }} +{%- endfor %} {} +{% for field in struct.fields %} + {{ field|name }} {{ field.mojom_name }}_; +{%- endfor %} +}; +{% endfor %} + +{#- +What to do about the #includes? Should we forward-declare, or +require {{module_name}}.h to include the required headers? +For now I'm going for the latter, coz this header essentially +extends {{module_name}}.h. +#} +class {{ interface_name }} : public IPAInterface +{ +public: + virtual ~{{interface_name}}() {}; +{% for method in interface_main.methods %} + virtual {{method|method_return_value}} {{method.mojom_name}}( +{%- for param in method|method_parameters %} + {{param}}{{- "," if not loop.last}} +{%- endfor -%} +) = 0; +{% endfor %} + +{%- for method in interface_cb.methods %} + Signal< +{%- for param in method.parameters -%} + const {{param|name}}{{" &" if not param|is_pod}} + {{- ", " if not loop.last }} +{%- endfor -%} +> {{method.mojom_name}}; +{% endfor -%} +}; + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_IPA_INTERFACE_{{ module_name|upper }}_GENERATED_H__ */ diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl new file mode 100644 index 00000000..bc8b34e5 --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl @@ -0,0 +1,219 @@ +{%- import "proxy_functions.tmpl" as proxy_funcs -%} + +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) {{ year }}, Google Inc. + * + * ipa_proxy_{{ module_name }}.cpp - Image Processing Algorithm proxy for {{ module_name }} + */ + +// automatically generated by custom compiler + +#include + +#include +#include +#include +#include +#include + +#include "libcamera/internal/control_serializer.h" +#include "libcamera/internal/ipa_ipc.h" +#include "libcamera/internal/ipa_ipc_unixsocket.h" +#include "libcamera/internal/ipa_data_serializer.h" +#include "libcamera/internal/ipa_module.h" +#include "libcamera/internal/ipa_proxy.h" +#include "libcamera/internal/ipc_unixsocket.h" +#include "libcamera/internal/log.h" +#include "libcamera/internal/process.h" +#include "libcamera/internal/thread.h" + +#include + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPAProxy) + +{{proxy_name}}::{{proxy_name}}(IPAModule *ipam, bool isolate) + : IPAProxy(ipam), running_(false), + isolate_(isolate) +{ + LOG(IPAProxy, Debug) + << "initializing dummy proxy: loading IPA from " + << ipam->path(); + + if (isolate_) { + const std::string proxy_worker_path = resolvePath("ipa_proxy_{{module_name}}"); + if (proxy_worker_path.empty()) { + LOG(IPAProxy, Error) + << "Failed to get proxy worker path"; + return; + } + + ipc_ = std::make_unique(ipam->path().c_str(), proxy_worker_path.c_str()); + if (!ipc_->isValid()) { + LOG(IPAProxy, Error) << "Failed to create IPAIPC"; + return; + } + + ipc_->recvIPC.connect(this, &{{proxy_name}}::recvIPC); + + valid_ = true; + return; + } + + if (!ipam->load()) + return; + + IPAInterface *ipai = ipam->createInterface(); + if (!ipai) { + LOG(IPAProxy, Error) + << "Failed to create IPA context for " << ipam->path(); + return; + } + + ipa_ = std::unique_ptr<{{interface_name}}>(dynamic_cast<{{interface_name}} *>(ipai)); + proxy_.setIPA(ipa_.get()); + +{% for method in interface_cb.methods %} + ipa_->{{method.mojom_name}}.connect(this, &{{proxy_name}}::{{method.mojom_name}}Thread); +{%- endfor %} + + valid_ = true; +} + +{{proxy_name}}::~{{proxy_name}}() +{ + if (isolate_) + ipc_->sendAsync(CMD_EXIT, {}, {}); +} + +void {{proxy_name}}::recvIPC(std::vector &data, std::vector &fds) +{ + uint32_t cmd = (data[0] & 0xff) | + ((data[1] & 0xff) << 8) | + ((data[2] & 0xff) << 16) | + ((data[3] & 0xff) << 24); + + /* Need to skip another 4 bytes for the sequence number. */ + std::vector vec(data.begin() + 8, data.end()); +{% if interface_cb.methods|length > 0 %} + switch (cmd) { +{%- for method in interface_cb.methods %} + case CMD_{{method.mojom_name|upper}}: { + {{method.mojom_name}}IPC(vec, fds); + break; + } +{%- endfor %} + } +{%- endif %} +} + +{% for method in interface_main.methods %} +{{proxy_funcs.func_sig(proxy_name, method)}} +{ + if (isolate_) + {{"return " if method|method_return_value != "void"}}{{method.mojom_name}}IPC( +{%- for param in method|method_param_names -%} + {{param}}{{- ", " if not loop.last}} +{%- endfor -%} +); + else + {{"return " if method|method_return_value != "void"}}{{method.mojom_name}}Thread( +{%- for param in method|method_param_names -%} + {{param}}{{- ", " if not loop.last}} +{%- endfor -%} +); +} + +{{proxy_funcs.func_sig(proxy_name, method, "Thread")}} +{ +{%- if method.mojom_name == "init" %} + {{proxy_funcs.init_thread_body()}} +{% elif method.mojom_name == "start" %} + {{proxy_funcs.start_thread_body()}} +{% elif method.mojom_name == "stop" %} + {{proxy_funcs.stop_thread_body()}} +{% elif not method|is_async %} + ipa_->{{method.mojom_name}}( + {%- for param in method|method_param_names -%} + {{param}}{{- ", " if not loop.last}} + {%- endfor -%} +); +{% elif method|is_async %} + proxy_.invokeMethod(&ThreadProxy::{{method.mojom_name}}, ConnectionTypeQueued, + {%- for param in method|method_param_names -%} + {{param}}{{- ", " if not loop.last}} + {%- endfor -%} +); +{%- endif %} +} + +{{proxy_funcs.func_sig(proxy_name, method, "IPC")}} +{ +{%- set has_input = true if method|method_param_inputs|length > 0 %} +{%- set has_output = true if method|method_param_outputs|length > 0 or method|method_return_value != "void" %} +{% if has_input %} + std::vector _ipcInputBuf; +{%- if method|method_input_has_fd %} + std::vector _ipcInputFds; +{%- endif %} +{%- endif %} +{%- if has_output %} + std::vector _ipcOutputBuf; +{%- if method|method_output_has_fd %} + std::vector _ipcOutputFds; +{%- endif %} +{%- endif %} + +{{proxy_funcs.serialize_call(method|method_param_inputs, '_ipcInputBuf', '_ipcInputFds')}} + +{%- set input_buf = "_ipcInputBuf" if has_input else "{}" %} +{%- set fds_buf = "_ipcInputFds" if method|method_input_has_fd else "{}" %} +{%- set cmd = "CMD_" + method.mojom_name|upper %} +{% if method|is_async %} + int ret = ipc_->sendAsync({{cmd}}, {{input_buf}}, {{fds_buf}}); +{%- else %} + int ret = ipc_->sendSync({{cmd}}, {{input_buf}}, {{fds_buf}} +{{- ", &_ipcOutputBuf" if has_output -}} +{{- ", &_ipcOutputFds" if has_output and method|method_output_has_fd -}} +); +{%- endif %} + if (ret < 0) { + LOG(IPAProxy, Error) << "Failed to call {{method.mojom_name}}"; +{%- if method|method_return_value == "void" %} + return; +{%- else %} + return static_cast<{{method|method_return_value}}>(ret); +{%- endif %} + } + LOG(IPAProxy, Debug) << "Done calling {{method.mojom_name}}"; +{% if method|method_return_value != "void" %} + return static_cast<{{method.response_parameters|first|name}}>(readUInt<{{method.response_parameters|first|name}}>(_ipcOutputBuf, 0)); +{% elif method|method_param_outputs|length > 0 %} +{{proxy_funcs.deserialize_call(method|method_param_outputs, '_ipcOutputBuf', '_ipcOutputFds')}} +{% endif -%} +} + +{% endfor %} + + +{% for method in interface_cb.methods %} +{{proxy_funcs.func_sig(proxy_name, method, "Thread")}} +{ + {{method.mojom_name}}.emit({{method.parameters|params_comma_sep}}); +} + +void {{proxy_name}}::{{method.mojom_name}}IPC( + std::vector &data, + [[maybe_unused]] std::vector &fds) +{ +{%- for param in method.parameters %} + {{param|name}} {{param.mojom_name}}; +{%- endfor %} +{{proxy_funcs.deserialize_call(method.parameters, 'data', 'fds', false)}} + {{method.mojom_name}}.emit({{method.parameters|params_comma_sep}}); +} +{% endfor %} + +} /* namespace libcamera */ diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl new file mode 100644 index 00000000..432fefa9 --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl @@ -0,0 +1,108 @@ +{%- import "proxy_functions.tmpl" as proxy_funcs -%} + +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) {{ year }}, Google Inc. + * + * ipa_proxy_{{ module_name }}.h - Image Processing Algorithm proxy for {{ module_name }} + */ + +// automatically generated by custom compiler + +#ifndef __LIBCAMERA_INTERNAL_IPA_PROXY_{{ module_name|upper }}_H__ +#define __LIBCAMERA_INTERNAL_IPA_PROXY_{{ module_name|upper }}_H__ + +// TODO move this to a proxy header directory + +#include +#include +#include + +#include "libcamera/internal/control_serializer.h" +#include "libcamera/internal/ipa_ipc.h" +#include "libcamera/internal/ipa_ipc_unixsocket.h" +#include "libcamera/internal/ipa_proxy.h" +#include "libcamera/internal/ipc_unixsocket.h" +#include "libcamera/internal/thread.h" + +namespace libcamera { + +class {{proxy_name}} : public IPAProxy, public {{interface_name}}, public Object +{ +public: + {{proxy_name}}(IPAModule *ipam, bool isolate); + ~{{proxy_name}}(); + +{% for method in interface_main.methods %} +{{proxy_funcs.func_sig(proxy_name, method, "", false, true)|indent(8, true)}}; +{% endfor %} + +{%- for method in interface_cb.methods %} + Signal< +{%- for param in method.parameters -%} + const {{param|name}}{{" &" if not param|is_pod}} + {{- ", " if not loop.last }} +{%- endfor -%} +> {{method.mojom_name}}; +{% endfor %} + +private: + void recvIPC(std::vector &data, std::vector &fds); + +{% for method in interface_main.methods %} +{{proxy_funcs.func_sig(proxy_name, method, "Thread", false)|indent(8, true)}}; +{{proxy_funcs.func_sig(proxy_name, method, "IPC", false)|indent(8, true)}}; +{% endfor %} +{% for method in interface_cb.methods %} +{{proxy_funcs.func_sig(proxy_name, method, "Thread", false)|indent(8, true)}}; + void {{method.mojom_name}}IPC( + std::vector &data, + std::vector &fds); +{% endfor %} + + /* Helper class to invoke async functions in another thread. */ + class ThreadProxy : public Object + { + public: + void setIPA(IPARPiInterface *ipa) + { + ipa_ = ipa; + } + + int start() + { + return ipa_->start(); + } + + void stop() + { + ipa_->stop(); + } +{% for method in interface_main.methods %} +{%- if method|is_async %} + {{proxy_funcs.func_sig(proxy_name, method, "", false)|indent(16)}} + { + ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}}); + } +{%- endif %} +{%- endfor %} + + private: + {{interface_name}} *ipa_; + }; + + bool running_; + Thread thread_; + ThreadProxy proxy_; + std::unique_ptr<{{interface_name}}> ipa_; + + const bool isolate_; + + std::unique_ptr ipc_; + + ControlSerializer controlSerializer_; +}; + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_{{ module_name|upper }}_H__ */ diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl new file mode 100644 index 00000000..22d0ca1f --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl @@ -0,0 +1,167 @@ +{%- import "proxy_functions.tmpl" as proxy_funcs -%} + +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) {{ year }}, Google Inc. + * + * ipa_proxy_{{ module_name }}_worker.cpp - Image Processing Algorithm proxy worker for {{ module_name }} + */ + +// automatically generated by custom compiler + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/control_serializer.h" +#include "libcamera/internal/ipa_data_serializer.h" +#include "libcamera/internal/ipa_ipc_unixsocket.h" +#include "libcamera/internal/ipa_module.h" +#include "libcamera/internal/ipa_proxy.h" +#include "libcamera/internal/ipc_unixsocket.h" +#include "libcamera/internal/log.h" +#include "libcamera/internal/thread.h" + +using namespace libcamera; + +LOG_DEFINE_CATEGORY({{proxy_name}}Worker) + +struct CallData { + IPCUnixSocket::Payload *response; + bool done; +}; + +IPARPiInterface *ipa_; +IPCUnixSocket socket_; +std::map callData_; + +ControlSerializer controlSerializer_; + +bool exit_ = false; + +void readyRead(IPCUnixSocket *socket) +{ + IPCUnixSocket::Payload _message, _response; + int _ret = socket->receive(&_message); + if (_ret) { + LOG({{proxy_name}}Worker, Error) + << "Receive message failed" << _ret; + return; + } + + uint32_t _cmd, _seq; + std::tie(_cmd, _seq) = readHeader(_message); + eraseHeader(_message); + + switch (_cmd) { + case CMD_EXIT: { + exit_ = true; + break; + } + +{% for method in interface_main.methods %} + case CMD_{{method.mojom_name|upper}}: { + {{proxy_funcs.deserialize_call(method|method_param_inputs, '_message.data', '_message.fds', false, true)|indent(8, true)}} +{% for param in method|method_param_outputs %} + {{param|name}} {{param.mojom_name}}; +{% endfor %} + ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}} +{{- ", " if method|method_param_outputs|params_comma_sep-}} +{%- for param in method|method_param_outputs -%} +&{{param.mojom_name}}{{", " if not loop.last}} +{%- endfor -%} +); +{% if not method|is_async %} + writeHeader(_response, _cmd, _seq); + {{proxy_funcs.serialize_call(method|method_param_outputs, "_response.data", "_response.fds")|indent(8, true)}} + int _ret = socket_.send(_response); + if (_ret < 0) { + LOG({{proxy_name}}Worker, Error) + << "Reply to {{method.mojom_name}}() failed" << _ret; + } + LOG({{proxy_name}}Worker, Debug) << "Done replying to {{method.mojom_name}}()"; + + break; +{%- endif %} + } +{% endfor %} + } +} + +{% for method in interface_cb.methods %} +{{proxy_funcs.func_sig(proxy_name, method, "", false)}} +{ + IPCUnixSocket::Payload _message; + + writeHeader(_message, CMD_{{method.mojom_name|upper}}, 0); + {{proxy_funcs.serialize_call(method|method_param_inputs, "_message.data", "_message.fds")}} + + socket_.send(_message); + + LOG(IPAProxyRPiWorker, Debug) << "{{method.mojom_name}} done"; +} +{%- endfor %} + +int main(int argc, char **argv) +{ + /* Uncomment this for debugging. */ +#if 0 + std::string logPath = "/tmp/libcamera.worker." + + std::to_string(getpid()) + ".log"; + logSetFile(logPath.c_str()); +#endif + + if (argc < 3) { + LOG({{proxy_name}}Worker, Debug) + << "Tried to start worker with no args"; + return EXIT_FAILURE; + } + + int fd = std::stoi(argv[2]); + LOG({{proxy_name}}Worker, Debug) + << "Starting worker for IPA module " << argv[1] + << " with IPC fd = " << fd; + + std::unique_ptr ipam = std::make_unique(argv[1]); + if (!ipam->isValid() || !ipam->load()) { + LOG({{proxy_name}}Worker, Error) + << "IPAModule " << argv[1] << " should be valid but isn't"; + return EXIT_FAILURE; + } + + if (socket_.bind(fd) < 0) { + LOG({{proxy_name}}Worker, Error) << "IPC socket binding failed"; + return EXIT_FAILURE; + } + socket_.readyRead.connect(&readyRead); + + ipa_ = dynamic_cast<{{interface_name}} *>(ipam->createInterface()); + if (!ipa_) { + LOG({{proxy_name}}Worker, Error) << "Failed to create IPA interface instance"; + return EXIT_FAILURE; + } +{% for method in interface_cb.methods %} + ipa_->{{method.mojom_name}}.connect(&{{method.mojom_name}}); +{%- endfor %} + + LOG({{proxy_name}}Worker, Debug) << "Proxy worker successfully started"; + + /* \todo upgrade listening loop */ + EventDispatcher *dispatcher = Thread::current()->eventDispatcher(); + while (!exit_) + dispatcher->processEvents(); + + delete ipa_; + socket_.close(); + + return 0; +} diff --git a/utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl b/utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl new file mode 100644 index 00000000..8e3c24d6 --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl @@ -0,0 +1,43 @@ +{%- import "serializer.tmpl" as serializer -%} + +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) {{ year }}, Google Inc. + * + * {{ module_name }}_serializer.h - Image Processing Algorithm data serializer for {{ module_name }} + */ + +// automatically generated by custom compiler + +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{ module_name|upper }}_H__ +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{ module_name|upper }}_H__ + +#include +#include + +#include "libcamera/internal/control_serializer.h" +#include "libcamera/internal/ipa_data_serializer.h" + +#include +#include + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPADataSerializer) +{% for struct in structs_nonempty %} +template<> +class IPADataSerializer<{{ struct.mojom_name }}> +{ +public: +{{- serializer.serializer(struct, base_controls) }} +{%- if struct|has_fd %} +{{ serializer.deserializer_fd(struct) }} +{%- else %} +{{ serializer.deserializer_no_fd(struct) }} +{%- endif %} +}; +{% endfor %} + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{ module_name|upper }}_H__ */ diff --git a/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl new file mode 100644 index 00000000..5ddc715b --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl @@ -0,0 +1,180 @@ +{# + # \brief Generate fuction prototype + # + # \param class Class name + # \param method mojom Method object + # \param suffix Suffix to append to \a method function name + # \param need_class_name True to generate class name with function + # \param override True to generate override tag after the function prototype + #} +{%- macro func_sig(class, method, suffix, need_class_name = true, override = false) -%} +{{method|method_return_value}} {{ class + "::" if need_class_name }}{{method.mojom_name}}{{suffix}}( +{%- for param in method|method_parameters %} + {{param}}{{- "," if not loop.last}} +{%- endfor -%} +){{" override" if override}} +{%- endmacro -%} + +{# + # \brief Generate function body for IPA init() function for thread + #} +{%- macro init_thread_body() -%} + int ret = ipa_->init(settings); + if (ret) + return ret; + + proxy_.moveToThread(&thread_); + + return 0; +{%- endmacro -%} + +{# + # \brief Generate function body for IPA start() function for thread + #} +{%- macro start_thread_body() -%} + running_ = true; + thread_.start(); + + return proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking); +{%- endmacro -%} + +{# + # \brief Generate function body for IPA stop() function for thread + #} +{%- macro stop_thread_body() -%} + if (!running_) + return; + + running_ = false; + + proxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking); + + thread_.exit(); + thread_.wait(); +{%- endmacro -%} + + +{# + # \brief Serialize multiple objects into data buffer and fd vector + # + # Generate code to serialize multiple objects, as specified in \a params + # (which are the parameters to some function), into \a buf data buffer and + # \a fds fd vector. + # This code is meant to be used by the proxy, for serializing prior to IPC calls. + #} +{%- macro serialize_call(params, buf, fds) %} +{%- for param in params %} + std::vector {{param.mojom_name}}Buf; +{%- if param|has_fd %} + std::vector {{param.mojom_name}}Fds; + std::tie({{param.mojom_name}}Buf, {{param.mojom_name}}Fds) = +{%- else %} + std::tie({{param.mojom_name}}Buf, std::ignore) = +{%- endif %} + IPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}} +{{- ", &controlSerializer_" if param|needs_control_serializer -}} +); +{%- endfor %} + +{%- if params|length > 1 %} +{%- for param in params %} + appendUInt({{buf}}, {{param.mojom_name}}Buf.size()); +{%- if param|has_fd %} + appendUInt({{buf}}, {{param.mojom_name}}Fds.size()); +{%- endif %} +{%- endfor %} +{%- endif %} + +{%- for param in params %} + {{buf}}.insert({{buf}}.end(), {{param.mojom_name}}Buf.begin(), {{param.mojom_name}}Buf.end()); +{%- endfor %} + +{%- for param in params %} +{%- if param|has_fd %} + {{fds}}.insert({{fds}}.end(), {{param.mojom_name}}Fds.begin(), {{param.mojom_name}}Fds.end()); +{%- endif %} +{%- endfor %} +{%- endmacro -%} + + +{# + # \brief Deserialize a single object from data buffer and fd vector + # + # \param pointer True deserializes the object into a dereferenced pointer + # + # Generate code to deserialize a single object, as specified in \a param, + # from \a buf data buffer and \a fds fd vector. + # This code is meant to be used by macro deserialize_call. + #} +{%- macro deserialize_param(param, pointer, loop, buf, fds) -%} +{{"*" if pointer}}{{param.mojom_name}} = IPADataSerializer<{{param|name}}>::deserialize( + {{buf}}.begin() + {{param.mojom_name}}Start, +{%- if loop.last %} + {{buf}}.end() +{%- else %} + {{buf}}.begin() + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize +{%- endif -%} +{{- "," if param|has_fd }} +{%- if param|has_fd %} + {{fds}}.begin() + {{param.mojom_name}}FdStart, +{%- if loop.last %} + {{fds}}.end() +{%- else %} + {{fds}}.begin() + {{param.mojom_name}}FdStart + {{param.mojom_name}}FdsSize +{%- endif -%} +{%- endif -%} +{{- "," if param|needs_control_serializer }} +{%- if param|needs_control_serializer %} + &controlSerializer_ +{%- endif -%} +); +{%- endmacro -%} + + +{# + # \brief Deserialize multiple objects from data buffer and fd vector + # + # \param pointer True deserializes objects into pointers, and adds a null check. + # \param declare True declares the objects in addition to deserialization. + # + # Generate code to deserialize multiple objects, as specified in \a params + # (which are the parameters to some function), from \a buf data buffer and + # \a fds fd vector. + # This code is meant to be used by the proxy, for serializing prior to IPC calls. + #} +{%- macro deserialize_call(params, buf, fds, pointer = true, declare = false) -%} +{% set ns = namespace(size_offset = 0) %} +{%- if params|length > 1 %} +{%- for param in params %} + [[maybe_unused]] size_t {{param.mojom_name}}BufSize = readUInt({{buf}}, {{ns.size_offset}}); + {%- set ns.size_offset = ns.size_offset + 4 %} +{%- if param|has_fd %} + [[maybe_unused]] size_t {{param.mojom_name}}FdsSize = readUInt({{buf}}, {{ns.size_offset}}); + {%- set ns.size_offset = ns.size_offset + 4 %} +{%- endif %} +{%- endfor %} +{%- endif %} +{% for param in params %} +{%- if loop.first %} + size_t {{param.mojom_name}}Start = {{ns.size_offset}}; +{%- else %} + size_t {{param.mojom_name}}Start = {{loop.previtem.mojom_name}}Start + {{loop.previtem.mojom_name}}BufSize; +{%- endif %} +{%- endfor %} +{% for param in params|with_fds %} +{%- if loop.first %} + size_t {{param.mojom_name}}FdStart = 0; +{%- elif not loop.last %} + size_t {{param.mojom_name}}FdStart = {{loop.previtem.mojom_name}}FdStart + {{loop.previtem.mojom_name}}FdsSize; +{%- endif %} +{%- endfor %} +{% for param in params %} + {%- if pointer %} + if ({{param.mojom_name}}) { +{{deserialize_param(param, pointer, loop, buf, fds)|indent(16, True)}} + } + {%- else %} + {{param|name + " " if declare}}{{deserialize_param(param, pointer, loop, buf, fds)|indent(8)}} + {%- endif %} +{% endfor %} +{%- endmacro -%} diff --git a/utils/ipc/generators/libcamera_templates/serializer.tmpl b/utils/ipc/generators/libcamera_templates/serializer.tmpl new file mode 100644 index 00000000..a1b7b9f8 --- /dev/null +++ b/utils/ipc/generators/libcamera_templates/serializer.tmpl @@ -0,0 +1,262 @@ +{# Turn this into a C macro? #} +{# + # \brief Verify that there is enough bytes to deserialize + # + # Generate code that verifies that \a size is not greater than \a dataSize. + # Otherwise log an error with \a name and \a typename. + #} +{%- macro check_data_size(size, dataSize, name, typename) %} + if ({{size}} > {{dataSize}}) { + LOG(IPADataSerializer, Error) + << "Failed to deserialize {{name}}: not enough {{typename}}, expected " + << ({{size}}) << ", got " << ({{dataSize}}); + return ret; + } +{%- endmacro %} + + +{# + # \brief Serialize some field into return vector + # + # Generate code to serialize \a field into ret_data, including size of the + # field and fds (where appropriate). \a base_controls indicates the + # default ControlInfoMap in the event that the ControlList does not have one. + # This code is meant to be used by the IPADataSerializer specialization. + #} +{%- macro serializer_field(field, base_controls, loop) %} +{%- if field|is_pod %} + appendUInt<{{field|name}}>(ret_data, static_cast(data.{{field.mojom_name}}_)); +{%- elif field|is_enum %} + appendUInt(ret_data, static_cast(data.{{field.mojom_name}}_)); +{%- elif field|is_fd %} + std::vector {{field.mojom_name}}; + std::vector {{field.mojom_name}}Fds; + std::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) = + IPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_); + ret_data.insert(ret_data.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end()); + ret_fds.insert(ret_fds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end()); +{%- elif field|is_controls %} + if (data.{{field.mojom_name}}_.size() > 0) { + std::vector {{field.mojom_name}}; + std::tie({{field.mojom_name}}, std::ignore) = + IPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, + data.{{field.mojom_name}}_.infoMap() ? *data.{{field.mojom_name}}_.infoMap() : {{base_controls}}, + cs); + appendUInt(ret_data, {{field.mojom_name}}.size()); + ret_data.insert(ret_data.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end()); + } else { + appendUInt(ret_data, 0); + } +{%- elif field|is_plain_struct or field|is_array or field|is_map %} + std::vector {{field.mojom_name}}; + {%- if field|has_fd %} + std::vector {{field.mojom_name}}Fds; + std::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) = + {%- else %} + std::tie({{field.mojom_name}}, std::ignore) = + {%- endif %} + IPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, cs); + appendUInt(ret_data, {{field.mojom_name}}.size()); + {%- if field|has_fd %} + appendUInt(ret_data, {{field.mojom_name}}Fds.size()); + {%- endif %} + ret_data.insert(ret_data.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end()); + {%- if field|has_fd %} + ret_fds.insert(ret_fds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end()); + {%- endif %} +{%- else %} + /* Unknown serialization for {{field.mojom_name}}. */ +{%- endif %} +{%- endmacro %} + + +{# + # \brief Deserialize some field into return struct + # + # Generate code to deserialize \a field into object ret. + # This code is meant to be used by the IPADataSerializer specialization. + #} +{%- macro deserializer_field(field, loop) %} +{% if field|is_pod or field|is_enum %} + {%- set field_size = (field|bit_width|int / 8)|int %} + {{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data') }} + {%- if field|is_pod %} + ret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(readUInt<{{field|name}}>(m)); + {%- else %} + ret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(readUInt(m)); + {%- endif %} + {%- if not loop.last %} + m += {{field_size}}; + dataSize -= {{field_size}}; + {%- endif %} +{% elif field|is_fd %} + {%- set field_size = 1 %} + {{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data') }} + ret.{{field.mojom_name}}_ = IPADataSerializer<{{field|name}}>::deserialize(m, m + 1, n, n + 1, cs); + {%- if not loop.last %} + m += {{field_size}}; + dataSize -= {{field_size}}; + n += ret.{{field.mojom_name}}_.isValid() ? 1 : 0; + fdsSize -= ret.{{field.mojom_name}}_.isValid() ? 1 : 0; + {%- endif %} +{% elif field|is_controls %} + {%- set field_size = 4 %} + {{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data') }} + size_t {{field.mojom_name}}Size = readUInt(m); + {%- set field_size = '4 + ' + field.mojom_name + 'Size' -%} + {{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data') }} + if ({{field.mojom_name}}Size > 0) + ret.{{field.mojom_name}}_ = + IPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs); + {%- if not loop.last %} + m += {{field_size}}; + dataSize -= {{field_size}}; + {%- endif %} +{% elif field|is_plain_struct or field|is_array or field|is_map%} + {%- set field_size = 4 %} + {{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data') }} + size_t {{field.mojom_name}}Size = readUInt(m); + {%- if field|has_fd %} + {%- set field_size = 8 %} + {{- check_data_size(field_size, 'dataSize', field.mojom_name + 'FdsSize', 'data') }} + size_t {{field.mojom_name}}FdsSize = readUInt(m + 4); + {{- check_data_size(field.mojom_name + 'FdsSize', 'fdsSize', field.mojom_name, 'fds') }} + {%- endif %} + {%- set field_size = field|has_fd|choose('8 + ', '4 + ') + field.mojom_name + 'Size' -%} + {{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data') }} + ret.{{field.mojom_name}}_ = + {%- if field|has_fd %} + IPADataSerializer<{{field|name}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs); + {%- else %} + IPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs); + {%- endif %} + {%- if not loop.last %} + m += {{field_size}}; + dataSize -= {{field_size}}; + {%- if field|has_fd %} + n += {{field.mojom_name}}FdsSize; + fdsSize -= {{field.mojom_name}}FdsSize; + {%- endif %} + {%- endif %} +{% else %} + /* Unknown deserialization for {{field.mojom_name}}. */ +{%- endif %} +{%- endmacro %} + + +{# + # \brief Serialize a struct + # + # Generate code for IPADataSerializer specialization, for serializing + # \a struct. \a base_controls indicates the default ControlInfoMap + # in the event that the ControlList does not have one. + #} +{%- macro serializer(struct, base_controls) %} + static std::tuple, std::vector> + serialize(const {{ struct.mojom_name }} &data, +{%- if struct|needs_control_serializer %} + ControlSerializer *cs) +{%- else %} + [[maybe_unused]] ControlSerializer *cs = nullptr) +{%- endif %} + { + std::vector ret_data; +{%- if struct|has_fd %} + std::vector ret_fds; +{%- endif %} +{%- for field in struct.fields %} +{{serializer_field(field, base_controls, loop)}} +{%- endfor %} +{% if struct|has_fd %} + return {ret_data, ret_fds}; +{%- else %} + return {ret_data, {}}; +{%- endif %} + } +{%- endmacro %} + + +{# + # \brief Deserialize a struct that has fds + # + # Generate code for IPADataSerializer specialization, for deserializing + # \a struct, in the case that \a struct has file descriptors. + #} +{%- macro deserializer_fd(struct) %} + static {{ struct.mojom_name }} + deserialize(std::vector &data, + std::vector &fds, +{%- if struct|needs_control_serializer %} + ControlSerializer *cs) +{%- else %} + [[maybe_unused]] ControlSerializer *cs = nullptr) +{%- endif %} + { + {{struct.mojom_name}} ret; + std::vector::iterator m = data.begin(); + std::vector::iterator n = fds.begin(); + + size_t dataSize = data.size(); + size_t fdsSize = fds.size(); +{%- for field in struct.fields -%} +{{deserializer_field(field, loop)}} +{%- endfor %} + return ret; + } + + static {{ struct.mojom_name }} + deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + std::vector::iterator fds_it1, + std::vector::iterator fds_it2, +{%- if struct|needs_control_serializer %} + ControlSerializer *cs) +{%- else %} + [[maybe_unused]] ControlSerializer *cs = nullptr) +{%- endif %} + { + std::vector data(data_it1, data_it2); + std::vector fds(fds_it1, fds_it2); + return IPADataSerializer<{{struct.mojom_name}}>::deserialize(data, fds, cs); + } +{%- endmacro %} + + +{# + # \brief Deserialize a struct that has no fds + # + # Generate code for IPADataSerializer specialization, for deserializing + # \a struct, in the case that \a struct does not have file descriptors. + #} +{%- macro deserializer_no_fd(struct) %} + static {{ struct.mojom_name }} + deserialize(std::vector &data, +{%- if struct|needs_control_serializer %} + ControlSerializer *cs) +{%- else %} + [[maybe_unused]] ControlSerializer *cs = nullptr) +{%- endif %} + { + {{struct.mojom_name}} ret; + std::vector::iterator m = data.begin(); + + size_t dataSize = data.size(); +{%- for field in struct.fields -%} +{{deserializer_field(field, loop)}} +{%- endfor %} + return ret; + } + + static {{ struct.mojom_name }} + deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, +{%- if struct|needs_control_serializer %} + ControlSerializer *cs) +{%- else %} + [[maybe_unused]] ControlSerializer *cs = nullptr) +{%- endif %} + { + std::vector data(data_it1, data_it2); + return IPADataSerializer<{{struct.mojom_name}}>::deserialize(data, cs); + } +{%- endmacro %} diff --git a/utils/ipc/generators/mojom_libcamera_generator.py b/utils/ipc/generators/mojom_libcamera_generator.py new file mode 100644 index 00000000..6a65dbc2 --- /dev/null +++ b/utils/ipc/generators/mojom_libcamera_generator.py @@ -0,0 +1,403 @@ +'''Generates libcamera files from a mojom.Module.''' + +import argparse +import ast +import datetime +import contextlib +import os +import re +import shutil +import sys +import tempfile + +from jinja2 import contextfilter + +import mojom.fileutil as fileutil +import mojom.generate.generator as generator +import mojom.generate.module as mojom +from mojom.generate.template_expander import UseJinja + + +GENERATOR_PREFIX = 'libcamera' + +_kind_to_cpp_type = { + mojom.BOOL: 'bool', + mojom.INT8: 'int8_t', + mojom.UINT8: 'uint8_t', + mojom.INT16: 'int16_t', + mojom.UINT16: 'uint16_t', + mojom.INT32: 'int32_t', + mojom.UINT32: 'uint32_t', + mojom.FLOAT: 'float', + mojom.INT64: 'int64_t', + mojom.UINT64: 'uint64_t', + mojom.DOUBLE: 'double', +} + +_bit_widths = { + mojom.BOOL: '8', + mojom.INT8: '8', + mojom.UINT8: '8', + mojom.INT16: '16', + mojom.UINT16: '16', + mojom.INT32: '32', + mojom.UINT32: '32', + mojom.FLOAT: '32', + mojom.INT64: '64', + mojom.UINT64: '64', + mojom.DOUBLE: '64', +} + +def ModuleName(path): + return path.split('/')[-1].split('.')[0] + +def ModuleClassName(module): + s = re.sub(r'^IPA', '', module.interfaces[0].mojom_name) + return re.sub(r'Interface$', '', s) + +def UpperCamelCase(name): + return ''.join([x.capitalize() for x in generator.SplitCamelCase(name)]) + +def CamelCase(name): + uccc = UpperCamelCase(name) + return uccc[0].lower() + uccc[1:] + +def ConstantStyle(name): + return generator.ToUpperSnakeCase(name) + +def Choose(cond, t, f): + return t if cond else f + +def CommaSep(l): + return ', '.join([m for m in l]) + +def ParamsCommaSep(l): + return ', '.join([m.mojom_name for m in l]) + +def GetDefaultValue(element): + if element.default is not None: + return element.default + if type(element.kind) == mojom.Kind: + return '0' + if mojom.IsEnumKind(element.kind): + return f'static_cast<{element.kind.mojom_name}>(0)' + if (isinstance(element.kind, mojom.Struct) and + element.kind.mojom_name == 'FileDescriptor'): + return '-1' + return '' + +def WithDefaultValues(element): + return [x for x in element if HasDefaultValue(x)] + +def WithFds(element): + return [x for x in element if HasFd(x)] + +def HasDefaultValue(element): + return GetDefaultValue(element) != '' + +def HasDefaultFields(element): + return True in [HasDefaultValue(x) for x in element.fields] + +def GetAllTypes(element): + if mojom.IsArrayKind(element): + return GetAllTypes(element.kind) + if mojom.IsMapKind(element): + return GetAllTypes(element.key_kind) + GetAllTypes(element.value_kind) + if isinstance(element, mojom.Parameter): + return GetAllTypes(element.kind) + if mojom.IsEnumKind(element): + return [element.mojom_name] + if not mojom.IsStructKind(element): + return [element.spec] + if len(element.fields) == 0: + return [element.mojom_name] + ret = [GetAllTypes(x.kind) for x in element.fields] + ret = [x for sublist in ret for x in sublist] + return list(set(ret)) + +def GetAllAttrs(element): + if mojom.IsArrayKind(element): + return GetAllAttrs(element.kind) + if mojom.IsMapKind(element): + return {**GetAllAttrs(element.key_kind), **GetAllAttrs(element.value_kind)} + if isinstance(element, mojom.Parameter): + return GetAllAttrs(element.kind) + if mojom.IsEnumKind(element): + return element.attributes if element.attributes is not None else {} + if mojom.IsStructKind(element) and len(element.fields) == 0: + return element.attributes if element.attributes is not None else {} + if not mojom.IsStructKind(element): + return {} + attrs = [(x.attributes) for x in element.fields] + ret = {} + for d in attrs: + if d is not None: + ret = {**ret, **d} + return ret + +def NeedsControlSerializer(element): + types = GetAllTypes(element) + return "ControlList" in types or "ControlInfoMap" in types + +def HasFd(element): + attrs = GetAllAttrs(element) + if isinstance(element, mojom.Kind): + types = GetAllTypes(element) + else: + types = GetAllTypes(element.kind) + return "FileDescriptor" in types or (attrs is not None and "hasFd" in attrs) + +def MethodInputHasFd(method): + if len([x for x in method.parameters if HasFd(x)]) > 0: + return True + return False + +def MethodOutputHasFd(method): + if (MethodReturnValue(method) != 'void' or + method.response_parameters is None): + return False + if len([x for x in method.parameters if HasFd(x)]) > 0: + return True + return False + +def MethodParamInputs(method): + return method.parameters + +def MethodParamOutputs(method): + if (MethodReturnValue(method) != 'void' or + method.response_parameters is None): + return [] + return method.response_parameters + +def MethodParamNames(method): + params = [] + for param in method.parameters: + params.append(param.mojom_name) + if MethodReturnValue(method) == 'void': + if method.response_parameters is None: + return params + for param in method.response_parameters: + params.append(param.mojom_name) + return params + +def MethodParameters(method): + params = [] + for param in method.parameters: + params.append('const %s %s%s' % (GetNameForElement(param), + ('&' if not IsPod(param) else ''), + param.mojom_name)) + if MethodReturnValue(method) == 'void': + if method.response_parameters is None: + return params + for param in method.response_parameters: + params.append(f'{GetNameForElement(param)} *{param.mojom_name}') + return params + +def MethodReturnValue(method): + if method.response_parameters is None: + return 'void' + if len(method.response_parameters) == 1 and IsPod(method.response_parameters[0]): + return GetNameForElement(method.response_parameters[0]) + return 'void' + +def IsAsync(method): + # callbacks are always async + if re.match("^IPA.*CallbackInterface$", method.interface.mojom_name): + return True + elif re.match("^IPA.*Interface$", method.interface.mojom_name): + if method.attributes is None: + return False + elif 'async' in method.attributes and method.attributes['async']: + return True + return False + +def IsArray(element): + return mojom.IsArrayKind(element.kind) + +def IsControls(element): + return mojom.IsStructKind(element.kind) and (element.kind.mojom_name == "ControlList" or + element.kind.mojom_name == "ControlInfoMap") + +def IsEnum(element): + return mojom.IsEnumKind(element.kind) + +def IsFd(element): + return mojom.IsStructKind(element.kind) and element.kind.mojom_name == "FileDescriptor" + +def IsMap(element): + return mojom.IsMapKind(element.kind) + +def IsPlainStruct(element): + return mojom.IsStructKind(element.kind) and not IsControls(element) and not IsFd(element) + +def IsPod(element): + return element.kind in _kind_to_cpp_type + +def BitWidth(element): + if element.kind in _bit_widths: + return _bit_widths[element.kind] + if mojom.IsEnumKind(element.kind): + return '32' + return '' + +def GetNameForElement(element): + if (mojom.IsEnumKind(element) or + mojom.IsInterfaceKind(element) or + mojom.IsStructKind(element)): + return element.mojom_name + if (mojom.IsArrayKind(element)): + elem_name = GetNameForElement(element.kind) + return f'std::vector<{elem_name}>' + if (mojom.IsMapKind(element)): + key_name = GetNameForElement(element.key_kind) + value_name = GetNameForElement(element.value_kind) + return f'std::map<{key_name}, {value_name}>' + if isinstance(element, (mojom.Field, mojom.Method, mojom.Parameter)): + if (mojom.IsArrayKind(element.kind) or mojom.IsMapKind(element.kind)): + return GetNameForElement(element.kind) + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type): + return _kind_to_cpp_type[element.kind] + return element.kind.mojom_name + if isinstance(element, mojom.EnumValue): + return (GetNameForElement(element.enum) + '.' + + ConstantStyle(element.name)) + if isinstance(element, (mojom.NamedValue, + mojom.Constant, + mojom.EnumField)): + return ConstantStyle(element.name) + if (hasattr(element, '__hash__') and element in _kind_to_cpp_type): + return _kind_to_cpp_type[element] + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type): + return _kind_to_cpp_type[element.kind] + if (hasattr(element, 'spec')): + return element.spec.split(':')[-1] + if (mojom.IsInterfaceRequestKind(element) or + mojom.IsAssociatedKind(element) or + mojom.IsPendingRemoteKind(element) or + mojom.IsPendingReceiverKind(element) or + mojom.IsUnionKind(element)): + raise Exception('Unsupported element: %s' % element) + raise Exception('Unexpected element: %s' % element) + + +class Generator(generator.Generator): + + @staticmethod + def GetTemplatePrefix(): + return 'libcamera_templates' + + def GetFilters(self): + libcamera_filters = { + 'all_types': GetAllTypes, + 'bit_width': BitWidth, + 'choose': Choose, + 'comma_sep': CommaSep, + 'default_value': GetDefaultValue, + 'has_default_fields': HasDefaultFields, + 'has_fd': HasFd, + 'is_async': IsAsync, + 'is_array': IsArray, + 'is_controls': IsControls, + 'is_enum': IsEnum, + 'is_fd': IsFd, + 'is_map': IsMap, + 'is_plain_struct': IsPlainStruct, + 'is_pod': IsPod, + 'method_input_has_fd': MethodInputHasFd, + 'method_output_has_fd': MethodOutputHasFd, + 'method_param_names': MethodParamNames, + 'method_param_inputs': MethodParamInputs, + 'method_param_outputs': MethodParamOutputs, + 'method_parameters': MethodParameters, + 'method_return_value': MethodReturnValue, + 'name': GetNameForElement, + 'needs_control_serializer': NeedsControlSerializer, + 'params_comma_sep': ParamsCommaSep, + 'with_default_values': WithDefaultValues, + 'with_fds': WithFds, + } + return libcamera_filters + + def _GetJinjaExports(self): + return { + 'base_controls': '%sControls' % ModuleClassName(self.module), + 'cmd_enum_name': '%sCMD' % ModuleClassName(self.module), + 'enums': self.module.enums, + 'has_array': len([x for x in self.module.kinds.keys() if x[0] == 'a']) > 0, + 'has_map': len([x for x in self.module.kinds.keys() if x[0] == 'm']) > 0, + 'has_namespace': self.module.namespace is not None, + 'imports': self.module.imports, + 'interface_cb': self.module.interfaces[1], + 'interface_main': self.module.interfaces[0], + 'interface_name': 'IPA%sInterface' % ModuleClassName(self.module), + 'ipc_name': 'IPAIPCUnixSocket', + 'kinds': self.module.kinds, + 'module': self.module, + 'module_name': ModuleName(self.module.path), + 'module_class_name': ModuleClassName(self.module), + 'namespace': self.module.namespace, + 'proxy_name': 'IPAProxy%s' % ModuleClassName(self.module), + 'proxy_worker_name': 'IPAProxy%sWorker' % ModuleClassName(self.module), + 'structs_nonempty': [x for x in self.module.structs if len(x.fields) > 0], + 'year': datetime.datetime.now().year, + } + + + @UseJinja('module_generated.h.tmpl') + def _GenerateDataHeader(self): + return self._GetJinjaExports() + + @UseJinja('module_serializer.h.tmpl') + def _GenerateSerializer(self): + return self._GetJinjaExports() + + @UseJinja('module_ipa_proxy.cpp.tmpl') + def _GenerateProxyCpp(self): + return self._GetJinjaExports() + + @UseJinja('module_ipa_proxy.h.tmpl') + def _GenerateProxyHeader(self): + return self._GetJinjaExports() + + @UseJinja('module_ipa_proxy_worker.cpp.tmpl') + def _GenerateProxyWorker(self): + return self._GetJinjaExports() + + def GenerateFiles(self, unparsed_args): + parser = argparse.ArgumentParser() + parser.add_argument('--libcamera_generate_header', action='store_true') + parser.add_argument('--libcamera_generate_serializer', action='store_true') + parser.add_argument('--libcamera_generate_proxy_cpp', action='store_true') + parser.add_argument('--libcamera_generate_proxy_h', action='store_true') + parser.add_argument('--libcamera_generate_proxy_worker', action='store_true') + parser.add_argument('--libcamera_output_path') + args = parser.parse_args(unparsed_args) + + fileutil.EnsureDirectoryExists(os.path.dirname(args.libcamera_output_path)) + + module_name = ModuleName(self.module.path) + + if args.libcamera_generate_header: + self.Write(self._GenerateDataHeader(), + args.libcamera_output_path) + #f'{module_name}_generated.h') + + if args.libcamera_generate_serializer: + self.Write(self._GenerateSerializer(), + args.libcamera_output_path) + #f'{module_name}_serializer.h') + + if args.libcamera_generate_proxy_cpp: + self.Write(self._GenerateProxyCpp(), + args.libcamera_output_path) + #f'ipa_proxy_{module_name}.cpp') + + if args.libcamera_generate_proxy_h: + self.Write(self._GenerateProxyHeader(), + args.libcamera_output_path) + #f'ipa_proxy_{module_name}.h') + + if args.libcamera_generate_proxy_worker: + self.Write(self._GenerateProxyWorker(), + args.libcamera_output_path) + #f'ipa_proxy_{module_name}_worker.cpp') From patchwork Tue Sep 15 14:20:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9613 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D65A3BF01C for ; Tue, 15 Sep 2020 14:21:06 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A411762E1B; Tue, 15 Sep 2020 16:21:06 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="pyeEXDvp"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id F135662D27 for ; Tue, 15 Sep 2020 16:21:02 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 26F4E10C6; Tue, 15 Sep 2020 16:21:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179662; bh=IX+V+06FXBOgZFhVffEcGAc+9RgidtxOYH3rGg7MZs8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pyeEXDvp5QykVICsW5dXo0SXc2OrzZXvLUE5KIpbvdE2tIkalTr4seJ9yyZWEOhEM J6jcamDE5OyoHMc/2dQtlmsouF3yWvaxioxnj0Tq8yiedpumXTAggnG15DoftF12Vq 9TXHJOSSfxIt9Qk9X0KHy9rDCFCvowjcvzpl5dq0= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:19 +0900 Message-Id: <20200915142038.28757-5-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 04/23] utils: ipc: add generator script X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We want to avoid changing our copy of mojo to make updates easier. Some parameters in the mojo generator script needs to be changed though; add a wrapper script that sets these parameters. Signed-off-by: Paul Elder --- utils/ipc/generate.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 utils/ipc/generate.py diff --git a/utils/ipc/generate.py b/utils/ipc/generate.py new file mode 100755 index 00000000..39160aa2 --- /dev/null +++ b/utils/ipc/generate.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import os +import sys +import mojo.public.tools.bindings.mojom_bindings_generator as generator + +def _GetModulePath(path, output_dir): + return os.path.join(output_dir, path.relative_path()) + +generator._BUILTIN_GENERATORS = {'libcamera': 'mojom_libcamera_generator'} +generator._GetModulePath = _GetModulePath +sys.path.append(os.path.join(os.path.dirname(__file__), 'utils/ipc')) + +generator.main() + +#./bindings/mojom_bindings_generator.py generate -g libcamera --bytecode_path . ./raspberrypi.mojom From patchwork Tue Sep 15 14:20:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9614 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 5CB08BF01C for ; Tue, 15 Sep 2020 14:21:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 2CA0562E1B; Tue, 15 Sep 2020 16:21:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="tRobn0uj"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6178E62E17 for ; Tue, 15 Sep 2020 16:21:05 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7D912AEA; Tue, 15 Sep 2020 16:21:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179665; bh=lyec7IuQNDjIoeYJuLdzbvq+kOvt4iSbPVo3OIP9voA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tRobn0ujZ7BPnUTv9fZUj1TXLd7fTkpwM/+cLI+xGcMDjvJEThVqQdY7SkO8ZfQjx xnHe0Voe0CaDB58UPUVPt/q4rBjvvTLTwa7ifQ4R2vwA23gc0NVbu23ilsGgfH1mQQ BOm4euWSR2vAr3lcDSthOn7q80v2qcW28+STJW74= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:20 +0900 Message-Id: <20200915142038.28757-6-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 05/23] meson: ipa, proxy: Generate headers and proxy with mojo X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Run mojo from meson to generate the header, serializer, and proxy files for every pipeline's mojom data definition file. So far we only have the raspberrypi mojom file. Signed-off-by: Paul Elder --- Dictionaries with non-literal keys is apparently a meson 0.53 feature. include/libcamera/ipa/meson.build | 87 ++++++++++++++++++++++++++ src/libcamera/proxy/meson.build | 21 +++++++ src/libcamera/proxy/worker/meson.build | 25 +++++++- 3 files changed, 131 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index 508c6bd1..f61a5a8f 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -8,3 +8,90 @@ libcamera_ipa_headers = files([ install_headers(libcamera_ipa_headers, subdir: join_paths(libcamera_include_dir, 'ipa')) + +# +# Prepare IPA/IPC generation components +# + +ipa_mojom_files = [] + +mojom_generator = find_program('../../../utils/ipc/generate.py') + +mojom_template = custom_target('mojom_template', + output : 'dummy', + command : [mojom_generator, 'precompile']) + +mojom_parser = find_program('../../../utils/ipc/mojo/public/tools/mojom/mojom_parser.py') + +# {pipeline}.mojom-module +ipa_mojoms = {} + +foreach file : ipa_mojom_files + name = file.split('.')[0] + mojom = custom_target(file.split('.')[0] + '_mojom_module', + input : file, + output : file + '-module', + command : [mojom_parser, + '--output-root', meson.build_root(), + '--input-root', meson.source_root(), + '--mojoms', '@INPUT@' + ]) + ipa_mojoms += { name: mojom } +endforeach + +# +# Generate headers from templates. +# + +# {pipeline}_generated.h +# generate {pipeline}_serializer.h +# generate ipa_proxy_{pipeline}.h +ipa_headers = {} +ipa_serializers = {} +ipa_proxy_headers = {} + +foreach name, mojom : ipa_mojoms + header = custom_target(name + '_generated_h', + input : mojom, + output : name + '_generated.h', + depends : mojom_template, + command : [mojom_generator, 'generate', + '-g', 'libcamera', + '--bytecode_path', '.', + '--libcamera_generate_header', + '--libcamera_output_path=@OUTPUT@', + './' +'@INPUT@' + ]) + + serializer = custom_target(name + '_serializer_h', + input : mojom, + output : name + '_serializer.h', + depends : mojom_template, + command : [mojom_generator, 'generate', + '-g', 'libcamera', + '--bytecode_path', '.', + '--libcamera_generate_serializer', + '--libcamera_output_path=@OUTPUT@', + './' +'@INPUT@' + ]) + + proxy_header = custom_target(name + '_proxy_h', + input : mojom, + output : 'ipa_proxy_' + name + '.h', + depends : mojom_template, + command : [mojom_generator, 'generate', + '-g', 'libcamera', + '--bytecode_path', '.', + '--libcamera_generate_proxy_h', + '--libcamera_output_path=@OUTPUT@', + './' +'@INPUT@' + ]) + + ipa_headers += { name: header } + ipa_serializers += { name: serializer } + ipa_proxy_headers += { name: proxy_header } +endforeach + +libcamera_internal_headers += ipa_proxy_headers +libcamera_internal_headers += ipa_headers +libcamera_internal_headers += ipa_serializers diff --git a/src/libcamera/proxy/meson.build b/src/libcamera/proxy/meson.build index bd804750..b0d35646 100644 --- a/src/libcamera/proxy/meson.build +++ b/src/libcamera/proxy/meson.build @@ -4,3 +4,24 @@ libcamera_sources += files([ 'ipa_proxy_linux.cpp', 'ipa_proxy_thread.cpp', ]) + +# generate ipa_proxy_{pipeline}.cpp +ipa_proxy_sources = [] + +foreach name, mojom : ipa_mojoms + ipa_proxy_sources += custom_target(name + '_proxy_cpp', + input : mojom, + output : 'ipa_proxy_' + name + '.cpp', + depends : [mojom_template, + ipa_headers[name], + ipa_serializers[name], + ipa_proxy_headers[name]], + command : [mojom_generator, 'generate', + '-g', 'libcamera', + '--bytecode_path', '.', + '--libcamera_generate_proxy_cpp', + '--libcamera_output_path=@OUTPUT@', + './' + '@INPUT@']) +endforeach + +libcamera_sources += ipa_proxy_sources diff --git a/src/libcamera/proxy/worker/meson.build b/src/libcamera/proxy/worker/meson.build index ac0310a7..c35be70c 100644 --- a/src/libcamera/proxy/worker/meson.build +++ b/src/libcamera/proxy/worker/meson.build @@ -4,10 +4,31 @@ ipa_proxy_sources = [ ['ipa_proxy_linux', 'ipa_proxy_linux_worker.cpp'] ] +# generate ipa_proxy_{pipeline}_worker.cpp +ipa_proxy_sources = {} + +foreach name, mojom : ipa_mojoms + worker = custom_target(name + '_proxy_worker', + input : mojom, + output : 'ipa_proxy_' + name + '_worker.cpp', + depends : [mojom_template, + ipa_headers[name], + ipa_serializers[name], + ipa_proxy_headers[name]], + command : [mojom_generator, 'generate', + '-g', 'libcamera', + '--bytecode_path', '.', + '--libcamera_generate_proxy_worker', + '--libcamera_output_path=@OUTPUT@', + './' + '@INPUT@' + ]) + ipa_proxy_sources += { 'ipa_proxy_' + name : worker } +endforeach + proxy_install_dir = join_paths(get_option('libexecdir'), 'libcamera') -foreach t : ipa_proxy_sources - proxy = executable(t[0], t[1], +foreach k, t : ipa_proxy_sources + proxy = executable(k, t, install : true, install_dir : proxy_install_dir, dependencies : libcamera_dep) From patchwork Tue Sep 15 14:20:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9615 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id CB64EBF01C for ; Tue, 15 Sep 2020 14:21:09 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 96B1862E30; Tue, 15 Sep 2020 16:21:09 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Q5Ies8Mf"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 7A35762E13 for ; Tue, 15 Sep 2020 16:21:07 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id BB62A276; Tue, 15 Sep 2020 16:21:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179667; bh=Df0owFvela1+ZFtd72Fed89eR96xckPniG6lY6aY7N0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q5Ies8MfYFdHeHHdCHRGQolvqj3sw++DYlDhC6e3vT7bwphRlTANHPW5puboyOsSE dd9R8T3vWDWx8w62owW+eWFQI9MFP9C3s4CBu5ftCyrwL07CGCJz0B/GRMsbaqLzNQ r7v2zLvv+UDkprIbdFnQvgNO9jAgN1cZgztkfl44= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:21 +0900 Message-Id: <20200915142038.28757-7-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 06/23] libcamera: Add IPADataSerializer X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add an IPADataSerializer which implments de/serialization of built-in and libcamera data structures. This is intended to be used by the proxy and the proxy worker in the IPC layer. Signed-off-by: Paul Elder --- .../libcamera/internal/ipa_data_serializer.h | 876 ++++++++++++++++++ src/libcamera/ipa_data_serializer.cpp | 29 + src/libcamera/meson.build | 1 + 3 files changed, 906 insertions(+) create mode 100644 include/libcamera/internal/ipa_data_serializer.h create mode 100644 src/libcamera/ipa_data_serializer.cpp diff --git a/include/libcamera/internal/ipa_data_serializer.h b/include/libcamera/internal/ipa_data_serializer.h new file mode 100644 index 00000000..6765e4e2 --- /dev/null +++ b/include/libcamera/internal/ipa_data_serializer.h @@ -0,0 +1,876 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_data_serializer.h - Image Processing Algorithm data serializer + */ +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_H__ +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_H__ + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "libcamera/internal/byte_stream_buffer.h" +#include "libcamera/internal/camera_sensor.h" +#include "libcamera/internal/control_serializer.h" +#include "libcamera/internal/log.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPADataSerializer) + +template +void appendUInt(std::vector &vec, T val) +{ + size_t byteWidth = sizeof(val); + for (size_t i = 0; i < byteWidth; i++) + vec.push_back(static_cast((val >> 8*i) & 0xff)); +} + +template +T readUInt(std::vector &vec, size_t pos) +{ + T ret = 0; + size_t byteWidth = sizeof(ret); + if (pos + byteWidth > vec.size()) + return ret; + + for (size_t i = 0; i < byteWidth; i++) + ret |= vec[pos + i] << 8*i; + return ret; +} + +template +T readUInt(std::vector::iterator it) +{ + T ret = 0; + size_t byteWidth = sizeof(ret); + for (size_t i = 0; i < byteWidth; i++) + ret |= *(it + i) << 8*i; + return ret; +} + +template +class IPADataSerializer +{ +}; + +template +class IPADataSerializer> +{ +public: + static std::tuple, std::vector> + serialize(const std::vector &data, ControlSerializer *cs = nullptr) + { + std::vector data_vec; + std::vector fds_vec; + + // serialize the length + uint32_t vec_len = data.size(); + appendUInt(data_vec, vec_len); + + // serialize the members + for (auto it = data.begin(); it != data.end(); ++it) { + std::vector dvec; + std::vector fvec; + + std::tie(dvec, fvec) = + IPADataSerializer::serialize(*it, cs); + + appendUInt(data_vec, dvec.size()); + appendUInt(data_vec, fvec.size()); + + data_vec.insert(data_vec.end(), dvec.begin(), dvec.end()); + fds_vec.insert(fds_vec.end(), fvec.begin(), fvec.end()); + } + + return {data_vec, fds_vec}; + } + + static std::vector deserialize(std::vector &data, ControlSerializer *cs = nullptr) + { + return IPADataSerializer>::deserialize(data.begin(), data.end(), cs); + } + + static std::vector deserialize(std::vector::iterator it1, + std::vector::iterator it2, + ControlSerializer *cs = nullptr) + { + std::vector fds; + return IPADataSerializer>::deserialize(it1, it2, + fds.begin(), fds.end(), + cs); + } + + static std::vector deserialize(std::vector &data, std::vector &fds, + ControlSerializer *cs = nullptr) + { + return IPADataSerializer>::deserialize(data.begin(), data.end(), + fds.begin(), fds.end(), + cs); + } + + static std::vector deserialize(std::vector::iterator data_it1, + [[maybe_unused]] std::vector::iterator data_it2, + std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + ControlSerializer *cs = nullptr) + { + uint32_t vec_len = readUInt(data_it1); + std::vector ret(vec_len); + + std::vector::iterator data_it = data_it1 + 4; + std::vector::iterator fd_it = fds_it1; + for (uint32_t i = 0; i < vec_len; i++) { + uint32_t sizeof_data = readUInt(data_it); + uint32_t sizeof_fds = readUInt(data_it + 4); + + ret[i] = IPADataSerializer::deserialize(data_it + 8, + data_it + 8 + sizeof_data, + fd_it, + fd_it + sizeof_fds, + cs); + + data_it += 8 + sizeof_data; + fd_it += sizeof_fds; + } + + return ret; + } +}; + +template +class IPADataSerializer> +{ +public: + static std::tuple, std::vector> + serialize(const std::map &data, ControlSerializer *cs = nullptr) + { + std::vector data_vec; + std::vector fds_vec; + + // serialize the length + uint32_t map_len = data.size(); + appendUInt(data_vec, map_len); + + // serialize the members + for (auto it = data.begin(); it != data.end(); ++it) { + std::vector dvec; + std::vector fvec; + + std::tie(dvec, fvec) = + IPADataSerializer::serialize(it->first, cs); + + appendUInt(data_vec, dvec.size()); + appendUInt(data_vec, fvec.size()); + + data_vec.insert(data_vec.end(), dvec.begin(), dvec.end()); + fds_vec.insert(fds_vec.end(), fvec.begin(), fvec.end()); + + std::tie(dvec, fvec) = + IPADataSerializer::serialize(it->second, cs); + + appendUInt(data_vec, dvec.size()); + appendUInt(data_vec, fvec.size()); + + data_vec.insert(data_vec.end(), dvec.begin(), dvec.end()); + fds_vec.insert(fds_vec.end(), fvec.begin(), fvec.end()); + } + + return {data_vec, fds_vec}; + } + + static std::map deserialize(std::vector &data, ControlSerializer *cs = nullptr) + { + return IPADataSerializer>::deserialize(data.begin(), data.end(), cs); + } + + static std::map deserialize(std::vector::iterator it1, + std::vector::iterator it2, + ControlSerializer *cs = nullptr) + { + std::vector fds; + return IPADataSerializer>::deserialize(it1, it2, + fds.begin(), fds.end(), + cs); + } + + static std::map deserialize(std::vector &data, std::vector &fds, + ControlSerializer *cs = nullptr) + { + return IPADataSerializer>::deserialize(data.begin(), data.end(), + fds.begin(), fds.end(), + cs); + } + + static std::map deserialize(std::vector::iterator data_it1, + [[maybe_unused]] std::vector::iterator data_it2, + std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + ControlSerializer *cs = nullptr) + { + std::map ret; + + uint32_t map_len = readUInt(data_it1); + + std::vector::iterator data_it = data_it1 + 4; + std::vector::iterator fd_it = fds_it1; + for (uint32_t i = 0; i < map_len; i++) { + uint32_t sizeof_data = readUInt(data_it); + uint32_t sizeof_fds = readUInt(data_it + 4); + + K key = IPADataSerializer::deserialize(data_it + 8, + data_it + 8 + sizeof_data, + fd_it, + fd_it + sizeof_fds, + cs); + + data_it += 8 + sizeof_data; + fd_it += sizeof_fds; + sizeof_data = readUInt(data_it); + sizeof_fds = readUInt(data_it + 4); + + const V value = IPADataSerializer::deserialize(data_it + 8, + data_it + 8 + sizeof_data, + fd_it, + fd_it + sizeof_fds, + cs); + ret.insert({key, value}); + + data_it += 8 + sizeof_data; + fd_it += sizeof_fds; + } + + return ret; + } +}; + +// TODO implement this for all primitives +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const uint32_t data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec; + appendUInt(data_vec, data); + + return {data_vec, {}}; + } + + static uint32_t deserialize(std::vector &data, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static uint32_t deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return readUInt(it1); + } + + static uint32_t deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static uint32_t deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data_it1, data_it2); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const FileDescriptor &data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec = { data.isValid() }; + std::vector fd_vec; + if (data.isValid()) + fd_vec.push_back(data.fd()); + + return {data_vec, fd_vec}; + } + + static FileDescriptor deserialize(std::vector &data, std::vector &fds, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), + fds.begin(), fds.end()); + } + + static FileDescriptor deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + std::vector::iterator fds_it1, + std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + if (std::distance(data_it1, data_it2) < 1) + LOG(IPADataSerializer, Fatal) + << "Invalid data to deserialize FileDescriptor"; + + bool valid = *data_it1; + + if (valid && std::distance(fds_it1, fds_it2) < 1) + LOG(IPADataSerializer, Fatal) + << "Invalid fds to deserialize FileDescriptor"; + + return valid ? FileDescriptor(*fds_it1) : FileDescriptor(); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const IPASettings &data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec(data.configurationFile.begin(), + data.configurationFile.end()); + + return {data_vec, {}}; + } + + static IPASettings deserialize(std::vector &data, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static IPASettings deserialize(std::vector::iterator it1, + std::vector::iterator it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::string str(it1, it2); + + IPASettings ret; + ret.configurationFile = str; + + return ret; + } + + static IPASettings deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static IPASettings deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data_it1, data_it2); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const CameraSensorInfo &data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec; + + uint32_t str_len = data.model.size(); + appendUInt(data_vec, str_len); + + data_vec.insert(data_vec.end(), data.model.begin(), data.model.end()); + + appendUInt(data_vec, data.bitsPerPixel); + + appendUInt(data_vec, data.activeAreaSize.width); + appendUInt(data_vec, data.activeAreaSize.height); + + appendUInt(data_vec, static_cast(data.analogCrop.x)); + appendUInt(data_vec, static_cast(data.analogCrop.y)); + appendUInt(data_vec, data.analogCrop.width); + appendUInt(data_vec, data.analogCrop.height); + + appendUInt(data_vec, data.outputSize.width); + appendUInt(data_vec, data.outputSize.height); + + appendUInt(data_vec, data.pixelRate); + + appendUInt(data_vec, data.lineLength); + + return {data_vec, {}}; + } + + static CameraSensorInfo deserialize(std::vector &data, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static CameraSensorInfo deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + CameraSensorInfo ret; + + uint32_t str_len = readUInt(it1); + std::string str(it1 + 4, it1 + 4 + str_len); + ret.model = str; + + std::vector::iterator it = it1 + 4 + str_len; + + ret.bitsPerPixel = readUInt(it); + + ret.activeAreaSize.width = readUInt(it + 4); + ret.activeAreaSize.height = readUInt(it + 8); + + ret.analogCrop.x = static_cast(readUInt(it + 12)); + ret.analogCrop.y = static_cast(readUInt(it + 16)); + ret.analogCrop.width = readUInt(it + 20); + ret.analogCrop.height = readUInt(it + 24); + + ret.outputSize.width = readUInt(it + 28); + ret.outputSize.height = readUInt(it + 32); + + ret.pixelRate = readUInt(it + 36); + + ret.lineLength = readUInt(it + 44); + + return ret; + } + + static CameraSensorInfo deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static CameraSensorInfo deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data_it1, data_it2); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const IPAStream &data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec; + + appendUInt(data_vec, data.pixelFormat); + + appendUInt(data_vec, data.size.width); + appendUInt(data_vec, data.size.height); + + return {data_vec, {}}; + } + + static IPAStream deserialize(std::vector &data, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static IPAStream deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + IPAStream ret; + + ret.pixelFormat = readUInt(it1); + + ret.size.width = readUInt(it1 + 4); + ret.size.height = readUInt(it1 + 8); + + return ret; + } + + static IPAStream deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end()); + } + + static IPAStream deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data_it1, data_it2); + } +}; + +template<> +class IPADataSerializer +{ +public: + // map arg will be generated, since it's per-pipeline anyway + static std::tuple, std::vector> + serialize(const ControlList &data, const ControlInfoMap &map, + ControlSerializer *cs) + { + if (!cs) + LOG(IPADataSerializer, Fatal) + << "ControlSerializer not provided for serialization of ControlList"; + + size_t size = cs->binarySize(map); + std::vector infoData(size); + ByteStreamBuffer buffer(infoData.data(), infoData.size()); + int ret = cs->serialize(map, buffer); + + if (ret < 0 || buffer.overflow()) { + LOG(IPADataSerializer, Error) << "Failed to serialize ControlList's ControlInfoMap"; + return {{}, {}}; + } + + size = cs->binarySize(data); + std::vector listData(size); + buffer = ByteStreamBuffer(listData.data(), listData.size()); + ret = cs->serialize(data, buffer); + + if (ret < 0 || buffer.overflow()) { + LOG(IPADataSerializer, Error) << "Failed to serialize ControlList"; + return {{}, {}}; + } + + std::vector data_vec; + appendUInt(data_vec, infoData.size()); + appendUInt(data_vec, listData.size()); + data_vec.insert(data_vec.end(), infoData.begin(), infoData.end()); + data_vec.insert(data_vec.end(), listData.begin(), listData.end()); + + return {data_vec, {}}; + } + + static ControlList deserialize(std::vector &data, ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), cs); + } + + static ControlList deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + ControlSerializer *cs) + { + if (!cs) + LOG(IPADataSerializer, Fatal) + << "ControlSerializer not provided for deserialization of ControlList"; + + uint32_t infoDataSize = readUInt(it1); + uint32_t listDataSize = readUInt(it1 + 4); + + std::vector::iterator it = it1 + 8; + + std::vector infoData(it, it + infoDataSize); + std::vector listData(it + infoDataSize, it + infoDataSize + listDataSize); + + ByteStreamBuffer buffer(const_cast(infoData.data()), infoData.size()); + ControlInfoMap map = cs->deserialize(buffer); + /* It's fine if map is empty. */ + if (buffer.overflow()) { + LOG(IPADataSerializer, Error) + << "Failed to deserialize ControlLists's ControlInfoMap: buffer overflow"; + return ControlList(); + } + + buffer = ByteStreamBuffer(const_cast(listData.data()), listData.size()); + ControlList list = cs->deserialize(buffer); + if (buffer.overflow()) + LOG(IPADataSerializer, Error) << "Failed to deserialize ControlList: buffer overflow"; + if (list.empty()) + LOG(IPADataSerializer, Error) << "Failed to deserialize ControlList: empty list"; + + return list; + } + + static ControlList deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), cs); + } + + static ControlList deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data_it1, data_it2, cs); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const ControlInfoMap &map, ControlSerializer *cs) + { + if (!cs) + LOG(IPADataSerializer, Fatal) + << "ControlSerializer not provided for serialization of ControlInfoMap"; + + size_t size = cs->binarySize(map); + std::vector infoData(size); + ByteStreamBuffer buffer(infoData.data(), infoData.size()); + int ret = cs->serialize(map, buffer); + + if (ret < 0 || buffer.overflow()) + return {{}, {}}; + + std::vector data_vec; + appendUInt(data_vec, infoData.size()); + data_vec.insert(data_vec.end(), infoData.begin(), infoData.end()); + + return {data_vec, {}}; + } + + static const ControlInfoMap deserialize(std::vector &data, + ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), cs); + } + + static const ControlInfoMap deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + ControlSerializer *cs) + { + if (!cs) + LOG(IPADataSerializer, Fatal) + << "ControlSerializer not provided for deserialization of ControlInfoMap"; + + uint32_t infoDataSize = readUInt(it1); + + std::vector::iterator it = it1 + 4; + + std::vector infoData(it, it + infoDataSize); + + ByteStreamBuffer buffer(const_cast(infoData.data()), infoData.size()); + const ControlInfoMap map = cs->deserialize(buffer); + + /* + ControlInfoMap::Map ctrls; + for (auto pair : map) + ctrls.emplace(controls::controls.at(pair.first->id()), + pair.second); + + ControlInfoMap ret = std::move(ctrls); + return ret; + */ + + return map; + } + + static const ControlInfoMap deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), cs); + } + + static const ControlInfoMap deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + ControlSerializer *cs) + { + return IPADataSerializer::deserialize(data_it1, data_it2, cs); + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize([[maybe_unused]] const ControlInfoMap &map, [[maybe_unused]] ControlSerializer *cs) + { + const ControlInfoMap &map_const = const_cast(map); + + std::vector data_vec; + std::tie(data_vec, std::ignore) = + IPADataSerializer::serialize(map_const, cs); + + return {data_vec, {}}; + } + + static ControlInfoMap deserialize(std::vector &data, + ControlSerializer *cs) + { + ControlInfoMap ret; + const ControlInfoMap &map = ret; + const_cast(map) = + IPADataSerializer::deserialize(data, cs); + + return ret; + } + + static ControlInfoMap deserialize(std::vector::iterator it1, + [[maybe_unused]] std::vector::iterator it2, + ControlSerializer *cs) + { + ControlInfoMap ret; + const ControlInfoMap &map = ret; + const_cast(map) = + IPADataSerializer::deserialize(it1, it2, cs); + + return ret; + } + + static ControlInfoMap deserialize(std::vector &data, + [[maybe_unused]] std::vector &fds, + ControlSerializer *cs) + { + ControlInfoMap ret; + const ControlInfoMap &map = ret; + const_cast(map) = + IPADataSerializer::deserialize(data, fds, cs); + + return ret; + } + + static ControlInfoMap deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + [[maybe_unused]] std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + ControlSerializer *cs) + { + ControlInfoMap ret; + const ControlInfoMap &map = ret; + const_cast(map) = + IPADataSerializer::deserialize(data_it1, data_it2, + fds_it1, fds_it2, cs); + + return ret; + } +}; + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const FrameBuffer::Plane &data, [[maybe_unused]] ControlSerializer *cs = nullptr) + { + std::vector data_vec; + std::vector fds_vec; + + // fd + std::vector fdBuf; + std::vector fdFds; + std::tie(fdBuf, fdFds) = + IPADataSerializer::serialize(data.fd); + data_vec.insert(data_vec.end(), fdBuf.begin(), fdBuf.end()); + fds_vec.insert(fds_vec.end(), fdFds.begin(), fdFds.end()); + + // length + appendUInt(data_vec, data.length); + + return {data_vec, fds_vec}; + } + + static FrameBuffer::Plane deserialize(std::vector &data, + std::vector &fds, + ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), + fds.begin(), fds.end(), + cs); + } + + static FrameBuffer::Plane deserialize(std::vector::iterator data_it1, + [[maybe_unused]] std::vector::iterator data_it2, + std::vector::iterator fds_it1, + [[maybe_unused]] std::vector::iterator fds_it2, + [[maybe_unused]] ControlSerializer *cs = nullptr) + { + FrameBuffer::Plane ret; + + ret.fd = IPADataSerializer::deserialize(data_it1, data_it1 + 1, + fds_it1, fds_it1 + 1); + ret.length = readUInt(data_it1 + 1); + + return ret; + } +}; + + +template<> +class IPADataSerializer +{ +public: + static std::tuple, std::vector> + serialize(const IPABuffer &data, ControlSerializer *cs = nullptr) + { + std::vector data_vec; + + appendUInt(data_vec, data.id); + + std::vector planes_data_vec; + std::vector planes_fds_vec; + std::tie(planes_data_vec, planes_fds_vec) = + IPADataSerializer>::serialize(data.planes, cs); + + data_vec.insert(data_vec.end(), planes_data_vec.begin(), planes_data_vec.end()); + + return {data_vec, planes_fds_vec}; + } + + static IPABuffer deserialize(std::vector &data, + std::vector &fds, + ControlSerializer *cs = nullptr) + { + return IPADataSerializer::deserialize(data.begin(), data.end(), + fds.begin(), fds.end(), cs); + } + + static IPABuffer deserialize(std::vector::iterator data_it1, + std::vector::iterator data_it2, + std::vector::iterator fds_it1, + std::vector::iterator fds_it2, + ControlSerializer *cs = nullptr) + { + IPABuffer ret; + + ret.id = readUInt(data_it1); + + ret.planes = + IPADataSerializer>::deserialize( + data_it1 + 4, data_it2, fds_it1, fds_it2, cs); + + return ret; + } +}; + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_H__ */ diff --git a/src/libcamera/ipa_data_serializer.cpp b/src/libcamera/ipa_data_serializer.cpp new file mode 100644 index 00000000..86332abc --- /dev/null +++ b/src/libcamera/ipa_data_serializer.cpp @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_data_serializer.cpp - Image Processing Algorithm data serializer + */ + +#include "libcamera/internal/ipa_data_serializer.h" + +#include "libcamera/internal/log.h" + +/** + * \file ipa_ipa_data_serializer.h + * \brief IPA Data Serializer + */ + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPADataSerializer) + +/** + * \class IPADataSerializer + * \brief IPA Data Serializer + * + */ + +// TODO the rest of the documentation + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index af2f3d95..91710520 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -23,6 +23,7 @@ libcamera_sources = files([ 'geometry.cpp', 'ipa_context_wrapper.cpp', 'ipa_controls.cpp', + 'ipa_data_serializer.cpp', 'ipa_interface.cpp', 'ipa_manager.cpp', 'ipa_module.cpp', From patchwork Tue Sep 15 14:20:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9617 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 40B78BF01C for ; Tue, 15 Sep 2020 14:21:12 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0E48362E30; Tue, 15 Sep 2020 16:21:12 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="SduAagn7"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id B63C462E39 for ; Tue, 15 Sep 2020 16:21:09 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 04FD2275; Tue, 15 Sep 2020 16:21:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179669; bh=Z/AHfzgfocuemZe2HFL9IogoH3u+3OGB2vD8GyYcFZo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SduAagn7+sgA+aaYPQrvlPdcvgPa7PT/0J/WEP/ZUT9A9Uc595U3iztwfW3/oYVdP uA/SGXPCNpvOockZpaHqB41UPBveB/2xRuydgMGPkcJ8pGghyhSkLyZijRn9kvgOpw 1ysaKvB35Kk6UHUz0WJB3dFvnQpBCVzezNepE7aQ= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:22 +0900 Message-Id: <20200915142038.28757-8-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 07/23] libcamera: Add IPAIPC X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Create a virtual IPAIPC class that models an IPC/RPC system. IPA proxies and proxy workers will call into the IPAIPC, rather than implementing the IPC themselves. Signed-off-by: Paul Elder --- include/libcamera/internal/ipa_ipc.h | 41 ++++++++++++++++++++++++++++ src/libcamera/ipa_ipc.cpp | 36 ++++++++++++++++++++++++ src/libcamera/meson.build | 1 + 3 files changed, 78 insertions(+) create mode 100644 include/libcamera/internal/ipa_ipc.h create mode 100644 src/libcamera/ipa_ipc.cpp diff --git a/include/libcamera/internal/ipa_ipc.h b/include/libcamera/internal/ipa_ipc.h new file mode 100644 index 00000000..34ca6eda --- /dev/null +++ b/include/libcamera/internal/ipa_ipc.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_ipc.h - Image Processing Algorithm IPC module for IPA proxies + */ +#ifndef __LIBCAMERA_INTERNAL_IPA_IPC_H__ +#define __LIBCAMERA_INTERNAL_IPA_IPC_H__ + +#include + +namespace libcamera { + +class IPAIPC +{ +public: + IPAIPC([[maybe_unused]] const char *ipa_module_path, + [[maybe_unused]] const char *ipa_proxy_worker_path); + virtual ~IPAIPC(); + + bool isValid() const { return valid_; } + + virtual int sendSync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in, + std::vector *data_out = nullptr, + std::vector *fds_out = nullptr) = 0; + + virtual int sendAsync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in) = 0; + + Signal &, std::vector &> recvIPC; + +protected: + bool valid_; +}; + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_INTERNAL_IPA_IPC_H__ */ diff --git a/src/libcamera/ipa_ipc.cpp b/src/libcamera/ipa_ipc.cpp new file mode 100644 index 00000000..703ec0a0 --- /dev/null +++ b/src/libcamera/ipa_ipc.cpp @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_ipc.cpp - Image Processing Algorithm IPC module for IPA proxies + */ + +#include + +#include "libcamera/internal/ipc_unixsocket.h" +#include "libcamera/internal/log.h" +#include "libcamera/internal/process.h" +#include "libcamera/internal/thread.h" + +#include +#include + +#include "libcamera/internal/ipa_ipc.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPAIPC) + +IPAIPC::IPAIPC([[maybe_unused]] const char *ipa_module_path, + [[maybe_unused]] const char *ipa_proxy_worker_path) + : valid_(false) +{ +} + +IPAIPC::~IPAIPC() +{ +} + +// TODO documentation, obviously + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 91710520..2e7c3b4d 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -24,6 +24,7 @@ libcamera_sources = files([ 'ipa_context_wrapper.cpp', 'ipa_controls.cpp', 'ipa_data_serializer.cpp', + 'ipa_ipc.cpp', 'ipa_interface.cpp', 'ipa_manager.cpp', 'ipa_module.cpp', From patchwork Tue Sep 15 14:20:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9618 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8CC43BF01C for ; Tue, 15 Sep 2020 14:21:13 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 59EB362E3B; Tue, 15 Sep 2020 16:21:13 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="bDUOKSLV"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 21D0262E1B for ; Tue, 15 Sep 2020 16:21:12 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 428D414DF; Tue, 15 Sep 2020 16:21:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179671; bh=TM5rZYgsDjQjRq5VJaUpNdlT8yWE1pDRGcP8KHx/9WI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bDUOKSLVUBT4REhXXKkcxx/220JIv2hBLjlrwmWF4CcRldkrf3jvEIV+4prEKH1Lq nDK4GPu3JysBqKGFXZaJ7RXsg8YVvRKp53j6NZOrvYdzHbf55cZPjbLIGJgvojBATY 2lkynvPJHMJc72Wd/Vp05jx5sAz56U6jAock48bA= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:23 +0900 Message-Id: <20200915142038.28757-9-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 08/23] libcamera: Add IPAIPC implementation based on unix socket X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add an implementation of IPAIPC using unix socket. Signed-off-by: Paul Elder --- .../libcamera/internal/ipa_ipc_unixsocket.h | 115 ++++++++++++ src/libcamera/ipa_ipc_unixsocket.cpp | 175 ++++++++++++++++++ src/libcamera/meson.build | 1 + 3 files changed, 291 insertions(+) create mode 100644 include/libcamera/internal/ipa_ipc_unixsocket.h create mode 100644 src/libcamera/ipa_ipc_unixsocket.cpp diff --git a/include/libcamera/internal/ipa_ipc_unixsocket.h b/include/libcamera/internal/ipa_ipc_unixsocket.h new file mode 100644 index 00000000..de8af35b --- /dev/null +++ b/include/libcamera/internal/ipa_ipc_unixsocket.h @@ -0,0 +1,115 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_ipc_unixsocket.h - Image Processing Algorithm IPC module using unix socket + */ +#ifndef __LIBCAMERA_INTERNAL_IPA_IPC_UNIXSOCKET_H__ +#define __LIBCAMERA_INTERNAL_IPA_IPC_UNIXSOCKET_H__ + +#include + +#include + +#include "libcamera/internal/ipa_ipc.h" +#include "libcamera/internal/ipa_module.h" +#include "libcamera/internal/ipc_unixsocket.h" + +namespace libcamera { + +class Process; + +inline void writeHeader(IPCUnixSocket::Payload &payload, uint32_t cmd, uint32_t seq) +{ + uint8_t cmd_arr[] = {static_cast(cmd & 0xff), + static_cast((cmd >> 8 ) & 0xff), + static_cast((cmd >> 16) & 0xff), + static_cast((cmd >> 24) & 0xff)}; + uint8_t seq_arr[] = {static_cast(seq & 0xff), + static_cast((seq >> 8 ) & 0xff), + static_cast((seq >> 16) & 0xff), + static_cast((seq >> 24) & 0xff)}; + payload.data.insert(payload.data.begin(), cmd_arr, cmd_arr+4); + payload.data.insert(payload.data.begin() + 4, seq_arr, seq_arr+4); +} + +inline std::tuple readHeader(IPCUnixSocket::Payload &payload) +{ + uint32_t cmd = (payload.data[0] & 0xff) | + ((payload.data[1] & 0xff) << 8) | + ((payload.data[2] & 0xff) << 16) | + ((payload.data[3] & 0xff) << 24); + uint32_t seq = (payload.data[4] & 0xff) | + ((payload.data[5] & 0xff) << 8) | + ((payload.data[6] & 0xff) << 16) | + ((payload.data[7] & 0xff) << 24); + + return {cmd, seq}; +} + +inline void eraseHeader(IPCUnixSocket::Payload &payload) +{ + payload.data.erase(payload.data.begin(), payload.data.begin() + 8); +} + +inline void writeUInt32(IPCUnixSocket::Payload &payload, uint32_t val, uint32_t pos) +{ + if (pos + 4 > payload.data.size()) + return; + + uint8_t arr[] = {static_cast(val & 0xff), + static_cast(val & (0xff << 8)), + static_cast(val & (0xff << 16)), + static_cast(val & (0xff << 24))}; + std::copy(arr, arr + 4, payload.data.begin() + pos); +} + +inline uint32_t readUInt32(IPCUnixSocket::Payload &payload, uint32_t pos) +{ + if (pos + 4 > payload.data.size()) + return 0; + + return payload.data[pos] & (payload.data[pos + 1] << 8) & + (payload.data[pos + 2] << 16) & (payload.data[pos + 3] << 24); +} + + +class IPAIPCUnixSocket : public IPAIPC +{ +public: + IPAIPCUnixSocket(const char *ipa_module_path, const char *ipa_proxy_worker_path); + ~IPAIPCUnixSocket(); + + bool isValid() const { return valid_; } + + int sendSync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in, + std::vector *data_out = nullptr, + std::vector *fds_out = nullptr) override; + + int sendAsync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in) override; + +private: + struct CallData { + IPCUnixSocket::Payload *response; + bool done; + }; + + void readyRead(IPCUnixSocket *socket); + int call(const IPCUnixSocket::Payload &message, IPCUnixSocket::Payload *response, uint32_t seq); + + uint32_t seq_; + + Process *proc_; + + IPCUnixSocket *socket_; + + std::map callData_; +}; + +} /* namespace libcamera */ + +#endif /* __LIBCAMERA_INTERNAL_IPA_IPC_UNIXSOCKET_H__ */ diff --git a/src/libcamera/ipa_ipc_unixsocket.cpp b/src/libcamera/ipa_ipc_unixsocket.cpp new file mode 100644 index 00000000..c1997e96 --- /dev/null +++ b/src/libcamera/ipa_ipc_unixsocket.cpp @@ -0,0 +1,175 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipa_ipc_unixsocket.cpp - Image Processing Algorithm IPC module using unix socket + */ + +#include + +#include "libcamera/internal/ipc_unixsocket.h" +#include "libcamera/internal/log.h" +#include "libcamera/internal/process.h" +#include "libcamera/internal/thread.h" + +#include +#include + +#include "libcamera/internal/ipa_ipc.h" +#include "libcamera/internal/ipa_ipc_unixsocket.h" + +namespace libcamera { + +LOG_DECLARE_CATEGORY(IPAIPC) + +IPAIPCUnixSocket::IPAIPCUnixSocket(const char *ipa_module_path, + const char *ipa_proxy_worker_path) + : IPAIPC(ipa_module_path, ipa_proxy_worker_path), seq_(0), + proc_(nullptr), socket_(nullptr) +{ + std::vector fds; + std::vector args; + args.push_back(ipa_module_path); + + socket_ = new IPCUnixSocket(); + int fd = socket_->create(); + if (fd < 0) { + LOG(IPAIPC, Error) + << "Failed to create socket"; + return; + } + socket_->readyRead.connect(this, &IPAIPCUnixSocket::readyRead); + args.push_back(std::to_string(fd)); + fds.push_back(fd); + + proc_ = new Process(); + int ret = proc_->start(ipa_proxy_worker_path, args, fds); + if (ret) { + LOG(IPAIPC, Error) + << "Failed to start proxy worker process"; + return; + } + + valid_ = true; +} + +IPAIPCUnixSocket::~IPAIPCUnixSocket() +{ + delete proc_; + delete socket_; +} + +int IPAIPCUnixSocket::sendSync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in, + std::vector *data_out, + std::vector *fds_out) +{ + IPCUnixSocket::Payload message, response; + int ret; + + /* It's fine if seq_ overflows; that'll just be the new epoch. */ + seq_++; + writeHeader(message, cmd, seq_); + message.data.insert(message.data.end(), data_in.begin(), data_in.end()); + + message.fds = const_cast &>(fds_in); + + ret = call(message, &response, seq_); + if (ret) { + LOG(IPAIPC, Error) << "Failed to call sync"; + callData_.erase(seq_); + return ret; + } + + if (data_out) + data_out->insert(data_out->end(), response.data.begin(), response.data.end()); + + if (fds_out) + fds_out->insert(fds_out->end(), response.fds.begin(), response.fds.end()); + + return 0; +} + +int IPAIPCUnixSocket::sendAsync(uint32_t cmd, + const std::vector &data_in, + const std::vector &fds_in) +{ + IPCUnixSocket::Payload message; + int ret; + + writeHeader(message, cmd, 0); + message.data.insert(message.data.end(), data_in.begin(), data_in.end()); + + message.fds = const_cast &>(fds_in); + + ret = socket_->send(message); + if (ret) { + LOG(IPAIPC, Error) << "Failed to call async"; + return ret; + } + + return 0; +} + +void IPAIPCUnixSocket::readyRead(IPCUnixSocket *socket) +{ + IPCUnixSocket::Payload message; + int ret = socket->receive(&message); + if (ret) { + LOG(IPAIPC, Error) << "Receive message failed" << ret; + return; + } + + uint32_t cmd, seq; + std::tie(cmd, seq) = readHeader(message); + + auto callData = callData_.find(seq); + if (callData != callData_.end()) { + eraseHeader(message); + /* Is there any way to avoid this copy? */ + *callData->second.response = message; + callData->second.done = true; + return; + } + + /* + * Received unexpected data, this means it's a call from the IPA. + * We can't return anything to the IPA (gotta keep them under *our* + * control, plus returning would require blocking the caller, and we + * can't afford to do that). Let the proxy do switch-case on cmd. + */ + recvIPC.emit(message.data, message.fds); + + return; +} + +int IPAIPCUnixSocket::call(const IPCUnixSocket::Payload &message, IPCUnixSocket::Payload *response, uint32_t seq) +{ + Timer timeout; + int ret; + + callData_[seq].response = response; + callData_[seq].done = false; + + ret = socket_->send(message); + if (ret) + return ret; + + timeout.start(200); + while (!callData_[seq].done) { + if (!timeout.isRunning()) { + LOG(IPAIPC, Error) << "Call timeout!"; + callData_.erase(seq); + return -ETIMEDOUT; + } + + Thread::current()->eventDispatcher()->processEvents(); + } + + callData_.erase(seq); + + return 0; +} + +} /* namespace libcamera */ diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 2e7c3b4d..9d1abeb0 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -25,6 +25,7 @@ libcamera_sources = files([ 'ipa_controls.cpp', 'ipa_data_serializer.cpp', 'ipa_ipc.cpp', + 'ipa_ipc_unixsocket.cpp', 'ipa_interface.cpp', 'ipa_manager.cpp', 'ipa_module.cpp', From patchwork Tue Sep 15 14:20:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9619 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 1C936BF01C for ; Tue, 15 Sep 2020 14:21:15 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id DA9FF62E35; Tue, 15 Sep 2020 16:21:14 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="lFNYmQXb"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6532C62E1B for ; Tue, 15 Sep 2020 16:21:14 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 997D41620; Tue, 15 Sep 2020 16:21:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179674; bh=d4epbIXHUE2oG37YlrxTSoC09mDXG3JTGpY6qLn0Ugk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lFNYmQXb6AU8q7euZAg0RYyADVx/m/+zl58RbUc+JcTZp4rBMfUmKhX/aNJpmN77U HeP4MiWW7HKySyykWN1RVvC6zC3GLzPWEbuvUx/L67yaBOpYmn3rknmKUQpKuCMNpU nklBJk8CZ/m+jJ8qUva5Ldb9IZ6T/ckfiRLMFeQU= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:24 +0900 Message-Id: <20200915142038.28757-10-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 09/23] libcamera: IPAModule: Replace ipa_context with IPAInterface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With the new IPC infrastructure, we no longer need the C interface as provided by struct ipa_context. Make ipaCreate_() and createInterface() return IPAInterface. Signed-off-by: Paul Elder --- include/libcamera/internal/ipa_module.h | 4 ++-- src/libcamera/ipa_module.cpp | 16 +--------------- src/libcamera/meson.build | 1 - 3 files changed, 3 insertions(+), 18 deletions(-) diff --git a/include/libcamera/internal/ipa_module.h b/include/libcamera/internal/ipa_module.h index c2df2476..19fc5827 100644 --- a/include/libcamera/internal/ipa_module.h +++ b/include/libcamera/internal/ipa_module.h @@ -33,7 +33,7 @@ public: bool load(); - struct ipa_context *createContext(); + IPAInterface *createInterface(); bool match(PipelineHandler *pipe, uint32_t minVersion, uint32_t maxVersion) const; @@ -52,7 +52,7 @@ private: bool loaded_; void *dlHandle_; - typedef struct ipa_context *(*IPAIntfFactory)(); + typedef IPAInterface *(*IPAIntfFactory)(void); IPAIntfFactory ipaCreate_; }; diff --git a/src/libcamera/ipa_module.cpp b/src/libcamera/ipa_module.cpp index de512a7f..008325c4 100644 --- a/src/libcamera/ipa_module.cpp +++ b/src/libcamera/ipa_module.cpp @@ -438,21 +438,7 @@ bool IPAModule::load() return true; } -/** - * \brief Instantiate an IPA context - * - * After loading the IPA module with load(), this method creates an instance of - * the IPA module context. Ownership of the context is passed to the caller, and - * the context shall be destroyed by calling the \ref ipa_context_ops::destroy - * "ipa_context::ops::destroy()" function. - * - * Calling this function on a module that has not yet been loaded, or an - * invalid module (as returned by load() and isValid(), respectively) is - * an error. - * - * \return The IPA context on success, or nullptr on error - */ -struct ipa_context *IPAModule::createContext() +IPAInterface *IPAModule::createInterface() { if (!valid_ || !loaded_) return nullptr; diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build index 9d1abeb0..a2a6ed04 100644 --- a/src/libcamera/meson.build +++ b/src/libcamera/meson.build @@ -21,7 +21,6 @@ libcamera_sources = files([ 'formats.cpp', 'framebuffer_allocator.cpp', 'geometry.cpp', - 'ipa_context_wrapper.cpp', 'ipa_controls.cpp', 'ipa_data_serializer.cpp', 'ipa_ipc.cpp', From patchwork Tue Sep 15 14:20:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9620 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 6F36FBF01C for ; Tue, 15 Sep 2020 14:21:18 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 3D7EA62E30; Tue, 15 Sep 2020 16:21:18 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="Z0auLaYw"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 9DDA162E17 for ; Tue, 15 Sep 2020 16:21:16 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id D54DA10C6; Tue, 15 Sep 2020 16:21:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179676; bh=3Tmzp3GTPNTq8PBgF2z8zvV2ry6agt6dvbi+OPuAp20=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z0auLaYwbFsMakZg1CYs9IbWWJV/d0m2Etjti4/pCX4gZ+ba4zKAgP0f6c008h+2G PUObuSZ/Kr2T6SpfCcE7oIa9OU1nPaXumd0bhwtWUw0C3RxkEST0njgAFMOSeZ7mwm SbiJ7sND6T/F5J8O1T8HnhC/ywXPXxyo/e48T10k= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:25 +0900 Message-Id: <20200915142038.28757-11-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 10/23] libcamera: IPAProxy: Remove stop() override X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since stop() is part of the IPA interface, and the IPA interface is now generated based on the data definition file per pipeline, this no longer needs to be overrided by the base IPAProxy. Remove it. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/internal/ipa_proxy.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/libcamera/internal/ipa_proxy.h b/include/libcamera/internal/ipa_proxy.h index b429ce5a..aec8f04f 100644 --- a/include/libcamera/internal/ipa_proxy.h +++ b/include/libcamera/internal/ipa_proxy.h @@ -27,8 +27,6 @@ public: std::string configurationFile(const std::string &file) const; - void stop() override = 0; - protected: std::string resolvePath(const std::string &file) const; From patchwork Tue Sep 15 14:20:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9621 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id C7767BF01C for ; Tue, 15 Sep 2020 14:21:20 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 90F9662E17; Tue, 15 Sep 2020 16:21:20 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="WKg2pIk3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id C21B162E1B for ; Tue, 15 Sep 2020 16:21:18 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0A88FAEA; Tue, 15 Sep 2020 16:21:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179678; bh=ILA9qGSrVUo7ScoEwEt/hWzj4MgFRkfuDHEvjOeEnOo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WKg2pIk35pnnN0DOoILADMQ6mwFhtxyyCY6X+KS/H/u1xYTr6sERNgQwNhFl/CF8Y ye4NatSgMndU01QAT8M/NvW0VXhbo1MwuuPFW2xY8b4wuYuCOdFdFxEccwmtxSwaDI vFAP6cVGIIfuDkTHooNN1g6sElNkvZf9uQAInLSg= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:26 +0900 Message-Id: <20200915142038.28757-12-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 11/23] libcamera: IPAProxy: Add isolate parameter to create() X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since IPAProxy implementations now always encapsulate IPA modules, add a parameter to create() to signal if the proxy should isolate the IPA or not. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/internal/ipa_proxy.h | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/include/libcamera/internal/ipa_proxy.h b/include/libcamera/internal/ipa_proxy.h index aec8f04f..1903150e 100644 --- a/include/libcamera/internal/ipa_proxy.h +++ b/include/libcamera/internal/ipa_proxy.h @@ -42,7 +42,7 @@ public: IPAProxyFactory(const char *name); virtual ~IPAProxyFactory() {} - virtual std::unique_ptr create(IPAModule *ipam) = 0; + virtual std::unique_ptr create(IPAModule *ipam, bool isolate) = 0; const std::string &name() const { return name_; } @@ -53,16 +53,16 @@ private: std::string name_; }; -#define REGISTER_IPA_PROXY(proxy) \ -class proxy##Factory final : public IPAProxyFactory \ -{ \ -public: \ - proxy##Factory() : IPAProxyFactory(#proxy) {} \ - std::unique_ptr create(IPAModule *ipam) \ - { \ - return std::make_unique(ipam); \ - } \ -}; \ +#define REGISTER_IPA_PROXY(proxy) \ +class proxy##Factory final : public IPAProxyFactory \ +{ \ +public: \ + proxy##Factory() : IPAProxyFactory(#proxy) {} \ + std::unique_ptr create(IPAModule *ipam, bool isolate) \ + { \ + return std::make_unique(ipam, isolate); \ + } \ +}; \ static proxy##Factory global_##proxy##Factory; } /* namespace libcamera */ From patchwork Tue Sep 15 14:20:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9622 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 27282BF01C for ; Tue, 15 Sep 2020 14:21:22 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id E724A62E32; Tue, 15 Sep 2020 16:21:21 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="CYQVTGqo"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DCF6862E23 for ; Tue, 15 Sep 2020 16:21:20 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 2C97C14DF; Tue, 15 Sep 2020 16:21:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179680; bh=sHQZxUf464cUUxWvh5t0rO7Wh4wGbbpd0kdCkgnFrhk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CYQVTGqocdXXlGIr7ObEIwhwo5WYu5wV9ds0OoU0O+BXCykMnULuMI2KpXSchXXFK 0P0GXkpEPQ38NzlWhEJWj935zhsfW9/IRwz1/eKgpxxZXg6b/9BgNFbvtixiJY4liO jIWLQmnOn3EuQiA09pYm3skq9R0IEY0bNgFnQVz0= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:27 +0900 Message-Id: <20200915142038.28757-13-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 12/23] libcamera: PipelineHandler: Remove IPA from base class X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since pipline handlers now have their own IPA interface types, it can no longer be defined in the base class, and each pipline handler implementation must declare it and its type themselves. Remove it from the base class. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/internal/pipeline_handler.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/libcamera/internal/pipeline_handler.h b/include/libcamera/internal/pipeline_handler.h index a4e1b529..2018bef0 100644 --- a/include/libcamera/internal/pipeline_handler.h +++ b/include/libcamera/internal/pipeline_handler.h @@ -47,7 +47,6 @@ public: std::list queuedRequests_; ControlInfoMap controlInfo_; ControlList properties_; - std::unique_ptr ipa_; private: CameraData(const CameraData &) = delete; From patchwork Tue Sep 15 14:20:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9623 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 7D35FBF01C for ; Tue, 15 Sep 2020 14:21:24 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 47EE962E30; Tue, 15 Sep 2020 16:21:24 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="t/5rP/eI"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 2934B603EE for ; Tue, 15 Sep 2020 16:21:23 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 686CF1620; Tue, 15 Sep 2020 16:21:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179682; bh=wzGdd1JnYp6naLTSildYQuu4PfeQ/BwsU5DGLaEbhsI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t/5rP/eIz02d1jf9jL1rngkVdgVtt6+14174J2iAds+WyhNqUX3/uRgojp6xmXirD E+lVilxKF5rVZOxOMf9Mft/WormhH1JPYt81Vsz2VhX8DcFJQ3QCQRTUbmLXcVGucD dt1r/9hjMw/uMwcEHCvFY+BU5LnAL5wqYQhDaCZw= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:28 +0900 Message-Id: <20200915142038.28757-14-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 13/23] libcamera: IPAInterface: Remove all functions from IPAInterface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that all the functions in the IPA interface are defined in the data definition file and a specialized IPAInterface is generated per pipeline handler, remove all the functions from the case IPAInterface. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/ipa/ipa_interface.h | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/include/libcamera/ipa/ipa_interface.h b/include/libcamera/ipa/ipa_interface.h index 5016ec25..cbe325ea 100644 --- a/include/libcamera/ipa/ipa_interface.h +++ b/include/libcamera/ipa/ipa_interface.h @@ -151,22 +151,6 @@ class IPAInterface { public: virtual ~IPAInterface() {} - - virtual int init(const IPASettings &settings) = 0; - virtual int start() = 0; - virtual void stop() = 0; - - virtual void configure(const CameraSensorInfo &sensorInfo, - const std::map &streamConfig, - const std::map &entityControls, - const IPAOperationData &ipaConfig, - IPAOperationData *result) = 0; - - virtual void mapBuffers(const std::vector &buffers) = 0; - virtual void unmapBuffers(const std::vector &ids) = 0; - - virtual void processEvent(const IPAOperationData &data) = 0; - Signal queueFrameAction; }; } /* namespace libcamera */ From patchwork Tue Sep 15 14:20:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9624 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id D8BCBBF01C for ; Tue, 15 Sep 2020 14:21:26 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id A502062E23; Tue, 15 Sep 2020 16:21:26 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="s4fR8sox"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 6EF7862E17 for ; Tue, 15 Sep 2020 16:21:25 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A9076276; Tue, 15 Sep 2020 16:21:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179685; bh=OcghZ9l+sP+9kXxJlJJSLmso5fjqijVhZW9Op5VXJ4c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=s4fR8soxTBLUIF7hoGW2Nk6V8Dd6FhwezDKM6Ec64r/lqKfgWCXyf3P5QuZLwXga/ q3hLYNsNy2kz/kn1wjXpSrRrVtckh57Wm4+8NwATOKO94i1+GbZHd7ECQ4sigfkc8G FkjA7vV1KQEzsspNST735gZShl1jaXQxJZYe4uxM= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:29 +0900 Message-Id: <20200915142038.28757-15-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 14/23] libcamera: IPAInterface: make ipaCreate return IPAInterface X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" With the new IPC infrastructure, we no longer need the C interface as provided by struct ipa_context. Make ipaCreate return IPAinterface. Signed-off-by: Paul Elder --- include/libcamera/ipa/ipa_interface.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/include/libcamera/ipa/ipa_interface.h b/include/libcamera/ipa/ipa_interface.h index cbe325ea..6efe2e99 100644 --- a/include/libcamera/ipa/ipa_interface.h +++ b/include/libcamera/ipa/ipa_interface.h @@ -110,8 +110,6 @@ struct ipa_context_ops { const struct ipa_operation_data *data); }; -struct ipa_context *ipaCreate(); - #ifdef __cplusplus } @@ -154,6 +152,11 @@ public: }; } /* namespace libcamera */ + +extern "C" { +libcamera::IPAInterface *ipaCreate(); +} + #endif #endif /* __LIBCAMERA_IPA_INTERFACE_H__ */ From patchwork Tue Sep 15 14:20:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9625 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 3E7D1BF01C for ; Tue, 15 Sep 2020 14:21:29 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 0A24862E3C; Tue, 15 Sep 2020 16:21:29 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ZnVms5a3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id DB5EB62E17 for ; Tue, 15 Sep 2020 16:21:27 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E5B3314DF; Tue, 15 Sep 2020 16:21:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179687; bh=mENnW9UmAMtZuYE04IPh/wzybnVDFzxLOipDE9iDS6o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZnVms5a3yhzPXdjGBIhijX08tVogNTYsa9KSLeGXmXjv/uHKcpMW1L9liFXb5lZLP hPL6fFBehiMkVSaE+/ciuwr9cqCCow/JQgS4c/m9B81BNxFvGuhaFFUlfXOxDVxiLO dH4S92wPf1hdp4nOEf7ALrcr7Veo9Tg1g9muSGVA= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:30 +0900 Message-Id: <20200915142038.28757-16-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 15/23] libcamera: IPAManager: Fetch IPAProxy corresponding to pipeline X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that each pipeline handler has its own IPAProxy implementation, make the IPAManager fetch the IPAProxy based on the pipeline handler name. Also, since the IPAProxy is used regardless of isolation or no isolation, remove the isolation check from the proxy selection. Signed-off-by: Paul Elder --- src/libcamera/ipa_manager.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libcamera/ipa_manager.cpp b/src/libcamera/ipa_manager.cpp index 046fd5c6..2d0ea242 100644 --- a/src/libcamera/ipa_manager.cpp +++ b/src/libcamera/ipa_manager.cpp @@ -275,8 +275,8 @@ std::unique_ptr IPAManager::createIPA(PipelineHandler *pipe, * * \todo Implement a better proxy selection */ - const char *proxyName = self_->isSignatureValid(m) - ? "IPAProxyThread" : "IPAProxyLinux"; + std::string pipeName(pipe->name()); + const char *proxyName = pipeName.replace(0, 15, "IPAProxy").c_str(); IPAProxyFactory *pf = nullptr; for (IPAProxyFactory *factory : IPAProxyFactory::factories()) { From patchwork Tue Sep 15 14:20:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9626 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 91B06BF01C for ; Tue, 15 Sep 2020 14:21:31 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 5DB6E62E27; Tue, 15 Sep 2020 16:21:31 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UjGqCt7I"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 36DEB62E17 for ; Tue, 15 Sep 2020 16:21:30 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6195C1620; Tue, 15 Sep 2020 16:21:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179690; bh=QjT7FPHIivPKBBoHciyl3yu0BFsUF5GXIgvLXLy20Ng=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UjGqCt7ImWtv0nJno40x1U+x88yOfauBrB8WoEsiQ4ic5MLLF6Rwkcc6Aux1zBAP9 jsJZ87qT4SVI1cw4kgW8VRgYaEaGWE+t1VqFxPIKAIND6AkXX0oy8ktdtdm+U/xerZ qsaBdybYfmSw72E6taEr3gFi0EBKQq8iCoqQ8lSo= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:31 +0900 Message-Id: <20200915142038.28757-17-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 16/23] libcamera: IPAManager: add isolation flag to proxy creation X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" When the IPA proxy is created, it needs to know whether to isolate or not. Feed the flag at creation of the IPA proxy. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- src/libcamera/ipa_manager.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/libcamera/ipa_manager.cpp b/src/libcamera/ipa_manager.cpp index 2d0ea242..26458153 100644 --- a/src/libcamera/ipa_manager.cpp +++ b/src/libcamera/ipa_manager.cpp @@ -291,7 +291,8 @@ std::unique_ptr IPAManager::createIPA(PipelineHandler *pipe, return nullptr; } - std::unique_ptr proxy = pf->create(m); + std::unique_ptr proxy = + pf->create(m, !self_->isSignatureValid(m)); if (!proxy->isValid()) { LOG(IPAManager, Error) << "Failed to load proxy"; return nullptr; From patchwork Tue Sep 15 14:20:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9627 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 06CB3BF01C for ; Tue, 15 Sep 2020 14:21:36 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C59BA62E23; Tue, 15 Sep 2020 16:21:35 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="ndJeS/Bz"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id E4B8B62E17 for ; Tue, 15 Sep 2020 16:21:33 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B7F34275; Tue, 15 Sep 2020 16:21:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179692; bh=ntq4+1Mu3zSwO2v8sDzUUhj9oQrrbn70H0T2eCC18Bs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ndJeS/Bz61iMNEI5NFXuZTJFvFDbD0M1Fjze+Y0gIxRUNpJ4J3P6ngiKArVVbKbaY WUWhDE6m/ILdHsI1MQo24S66F78+9nviDC6zOqfffYn28kzYIg30rf38h0fvgJedbk 3jyKickMc+3BXgOmiMYjTloaDo96InLl8fozenVk= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:32 +0900 Message-Id: <20200915142038.28757-18-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 17/23] ipa: raspberrypi: Add mojom data definition file X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Add a mojom data definition for raspberrypi pipeline handler's IPAs. This is a direct translation of what the raspberrypi pipeline handler and IPA was using before with IPAOperationData. Also move the enums from raspberrypi.h to raspberrypi.mojom Signed-off-by: Paul Elder --- include/libcamera/ipa/meson.build | 4 +- include/libcamera/ipa/raspberrypi.h | 18 ---- include/libcamera/ipa/raspberrypi.mojom | 110 ++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 19 deletions(-) create mode 100644 include/libcamera/ipa/raspberrypi.mojom diff --git a/include/libcamera/ipa/meson.build b/include/libcamera/ipa/meson.build index f61a5a8f..c80594d7 100644 --- a/include/libcamera/ipa/meson.build +++ b/include/libcamera/ipa/meson.build @@ -13,7 +13,9 @@ install_headers(libcamera_ipa_headers, # Prepare IPA/IPC generation components # -ipa_mojom_files = [] +ipa_mojom_files = [ + 'raspberrypi.mojom', +] mojom_generator = find_program('../../../utils/ipc/generate.py') diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index ca62990e..ed2b12d5 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -10,24 +10,6 @@ #include #include -enum RPiConfigParameters { - RPI_IPA_CONFIG_LS_TABLE = (1 << 0), - RPI_IPA_CONFIG_STAGGERED_WRITE = (1 << 1), - RPI_IPA_CONFIG_SENSOR = (1 << 2), -}; - -enum RPiOperations { - RPI_IPA_ACTION_V4L2_SET_STAGGERED = 1, - RPI_IPA_ACTION_V4L2_SET_ISP, - RPI_IPA_ACTION_STATS_METADATA_COMPLETE, - RPI_IPA_ACTION_RUN_ISP, - RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME, - RPI_IPA_ACTION_EMBEDDED_COMPLETE, - RPI_IPA_EVENT_SIGNAL_STAT_READY, - RPI_IPA_EVENT_SIGNAL_ISP_PREPARE, - RPI_IPA_EVENT_QUEUE_REQUEST, -}; - enum RPiIpaMask { ID = 0x0ffff, STATS = 0x10000, diff --git a/include/libcamera/ipa/raspberrypi.mojom b/include/libcamera/ipa/raspberrypi.mojom new file mode 100644 index 00000000..6e1ce8c0 --- /dev/null +++ b/include/libcamera/ipa/raspberrypi.mojom @@ -0,0 +1,110 @@ +// these should probably should be in a mojo header +struct CameraSensorInfo {}; +struct ControlInfoMap {}; +struct ControlList {}; +struct FileDescriptor {}; +// for libcamera types, hasFd attr is needed to notify compiler that struct has fd +[hasFd] struct IPABuffer {}; +struct IPASettings {}; +struct IPAStream {}; + +// automatically add const and & to all inputs (that are not primitives) +// automatically add pointer to all outputs (that are not primitives) +// if return value is single and primitive, then return as ret value +// if return value is multiple or non-primitive, then return as output param +// const not allowed in array/map due to mojo parser +// default sync +interface IPARPiInterface { + init(IPASettings settings) => (int32 ret); + start() => (int32 ret); + stop(); + + configure(CameraSensorInfo sensorInfo, + map streamConfig, + map entityControls, + RPiConfigureParams ipaConfig) + => (RPiConfigureParams results); + + // arrays get turned into vectors + mapBuffers(array buffers); + unmapBuffers(array ids); + + [async] processEvent(RPiEventParams data); +}; + +// these shall not return anything +// only async allowed (sync not allowed) +interface IPARPiCallbackInterface { + queueFrameAction(uint32 frame, RPiActionParams action); +}; + +enum RPiConfigParameters { + RPI_IPA_CONFIG_LS_TABLE, + RPI_IPA_CONFIG_STAGGERED_WRITE, + RPI_IPA_CONFIG_SENSOR, + RPI_IPA_CONFIG_SEND_FD, +}; + +enum RPiEvents { + RPI_IPA_EVENT_SIGNAL_STAT_READY, + RPI_IPA_EVENT_SIGNAL_ISP_PREPARE, + RPI_IPA_EVENT_QUEUE_REQUEST, + RPI_IPA_EVENT_SEND_FD, +}; + +enum RPiActions { + RPI_IPA_ACTION_V4L2_SET_STAGGERED, + RPI_IPA_ACTION_V4L2_SET_ISP, + RPI_IPA_ACTION_STATS_METADATA_COMPLETE, + RPI_IPA_ACTION_RUN_ISP, + RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME, + RPI_IPA_ACTION_EMBEDDED_COMPLETE, +}; + +// Custom Data containers + +struct RPiStaggeredWritePayload { + uint32 gainDelay; + uint32 exposureDelay; + uint32 sensorMetadata; +}; + +struct RPiIspPreparePayload { + uint32 embeddedbufferId; + uint32 bayerbufferId; +}; + +struct RPiStatsCompletePayload { + uint32 bufferId; + ControlList controls; +}; + +struct RPiConfigurePayload { + RPiConfigParameters op; + FileDescriptor lsTableHandle; + int32 lsTableHandleStatic = -1; + RPiStaggeredWritePayload staggeredWriteResult; + ControlList controls; + int32 bufferFd; +}; + +// First level payload + +struct RPiConfigureParams { + array payload; +}; + +struct RPiEventParams { + RPiEvents ev; + uint32 bufferId; + RPiIspPreparePayload ispPrepare; + ControlList controls; + int32 bufferFd; +}; + +struct RPiActionParams { + RPiActions op; + uint32 bufferId; + RPiStatsCompletePayload statsComplete; + ControlList controls; +}; From patchwork Tue Sep 15 14:20:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9628 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 8BB5DBF01C for ; Tue, 15 Sep 2020 14:21:37 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 591EC62E3F; Tue, 15 Sep 2020 16:21:37 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="nbwpCeOv"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 320EA62E3E for ; Tue, 15 Sep 2020 16:21:36 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 6B4C7AEA; Tue, 15 Sep 2020 16:21:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179695; bh=uMPfma4yXNr9kvE46CXXDdJ4jptUJhDG6s2I8cuDxjg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nbwpCeOvzduLH8pqsKn7mZ7XGg3NndRj+/1m3ZlxgLsiPrXKnfsH22D50bzSchOtF HI1svTR0Ib06qFsMHkn8IGqQh+8ljIkJXVx3peg4o3QSvZSnGlczPmWQV9wVuG2ceI k5Y5EzB1bJVLG2wxb9Vts0aY0S83HLjbEdh1XmUk= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:33 +0900 Message-Id: <20200915142038.28757-19-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 18/23] libcamera: pipeline, ipa: raspberrypi: Use new data definition X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Now that we can generate custom functions and data structures with mojo, switch the raspberrypi pipeline handler and IPA to use the custom data structures as defined in the mojom data definition file. Signed-off-by: Paul Elder --- include/libcamera/ipa/raspberrypi.h | 37 +++--- src/ipa/raspberrypi/raspberrypi.cpp | 108 +++++++++------- .../pipeline/raspberrypi/raspberrypi.cpp | 120 +++++++++++------- 3 files changed, 155 insertions(+), 110 deletions(-) diff --git a/include/libcamera/ipa/raspberrypi.h b/include/libcamera/ipa/raspberrypi.h index ed2b12d5..e58d1cbc 100644 --- a/include/libcamera/ipa/raspberrypi.h +++ b/include/libcamera/ipa/raspberrypi.h @@ -23,22 +23,27 @@ enum RPiIpaMask { namespace libcamera { /* List of controls handled by the Raspberry Pi IPA */ -static const ControlInfoMap RPiControls = { - { &controls::AeEnable, ControlInfo(false, true) }, - { &controls::ExposureTime, ControlInfo(0, 999999) }, - { &controls::AnalogueGain, ControlInfo(1.0f, 32.0f) }, - { &controls::AeMeteringMode, ControlInfo(0, static_cast(controls::MeteringModeMax)) }, - { &controls::AeConstraintMode, ControlInfo(0, static_cast(controls::ConstraintModeMax)) }, - { &controls::AeExposureMode, ControlInfo(0, static_cast(controls::ExposureModeMax)) }, - { &controls::ExposureValue, ControlInfo(0.0f, 16.0f) }, - { &controls::AwbEnable, ControlInfo(false, true) }, - { &controls::ColourGains, ControlInfo(0.0f, 32.0f) }, - { &controls::AwbMode, ControlInfo(0, static_cast(controls::AwbModeMax)) }, - { &controls::Brightness, ControlInfo(-1.0f, 1.0f) }, - { &controls::Contrast, ControlInfo(0.0f, 32.0f) }, - { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, - { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, - { &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) }, +static ControlInfoMap RPiControls; + +inline void initializeRPiControls() +{ + RPiControls = { + { &controls::AeEnable, ControlInfo(false, true) }, + { &controls::ExposureTime, ControlInfo(0, 999999) }, + { &controls::AnalogueGain, ControlInfo(1.0f, 32.0f) }, + { &controls::AeMeteringMode, ControlInfo(0, static_cast(controls::MeteringModeMax)) }, + { &controls::AeConstraintMode, ControlInfo(0, static_cast(controls::ConstraintModeMax)) }, + { &controls::AeExposureMode, ControlInfo(0, static_cast(controls::ExposureModeMax)) }, + { &controls::ExposureValue, ControlInfo(0.0f, 16.0f) }, + { &controls::AwbEnable, ControlInfo(false, true) }, + { &controls::ColourGains, ControlInfo(0.0f, 32.0f) }, + { &controls::AwbMode, ControlInfo(0, static_cast(controls::AwbModeMax)) }, + { &controls::Brightness, ControlInfo(-1.0f, 1.0f) }, + { &controls::Contrast, ControlInfo(0.0f, 32.0f) }, + { &controls::Saturation, ControlInfo(0.0f, 32.0f) }, + { &controls::Sharpness, ControlInfo(0.0f, 16.0f, 1.0f) }, + { &controls::ColourCorrectionMatrix, ControlInfo(-16.0f, 16.0f) }, + }; }; } /* namespace libcamera */ diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index 4557016c..e09caa8d 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -60,7 +61,7 @@ namespace libcamera { LOG_DEFINE_CATEGORY(IPARPI) -class IPARPi : public IPAInterface +class IPARPi : public IPARPiInterface { public: IPARPi() @@ -68,6 +69,7 @@ public: frame_count_(0), check_count_(0), hide_count_(0), mistrust_count_(0), lsTable_(nullptr) { + initializeRPiControls(); } ~IPARPi() @@ -82,12 +84,12 @@ public: void configure(const CameraSensorInfo &sensorInfo, const std::map &streamConfig, - const std::map &entityControls, - const IPAOperationData &data, - IPAOperationData *response) override; + const std::map &entityControls, + const RPiConfigureParams &data, + RPiConfigureParams *response) override; void mapBuffers(const std::vector &buffers) override; void unmapBuffers(const std::vector &ids) override; - void processEvent(const IPAOperationData &event) override; + void processEvent(const RPiEventParams &event) override; private: void setMode(const CameraSensorInfo &sensorInfo); @@ -143,6 +145,11 @@ private: unsigned int mistrust_count_; /* LS table allocation passed in from the pipeline handler. */ FileDescriptor lsTableHandle_; + /* + * LS table allocation passed in from the pipeline handler, + * in the context of the pipeline handler. + */ + int32_t lsTableHandlePH_; void *lsTable_; }; @@ -192,15 +199,13 @@ void IPARPi::setMode(const CameraSensorInfo &sensorInfo) void IPARPi::configure(const CameraSensorInfo &sensorInfo, [[maybe_unused]] const std::map &streamConfig, - const std::map &entityControls, - const IPAOperationData &ipaConfig, - IPAOperationData *result) + const std::map &entityControls, + const RPiConfigureParams &ipaConfig, + RPiConfigureParams *result) { if (entityControls.empty()) return; - result->operation = 0; - unicam_ctrls_ = entityControls.at(0); isp_ctrls_ = entityControls.at(1); /* Setup a metadata ControlList to output metadata. */ @@ -222,11 +227,13 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, helper_->GetDelays(exposureDelay, gainDelay); sensorMetadata = helper_->SensorEmbeddedDataPresent(); - result->data.push_back(gainDelay); - result->data.push_back(exposureDelay); - result->data.push_back(sensorMetadata); + RPiConfigurePayload payload = {}; + payload.op_ = RPI_IPA_CONFIG_STAGGERED_WRITE; + payload.staggeredWriteResult_.gainDelay_ = gainDelay; + payload.staggeredWriteResult_.exposureDelay_ = exposureDelay; + payload.staggeredWriteResult_.sensorMetadata_ = sensorMetadata; - result->operation |= RPI_IPA_CONFIG_STAGGERED_WRITE; + result->payload_.push_back(payload); } /* Re-assemble camera mode using the sensor info. */ @@ -274,15 +281,23 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, if (agcStatus.shutter_time != 0.0 && agcStatus.analogue_gain != 0.0) { ControlList ctrls(unicam_ctrls_); applyAGC(&agcStatus, ctrls); - result->controls.push_back(ctrls); - result->operation |= RPI_IPA_CONFIG_SENSOR; + RPiConfigurePayload payload = {}; + payload.op_ = RPI_IPA_CONFIG_SENSOR; + payload.controls_ = ctrls; + + result->payload_.push_back(payload); } lastMode_ = mode_; /* Store the lens shading table pointer and handle if available. */ - if (ipaConfig.operation & RPI_IPA_CONFIG_LS_TABLE) { + auto lens = std::find_if(ipaConfig.payload_.begin(), + ipaConfig.payload_.end(), + [] (const RPiConfigurePayload &p) { + return p.op_ == RPI_IPA_CONFIG_LS_TABLE; + }); + if (lens != ipaConfig.payload_.end()) { /* Remove any previous table, if there was one. */ if (lsTable_) { munmap(lsTable_, MAX_LS_GRID_SIZE); @@ -290,7 +305,8 @@ void IPARPi::configure(const CameraSensorInfo &sensorInfo, } /* Map the LS table buffer into user space. */ - lsTableHandle_ = FileDescriptor(ipaConfig.data[0]); + lsTableHandle_ = FileDescriptor(lens->lsTableHandle_); + lsTableHandlePH_ = lens->lsTableHandleStatic_; if (lsTableHandle_.isValid()) { lsTable_ = mmap(nullptr, MAX_LS_GRID_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, lsTableHandle_.fd(), 0); @@ -335,11 +351,11 @@ void IPARPi::unmapBuffers(const std::vector &ids) } } -void IPARPi::processEvent(const IPAOperationData &event) +void IPARPi::processEvent(const RPiEventParams &event) { - switch (event.operation) { + switch (event.ev_) { case RPI_IPA_EVENT_SIGNAL_STAT_READY: { - unsigned int bufferId = event.data[0]; + unsigned int bufferId = event.bufferId_; if (++check_count_ != frame_count_) /* assert here? */ LOG(IPARPI, Error) << "WARNING: Prepare/Process mismatch!!!"; @@ -348,17 +364,17 @@ void IPARPi::processEvent(const IPAOperationData &event) reportMetadata(); - IPAOperationData op; - op.operation = RPI_IPA_ACTION_STATS_METADATA_COMPLETE; - op.data = { bufferId & RPiIpaMask::ID }; - op.controls = { libcameraMetadata_ }; + RPiActionParams op; + op.op_ = RPI_IPA_ACTION_STATS_METADATA_COMPLETE; + op.statsComplete_.bufferId_ = { bufferId & RPiIpaMask::ID }; + op.statsComplete_.controls_ = { libcameraMetadata_ }; queueFrameAction.emit(0, op); break; } case RPI_IPA_EVENT_SIGNAL_ISP_PREPARE: { - unsigned int embeddedbufferId = event.data[0]; - unsigned int bayerbufferId = event.data[1]; + unsigned int embeddedbufferId = event.ispPrepare_.embeddedbufferId_; + unsigned int bayerbufferId = event.ispPrepare_.bayerbufferId_; /* * At start-up, or after a mode-switch, we may want to @@ -368,23 +384,23 @@ void IPARPi::processEvent(const IPAOperationData &event) prepareISP(embeddedbufferId); /* Ready to push the input buffer into the ISP. */ - IPAOperationData op; + RPiActionParams op; if (++frame_count_ > hide_count_) - op.operation = RPI_IPA_ACTION_RUN_ISP; + op.op_ = RPI_IPA_ACTION_RUN_ISP; else - op.operation = RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME; - op.data = { bayerbufferId & RPiIpaMask::ID }; + op.op_ = RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME; + op.bufferId_ = { bayerbufferId & RPiIpaMask::ID }; queueFrameAction.emit(0, op); break; } case RPI_IPA_EVENT_QUEUE_REQUEST: { - queueRequest(event.controls[0]); + queueRequest(event.controls_); break; } default: - LOG(IPARPI, Error) << "Unknown event " << event.operation; + LOG(IPARPI, Error) << "Unknown event " << event.ev_; break; } } @@ -489,6 +505,8 @@ void IPARPi::queueRequest(const ControlList &controls) /* Clear the return metadata buffer. */ libcameraMetadata_.clear(); + LOG(IPARPI, Info) << "Request ctrl length: " << controls.size(); + for (auto const &ctrl : controls) { LOG(IPARPI, Info) << "Request ctrl: " << controls::controls.at(ctrl.first)->name() @@ -698,9 +716,9 @@ void IPARPi::queueRequest(const ControlList &controls) void IPARPi::returnEmbeddedBuffer(unsigned int bufferId) { - IPAOperationData op; - op.operation = RPI_IPA_ACTION_EMBEDDED_COMPLETE; - op.data = { bufferId & RPiIpaMask::ID }; + RPiActionParams op; + op.op_ = RPI_IPA_ACTION_EMBEDDED_COMPLETE; + op.bufferId_ = { bufferId & RPiIpaMask::ID }; queueFrameAction.emit(0, op); } @@ -763,9 +781,9 @@ void IPARPi::prepareISP(unsigned int bufferId) applyDPC(dpcStatus, ctrls); if (!ctrls.empty()) { - IPAOperationData op; - op.operation = RPI_IPA_ACTION_V4L2_SET_ISP; - op.controls.push_back(ctrls); + RPiActionParams op; + op.op_ = RPI_IPA_ACTION_V4L2_SET_ISP; + op.controls_ = ctrls; queueFrameAction.emit(0, op); } } @@ -823,9 +841,9 @@ void IPARPi::processStats(unsigned int bufferId) ControlList ctrls(unicam_ctrls_); applyAGC(&agcStatus, ctrls); - IPAOperationData op; - op.operation = RPI_IPA_ACTION_V4L2_SET_STAGGERED; - op.controls.push_back(ctrls); + RPiActionParams op; + op.op_ = RPI_IPA_ACTION_V4L2_SET_STAGGERED; + op.controls_ = ctrls; queueFrameAction.emit(0, op); } } @@ -1056,7 +1074,7 @@ void IPARPi::applyLS(const struct AlscStatus *lsStatus, ControlList &ctrls) .grid_width = w, .grid_stride = w, .grid_height = h, - .dmabuf = lsTableHandle_.fd(), + .dmabuf = lsTableHandlePH_, .ref_transform = 0, .corner_sampled = 1, .gain_format = GAIN_FORMAT_U4P10 @@ -1136,9 +1154,9 @@ const struct IPAModuleInfo ipaModuleInfo = { "raspberrypi", }; -struct ipa_context *ipaCreate() +IPAInterface *ipaCreate() { - return new IPAInterfaceWrapper(std::make_unique()); + return new IPARPi(); } }; /* extern "C" */ diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index ce43af34..70bb6fcc 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -15,7 +15,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -296,7 +298,7 @@ public: int loadIPA(); int configureIPA(); - void queueFrameAction(unsigned int frame, const IPAOperationData &action); + void queueFrameAction(unsigned int frame, const RPiActionParams &action); /* bufferComplete signal handlers. */ void unicamBufferDequeue(FrameBuffer *buffer); @@ -307,6 +309,8 @@ public: void handleStreamBuffer(FrameBuffer *buffer, const RPiStream *stream); void handleState(); + std::unique_ptr ipa_; + CameraSensor *sensor_; /* Array of Unicam and ISP device streams and associated buffers/streams. */ RPiDevice unicam_; @@ -506,6 +510,7 @@ CameraConfiguration::Status RPiCameraConfiguration::validate() PipelineHandlerRPi::PipelineHandlerRPi(CameraManager *manager) : PipelineHandler(manager), unicam_(nullptr), isp_(nullptr) { + initializeRPiControls(); } CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera, @@ -1103,7 +1108,9 @@ void RPiCameraData::frameStarted(uint32_t sequence) int RPiCameraData::loadIPA() { - ipa_ = IPAManager::createIPA(pipe_, 1, 1); + std::unique_ptr ptr = IPAManager::createIPA(pipe_, 1, 1); + ipa_ = std::unique_ptr{static_cast(std::move(ptr).release())}; + if (!ipa_) return -ENOENT; @@ -1119,8 +1126,8 @@ int RPiCameraData::loadIPA() int RPiCameraData::configureIPA() { std::map streamConfig; - std::map entityControls; - IPAOperationData ipaConfig = {}; + std::map entityControls; + RPiConfigureParams ipaConfig; /* Get the device format to pass to the IPA. */ V4L2DeviceFormat sensorFormat; @@ -1145,8 +1152,11 @@ int RPiCameraData::configureIPA() return -ENOMEM; /* Allow the IPA to mmap the LS table via the file descriptor. */ - ipaConfig.operation = RPI_IPA_CONFIG_LS_TABLE; - ipaConfig.data = { static_cast(lsTable_.fd()) }; + RPiConfigurePayload payload; + payload.op_ = RPI_IPA_CONFIG_LS_TABLE; + payload.lsTableHandle_ = lsTable_; + payload.lsTableHandleStatic_ = lsTable_.fd(); + ipaConfig.payload_.push_back(payload); } CameraSensorInfo sensorInfo = {}; @@ -1157,53 +1167,68 @@ int RPiCameraData::configureIPA() } /* Ready the IPA - it must know about the sensor resolution. */ - IPAOperationData result; + RPiConfigureParams results; ipa_->configure(sensorInfo, streamConfig, entityControls, ipaConfig, - &result); + &results); - if (result.operation & RPI_IPA_CONFIG_STAGGERED_WRITE) { - /* - * Setup our staggered control writer with the sensor default - * gain and exposure delays. - */ - if (!staggeredCtrl_) { - staggeredCtrl_.init(unicam_[Unicam::Image].dev(), - { { V4L2_CID_ANALOGUE_GAIN, result.data[0] }, - { V4L2_CID_EXPOSURE, result.data[1] } }); - sensorMetadata_ = result.data[2]; + for (RPiConfigurePayload &result : results.payload_) { + if (result.op_ == RPI_IPA_CONFIG_STAGGERED_WRITE) { + + /* + * Setup our staggered control writer with the sensor default + * gain and exposure delays. + */ + if (!staggeredCtrl_) { + staggeredCtrl_.init(unicam_[Unicam::Image].dev(), + { { V4L2_CID_ANALOGUE_GAIN, + result.staggeredWriteResult_.gainDelay_ }, + { V4L2_CID_EXPOSURE, + result.staggeredWriteResult_.exposureDelay_ } }); + sensorMetadata_ = result.staggeredWriteResult_.sensorMetadata_; + } + + /* Configure the H/V flip controls based on the sensor rotation. */ + ControlList ctrls(unicam_[Unicam::Image].dev()->controls()); + int32_t rotation = sensor_->properties().get(properties::Rotation); + ctrls.set(V4L2_CID_HFLIP, static_cast(!!rotation)); + ctrls.set(V4L2_CID_VFLIP, static_cast(!!rotation)); + unicam_[Unicam::Image].dev()->setControls(&ctrls); } - } - if (result.operation & RPI_IPA_CONFIG_SENSOR) { - const ControlList &ctrls = result.controls[0]; - if (!staggeredCtrl_.set(ctrls)) - LOG(RPI, Error) << "V4L2 staggered set failed"; + if (result.op_ == RPI_IPA_CONFIG_SENSOR) { + const ControlList &ctrls = result.controls_; + if (!staggeredCtrl_.set(ctrls)) + LOG(RPI, Error) << "V4L2 staggered set failed"; + } } return 0; } void RPiCameraData::queueFrameAction([[maybe_unused]] unsigned int frame, - const IPAOperationData &action) + const RPiActionParams &action) { /* * The following actions can be handled when the pipeline handler is in * a stopped state. */ - switch (action.operation) { + switch (action.op_) { case RPI_IPA_ACTION_V4L2_SET_STAGGERED: { - const ControlList &controls = action.controls[0]; + const ControlList &controls = action.controls_; if (!staggeredCtrl_.set(controls)) LOG(RPI, Error) << "V4L2 staggered set failed"; goto done; } case RPI_IPA_ACTION_V4L2_SET_ISP: { - ControlList controls = action.controls[0]; + ControlList controls = action.controls_; isp_[Isp::Input].dev()->setControls(&controls); goto done; } + + default: + break; } if (state_ == State::Stopped) @@ -1213,20 +1238,20 @@ void RPiCameraData::queueFrameAction([[maybe_unused]] unsigned int frame, * The following actions must not be handled when the pipeline handler * is in a stopped state. */ - switch (action.operation) { + switch (action.op_) { case RPI_IPA_ACTION_STATS_METADATA_COMPLETE: { - unsigned int bufferId = action.data[0]; + unsigned int bufferId = action.statsComplete_.bufferId_; FrameBuffer *buffer = isp_[Isp::Stats].getBuffers()->at(bufferId).get(); handleStreamBuffer(buffer, &isp_[Isp::Stats]); /* Fill the Request metadata buffer with what the IPA has provided */ - requestQueue_.front()->metadata() = std::move(action.controls[0]); + requestQueue_.front()->metadata() = std::move(action.statsComplete_.controls_); state_ = State::IpaComplete; break; } case RPI_IPA_ACTION_EMBEDDED_COMPLETE: { - unsigned int bufferId = action.data[0]; + unsigned int bufferId = action.bufferId_; FrameBuffer *buffer = unicam_[Unicam::Embedded].getBuffers()->at(bufferId).get(); handleStreamBuffer(buffer, &unicam_[Unicam::Embedded]); break; @@ -1234,20 +1259,17 @@ void RPiCameraData::queueFrameAction([[maybe_unused]] unsigned int frame, case RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME: case RPI_IPA_ACTION_RUN_ISP: { - unsigned int bufferId = action.data[0]; + unsigned int bufferId = action.bufferId_; FrameBuffer *buffer = unicam_[Unicam::Image].getBuffers()->at(bufferId).get(); - LOG(RPI, Debug) << "Input re-queue to ISP, buffer id " << buffer->cookie() - << ", timestamp: " << buffer->metadata().timestamp; - isp_[Isp::Input].dev()->queueBuffer(buffer); - dropFrame_ = (action.operation == RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME) ? true : false; + dropFrame_ = (action.op_ == RPI_IPA_ACTION_RUN_ISP_AND_DROP_FRAME) ? true : false; ispOutputCount_ = 0; break; } default: - LOG(RPI, Error) << "Unknown action " << action.operation; + LOG(RPI, Error) << "Unknown action " << action.op_; break; } @@ -1347,10 +1369,10 @@ void RPiCameraData::ispOutputDequeue(FrameBuffer *buffer) /* If this is a stats output, hand it to the IPA now. */ if (stream == &isp_[Isp::Stats]) { - IPAOperationData op; - op.operation = RPI_IPA_EVENT_SIGNAL_STAT_READY; - op.data = { RPiIpaMask::STATS | buffer->cookie() }; - ipa_->processEvent(op); + RPiEventParams ev; + ev.ev_ = RPI_IPA_EVENT_SIGNAL_STAT_READY; + ev.bufferId_ = { RPiIpaMask::STATS | buffer->cookie() }; + ipa_->processEvent(ev); } handleState(); @@ -1493,7 +1515,7 @@ void RPiCameraData::checkRequestCompleted() void RPiCameraData::tryRunPipeline() { FrameBuffer *bayerBuffer, *embeddedBuffer; - IPAOperationData op; + RPiEventParams ev; /* If any of our request or buffer queues are empty, we cannot proceed. */ if (state_ != State::Idle || requestQueue_.empty() || @@ -1548,9 +1570,9 @@ void RPiCameraData::tryRunPipeline() * queue the ISP output buffer listed in the request to start the HW * pipeline. */ - op.operation = RPI_IPA_EVENT_QUEUE_REQUEST; - op.controls = { request->controls() }; - ipa_->processEvent(op); + ev.ev_ = RPI_IPA_EVENT_QUEUE_REQUEST; + ev.controls_ = { request->controls() }; + ipa_->processEvent(ev); /* Queue up any ISP buffers passed into the request. */ for (auto &stream : isp_) { @@ -1569,10 +1591,10 @@ void RPiCameraData::tryRunPipeline() << " Bayer buffer id: " << bayerBuffer->cookie() << " Embedded buffer id: " << embeddedBuffer->cookie(); - op.operation = RPI_IPA_EVENT_SIGNAL_ISP_PREPARE; - op.data = { RPiIpaMask::EMBEDDED_DATA | embeddedBuffer->cookie(), - RPiIpaMask::BAYER_DATA | bayerBuffer->cookie() }; - ipa_->processEvent(op); + ev.ev_ = RPI_IPA_EVENT_SIGNAL_ISP_PREPARE; + ev.ispPrepare_.embeddedbufferId_ = RPiIpaMask::EMBEDDED_DATA | embeddedBuffer->cookie(); + ev.ispPrepare_.bayerbufferId_ = RPiIpaMask::BAYER_DATA | bayerBuffer->cookie(); + ipa_->processEvent(ev); } void RPiCameraData::tryFlushQueues() From patchwork Tue Sep 15 14:20:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9629 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id EF63CBF01C for ; Tue, 15 Sep 2020 14:21:39 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id C163862E32; Tue, 15 Sep 2020 16:21:39 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="QiHgRXxX"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 65BFD62D27 for ; Tue, 15 Sep 2020 16:21:38 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id A9338276; Tue, 15 Sep 2020 16:21:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179698; bh=urhzEubPDXC4MXe9xCrpU38vtq2k003CzNMsLvFNjTY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QiHgRXxX/Ba9f4SPg5n3BsiZixS/hqbJDR9QmmT9nyf9I/N6lRSMsa1OgoQVe+qeI rN/NJtyxfLmBodPC/jAKIcbI7fSrTif5prhKvoshh2edZtncWcmKzmIqXK+/J4gUdl NWLdOa27osNe2Xve19YcelcixBnfEgaDT+wF4ySE= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:34 +0900 Message-Id: <20200915142038.28757-20-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 19/23] libcamera: IPAProxy: Remove registration mechanism X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Implementations of IPA proxies use a registration mechanism to register themselves with the main IPA proxy factory. This registration declares static objects, causing a risk of things being constructed before the proper libcamera facilities are ready. Since each pipeline handler has its own IPA proxy and knows the type, it isn't necessary to have a proxy factory. Remove it to alleviate the risk of early construction. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/internal/ipa_proxy.h | 29 --------- src/libcamera/ipa_proxy.cpp | 85 -------------------------- 2 files changed, 114 deletions(-) diff --git a/include/libcamera/internal/ipa_proxy.h b/include/libcamera/internal/ipa_proxy.h index 1903150e..f651a3ae 100644 --- a/include/libcamera/internal/ipa_proxy.h +++ b/include/libcamera/internal/ipa_proxy.h @@ -36,35 +36,6 @@ private: IPAModule *ipam_; }; -class IPAProxyFactory -{ -public: - IPAProxyFactory(const char *name); - virtual ~IPAProxyFactory() {} - - virtual std::unique_ptr create(IPAModule *ipam, bool isolate) = 0; - - const std::string &name() const { return name_; } - - static void registerType(IPAProxyFactory *factory); - static std::vector &factories(); - -private: - std::string name_; -}; - -#define REGISTER_IPA_PROXY(proxy) \ -class proxy##Factory final : public IPAProxyFactory \ -{ \ -public: \ - proxy##Factory() : IPAProxyFactory(#proxy) {} \ - std::unique_ptr create(IPAModule *ipam, bool isolate) \ - { \ - return std::make_unique(ipam, isolate); \ - } \ -}; \ -static proxy##Factory global_##proxy##Factory; - } /* namespace libcamera */ #endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_H__ */ diff --git a/src/libcamera/ipa_proxy.cpp b/src/libcamera/ipa_proxy.cpp index ff4d7fd1..e7d4db06 100644 --- a/src/libcamera/ipa_proxy.cpp +++ b/src/libcamera/ipa_proxy.cpp @@ -229,89 +229,4 @@ std::string IPAProxy::resolvePath(const std::string &file) const * construction. */ -/** - * \class IPAProxyFactory - * \brief Registration of IPAProxy classes and creation of instances - * - * To facilitate discovery and instantiation of IPAProxy classes, the - * IPAProxyFactory class maintains a registry of IPAProxy classes. Each - * IPAProxy subclass shall register itself using the REGISTER_IPA_PROXY() - * macro, which will create a corresponding instance of a IPAProxyFactory - * subclass and register it with the static list of factories. - */ - -/** - * \brief Construct a IPAProxy factory - * \param[in] name Name of the IPAProxy class - * - * Creating an instance of the factory registers is with the global list of - * factories, accessible through the factories() function. - * - * The factory \a name is used for debugging and IPAProxy matching purposes - * and shall be unique. - */ -IPAProxyFactory::IPAProxyFactory(const char *name) - : name_(name) -{ - registerType(this); -} - -/** - * \fn IPAProxyFactory::create() - * \brief Create an instance of the IPAProxy corresponding to the factory - * \param[in] ipam The IPA module - * - * This virtual function is implemented by the REGISTER_IPA_PROXY() macro. - * It creates a IPAProxy instance that isolates an IPA interface designated - * by the IPA module \a ipam. - * - * \return A pointer to a newly constructed instance of the IPAProxy subclass - * corresponding to the factory - */ - -/** - * \fn IPAProxyFactory::name() - * \brief Retrieve the factory name - * \return The factory name - */ - -/** - * \brief Add a IPAProxy class to the registry - * \param[in] factory Factory to use to construct the IPAProxy - * - * The caller is responsible to guarantee the uniqueness of the IPAProxy name. - */ -void IPAProxyFactory::registerType(IPAProxyFactory *factory) -{ - std::vector &factories = IPAProxyFactory::factories(); - - factories.push_back(factory); - - LOG(IPAProxy, Debug) - << "Registered proxy \"" << factory->name() << "\""; -} - -/** - * \brief Retrieve the list of all IPAProxy factories - * - * The static factories map is defined inside the function to ensure it gets - * initialized on first use, without any dependency on link order. - * - * \return The list of pipeline handler factories - */ -std::vector &IPAProxyFactory::factories() -{ - static std::vector factories; - return factories; -} - -/** - * \def REGISTER_IPA_PROXY - * \brief Register a IPAProxy with the IPAProxy factory - * \param[in] proxy Class name of IPAProxy derived class to register - * - * Register a proxy subclass with the factory and make it available to - * isolate IPA modules. - */ - } /* namespace libcamera */ From patchwork Tue Sep 15 14:20:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9630 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 85D2DBF01C for ; Tue, 15 Sep 2020 14:21:41 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4C0EA62E42; Tue, 15 Sep 2020 16:21:41 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="UCjC5z9C"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id BC9E462D27 for ; Tue, 15 Sep 2020 16:21:40 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CDED910C6; Tue, 15 Sep 2020 16:21:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179700; bh=ldKNhfi+pIs61RD/4sI+K7VwOXfkwUa+VLvRY1itMfY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UCjC5z9C8CrFmGMGeU+DGIw8v+9WfRBX+Qr6dRUEmBScJ22qY3c/OpBegBlMxCR9d fvEzyVloJzj4a+/3KSc5hNoo8/YimrNJV4WXI9eCa4AqHAyTPm4SYYHlsFk81hQI4y 1RiCZFQ7vMqY47kPuvHAtaYC3UWQ9S5oWUtwkSnk= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:35 +0900 Message-Id: <20200915142038.28757-21-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 20/23] libcamera: proxy: Remove IPAProxyLinux and IPAProxyThread X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" We have now changed the proxy from per-IPC mechanism to per-pipeline. The per-IPC mechanism proxies are thus no longer needed; remove them. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- src/libcamera/proxy/ipa_proxy_linux.cpp | 103 ----------- src/libcamera/proxy/ipa_proxy_thread.cpp | 172 ------------------ src/libcamera/proxy/meson.build | 5 - .../proxy/worker/ipa_proxy_linux_worker.cpp | 90 --------- src/libcamera/proxy/worker/meson.build | 3 - 5 files changed, 373 deletions(-) delete mode 100644 src/libcamera/proxy/ipa_proxy_linux.cpp delete mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp delete mode 100644 src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp diff --git a/src/libcamera/proxy/ipa_proxy_linux.cpp b/src/libcamera/proxy/ipa_proxy_linux.cpp deleted file mode 100644 index b78a0e45..00000000 --- a/src/libcamera/proxy/ipa_proxy_linux.cpp +++ /dev/null @@ -1,103 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2019, Google Inc. - * - * ipa_proxy_linux.cpp - Default Image Processing Algorithm proxy for Linux - */ - -#include - -#include -#include - -#include "libcamera/internal/ipa_module.h" -#include "libcamera/internal/ipa_proxy.h" -#include "libcamera/internal/ipc_unixsocket.h" -#include "libcamera/internal/log.h" -#include "libcamera/internal/process.h" - -namespace libcamera { - -LOG_DECLARE_CATEGORY(IPAProxy) - -class IPAProxyLinux : public IPAProxy -{ -public: - IPAProxyLinux(IPAModule *ipam); - ~IPAProxyLinux(); - - int init([[maybe_unused]] const IPASettings &settings) override - { - return 0; - } - int start() override { return 0; } - void stop() override {} - void configure([[maybe_unused]] const CameraSensorInfo &sensorInfo, - [[maybe_unused]] const std::map &streamConfig, - [[maybe_unused]] const std::map &entityControls, - [[maybe_unused]] const IPAOperationData &ipaConfig, - [[maybe_unused]] IPAOperationData *result) override {} - void mapBuffers([[maybe_unused]] const std::vector &buffers) override {} - void unmapBuffers([[maybe_unused]] const std::vector &ids) override {} - void processEvent([[maybe_unused]] const IPAOperationData &event) override {} - -private: - void readyRead(IPCUnixSocket *ipc); - - Process *proc_; - - IPCUnixSocket *socket_; -}; - -IPAProxyLinux::IPAProxyLinux(IPAModule *ipam) - : IPAProxy(ipam), proc_(nullptr), socket_(nullptr) -{ - LOG(IPAProxy, Debug) - << "initializing dummy proxy: loading IPA from " - << ipam->path(); - - std::vector fds; - std::vector args; - args.push_back(ipam->path()); - const std::string path = resolvePath("ipa_proxy_linux"); - if (path.empty()) { - LOG(IPAProxy, Error) - << "Failed to get proxy worker path"; - return; - } - - socket_ = new IPCUnixSocket(); - int fd = socket_->create(); - if (fd < 0) { - LOG(IPAProxy, Error) - << "Failed to create socket"; - return; - } - socket_->readyRead.connect(this, &IPAProxyLinux::readyRead); - args.push_back(std::to_string(fd)); - fds.push_back(fd); - - proc_ = new Process(); - int ret = proc_->start(path, args, fds); - if (ret) { - LOG(IPAProxy, Error) - << "Failed to start proxy worker process"; - return; - } - - valid_ = true; -} - -IPAProxyLinux::~IPAProxyLinux() -{ - delete proc_; - delete socket_; -} - -void IPAProxyLinux::readyRead([[maybe_unused]] IPCUnixSocket *ipc) -{ -} - -REGISTER_IPA_PROXY(IPAProxyLinux) - -} /* namespace libcamera */ diff --git a/src/libcamera/proxy/ipa_proxy_thread.cpp b/src/libcamera/proxy/ipa_proxy_thread.cpp deleted file mode 100644 index eead2883..00000000 --- a/src/libcamera/proxy/ipa_proxy_thread.cpp +++ /dev/null @@ -1,172 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2020, Google Inc. - * - * ipa_proxy_thread.cpp - Proxy running an Image Processing Algorithm in a thread - */ - -#include - -#include -#include - -#include "libcamera/internal/ipa_context_wrapper.h" -#include "libcamera/internal/ipa_module.h" -#include "libcamera/internal/ipa_proxy.h" -#include "libcamera/internal/log.h" -#include "libcamera/internal/thread.h" - -namespace libcamera { - -LOG_DECLARE_CATEGORY(IPAProxy) - -class IPAProxyThread : public IPAProxy, public Object -{ -public: - IPAProxyThread(IPAModule *ipam); - - int init(const IPASettings &settings) override; - int start() override; - void stop() override; - - void configure(const CameraSensorInfo &sensorInfo, - const std::map &streamConfig, - const std::map &entityControls, - const IPAOperationData &ipaConfig, - IPAOperationData *result) override; - void mapBuffers(const std::vector &buffers) override; - void unmapBuffers(const std::vector &ids) override; - void processEvent(const IPAOperationData &event) override; - -private: - void queueFrameAction(unsigned int frame, const IPAOperationData &data); - - /* Helper class to invoke processEvent() in another thread. */ - class ThreadProxy : public Object - { - public: - void setIPA(IPAInterface *ipa) - { - ipa_ = ipa; - } - - int start() - { - return ipa_->start(); - } - - void stop() - { - ipa_->stop(); - } - - void processEvent(const IPAOperationData &event) - { - ipa_->processEvent(event); - } - - private: - IPAInterface *ipa_; - }; - - bool running_; - Thread thread_; - ThreadProxy proxy_; - std::unique_ptr ipa_; -}; - -IPAProxyThread::IPAProxyThread(IPAModule *ipam) - : IPAProxy(ipam), running_(false) -{ - if (!ipam->load()) - return; - - struct ipa_context *ctx = ipam->createContext(); - if (!ctx) { - LOG(IPAProxy, Error) - << "Failed to create IPA context for " << ipam->path(); - return; - } - - ipa_ = std::make_unique(ctx); - proxy_.setIPA(ipa_.get()); - - /* - * Proxy the queueFrameAction signal to dispatch it in the caller's - * thread. - */ - ipa_->queueFrameAction.connect(this, &IPAProxyThread::queueFrameAction); - - valid_ = true; -} - -int IPAProxyThread::init(const IPASettings &settings) -{ - int ret = ipa_->init(settings); - if (ret) - return ret; - - proxy_.moveToThread(&thread_); - - return 0; -} - -int IPAProxyThread::start() -{ - running_ = true; - thread_.start(); - - return proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking); -} - -void IPAProxyThread::stop() -{ - if (!running_) - return; - - running_ = false; - - proxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking); - - thread_.exit(); - thread_.wait(); -} - -void IPAProxyThread::configure(const CameraSensorInfo &sensorInfo, - const std::map &streamConfig, - const std::map &entityControls, - const IPAOperationData &ipaConfig, - IPAOperationData *result) -{ - ipa_->configure(sensorInfo, streamConfig, entityControls, ipaConfig, - result); -} - -void IPAProxyThread::mapBuffers(const std::vector &buffers) -{ - ipa_->mapBuffers(buffers); -} - -void IPAProxyThread::unmapBuffers(const std::vector &ids) -{ - ipa_->unmapBuffers(ids); -} - -void IPAProxyThread::processEvent(const IPAOperationData &event) -{ - if (!running_) - return; - - /* Dispatch the processEvent() call to the thread. */ - proxy_.invokeMethod(&ThreadProxy::processEvent, ConnectionTypeQueued, - event); -} - -void IPAProxyThread::queueFrameAction(unsigned int frame, const IPAOperationData &data) -{ - IPAInterface::queueFrameAction.emit(frame, data); -} - -REGISTER_IPA_PROXY(IPAProxyThread) - -} /* namespace libcamera */ diff --git a/src/libcamera/proxy/meson.build b/src/libcamera/proxy/meson.build index b0d35646..cfec0567 100644 --- a/src/libcamera/proxy/meson.build +++ b/src/libcamera/proxy/meson.build @@ -1,10 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 -libcamera_sources += files([ - 'ipa_proxy_linux.cpp', - 'ipa_proxy_thread.cpp', -]) - # generate ipa_proxy_{pipeline}.cpp ipa_proxy_sources = [] diff --git a/src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp b/src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp deleted file mode 100644 index 0c4687f7..00000000 --- a/src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp +++ /dev/null @@ -1,90 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2019, Google Inc. - * - * ipa_proxy_linux_worker.cpp - Default Image Processing Algorithm proxy worker for Linux - */ - -#include -#include -#include - -#include -#include -#include - -#include "libcamera/internal/ipa_module.h" -#include "libcamera/internal/ipc_unixsocket.h" -#include "libcamera/internal/log.h" -#include "libcamera/internal/thread.h" - -using namespace libcamera; - -LOG_DEFINE_CATEGORY(IPAProxyLinuxWorker) - -void readyRead(IPCUnixSocket *ipc) -{ - IPCUnixSocket::Payload message; - int ret; - - ret = ipc->receive(&message); - if (ret) { - LOG(IPAProxyLinuxWorker, Error) - << "Receive message failed: " << ret; - return; - } - - LOG(IPAProxyLinuxWorker, Debug) << "Received a message!"; -} - -int main(int argc, char **argv) -{ - /* Uncomment this for debugging. */ -#if 0 - std::string logPath = "/tmp/libcamera.worker." + - std::to_string(getpid()) + ".log"; - logSetFile(logPath.c_str()); -#endif - - if (argc < 3) { - LOG(IPAProxyLinuxWorker, Debug) - << "Tried to start worker with no args"; - return EXIT_FAILURE; - } - - int fd = std::stoi(argv[2]); - LOG(IPAProxyLinuxWorker, Debug) - << "Starting worker for IPA module " << argv[1] - << " with IPC fd = " << fd; - - std::unique_ptr ipam = std::make_unique(argv[1]); - if (!ipam->isValid() || !ipam->load()) { - LOG(IPAProxyLinuxWorker, Error) - << "IPAModule " << argv[1] << " should be valid but isn't"; - return EXIT_FAILURE; - } - - IPCUnixSocket socket; - if (socket.bind(fd) < 0) { - LOG(IPAProxyLinuxWorker, Error) << "IPC socket binding failed"; - return EXIT_FAILURE; - } - socket.readyRead.connect(&readyRead); - - struct ipa_context *ipac = ipam->createContext(); - if (!ipac) { - LOG(IPAProxyLinuxWorker, Error) << "Failed to create IPA context"; - return EXIT_FAILURE; - } - - LOG(IPAProxyLinuxWorker, Debug) << "Proxy worker successfully started"; - - /* \todo upgrade listening loop */ - EventDispatcher *dispatcher = Thread::current()->eventDispatcher(); - while (1) - dispatcher->processEvents(); - - ipac->ops->destroy(ipac); - - return 0; -} diff --git a/src/libcamera/proxy/worker/meson.build b/src/libcamera/proxy/worker/meson.build index c35be70c..db463065 100644 --- a/src/libcamera/proxy/worker/meson.build +++ b/src/libcamera/proxy/worker/meson.build @@ -1,8 +1,5 @@ # SPDX-License-Identifier: CC0-1.0 -ipa_proxy_sources = [ - ['ipa_proxy_linux', 'ipa_proxy_linux_worker.cpp'] -] # generate ipa_proxy_{pipeline}_worker.cpp ipa_proxy_sources = {} From patchwork Tue Sep 15 14:20:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9631 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 19A43BF01C for ; Tue, 15 Sep 2020 14:21:45 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id D819162E35; Tue, 15 Sep 2020 16:21:44 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="emUlahsz"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 111EE62D27 for ; Tue, 15 Sep 2020 16:21:43 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 496E1FD8; Tue, 15 Sep 2020 16:21:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179702; bh=CaMNBBUB7TnmWGd3aw5JUmKT74YiY1Sgfi5UpiBtBm4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=emUlahszO80fB0zz/WPOAp10WM6UKs5T9tSOieXFbvXhg9pUvZB7qBjtElMh3ENU8 xkNcWem7wiSJl4MTkY0Gp/Ivw43zTYUENHSy5ShVqsns8fj01hxDLDBai/rMegV/E2 5nO+d2YquVpqvayqZkR9X2WJoJgGkCyVe3eJZV+U= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:36 +0900 Message-Id: <20200915142038.28757-22-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 21/23] libcamera: IPAManager: Make createIPA return proxy directly X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since every pipeline knows the type of the proxy that it needs, and since all IPAs are to be wrapped in a proxy, IPAManager no longer needs to search in the factory list to fetch the proxy factory to construct a factory. Instead, we define createIPA as a template function, and the pipeline handler can declare the proxy type when it calls createIPA. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- include/libcamera/internal/ipa_manager.h | 31 ++++++++++-- src/libcamera/ipa_manager.cpp | 48 +------------------ .../pipeline/raspberrypi/raspberrypi.cpp | 3 +- 3 files changed, 30 insertions(+), 52 deletions(-) diff --git a/include/libcamera/internal/ipa_manager.h b/include/libcamera/internal/ipa_manager.h index 4a143b6a..297a8a58 100644 --- a/include/libcamera/internal/ipa_manager.h +++ b/include/libcamera/internal/ipa_manager.h @@ -14,20 +14,45 @@ #include #include "libcamera/internal/ipa_module.h" +#include "libcamera/internal/log.h" #include "libcamera/internal/pipeline_handler.h" #include "libcamera/internal/pub_key.h" namespace libcamera { +LOG_DECLARE_CATEGORY(IPAManager) + class IPAManager { public: IPAManager(); ~IPAManager(); - static std::unique_ptr createIPA(PipelineHandler *pipe, - uint32_t maxVersion, - uint32_t minVersion); + template + static std::unique_ptr

createIPA(PipelineHandler *pipe, + uint32_t maxVersion, + uint32_t minVersion) + { + IPAModule *m = nullptr; + + for (IPAModule *module : self_->modules_) { + if (module->match(pipe, minVersion, maxVersion)) { + m = module; + break; + } + } + + if (!m) + return nullptr; + + std::unique_ptr

proxy = std::make_unique

(m, !self_->isSignatureValid(m)); + if (!proxy->isValid()) { + LOG(IPAManager, Error) << "Failed to load proxy"; + return nullptr; + } + + return proxy; + } private: static IPAManager *self_; diff --git a/src/libcamera/ipa_manager.cpp b/src/libcamera/ipa_manager.cpp index 26458153..0d518443 100644 --- a/src/libcamera/ipa_manager.cpp +++ b/src/libcamera/ipa_manager.cpp @@ -245,6 +245,7 @@ unsigned int IPAManager::addDir(const char *libDir, unsigned int maxDepth) } /** + * \fn IPAManager::createIPA() * \brief Create an IPA proxy that matches a given pipeline handler * \param[in] pipe The pipeline handler that wants a matching IPA proxy * \param[in] minVersion Minimum acceptable version of IPA module @@ -253,53 +254,6 @@ unsigned int IPAManager::addDir(const char *libDir, unsigned int maxDepth) * \return A newly created IPA proxy, or nullptr if no matching IPA module is * found or if the IPA proxy fails to initialize */ -std::unique_ptr IPAManager::createIPA(PipelineHandler *pipe, - uint32_t maxVersion, - uint32_t minVersion) -{ - IPAModule *m = nullptr; - - for (IPAModule *module : self_->modules_) { - if (module->match(pipe, minVersion, maxVersion)) { - m = module; - break; - } - } - - if (!m) - return nullptr; - - /* - * Load and run the IPA module in a thread if it has a valid signature, - * or isolate it in a separate process otherwise. - * - * \todo Implement a better proxy selection - */ - std::string pipeName(pipe->name()); - const char *proxyName = pipeName.replace(0, 15, "IPAProxy").c_str(); - IPAProxyFactory *pf = nullptr; - - for (IPAProxyFactory *factory : IPAProxyFactory::factories()) { - if (!strcmp(factory->name().c_str(), proxyName)) { - pf = factory; - break; - } - } - - if (!pf) { - LOG(IPAManager, Error) << "Failed to get proxy factory"; - return nullptr; - } - - std::unique_ptr proxy = - pf->create(m, !self_->isSignatureValid(m)); - if (!proxy->isValid()) { - LOG(IPAManager, Error) << "Failed to load proxy"; - return nullptr; - } - - return proxy; -} bool IPAManager::isSignatureValid([[maybe_unused]] IPAModule *ipa) const { diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp index 70bb6fcc..19755e8c 100644 --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp @@ -1108,8 +1108,7 @@ void RPiCameraData::frameStarted(uint32_t sequence) int RPiCameraData::loadIPA() { - std::unique_ptr ptr = IPAManager::createIPA(pipe_, 1, 1); - ipa_ = std::unique_ptr{static_cast(std::move(ptr).release())}; + ipa_ = IPAManager::createIPA(pipe_, 1, 1); if (!ipa_) return -ENOENT; From patchwork Tue Sep 15 14:20:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9632 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 81E75BF01C for ; Tue, 15 Sep 2020 14:21:46 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id 4FDDD62E45; Tue, 15 Sep 2020 16:21:46 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="MBZA6lTo"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 4DB1F62E17 for ; Tue, 15 Sep 2020 16:21:45 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 871E9276; Tue, 15 Sep 2020 16:21:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179705; bh=FgZv15m/60L06YYMv+V1qKcNSrb6jWuHK4fTbsBBFAk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MBZA6lTok5PE6rRzaQsoD3WxzhVT32+KhtY1qlh2IiLralwMmrQg4/FP3Lb17RiQj gWVr7KLcAsyPYo7fATbNDUu/8BNa+eFsYlVkIRKXpvaZBGbYUqns+WQT7BcExaKtRT FsyIWciyqDb3WNOLL3hdJNIs8/K5FYOQxwB1ItDM= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:37 +0900 Message-Id: <20200915142038.28757-23-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 22/23] ipa: remove libipa X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" As every pipeline and have its own proxy, IPAInterfaceWrapper is no longer necessary. Since it's the only member of libipa, remove libipa completely. Signed-off-by: Paul Elder Reviewed-by: Niklas Söderlund --- src/ipa/libipa/ipa_interface_wrapper.cpp | 285 ----------------------- src/ipa/libipa/ipa_interface_wrapper.h | 61 ----- src/ipa/libipa/meson.build | 15 -- src/ipa/meson.build | 2 - src/ipa/raspberrypi/meson.build | 2 - src/ipa/raspberrypi/raspberrypi.cpp | 2 - 6 files changed, 367 deletions(-) delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.cpp delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.h delete mode 100644 src/ipa/libipa/meson.build diff --git a/src/ipa/libipa/ipa_interface_wrapper.cpp b/src/ipa/libipa/ipa_interface_wrapper.cpp deleted file mode 100644 index cee532e3..00000000 --- a/src/ipa/libipa/ipa_interface_wrapper.cpp +++ /dev/null @@ -1,285 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2019, Google Inc. - * - * ipa_interface_wrapper.cpp - Image Processing Algorithm interface wrapper - */ - -#include "ipa_interface_wrapper.h" - -#include -#include -#include -#include - -#include - -#include "libcamera/internal/byte_stream_buffer.h" -#include "libcamera/internal/camera_sensor.h" - -/** - * \file ipa_interface_wrapper.h - * \brief Image Processing Algorithm interface wrapper - */ - -namespace libcamera { - -/** - * \class IPAInterfaceWrapper - * \brief Wrap an IPAInterface and expose it as an ipa_context - * - * This class implements the ipa_context API based on a provided IPAInterface. - * It helps IPAs that implement the IPAInterface API to provide the external - * ipa_context API. - * - * To use the wrapper, an IPA module simple creates a new instance of its - * IPAInterface implementation, and passes it to the constructor of the - * IPAInterfaceWrapper. As IPAInterfaceWrapper inherits from ipa_context, the - * constructed wrapper can then be directly returned from the IPA module's - * ipaCreate() function. - * - * \code{.cpp} - * class MyIPA : public IPAInterface - * { - * ... - * }; - * - * struct ipa_context *ipaCreate() - * { - * return new IPAInterfaceWrapper(std::make_unique()); - * } - * \endcode - * - * The wrapper takes ownership of the IPAInterface and will automatically - * delete it when the wrapper is destroyed. - */ - -/** - * \brief Construct an IPAInterfaceWrapper wrapping \a interface - * \param[in] interface The interface to wrap - */ -IPAInterfaceWrapper::IPAInterfaceWrapper(std::unique_ptr interface) - : ipa_(std::move(interface)), callbacks_(nullptr), cb_ctx_(nullptr) -{ - ops = &operations_; - - ipa_->queueFrameAction.connect(this, &IPAInterfaceWrapper::queueFrameAction); -} - -void IPAInterfaceWrapper::destroy(struct ipa_context *_ctx) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - delete ctx; -} - -void *IPAInterfaceWrapper::get_interface(struct ipa_context *_ctx) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - return ctx->ipa_.get(); -} - -void IPAInterfaceWrapper::init(struct ipa_context *_ctx, - const struct ipa_settings *settings) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - IPASettings ipaSettings{ - .configurationFile = settings->configuration_file - }; - ctx->ipa_->init(ipaSettings); -} - -int IPAInterfaceWrapper::start(struct ipa_context *_ctx) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - return ctx->ipa_->start(); -} - -void IPAInterfaceWrapper::stop(struct ipa_context *_ctx) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - ctx->ipa_->stop(); -} - -void IPAInterfaceWrapper::register_callbacks(struct ipa_context *_ctx, - const struct ipa_callback_ops *callbacks, - void *cb_ctx) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - ctx->callbacks_ = callbacks; - ctx->cb_ctx_ = cb_ctx; -} - -void IPAInterfaceWrapper::configure(struct ipa_context *_ctx, - const struct ipa_sensor_info *sensor_info, - const struct ipa_stream *streams, - unsigned int num_streams, - const struct ipa_control_info_map *maps, - unsigned int num_maps) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - - ctx->serializer_.reset(); - - /* Translate the IPA sensor info. */ - CameraSensorInfo sensorInfo{}; - sensorInfo.model = sensor_info->model; - sensorInfo.bitsPerPixel = sensor_info->bits_per_pixel; - sensorInfo.activeAreaSize = { sensor_info->active_area.width, - sensor_info->active_area.height }; - sensorInfo.analogCrop = { sensor_info->analog_crop.left, - sensor_info->analog_crop.top, - sensor_info->analog_crop.width, - sensor_info->analog_crop.height }; - sensorInfo.outputSize = { sensor_info->output_size.width, - sensor_info->output_size.height }; - sensorInfo.pixelRate = sensor_info->pixel_rate; - sensorInfo.lineLength = sensor_info->line_length; - - /* Translate the IPA stream configurations map. */ - std::map ipaStreams; - - for (unsigned int i = 0; i < num_streams; ++i) { - const struct ipa_stream &stream = streams[i]; - - ipaStreams[stream.id] = { - stream.pixel_format, - Size(stream.width, stream.height), - }; - } - - /* Translate the IPA entity controls map. */ - std::map entityControls; - std::map infoMaps; - - for (unsigned int i = 0; i < num_maps; ++i) { - const struct ipa_control_info_map &ipa_map = maps[i]; - ByteStreamBuffer byteStream(ipa_map.data, ipa_map.size); - unsigned int id = ipa_map.id; - - infoMaps[id] = ctx->serializer_.deserialize(byteStream); - entityControls.emplace(id, infoMaps[id]); - } - - /* \todo Translate the ipaConfig and result. */ - IPAOperationData ipaConfig; - ctx->ipa_->configure(sensorInfo, ipaStreams, entityControls, ipaConfig, - nullptr); -} - -void IPAInterfaceWrapper::map_buffers(struct ipa_context *_ctx, - const struct ipa_buffer *_buffers, - size_t num_buffers) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - std::vector buffers(num_buffers); - - for (unsigned int i = 0; i < num_buffers; ++i) { - const struct ipa_buffer &_buffer = _buffers[i]; - IPABuffer &buffer = buffers[i]; - std::vector &planes = buffer.planes; - - buffer.id = _buffer.id; - - planes.resize(_buffer.num_planes); - for (unsigned int j = 0; j < _buffer.num_planes; ++j) { - planes[j].fd = FileDescriptor(_buffer.planes[j].dmabuf); - planes[j].length = _buffer.planes[j].length; - } - } - - ctx->ipa_->mapBuffers(buffers); -} - -void IPAInterfaceWrapper::unmap_buffers(struct ipa_context *_ctx, - const unsigned int *_ids, - size_t num_buffers) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - std::vector ids(_ids, _ids + num_buffers); - ctx->ipa_->unmapBuffers(ids); -} - -void IPAInterfaceWrapper::process_event(struct ipa_context *_ctx, - const struct ipa_operation_data *data) -{ - IPAInterfaceWrapper *ctx = static_cast(_ctx); - IPAOperationData opData; - - opData.operation = data->operation; - - opData.data.resize(data->num_data); - memcpy(opData.data.data(), data->data, - data->num_data * sizeof(*data->data)); - - opData.controls.resize(data->num_lists); - for (unsigned int i = 0; i < data->num_lists; ++i) { - const struct ipa_control_list *c_list = &data->lists[i]; - ByteStreamBuffer byteStream(c_list->data, c_list->size); - opData.controls[i] = ctx->serializer_.deserialize(byteStream); - } - - ctx->ipa_->processEvent(opData); -} - -void IPAInterfaceWrapper::queueFrameAction(unsigned int frame, - const IPAOperationData &data) -{ - if (!callbacks_) - return; - - struct ipa_operation_data c_data; - c_data.operation = data.operation; - c_data.data = data.data.data(); - c_data.num_data = data.data.size(); - - struct ipa_control_list control_lists[data.controls.size()]; - c_data.lists = control_lists; - c_data.num_lists = data.controls.size(); - - std::size_t listsSize = 0; - for (const auto &list : data.controls) - listsSize += serializer_.binarySize(list); - - std::vector binaryData(listsSize); - ByteStreamBuffer byteStreamBuffer(binaryData.data(), listsSize); - - unsigned int i = 0; - for (const auto &list : data.controls) { - struct ipa_control_list &c_list = control_lists[i]; - c_list.size = serializer_.binarySize(list); - - ByteStreamBuffer b = byteStreamBuffer.carveOut(c_list.size); - serializer_.serialize(list, b); - - c_list.data = b.base(); - } - - callbacks_->queue_frame_action(cb_ctx_, frame, c_data); -} - -#ifndef __DOXYGEN__ -/* - * This construct confuses Doygen and makes it believe that all members of the - * operations is a member of IPAInterfaceWrapper. It must thus be hidden. - */ -const struct ipa_context_ops IPAInterfaceWrapper::operations_ = { - .destroy = &IPAInterfaceWrapper::destroy, - .get_interface = &IPAInterfaceWrapper::get_interface, - .init = &IPAInterfaceWrapper::init, - .start = &IPAInterfaceWrapper::start, - .stop = &IPAInterfaceWrapper::stop, - .register_callbacks = &IPAInterfaceWrapper::register_callbacks, - .configure = &IPAInterfaceWrapper::configure, - .map_buffers = &IPAInterfaceWrapper::map_buffers, - .unmap_buffers = &IPAInterfaceWrapper::unmap_buffers, - .process_event = &IPAInterfaceWrapper::process_event, -}; -#endif - -} /* namespace libcamera */ diff --git a/src/ipa/libipa/ipa_interface_wrapper.h b/src/ipa/libipa/ipa_interface_wrapper.h deleted file mode 100644 index a1c70159..00000000 --- a/src/ipa/libipa/ipa_interface_wrapper.h +++ /dev/null @@ -1,61 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -/* - * Copyright (C) 2019, Google Inc. - * - * ipa_interface_wrapper.h - Image Processing Algorithm interface wrapper - */ -#ifndef __LIBCAMERA_IPA_INTERFACE_WRAPPER_H__ -#define __LIBCAMERA_IPA_INTERFACE_WRAPPER_H__ - -#include - -#include - -#include "libcamera/internal/control_serializer.h" - -namespace libcamera { - -class IPAInterfaceWrapper : public ipa_context -{ -public: - IPAInterfaceWrapper(std::unique_ptr interface); - -private: - static void destroy(struct ipa_context *ctx); - static void *get_interface(struct ipa_context *ctx); - static void init(struct ipa_context *ctx, - const struct ipa_settings *settings); - static int start(struct ipa_context *ctx); - static void stop(struct ipa_context *ctx); - static void register_callbacks(struct ipa_context *ctx, - const struct ipa_callback_ops *callbacks, - void *cb_ctx); - static void configure(struct ipa_context *ctx, - const struct ipa_sensor_info *sensor_info, - const struct ipa_stream *streams, - unsigned int num_streams, - const struct ipa_control_info_map *maps, - unsigned int num_maps); - static void map_buffers(struct ipa_context *ctx, - const struct ipa_buffer *c_buffers, - size_t num_buffers); - static void unmap_buffers(struct ipa_context *ctx, - const unsigned int *ids, - size_t num_buffers); - static void process_event(struct ipa_context *ctx, - const struct ipa_operation_data *data); - - static const struct ipa_context_ops operations_; - - void queueFrameAction(unsigned int frame, const IPAOperationData &data); - - std::unique_ptr ipa_; - const struct ipa_callback_ops *callbacks_; - void *cb_ctx_; - - ControlSerializer serializer_; -}; - -} /* namespace libcamera */ - -#endif /* __LIBCAMERA_IPA_INTERFACE_WRAPPER_H__ */ diff --git a/src/ipa/libipa/meson.build b/src/ipa/libipa/meson.build deleted file mode 100644 index 22626405..00000000 --- a/src/ipa/libipa/meson.build +++ /dev/null @@ -1,15 +0,0 @@ -# SPDX-License-Identifier: CC0-1.0 - -libipa_headers = files([ - 'ipa_interface_wrapper.h', -]) - -libipa_sources = files([ - 'ipa_interface_wrapper.cpp', -]) - -libipa_includes = include_directories('..') - -libipa = static_library('ipa', libipa_sources, - include_directories : ipa_includes, - dependencies : libcamera_dep) diff --git a/src/ipa/meson.build b/src/ipa/meson.build index 5a5de267..b11e5d23 100644 --- a/src/ipa/meson.build +++ b/src/ipa/meson.build @@ -15,8 +15,6 @@ config_h.set('IPA_CONFIG_DIR', config_h.set('IPA_MODULE_DIR', '"' + join_paths(get_option('prefix'), ipa_install_dir) + '"') -subdir('libipa') - ipa_sign = files('ipa-sign.sh') ipas = ['raspberrypi', 'rkisp1', 'vimc'] diff --git a/src/ipa/raspberrypi/meson.build b/src/ipa/raspberrypi/meson.build index 9445cd09..e912c668 100644 --- a/src/ipa/raspberrypi/meson.build +++ b/src/ipa/raspberrypi/meson.build @@ -10,7 +10,6 @@ rpi_ipa_deps = [ rpi_ipa_includes = [ ipa_includes, - libipa_includes, include_directories('controller') ] @@ -46,7 +45,6 @@ mod = shared_module(ipa_name, name_prefix : '', include_directories : rpi_ipa_includes, dependencies : rpi_ipa_deps, - link_with : libipa, install : true, install_dir : ipa_install_dir) diff --git a/src/ipa/raspberrypi/raspberrypi.cpp b/src/ipa/raspberrypi/raspberrypi.cpp index e09caa8d..ec04b23d 100644 --- a/src/ipa/raspberrypi/raspberrypi.cpp +++ b/src/ipa/raspberrypi/raspberrypi.cpp @@ -23,8 +23,6 @@ #include #include -#include - #include "libcamera/internal/camera_sensor.h" #include "libcamera/internal/log.h" #include "libcamera/internal/utils.h" From patchwork Tue Sep 15 14:20:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Elder X-Patchwork-Id: 9633 Return-Path: X-Original-To: parsemail@patchwork.libcamera.org Delivered-To: parsemail@patchwork.libcamera.org Received: from lancelot.ideasonboard.com (lancelot.ideasonboard.com [92.243.16.209]) by patchwork.libcamera.org (Postfix) with ESMTPS id 270E8BF01C for ; Tue, 15 Sep 2020 14:21:49 +0000 (UTC) Received: from lancelot.ideasonboard.com (localhost [IPv6:::1]) by lancelot.ideasonboard.com (Postfix) with ESMTP id EB0DC62E3D; Tue, 15 Sep 2020 16:21:48 +0200 (CEST) Authentication-Results: lancelot.ideasonboard.com; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="txHN8gA3"; dkim-atps=neutral Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lancelot.ideasonboard.com (Postfix) with ESMTPS id 67D2062E32 for ; Tue, 15 Sep 2020 16:21:47 +0200 (CEST) Received: from pyrite.rasen.tech (unknown [IPv6:2400:4051:61:600:2c71:1b79:d06d:5032]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id ABC4010C6; Tue, 15 Sep 2020 16:21:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1600179707; bh=iyjeV0BSwEVTg0w/laQbVt45nmFu0svAY8+ltV2mxXE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=txHN8gA3WSksjas/yYY9XzAkDbMBdhbMIbH8hLfz2oWoujGGz+BAzLvDtCi/netLk qGRX5jEn4oCddPq5RSab+fD6Y7+Ndzzbb08UrgGnovlSmpzAUaTZ3KZtWxlJS5qSFT mNTyJVzjbqQHG6HhydemYrEWDorh1/PrsFOthbb0= From: Paul Elder To: libcamera-devel@lists.libcamera.org Date: Tue, 15 Sep 2020 23:20:38 +0900 Message-Id: <20200915142038.28757-24-paul.elder@ideasonboard.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200915142038.28757-1-paul.elder@ideasonboard.com> References: <20200915142038.28757-1-paul.elder@ideasonboard.com> MIME-Version: 1.0 Subject: [libcamera-devel] [PATCH 23/23] libcamera: PipelineHandler: Move printing pipeline names to CameraManager X-BeenThere: libcamera-devel@lists.libcamera.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libcamera-devel-bounces@lists.libcamera.org Sender: "libcamera-devel" Since pipeline registration is done with declaring static factory objects, there is a risk that pipeline factories will be constructed before libcamera facilities are ready. For example, logging in the constructor of a pipeline handler factory may cause a segfault if threading isn't ready yet. Avoid this issue by moving printing the registration of the pipeline handler to the camera manager. Signed-off-by: Paul Elder Reviewed-by: Laurent Pinchart --- src/libcamera/camera_manager.cpp | 2 ++ src/libcamera/pipeline_handler.cpp | 3 --- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/src/libcamera/camera_manager.cpp b/src/libcamera/camera_manager.cpp index b8d3ccc8..e5af271d 100644 --- a/src/libcamera/camera_manager.cpp +++ b/src/libcamera/camera_manager.cpp @@ -142,6 +142,8 @@ void CameraManager::Private::createPipelineHandlers() PipelineHandlerFactory::factories(); for (PipelineHandlerFactory *factory : factories) { + LOG(Camera, Debug) + << "Found registered pipeline handler \"" << factory->name() << "\""; /* * Try each pipeline handler until it exhaust * all pipelines it can provide. diff --git a/src/libcamera/pipeline_handler.cpp b/src/libcamera/pipeline_handler.cpp index 918aea1e..894200ee 100644 --- a/src/libcamera/pipeline_handler.cpp +++ b/src/libcamera/pipeline_handler.cpp @@ -689,9 +689,6 @@ void PipelineHandlerFactory::registerType(PipelineHandlerFactory *factory) std::vector &factories = PipelineHandlerFactory::factories(); factories.push_back(factory); - - LOG(Pipeline, Debug) - << "Registered pipeline handler \"" << factory->name() << "\""; } /**