[{"id":13750,"web_url":"https://patchwork.libcamera.org/comment/13750/","msgid":"<20201117145540.2zxf4cefodqcck22@uno.localdomain>","date":"2020-11-17T14:55:40","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":3,"url":"https://patchwork.libcamera.org/api/people/3/","name":"Jacopo Mondi","email":"jacopo@jmondi.org"},"content":"Hi Paul,\n\nOn Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> Add templates to mojo to generate code for the IPC mechanism. These\n> templates generate:\n> - module header\n> - module serializer\n> - IPA proxy cpp, header, and worker\n>\n> Given an input data definition mojom file for a pipeline.\n>\n> Signed-off-by: Paul Elder \n>\n> ---\n> Changes in v4:\n> For the non-template files:\n> - rename IPA{pipeline_name}CallbackInterface to\n> IPA{pipeline_name}EventInterface\n> - to avoid the notion of \"callback\" and emphasize that it's an event\n> - add support for strings in custom structs\n> - add validation, that async methods must not have return values\n> - it throws exception and isn't very clear though...?\n> - rename controls to libcamera::{pipeline_name}::controls (controls is\n> now lowercase)\n> - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> - same for their corresponding template files\n> For the template files:\n> - fix spacing (now it's all {{var}} instead of some {{ var }})\n> - except if it's code, so code is still {{ code }}\n> - move inclusion of corresponding header to first in the inclusion list\n> - fix copy&paste errors\n> - change snake_case to camelCase in the generated code\n> - template code still uses snake_case\n> - change the generated command enums to an enum class, and make it\n> capitalized (instead of allcaps)\n> - add length checks to recvIPC (in proxy)\n> - fix some template spacing\n> - don't use const for PODs in function/signal parameters\n> - add the proper length checks to readPOD/appendPOD\n> - the helper functions for reading and writing PODs to and from\n> serialized data\n> - rename readUInt/appendUInt to readPOD/appendPOD\n> - add support for strings in custom structs\n>\n> Changes in v3:\n> - add support for namespaces\n> - fix enum assignment (used to have +1 for CMD applied to all enums)\n> - use readHeader, writeHeader, and eraseHeader as static class functions\n> of IPAIPCUnixSocket (in the proxy worker)\n> - add requirement that base controls *must* be defined in\n> libcamera::{pipeline_name}::Controls\n>\n> Changes in v2:\n> - mandate the main and callback interfaces, and init(), start(), stop()\n> and their parameters\n> - fix returning single pod value from IPC-called function\n> - add licenses\n> - improve auto-generated message\n> - other fixes related to serdes\n> ---\n> .../module_ipa_interface.h.tmpl | 113 ++++\n> .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> .../module_ipa_proxy.h.tmpl | 118 +++++\n> .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> .../module_ipa_serializer.h.tmpl | 44 ++\n> .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> 8 files changed, 1673 insertions(+)\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n>\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> new file mode 100644\n> index 00000000..a470b99e\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> @@ -0,0 +1,113 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n\nWondering if the copyright shouldn't be attribute to who writes the\ntemplate.\n\nApart from this, thanks for the monstrous patch, that's a template,\nI'm not reviewing it in detail\n\n> + *\n> + * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> +#define __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> +\n> +#include \n> +#include \n> +\n> +{% if has_map %}#include {% endif %}\n> +{% if has_array %}#include {% endif %}\n> +\n> +namespace libcamera {\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +enum class {{cmd_enum_name}} {\n> +\tExit = 0,\n> +{%- for method in interface_main.methods %}\n> +\t{{method.mojom_name|cap}} = {{loop.index}},\n> +{%- endfor %}\n> +};\n> +\n> +enum class {{cmd_event_enum_name}} {\n> +{%- for method in interface_cb.methods %}\n> +\t{{method.mojom_name|cap}} = {{loop.index}},\n> +{%- endfor %}\n> +};\n> +\n> +{% for enum in enums %}\n> +enum {{enum.mojom_name}} {\n> +{%- for field in enum.fields %}\n> +\t{{field.mojom_name}} = {{field.numeric_value}},\n> +{%- endfor %}\n> +};\n> +{% endfor %}\n> +\n> +{%- for struct in structs_nonempty %}\n> +struct {{struct.mojom_name}}\n> +{\n> +public:\n> +\t{{struct.mojom_name}}() {%- if struct|has_default_fields %}\n> +\t\t:{% endif %}\n> +{%- for field in struct.fields|with_default_values -%}\n> +{{\" \" if loop.first}}{{field.mojom_name}}_({{field|default_value}}){{\", \" if not loop.last}}\n> +{%- endfor %}\n> +\t{\n> +\t}\n> +\n> +\t~{{struct.mojom_name}}() {}\n> +\n> +\t{{struct.mojom_name}}(\n> +{%- for field in struct.fields -%}\n> +{{field|name}} {{field.mojom_name}}{{\", \" if not loop.last}}\n> +{%- endfor -%}\n> +)\n> +\t\t:\n> +{%- for field in struct.fields -%}\n> +{{\" \" if loop.first}}{{field.mojom_name}}_({{field.mojom_name}}){{\", \" if not loop.last}}\n> +{%- endfor %}\n> +\t{\n> +\t}\n> +{% for field in struct.fields %}\n> +\t{{field|name}} {{field.mojom_name}}_;\n> +{%- endfor %}\n> +};\n> +{% endfor %}\n> +\n> +{#-\n> +Any consts or #defines should be moved to the mojom file when possible.\n> +If anything needs to be #included, then {{module_name}}.h needs to have the\n> +#include.\n> +#}\n> +class {{interface_name}} : public IPAInterface\n> +{\n> +public:\n> +\tvirtual ~{{interface_name}}() {}\n> +{% for method in interface_main.methods %}\n> +\tvirtual {{method|method_return_value}} {{method.mojom_name}}(\n> +{%- for param in method|method_parameters %}\n> +\t\t{{param}}{{- \",\" if not loop.last}}\n> +{%- endfor -%}\n> +) = 0;\n> +{% endfor %}\n> +\n> +{%- for method in interface_cb.methods %}\n> +\tSignal<\n> +{%- for param in method.parameters -%}\n> +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> +\t\t{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +> {{method.mojom_name}};\n> +{% endfor -%}\n> +};\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> new file mode 100644\n> index 00000000..9328c7ca\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> @@ -0,0 +1,238 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}.cpp - Image Processing Algorithm proxy for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +#include \"libcamera/internal/ipa_module.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/log.h\"\n> +#include \"libcamera/internal/process.h\"\n> +#include \"libcamera/internal/thread.h\"\n> +\n> +namespace libcamera {\n> +\n> +LOG_DECLARE_CATEGORY(IPAProxy)\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +{{proxy_name}}::{{proxy_name}}(IPAModule *ipam, bool isolate)\n> +\t: IPAProxy(ipam), running_(false),\n> +\t isolate_(isolate)\n> +{\n> +\tLOG(IPAProxy, Debug)\n> +\t\t<< \"initializing {{module_name}} proxy: loading IPA from \"\n> +\t\t<< ipam->path();\n> +\n> +\tif (isolate_) {\n> +\t\tconst std::string proxyWorkerPath = resolvePath(\"ipa_proxy_{{module_name}}\");\n> +\t\tif (proxyWorkerPath.empty()) {\n> +\t\t\tLOG(IPAProxy, Error)\n> +\t\t\t\t<< \"Failed to get proxy worker path\";\n> +\t\t\treturn;\n> +\t\t}\n> +\n> +\t\tipc_ = std::make_unique(ipam->path().c_str(), proxyWorkerPath.c_str());\n> +\t\tif (!ipc_->isValid()) {\n> +\t\t\tLOG(IPAProxy, Error) << \"Failed to create IPAIPC\";\n> +\t\t\treturn;\n> +\t\t}\n> +\n> +{% if interface_cb.methods|length > 0 %}\n> +\t\tipc_->recvIPC.connect(this, &{{proxy_name}}::recvIPC);\n> +{% endif %}\n> +\n> +\t\tvalid_ = true;\n> +\t\treturn;\n> +\t}\n> +\n> +\tif (!ipam->load())\n> +\t\treturn;\n> +\n> +\tIPAInterface *ipai = ipam->createInterface();\n> +\tif (!ipai) {\n> +\t\tLOG(IPAProxy, Error)\n> +\t\t\t<< \"Failed to create IPA context for \" << ipam->path();\n> +\t\treturn;\n> +\t}\n> +\n> +\tipa_ = std::unique_ptr<{{interface_name}}>(dynamic_cast<{{interface_name}} *>(ipai));\n> +\tproxy_.setIPA(ipa_.get());\n> +\n> +{% for method in interface_cb.methods %}\n> +\tipa_->{{method.mojom_name}}.connect(this, &{{proxy_name}}::{{method.mojom_name}}Thread);\n> +{%- endfor %}\n> +\n> +\tvalid_ = true;\n> +}\n> +\n> +{{proxy_name}}::~{{proxy_name}}()\n> +{\n> +\tif (isolate_)\n> +\t\tipc_->sendAsync(static_cast({{cmd_enum_name}}::Exit), {}, {});\n> +}\n> +\n> +{% if interface_cb.methods|length > 0 %}\n> +void {{proxy_name}}::recvIPC(std::vector &data, std::vector &fds)\n> +{\n> +\tif (data.size() < 8) {\n> +\t\tLOG(IPAProxy, Error)\n> +\t\t\t<< \"Didn't receive enough bytes to parse event\";\n> +\t\treturn;\n> +\t}\n> +\n> +\t{{cmd_event_enum_name}} cmd = static_cast<{{cmd_event_enum_name}}>((\n> +\t\tdata[0]) | (data[1] << 8) | (data[2] << 16) | (data[3] << 24));\n> +\n> +\t/* Need to skip another 4 bytes for the sequence number. */\n> +\tstd::vector::iterator vec = data.begin() + 8;\n> +\tsize_t dataSize = data.size() - 8;\n> +\tswitch (cmd) {\n> +{%- for method in interface_cb.methods %}\n> +\tcase {{cmd_event_enum_name}}::{{method.mojom_name|cap}}: {\n> +\t\t{{method.mojom_name}}IPC(vec, dataSize, fds);\n> +\t\tbreak;\n> +\t}\n> +{%- endfor %}\n> +\tdefault:\n> +\t\tLOG(IPAProxy, Error) << \"Unknown command \" << static_cast(cmd);\n> +\t}\n> +}\n> +{%- endif %}\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method)}}\n> +{\n> +\tif (isolate_)\n> +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}IPC(\n> +{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +\telse\n> +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}Thread(\n> +{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +}\n> +\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> +{\n> +{%- if method.mojom_name == \"init\" %}\n> +\t{{proxy_funcs.init_thread_body()}}\n> +{%- elif method.mojom_name == \"start\" %}\n> +\t{{proxy_funcs.start_thread_body()}}\n> +{%- elif method.mojom_name == \"stop\" %}\n> +\t{{proxy_funcs.stop_thread_body()}}\n> +{%- elif not method|is_async %}\n> +\tipa_->{{method.mojom_name}}(\n> +\t{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +\t{%- endfor -%}\n> +);\n> +{% elif method|is_async %}\n> +\tproxy_.invokeMethod(&ThreadProxy::{{method.mojom_name}}, ConnectionTypeQueued,\n> +\t{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +\t{%- endfor -%}\n> +);\n> +{%- endif %}\n> +}\n> +\n> +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\")}}\n> +{\n> +{%- set has_input = true if method|method_param_inputs|length > 0 %}\n> +{%- set has_output = true if method|method_param_outputs|length > 0 or method|method_return_value != \"void\" %}\n> +{%- if has_input %}\n> +\tstd::vector _ipcInputBuf;\n> +{%- if method|method_input_has_fd %}\n> +\tstd::vector _ipcInputFds;\n> +{%- endif %}\n> +{%- endif %}\n> +{%- if has_output %}\n> +\tstd::vector _ipcOutputBuf;\n> +{%- if method|method_output_has_fd %}\n> +\tstd::vector _ipcOutputFds;\n> +{%- endif %}\n> +{%- endif %}\n> +\n> +{{proxy_funcs.serialize_call(method|method_param_inputs, '_ipcInputBuf', '_ipcInputFds', base_controls)}}\n> +\n> +{%- set input_buf = \"_ipcInputBuf\" if has_input else \"{}\" %}\n> +{%- set fds_buf = \"_ipcInputFds\" if method|method_input_has_fd else \"{}\" %}\n> +{%- set cmd = cmd_enum_name + \"::\" + method.mojom_name|cap %}\n> +{% if method|is_async %}\n> +\tint ret = ipc_->sendAsync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}});\n> +{%- else %}\n> +\tint ret = ipc_->sendSync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}}\n> +{{- \", &_ipcOutputBuf\" if has_output -}}\n> +{{- \", &_ipcOutputFds\" if has_output and method|method_output_has_fd -}}\n> +);\n> +{%- endif %}\n> +\tif (ret < 0) {\n> +\t\tLOG(IPAProxy, Error) << \"Failed to call {{method.mojom_name}}\";\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\treturn static_cast<{{method|method_return_value}}>(ret);\n> +{%- else %}\n> +\t\treturn;\n> +{%- endif %}\n> +\t}\n> +{% if method|method_return_value != \"void\" %}\n> +\treturn IPADataSerializer<{{method.response_parameters|first|name}}>::deserialize(_ipcOutputBuf, 0);\n> +{% elif method|method_param_outputs|length > 0 %}\n> +{{proxy_funcs.deserialize_call(method|method_param_outputs, '_ipcOutputBuf', '_ipcOutputFds')}}\n> +{% endif -%}\n> +}\n> +\n> +{% endfor %}\n> +\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> +{\n> +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> +}\n> +\n> +void {{proxy_name}}::{{method.mojom_name}}IPC(\n> +\tstd::vector::iterator data,\n> +\tsize_t dataSize,\n> +\t[[maybe_unused]] std::vector &fds)\n> +{\n> +{%- for param in method.parameters %}\n> +\t{{param|name}} {{param.mojom_name}};\n> +{%- endfor %}\n> +{{proxy_funcs.deserialize_call(method.parameters, 'data', 'fds', false, false, true, 'dataSize')}}\n> +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> +}\n> +{% endfor %}\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> new file mode 100644\n> index 00000000..3fb7192f\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> @@ -0,0 +1,118 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}.h - Image Processing Algorithm proxy for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> +#define __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/thread.h\"\n> +\n> +namespace libcamera {\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +class {{proxy_name}} : public IPAProxy, public {{interface_name}}, public Object\n> +{\n> +public:\n> +\t{{proxy_name}}(IPAModule *ipam, bool isolate);\n> +\t~{{proxy_name}}();\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"\", false, true)|indent(8, true)}};\n> +{% endfor %}\n> +\n> +{%- for method in interface_cb.methods %}\n> +\tSignal<\n> +{%- for param in method.parameters -%}\n> +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> +\t\t{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +> {{method.mojom_name}};\n> +{% endfor %}\n> +\n> +private:\n> +\tvoid recvIPC(std::vector &data, std::vector &fds);\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\", false)|indent(8, true)}};\n> +{% endfor %}\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> +\tvoid {{method.mojom_name}}IPC(\n> +\t\tstd::vector::iterator data,\n> +\t\tsize_t dataSize,\n> +\t\tstd::vector &fds);\n> +{% endfor %}\n> +\n> +\t/* Helper class to invoke async functions in another thread. */\n> +\tclass ThreadProxy : public Object\n> +\t{\n> +\tpublic:\n> +\t\tvoid setIPA({{interface_name}} *ipa)\n> +\t\t{\n> +\t\t\tipa_ = ipa;\n> +\t\t}\n> +\n> +\t\tint start()\n> +\t\t{\n> +\t\t\treturn ipa_->start();\n> +\t\t}\n> +\n> +\t\tvoid stop()\n> +\t\t{\n> +\t\t\tipa_->stop();\n> +\t\t}\n> +{% for method in interface_main.methods %}\n> +{%- if method|is_async %}\n> +\t\t{{proxy_funcs.func_sig(proxy_name, method, \"\", false)|indent(16)}}\n> +\t\t{\n> +\t\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> +\t\t}\n> +{%- endif %}\n> +{%- endfor %}\n> +\n> +\tprivate:\n> +\t\t{{interface_name}} *ipa_;\n> +\t};\n> +\n> +\tbool running_;\n> +\tThread thread_;\n> +\tThreadProxy proxy_;\n> +\tstd::unique_ptr<{{interface_name}}> ipa_;\n> +\n> +\tconst bool isolate_;\n> +\n> +\tstd::unique_ptr ipc_;\n> +\n> +\tControlSerializer controlSerializer_;\n> +};\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> new file mode 100644\n> index 00000000..dca4f99d\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> @@ -0,0 +1,187 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}_worker.cpp - Image Processing Algorithm proxy worker for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/camera_sensor.h\"\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_module.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/log.h\"\n> +#include \"libcamera/internal/thread.h\"\n> +\n> +using namespace libcamera;\n> +\n> +LOG_DEFINE_CATEGORY({{proxy_name}}Worker)\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace -%}\n> +using namespace {{ns}};\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +struct CallData {\n> +\tIPCUnixSocket::Payload *response;\n> +\tbool done;\n> +};\n> +\n> +{{interface_name}} *ipa_;\n> +IPCUnixSocket socket_;\n> +\n> +ControlSerializer controlSerializer_;\n> +\n> +bool exit_ = false;\n> +\n> +void readyRead(IPCUnixSocket *socket)\n> +{\n> +\tIPCUnixSocket::Payload _message, _response;\n> +\tint _retRecv = socket->receive(&_message);\n> +\tif (_retRecv) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"Receive message failed\" << _retRecv;\n> +\t\treturn;\n> +\t}\n> +\n> +\tuint32_t _cmdUint, _seq;\n> +\tstd::tie(_cmdUint, _seq) = IPAIPCUnixSocket::readHeader(_message);\n> +\tIPAIPCUnixSocket::eraseHeader(_message);\n> +\n> +\t{{cmd_enum_name}} _cmd = static_cast<{{cmd_enum_name}}>(_cmdUint);\n> +\n> +\tswitch (_cmd) {\n> +\tcase {{cmd_enum_name}}::Exit: {\n> +\t\texit_ = true;\n> +\t\tbreak;\n> +\t}\n> +\n> +{% for method in interface_main.methods %}\n> +\tcase {{cmd_enum_name}}::{{method.mojom_name|cap}}: {\n> +\t{{proxy_funcs.deserialize_call(method|method_param_inputs, '_message.data', '_message.fds', false, true)|indent(8, true)}}\n> +{% for param in method|method_param_outputs %}\n> +\t\t{{param|name}} {{param.mojom_name}};\n> +{% endfor %}\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\t{{method|method_return_value}} _callRet = ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> +{%- else %}\n> +\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}}\n> +{{- \", \" if method|method_param_outputs|params_comma_sep -}}\n> +{%- for param in method|method_param_outputs -%}\n> +&{{param.mojom_name}}{{\", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +{%- endif %}\n> +{% if not method|is_async %}\n> +\t\tIPAIPCUnixSocket::writeHeader(_response, _cmdUint, _seq);\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\tstd::vector _callRetBuf;\n> +\t\tstd::tie(_callRetBuf, std::ignore) =\n> +\t\t\tIPADataSerializer<{{method|method_return_value}}>::serialize(_callRet);\n> +\t\t_response.data.insert(_response.data.end(), _callRetBuf.begin(), _callRetBuf.end());\n> +{%- else %}\n> +\t{{proxy_funcs.serialize_call(method|method_param_outputs, \"_response.data\", \"_response.fds\", base_controls)|indent(8, true)}}\n> +{%- endif %}\n> +\t\tint _ret = socket_.send(_response);\n> +\t\tif (_ret < 0) {\n> +\t\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t\t<< \"Reply to {{method.mojom_name}}() failed\" << _ret;\n> +\t\t}\n> +\t\tLOG({{proxy_name}}Worker, Debug) << \"Done replying to {{method.mojom_name}}()\";\n> +{%- endif %}\n> +\t\tbreak;\n> +\t}\n> +{% endfor %}\n> +\tdefault:\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"Unknown command \" << _cmdUint;\n> +\t}\n> +}\n> +\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"\", false)}}\n> +{\n> +\tIPCUnixSocket::Payload _message;\n> +\n> +\tIPAIPCUnixSocket::writeHeader(_message, static_cast({{cmd_event_enum_name}}::{{method.mojom_name|cap}}), 0);\n> +\t{{proxy_funcs.serialize_call(method|method_param_inputs, \"_message.data\", \"_message.fds\", base_controls)}}\n> +\n> +\tsocket_.send(_message);\n> +\n> +\tLOG({{proxy_name}}Worker, Debug) << \"{{method.mojom_name}} done\";\n> +}\n> +{% endfor %}\n> +\n> +int main(int argc, char **argv)\n> +{\n> +\t/* Uncomment this for debugging. */\n> +#if 0\n> +\tstd::string logPath = \"/tmp/libcamera.worker.\" +\n> +\t\t\t std::to_string(getpid()) + \".log\";\n> +\tlogSetFile(logPath.c_str());\n> +#endif\n> +\n> +\tif (argc < 3) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"Tried to start worker with no args\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\n> +\tint fd = std::stoi(argv[2]);\n> +\tLOG({{proxy_name}}Worker, Info)\n> +\t\t<< \"Starting worker for IPA module \" << argv[1]\n> +\t\t<< \" with IPC fd = \" << fd;\n> +\n> +\tstd::unique_ptr ipam = std::make_unique(argv[1]);\n> +\tif (!ipam->isValid() || !ipam->load()) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"IPAModule \" << argv[1] << \" isn't valid\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\n> +\tif (socket_.bind(fd) < 0) {\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"IPC socket binding failed\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\tsocket_.readyRead.connect(&readyRead);\n> +\n> +\tipa_ = dynamic_cast<{{interface_name}} *>(ipam->createInterface());\n> +\tif (!ipa_) {\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"Failed to create IPA interface instance\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +{% for method in interface_cb.methods %}\n> +\tipa_->{{method.mojom_name}}.connect(&{{method.mojom_name}});\n> +{%- endfor %}\n> +\n> +\tLOG({{proxy_name}}Worker, Debug) << \"Proxy worker successfully started\";\n> +\n> +\t/* \\todo upgrade listening loop */\n> +\tEventDispatcher *dispatcher = Thread::current()->eventDispatcher();\n> +\twhile (!exit_)\n> +\t\tdispatcher->processEvents();\n> +\n> +\tdelete ipa_;\n> +\tsocket_.close();\n> +\n> +\treturn 0;\n> +}\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> new file mode 100644\n> index 00000000..675f9adf\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> @@ -0,0 +1,44 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"serializer.tmpl\" as serializer -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * {{module_name}}_serializer.h - Image Processing Algorithm data serializer for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> +\n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +\n> +#include \n> +#include \n> +\n> +namespace libcamera {\n> +\n> +LOG_DECLARE_CATEGORY(IPADataSerializer)\n> +{% for struct in structs_nonempty %}\n> +template<>\n> +class IPADataSerializer<{{struct|name_full(namespace_str)}}>\n> +{\n> +public:\n> +{{- serializer.serializer(struct, base_controls, namespace_str)}}\n> +{%- if struct|has_fd %}\n> +{{serializer.deserializer_fd(struct, namespace_str)}}\n> +{%- else %}\n> +{{serializer.deserializer_no_fd(struct, namespace_str)}}\n> +{%- endif %}\n> +};\n> +{% endfor %}\n> +\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> new file mode 100644\n> index 00000000..f6836034\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> @@ -0,0 +1,205 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{#\n> + # \\brief Generate fuction prototype\n> + #\n> + # \\param class Class name\n> + # \\param method mojom Method object\n> + # \\param suffix Suffix to append to \\a method function name\n> + # \\param need_class_name True to generate class name with function\n> + # \\param override True to generate override tag after the function prototype\n> + #}\n> +{%- macro func_sig(class, method, suffix, need_class_name = true, override = false) -%}\n> +{{method|method_return_value}} {{class + \"::\" if need_class_name}}{{method.mojom_name}}{{suffix}}(\n> +{%- for param in method|method_parameters %}\n> +\t{{param}}{{- \",\" if not loop.last}}\n> +{%- endfor -%}\n> +){{\" override\" if override}}\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA init() function for thread\n> + #}\n> +{%- macro init_thread_body() -%}\n> +\tint ret = ipa_->init(settings);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tproxy_.moveToThread(&thread_);\n> +\n> +\treturn 0;\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA start() function for thread\n> + #}\n> +{%- macro start_thread_body() -%}\n> +\trunning_ = true;\n> +\tthread_.start();\n> +\n> +\treturn proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking);\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA stop() function for thread\n> + #}\n> +{%- macro stop_thread_body() -%}\n> +\tif (!running_)\n> +\t\treturn;\n> +\n> +\trunning_ = false;\n> +\n> +\tproxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking);\n> +\n> +\tthread_.exit();\n> +\tthread_.wait();\n> +{%- endmacro -%}\n> +\n> +\n> +{#\n> + # \\brief Serialize multiple objects into data buffer and fd vector\n> + #\n> + # Generate code to serialize multiple objects, as specified in \\a params\n> + # (which are the parameters to some function), into \\a buf data buffer and\n> + # \\a fds fd vector.\n> + # This code is meant to be used by the proxy, for serializing prior to IPC calls.\n> + #}\n> +{%- macro serialize_call(params, buf, fds, base_controls) %}\n> +{%- for param in params %}\n> +\tstd::vector {{param.mojom_name}}Buf;\n> +{%- if param|has_fd %}\n> +\tstd::vector {{param.mojom_name}}Fds;\n> +\tstd::tie({{param.mojom_name}}Buf, {{param.mojom_name}}Fds) =\n> +{%- else %}\n> +\tstd::tie({{param.mojom_name}}Buf, std::ignore) =\n> +{%- endif %}\n> +{%- if param|is_controls %}\n> +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}, {{param.mojom_name}}.infoMap() ? *{{param.mojom_name}}.infoMap() : {{base_controls}}\n> +{%- else %}\n> +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}\n> +{%- endif %}\n> +{{- \", &controlSerializer_\" if param|needs_control_serializer -}}\n> +);\n> +{%- endfor %}\n> +\n> +{%- if params|length > 1 %}\n> +{%- for param in params %}\n> +\tappendPOD({{buf}}, {{param.mojom_name}}Buf.size());\n> +{%- if param|has_fd %}\n> +\tappendPOD({{buf}}, {{param.mojom_name}}Fds.size());\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endif %}\n> +\n> +{%- for param in params %}\n> +\t{{buf}}.insert({{buf}}.end(), {{param.mojom_name}}Buf.begin(), {{param.mojom_name}}Buf.end());\n> +{%- endfor %}\n> +\n> +{%- for param in params %}\n> +{%- if param|has_fd %}\n> +\t{{fds}}.insert({{fds}}.end(), {{param.mojom_name}}Fds.begin(), {{param.mojom_name}}Fds.end());\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endmacro -%}\n> +\n> +\n> +{#\n> + # \\brief Deserialize a single object from data buffer and fd vector\n> + #\n> + # \\param pointer True deserializes the object into a dereferenced pointer\n> + # \\param iter True treats \\a buf as an iterator instead of a vector\n> + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> + #\n> + # Generate code to deserialize a single object, as specified in \\a param,\n> + # from \\a buf data buffer and \\a fds fd vector.\n> + # This code is meant to be used by macro deserialize_call.\n> + #}\n> +{%- macro deserialize_param(param, pointer, loop, buf, fds, iter, data_size) -%}\n> +{{\"*\" if pointer}}{{param.mojom_name}} = IPADataSerializer<{{param|name}}>::deserialize(\n> +{%- if not iter %}\n> +\t{{buf}}.begin() + {{param.mojom_name}}Start,\n> +{%- else %}\n> +\t{{buf}} + {{param.mojom_name}}Start,\n> +{%- endif %}\n> +{%- if loop.last and not iter %}\n> +\t{{buf}}.end()\n> +{%- elif not iter %}\n> +\t{{buf}}.begin() + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> +{%- elif iter and loop.length == 1 %}\n> +\t{{buf}} + {{data_size}}\n> +{%- else %}\n> +\t{{buf}} + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> +{%- endif -%}\n> +{{- \",\" if param|has_fd}}\n> +{%- if param|has_fd %}\n> +\t{{fds}}.begin() + {{param.mojom_name}}FdStart,\n> +{%- if loop.last %}\n> +\t{{fds}}.end()\n> +{%- else %}\n> +\t{{fds}}.begin() + {{param.mojom_name}}FdStart + {{param.mojom_name}}FdsSize\n> +{%- endif -%}\n> +{%- endif -%}\n> +{{- \",\" if param|needs_control_serializer}}\n> +{%- if param|needs_control_serializer %}\n> +\t&controlSerializer_\n> +{%- endif -%}\n> +);\n> +{%- endmacro -%}\n> +\n> +\n> +{#\n> + # \\brief Deserialize multiple objects from data buffer and fd vector\n> + #\n> + # \\param pointer True deserializes objects into pointers, and adds a null check.\n> + # \\param declare True declares the objects in addition to deserialization.\n> + # \\param iter True treats \\a buf as an iterator instead of a vector\n> + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> + #\n> + # Generate code to deserialize multiple objects, as specified in \\a params\n> + # (which are the parameters to some function), from \\a buf data buffer and\n> + # \\a fds fd vector.\n> + # This code is meant to be used by the proxy, for deserializing after IPC calls.\n> + #}\n> +{%- macro deserialize_call(params, buf, fds, pointer = true, declare = false, iter = false, data_size = '') -%}\n> +{% set ns = namespace(size_offset = 0) %}\n> +{%- if params|length > 1 %}\n> +{%- for param in params %}\n> +\t[[maybe_unused]] size_t {{param.mojom_name}}BufSize = readPOD({{buf}}, {{ns.size_offset}}\n> +{%- if iter -%}\n> +, {{buf}} + {{data_size}}\n> +{%- endif -%}\n> +);\n> +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> +{%- if param|has_fd %}\n> +\t[[maybe_unused]] size_t {{param.mojom_name}}FdsSize = readPOD({{buf}}, {{ns.size_offset}}\n> +{%- if iter -%}\n> +, {{buf}} + {{data_size}}\n> +{%- endif -%}\n> +);\n> +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endif %}\n> +{% for param in params %}\n> +{%- if loop.first %}\n> +\tsize_t {{param.mojom_name}}Start = {{ns.size_offset}};\n> +{%- else %}\n> +\tsize_t {{param.mojom_name}}Start = {{loop.previtem.mojom_name}}Start + {{loop.previtem.mojom_name}}BufSize;\n> +{%- endif %}\n> +{%- endfor %}\n> +{% for param in params|with_fds %}\n> +{%- if loop.first %}\n> +\tsize_t {{param.mojom_name}}FdStart = 0;\n> +{%- elif not loop.last %}\n> +\tsize_t {{param.mojom_name}}FdStart = {{loop.previtem.mojom_name}}FdStart + {{loop.previtem.mojom_name}}FdsSize;\n> +{%- endif %}\n> +{%- endfor %}\n> +{% for param in params %}\n> +\t{%- if pointer %}\n> +\tif ({{param.mojom_name}}) {\n> +{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(16, True)}}\n> +\t}\n> +\t{%- else %}\n> +\t{{param|name + \" \" if declare}}{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(8)}}\n> +\t{%- endif %}\n> +{% endfor %}\n> +{%- endmacro -%}\n> diff --git a/utils/ipc/generators/libcamera_templates/serializer.tmpl b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> new file mode 100644\n> index 00000000..51dbeb0e\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> @@ -0,0 +1,280 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{# Turn this into a C macro? #}\n> +{#\n> + # \\brief Verify that there is enough bytes to deserialize\n> + #\n> + # Generate code that verifies that \\a size is not greater than \\a dataSize.\n> + # Otherwise log an error with \\a name and \\a typename.\n> + #}\n> +{%- macro check_data_size(size, dataSize, name, typename) %}\n> +\t\tif ({{size}} > {{dataSize}}) {\n> +\t\t\tLOG(IPADataSerializer, Error)\n> +\t\t\t\t<< \"Failed to deserialize {{name}}: not enough {{typename}}, expected \"\n> +\t\t\t\t<< ({{size}}) << \", got \" << ({{dataSize}});\n> +\t\t\treturn ret;\n> +\t\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Serialize some field into return vector\n> + #\n> + # Generate code to serialize \\a field into retData, including size of the\n> + # field and fds (where appropriate). \\a base_controls indicates the\n> + # default ControlInfoMap in the event that the ControlList does not have one.\n> + # This code is meant to be used by the IPADataSerializer specialization.\n> + #}\n> +{%- macro serializer_field(field, base_controls, namespace, loop) %}\n> +{%- if field|is_pod or field|is_enum %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t{%- if field|is_pod %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t{%- elif field|is_enum %}\n> +\t\t\tIPADataSerializer::serialize(data.{{field.mojom_name}}_);\n> +\t{%- endif %}\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +{%- elif field|is_fd %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t\tstd::vector {{field.mojom_name}}Fds;\n> +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> +{%- elif field|is_controls %}\n> +\t\tif (data.{{field.mojom_name}}_.size() > 0) {\n> +\t\t\tstd::vector {{field.mojom_name}};\n> +\t\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_,\n> +\t\t\t\t\tdata.{{field.mojom_name}}_.infoMap() ? *data.{{field.mojom_name}}_.infoMap() : {{base_controls}},\n> +\t\t\t\t\tcs);\n> +\t\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> +\t\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t\t} else {\n> +\t\t\tappendPOD(retData, 0);\n> +\t\t}\n> +{%- elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t{%- if field|has_fd %}\n> +\t\tstd::vector {{field.mojom_name}}Fds;\n> +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> +\t{%- else %}\n> +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t{%- endif %}\n> +\t{%- if field|is_array or field|is_map %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, cs);\n> +\t{%- elif field|is_str %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t{%- else %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::serialize(data.{{field.mojom_name}}_, cs);\n> +\t{%- endif %}\n> +\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> +\t{%- if field|has_fd %}\n> +\t\tappendPOD(retData, {{field.mojom_name}}Fds.size());\n> +\t{%- endif %}\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t{%- if field|has_fd %}\n> +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> +\t{%- endif %}\n> +{%- else %}\n> +\t\t/* Unknown serialization for {{field.mojom_name}}. */\n> +{%- endif %}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Deserialize some field into return struct\n> + #\n> + # Generate code to deserialize \\a field into object ret.\n> + # This code is meant to be used by the IPADataSerializer specialization.\n> + #}\n> +{%- macro deserializer_field(field, namespace, loop) %}\n> +{% if field|is_pod or field|is_enum %}\n> +\t{%- set field_size = (field|bit_width|int / 8)|int %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\t{%- if field|is_pod %}\n> +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer<{{field|name}}>::deserialize(m, m + {{field_size}}));\n> +\t\t{%- else %}\n> +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer::deserialize(m, m + {{field_size}}));\n> +\t\t{%- endif %}\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- endif %}\n> +{% elif field|is_fd %}\n> +\t{%- set field_size = 1 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tret.{{field.mojom_name}}_ = IPADataSerializer<{{field|name}}>::deserialize(m, m + 1, n, n + 1, cs);\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t\tn += ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> +\t\tfdsSize -= ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> +\t{%- endif %}\n> +{% elif field|is_controls %}\n> +\t{%- set field_size = 4 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> +\t{%- set field_size = '4 + ' + field.mojom_name + 'Size' -%}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tif ({{field.mojom_name}}Size > 0)\n> +\t\t\tret.{{field.mojom_name}}_ =\n> +\t\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- endif %}\n> +{% elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> +\t{%- set field_size = 4 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> +\t{%- if field|has_fd %}\n> +\t{%- set field_size = 8 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'FdsSize', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}FdsSize = readPOD(m, 4, data.end());\n> +\t\t{{- check_data_size(field.mojom_name + 'FdsSize', 'fdsSize', field.mojom_name, 'fds')}}\n> +\t{%- endif %}\n> +\t{%- set field_size = field|has_fd|choose('8 + ', '4 + ') + field.mojom_name + 'Size' -%}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tret.{{field.mojom_name}}_ =\n> +\t{%- if field|is_str %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size);\n> +\t{%- elif field|has_fd and (field|is_array or field|is_map) %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> +\t{%- elif field|has_fd and (not (field|is_array or field|is_map)) %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> +\t{%- elif (not field|has_fd) and (field|is_array or field|is_map) %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- else %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- endif %}\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- if field|has_fd %}\n> +\t\tn += {{field.mojom_name}}FdsSize;\n> +\t\tfdsSize -= {{field.mojom_name}}FdsSize;\n> +\t{%- endif %}\n> +\t{%- endif %}\n> +{% else %}\n> +\t\t/* Unknown deserialization for {{field.mojom_name}}. */\n> +{%- endif %}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Serialize a struct\n> + #\n> + # Generate code for IPADataSerializer specialization, for serializing\n> + # \\a struct. \\a base_controls indicates the default ControlInfoMap\n> + # in the event that the ControlList does not have one.\n> + #}\n> +{%- macro serializer(struct, base_controls, namespace) %}\n> +\tstatic std::tuple, std::vector>\n> +\tserialize(const {{struct|name_full(namespace)}} &data,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector retData;\n> +{%- if struct|has_fd %}\n> +\t\tstd::vector retFds;\n> +{%- endif %}\n> +{%- for field in struct.fields %}\n> +{{serializer_field(field, base_controls, namespace, loop)}}\n> +{%- endfor %}\n> +{% if struct|has_fd %}\n> +\t\treturn {retData, retFds};\n> +{%- else %}\n> +\t\treturn {retData, {}};\n> +{%- endif %}\n> +\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Deserialize a struct that has fds\n> + #\n> + # Generate code for IPADataSerializer specialization, for deserializing\n> + # \\a struct, in the case that \\a struct has file descriptors.\n> + #}\n> +{%- macro deserializer_fd(struct, namespace) %}\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector &data,\n> +\t\t std::vector &fds,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\t{{struct|name_full(namespace)}} ret;\n> +\t\tstd::vector::iterator m = data.begin();\n> +\t\tstd::vector::iterator n = fds.begin();\n> +\n> +\t\tsize_t dataSize = data.size();\n> +\t\tsize_t fdsSize = fds.size();\n> +{%- for field in struct.fields -%}\n> +{{deserializer_field(field, namespace, loop)}}\n> +{%- endfor %}\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector::iterator dataBegin,\n> +\t\t std::vector::iterator dataEnd,\n> +\t\t std::vector::iterator fdsBegin,\n> +\t\t std::vector::iterator fdsEnd,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector data(dataBegin, dataEnd);\n> +\t\tstd::vector fds(fdsBegin, fdsEnd);\n> +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, fds, cs);\n> +\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Deserialize a struct that has no fds\n> + #\n> + # Generate code for IPADataSerializer specialization, for deserializing\n> + # \\a struct, in the case that \\a struct does not have file descriptors.\n> + #}\n> +{%- macro deserializer_no_fd(struct, namespace) %}\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector &data,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\t{{struct|name_full(namespace)}} ret;\n> +\t\tstd::vector::iterator m = data.begin();\n> +\n> +\t\tsize_t dataSize = data.size();\n> +{%- for field in struct.fields -%}\n> +{{deserializer_field(field, namespace, loop)}}\n> +{%- endfor %}\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector::iterator dataBegin,\n> +\t\t std::vector::iterator dataEnd,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector data(dataBegin, dataEnd);\n> +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, cs);\n> +\t}\n> +{%- endmacro %}\n> diff --git a/utils/ipc/generators/mojom_libcamera_generator.py b/utils/ipc/generators/mojom_libcamera_generator.py\n> new file mode 100644\n> index 00000000..c4fbf9b3\n> --- /dev/null\n> +++ b/utils/ipc/generators/mojom_libcamera_generator.py\n> @@ -0,0 +1,488 @@\n> +'''Generates libcamera files from a mojom.Module.'''\n> +\n> +import argparse\n> +import ast\n> +import datetime\n> +import contextlib\n> +import os\n> +import re\n> +import shutil\n> +import sys\n> +import tempfile\n> +\n> +from jinja2 import contextfilter\n> +\n> +import mojom.fileutil as fileutil\n> +import mojom.generate.generator as generator\n> +import mojom.generate.module as mojom\n> +from mojom.generate.template_expander import UseJinja\n> +\n> +\n> +GENERATOR_PREFIX = 'libcamera'\n> +\n> +_kind_to_cpp_type = {\n> + mojom.BOOL: 'bool',\n> + mojom.INT8: 'int8_t',\n> + mojom.UINT8: 'uint8_t',\n> + mojom.INT16: 'int16_t',\n> + mojom.UINT16: 'uint16_t',\n> + mojom.INT32: 'int32_t',\n> + mojom.UINT32: 'uint32_t',\n> + mojom.FLOAT: 'float',\n> + mojom.INT64: 'int64_t',\n> + mojom.UINT64: 'uint64_t',\n> + mojom.DOUBLE: 'double',\n> +}\n> +\n> +_bit_widths = {\n> + mojom.BOOL: '8',\n> + mojom.INT8: '8',\n> + mojom.UINT8: '8',\n> + mojom.INT16: '16',\n> + mojom.UINT16: '16',\n> + mojom.INT32: '32',\n> + mojom.UINT32: '32',\n> + mojom.FLOAT: '32',\n> + mojom.INT64: '64',\n> + mojom.UINT64: '64',\n> + mojom.DOUBLE: '64',\n> +}\n> +\n> +def ModuleName(path):\n> + return path.split('/')[-1].split('.')[0]\n> +\n> +def ModuleClassName(module):\n> + s = re.sub(r'^IPA', '', module.interfaces[0].mojom_name)\n> + return re.sub(r'Interface$', '', s)\n> +\n> +def UpperCamelCase(name):\n> + return ''.join([x.capitalize() for x in generator.SplitCamelCase(name)])\n> +\n> +def CamelCase(name):\n> + uccc = UpperCamelCase(name)\n> + return uccc[0].lower() + uccc[1:]\n> +\n> +def Capitalize(name):\n> + return name[0].upper() + name[1:]\n> +\n> +def ConstantStyle(name):\n> + return generator.ToUpperSnakeCase(name)\n> +\n> +def Choose(cond, t, f):\n> + return t if cond else f\n> +\n> +def CommaSep(l):\n> + return ', '.join([m for m in l])\n> +\n> +def ParamsCommaSep(l):\n> + return ', '.join([m.mojom_name for m in l])\n> +\n> +def GetDefaultValue(element):\n> + if element.default is not None:\n> + return element.default\n> + if type(element.kind) == mojom.Kind:\n> + return '0'\n> + if mojom.IsEnumKind(element.kind):\n> + return f'static_cast<{element.kind.mojom_name}>(0)'\n> + if (isinstance(element.kind, mojom.Struct) and\n> + element.kind.mojom_name == 'FileDescriptor'):\n> + return '-1'\n> + return ''\n> +\n> +def WithDefaultValues(element):\n> + return [x for x in element if HasDefaultValue(x)]\n> +\n> +def WithFds(element):\n> + return [x for x in element if HasFd(x)]\n> +\n> +def HasDefaultValue(element):\n> + return GetDefaultValue(element) != ''\n> +\n> +def HasDefaultFields(element):\n> + return True in [HasDefaultValue(x) for x in element.fields]\n> +\n> +def GetAllTypes(element):\n> + if mojom.IsArrayKind(element):\n> + return GetAllTypes(element.kind)\n> + if mojom.IsMapKind(element):\n> + return GetAllTypes(element.key_kind) + GetAllTypes(element.value_kind)\n> + if isinstance(element, mojom.Parameter):\n> + return GetAllTypes(element.kind)\n> + if mojom.IsEnumKind(element):\n> + return [element.mojom_name]\n> + if not mojom.IsStructKind(element):\n> + return [element.spec]\n> + if len(element.fields) == 0:\n> + return [element.mojom_name]\n> + ret = [GetAllTypes(x.kind) for x in element.fields]\n> + ret = [x for sublist in ret for x in sublist]\n> + return list(set(ret))\n> +\n> +def GetAllAttrs(element):\n> + if mojom.IsArrayKind(element):\n> + return GetAllAttrs(element.kind)\n> + if mojom.IsMapKind(element):\n> + return {**GetAllAttrs(element.key_kind), **GetAllAttrs(element.value_kind)}\n> + if isinstance(element, mojom.Parameter):\n> + return GetAllAttrs(element.kind)\n> + if mojom.IsEnumKind(element):\n> + return element.attributes if element.attributes is not None else {}\n> + if mojom.IsStructKind(element) and len(element.fields) == 0:\n> + return element.attributes if element.attributes is not None else {}\n> + if not mojom.IsStructKind(element):\n> + return {}\n> + attrs = [(x.attributes) for x in element.fields]\n> + ret = {}\n> + for d in attrs:\n> + if d is not None:\n> + ret = {**ret, **d}\n> + return ret\n> +\n> +def NeedsControlSerializer(element):\n> + types = GetAllTypes(element)\n> + return \"ControlList\" in types or \"ControlInfoMap\" in types\n> +\n> +def HasFd(element):\n> + attrs = GetAllAttrs(element)\n> + if isinstance(element, mojom.Kind):\n> + types = GetAllTypes(element)\n> + else:\n> + types = GetAllTypes(element.kind)\n> + return \"FileDescriptor\" in types or (attrs is not None and \"hasFd\" in attrs)\n> +\n> +def MethodInputHasFd(method):\n> + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> + return True\n> + return False\n> +\n> +def MethodOutputHasFd(method):\n> + if (MethodReturnValue(method) != 'void' or\n> + method.response_parameters is None):\n> + return False\n> + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> + return True\n> + return False\n> +\n> +def MethodParamInputs(method):\n> + return method.parameters\n> +\n> +def MethodParamOutputs(method):\n> + if (MethodReturnValue(method) != 'void' or\n> + method.response_parameters is None):\n> + return []\n> + return method.response_parameters\n> +\n> +def MethodParamNames(method):\n> + params = []\n> + for param in method.parameters:\n> + params.append(param.mojom_name)\n> + if MethodReturnValue(method) == 'void':\n> + if method.response_parameters is None:\n> + return params\n> + for param in method.response_parameters:\n> + params.append(param.mojom_name)\n> + return params\n> +\n> +def MethodParameters(method):\n> + params = []\n> + for param in method.parameters:\n> + params.append('const %s %s%s' % (GetNameForElement(param),\n> + ('&' if not IsPod(param) else ''),\n> + param.mojom_name))\n> + if MethodReturnValue(method) == 'void':\n> + if method.response_parameters is None:\n> + return params\n> + for param in method.response_parameters:\n> + params.append(f'{GetNameForElement(param)} *{param.mojom_name}')\n> + return params\n> +\n> +def MethodReturnValue(method):\n> + if method.response_parameters is None:\n> + return 'void'\n> + if len(method.response_parameters) == 1 and IsPod(method.response_parameters[0]):\n> + return GetNameForElement(method.response_parameters[0])\n> + return 'void'\n> +\n> +def IsAsync(method):\n> + # callbacks are always async\n> + if re.match(\"^IPA.*EventInterface$\", method.interface.mojom_name):\n> + return True\n> + elif re.match(\"^IPA.*Interface$\", method.interface.mojom_name):\n> + if method.attributes is None:\n> + return False\n> + elif 'async' in method.attributes and method.attributes['async']:\n> + return True\n> + return False\n> +\n> +def IsArray(element):\n> + return mojom.IsArrayKind(element.kind)\n> +\n> +def IsControls(element):\n> + return mojom.IsStructKind(element.kind) and (element.kind.mojom_name == \"ControlList\" or\n> + element.kind.mojom_name == \"ControlInfoMap\")\n> +\n> +def IsEnum(element):\n> + return mojom.IsEnumKind(element.kind)\n> +\n> +def IsFd(element):\n> + return mojom.IsStructKind(element.kind) and element.kind.mojom_name == \"FileDescriptor\"\n> +\n> +def IsMap(element):\n> + return mojom.IsMapKind(element.kind)\n> +\n> +def IsPlainStruct(element):\n> + return mojom.IsStructKind(element.kind) and not IsControls(element) and not IsFd(element)\n> +\n> +def IsPod(element):\n> + return element.kind in _kind_to_cpp_type\n> +\n> +def IsStr(element):\n> + return element.kind.spec == 's'\n> +\n> +def BitWidth(element):\n> + if element.kind in _bit_widths:\n> + return _bit_widths[element.kind]\n> + if mojom.IsEnumKind(element.kind):\n> + return '32'\n> + return ''\n> +\n> +def GetNameForElement(element):\n> + if (mojom.IsEnumKind(element) or\n> + mojom.IsInterfaceKind(element) or\n> + mojom.IsStructKind(element)):\n> + return element.mojom_name\n> + if (mojom.IsArrayKind(element)):\n> + elem_name = GetNameForElement(element.kind)\n> + return f'std::vector<{elem_name}>'\n> + if (mojom.IsMapKind(element)):\n> + key_name = GetNameForElement(element.key_kind)\n> + value_name = GetNameForElement(element.value_kind)\n> + return f'std::map<{key_name}, {value_name}>'\n> + if isinstance(element, (mojom.Field, mojom.Method, mojom.Parameter)):\n> + if (mojom.IsArrayKind(element.kind) or mojom.IsMapKind(element.kind)):\n> + return GetNameForElement(element.kind)\n> + if (mojom.IsReferenceKind(element.kind) and element.kind.spec == 's'):\n> + return 'std::string'\n> + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> + return _kind_to_cpp_type[element.kind]\n> + return element.kind.mojom_name\n> + if isinstance(element, mojom.EnumValue):\n> + return (GetNameForElement(element.enum) + '.' +\n> + ConstantStyle(element.name))\n> + if isinstance(element, (mojom.NamedValue,\n> + mojom.Constant,\n> + mojom.EnumField)):\n> + return ConstantStyle(element.name)\n> + if (hasattr(element, '__hash__') and element in _kind_to_cpp_type):\n> + return _kind_to_cpp_type[element]\n> + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> + return _kind_to_cpp_type[element.kind]\n> + if (hasattr(element, 'spec')):\n> + if (element.spec == 's'):\n> + return 'std::string'\n> + return element.spec.split(':')[-1]\n> + if (mojom.IsInterfaceRequestKind(element) or\n> + mojom.IsAssociatedKind(element) or\n> + mojom.IsPendingRemoteKind(element) or\n> + mojom.IsPendingReceiverKind(element) or\n> + mojom.IsUnionKind(element)):\n> + raise Exception('Unsupported element: %s' % element)\n> + raise Exception('Unexpected element: %s' % element)\n> +\n> +def GetFullNameForElement(element, namespace_str):\n> + name = GetNameForElement(element)\n> + if namespace_str == '':\n> + return name\n> + return f'{namespace_str}::{name}'\n> +\n> +def ValidateZeroLength(l, s, cap=True):\n> + if l is None:\n> + return\n> + if len(l) > 0:\n> + raise Exception(f'{s.capitalize() if cap else s} should be empty')\n> +\n> +def ValidateSingleLength(l, s, cap=True):\n> + if len(l) > 1:\n> + raise Exception(f'Only one {s} allowed')\n> + if len(l) < 1:\n> + raise Exception(f'{s.capitalize() if cap else s} is required')\n> +\n> +def ValidateInterfaces(interfaces):\n> + # Validate presence of main interface\n> + intf = [x for x in interfaces\n> + if re.match(\"^IPA.*Interface\", x.mojom_name) and\n> + not re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> + ValidateSingleLength(intf, 'main interface')\n> + intf = intf[0]\n> +\n> + # Validate presence of callback interface\n> + cb = [x for x in interfaces if re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> + ValidateSingleLength(cb, 'event interface')\n> + cb = cb[0]\n> +\n> + # Validate required main interface functions\n> + f_init = [x for x in intf.methods if x.mojom_name == 'init']\n> + f_start = [x for x in intf.methods if x.mojom_name == 'start']\n> + f_stop = [x for x in intf.methods if x.mojom_name == 'stop']\n> +\n> + ValidateSingleLength(f_init, 'init()', False)\n> + ValidateSingleLength(f_start, 'start()', False)\n> + ValidateSingleLength(f_stop, 'stop()', False)\n> +\n> + f_init = f_init[0]\n> + f_start = f_start[0]\n> + f_stop = f_stop[0]\n> +\n> + # Validate parameters to init()\n> + ValidateSingleLength(f_init.parameters, 'input parameter to init()')\n> + ValidateSingleLength(f_init.response_parameters, 'output parameter from init()')\n> + if f_init.parameters[0].kind.mojom_name != 'IPASettings':\n> + raise Exception('init() must have single IPASettings input parameter')\n> + if f_init.response_parameters[0].kind.spec != 'i32':\n> + raise Exception('init() must have single int32 output parameter')\n> +\n> + # Validate parameters to start()\n> + ValidateZeroLength(f_start.parameters, 'input parameter to start()')\n> + ValidateSingleLength(f_start.response_parameters, 'output parameter from start()')\n> + if f_start.response_parameters[0].kind.spec != 'i32':\n> + raise Exception('start() must have single int32 output parameter')\n> +\n> + # Validate parameters to stop()\n> + ValidateZeroLength(f_stop.parameters, 'input parameter to stop()')\n> + ValidateZeroLength(f_stop.parameters, 'output parameter from stop()')\n> +\n> + # Validate that all async methods don't have return values\n> + intf_methods_async = [x for x in intf.methods if IsAsync(x)]\n> + for method in intf_methods_async:\n> + ValidateZeroLength(method.response_parameters,\n> + f'{method.mojom_name} response parameters', False)\n> +\n> + cb_methods_async = [x for x in cb.methods if IsAsync(x)]\n> + for method in cb_methods_async:\n> + ValidateZeroLength(method.response_parameters,\n> + f'{method.mojom_name} response parameters', False)\n> +\n> +class Generator(generator.Generator):\n> +\n> + @staticmethod\n> + def GetTemplatePrefix():\n> + return 'libcamera_templates'\n> +\n> + def GetFilters(self):\n> + libcamera_filters = {\n> + 'all_types': GetAllTypes,\n> + 'bit_width': BitWidth,\n> + 'cap': Capitalize,\n> + 'choose': Choose,\n> + 'comma_sep': CommaSep,\n> + 'default_value': GetDefaultValue,\n> + 'has_default_fields': HasDefaultFields,\n> + 'has_fd': HasFd,\n> + 'is_async': IsAsync,\n> + 'is_array': IsArray,\n> + 'is_controls': IsControls,\n> + 'is_enum': IsEnum,\n> + 'is_fd': IsFd,\n> + 'is_map': IsMap,\n> + 'is_plain_struct': IsPlainStruct,\n> + 'is_pod': IsPod,\n> + 'is_str': IsStr,\n> + 'method_input_has_fd': MethodInputHasFd,\n> + 'method_output_has_fd': MethodOutputHasFd,\n> + 'method_param_names': MethodParamNames,\n> + 'method_param_inputs': MethodParamInputs,\n> + 'method_param_outputs': MethodParamOutputs,\n> + 'method_parameters': MethodParameters,\n> + 'method_return_value': MethodReturnValue,\n> + 'name': GetNameForElement,\n> + 'name_full': GetFullNameForElement,\n> + 'needs_control_serializer': NeedsControlSerializer,\n> + 'params_comma_sep': ParamsCommaSep,\n> + 'with_default_values': WithDefaultValues,\n> + 'with_fds': WithFds,\n> + }\n> + return libcamera_filters\n> +\n> + def _GetJinjaExports(self):\n> + return {\n> + 'base_controls': '%s::controls' % ModuleClassName(self.module),\n> + 'cmd_enum_name': '_%sCmd' % ModuleClassName(self.module),\n> + 'cmd_event_enum_name': '_%sEventCmd' % ModuleClassName(self.module),\n> + 'enums': self.module.enums,\n> + 'has_array': len([x for x in self.module.kinds.keys() if x[0] == 'a']) > 0,\n> + 'has_map': len([x for x in self.module.kinds.keys() if x[0] == 'm']) > 0,\n> + 'has_namespace': self.module.mojom_namespace != '',\n> + 'imports': self.module.imports,\n> + 'interface_cb': self.module.interfaces[1],\n> + 'interface_main': self.module.interfaces[0],\n> + 'interface_name': 'IPA%sInterface' % ModuleClassName(self.module),\n> + 'ipc_name': 'IPAIPCUnixSocket',\n> + 'kinds': self.module.kinds,\n> + 'module': self.module,\n> + 'module_name': ModuleName(self.module.path),\n> + 'module_class_name': ModuleClassName(self.module),\n> + 'namespace': self.module.mojom_namespace.split('.'),\n> + 'namespace_str': self.module.mojom_namespace.replace('.', '::') if\n> + self.module.mojom_namespace is not None else '',\n> + 'proxy_name': 'IPAProxy%s' % ModuleClassName(self.module),\n> + 'proxy_worker_name': 'IPAProxy%sWorker' % ModuleClassName(self.module),\n> + 'structs_nonempty': [x for x in self.module.structs if len(x.fields) > 0],\n> + 'year': datetime.datetime.now().year,\n> + }\n> +\n> +\n> + @UseJinja('module_ipa_interface.h.tmpl')\n> + def _GenerateDataHeader(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_serializer.h.tmpl')\n> + def _GenerateSerializer(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy.cpp.tmpl')\n> + def _GenerateProxyCpp(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy.h.tmpl')\n> + def _GenerateProxyHeader(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy_worker.cpp.tmpl')\n> + def _GenerateProxyWorker(self):\n> + return self._GetJinjaExports()\n> +\n> + def GenerateFiles(self, unparsed_args):\n> + parser = argparse.ArgumentParser()\n> + parser.add_argument('--libcamera_generate_header', action='store_true')\n> + parser.add_argument('--libcamera_generate_serializer', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_cpp', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_h', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_worker', action='store_true')\n> + parser.add_argument('--libcamera_output_path')\n> + args = parser.parse_args(unparsed_args)\n> +\n> + ValidateInterfaces(self.module.interfaces)\n> +\n> + fileutil.EnsureDirectoryExists(os.path.dirname(args.libcamera_output_path))\n> +\n> + module_name = ModuleName(self.module.path)\n> +\n> + if args.libcamera_generate_header:\n> + self.Write(self._GenerateDataHeader(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_serializer:\n> + self.Write(self._GenerateSerializer(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_cpp:\n> + self.Write(self._GenerateProxyCpp(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_h:\n> + self.Write(self._GenerateProxyHeader(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_worker:\n> + self.Write(self._GenerateProxyWorker(),\n> + args.libcamera_output_path)\n\nA very cursory look:\nAcked-by: Jacopo Mondi \n\nThanks\n j\n\n> --\n> 2.27.0\n>\n> _______________________________________________\n> libcamera-devel mailing list\n> libcamera-devel@lists.libcamera.org\n> https://lists.libcamera.org/listinfo/libcamera-devel","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7970DBE081\n\tfor ;\n\tTue, 17 Nov 2020 14:55:40 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id DE60E63325;\n\tTue, 17 Nov 2020 15:55:39 +0100 (CET)","from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net\n\t[217.70.183.193])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 330E76033B\n\tfor ;\n\tTue, 17 Nov 2020 15:55:38 +0100 (CET)","from uno.localdomain (93-34-118-233.ip49.fastwebnet.it\n\t[93.34.118.233]) (Authenticated sender: jacopo@jmondi.org)\n\tby relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 9C4C2240002;\n\tTue, 17 Nov 2020 14:55:37 +0000 (UTC)"],"X-Originating-IP":"93.34.118.233","Date":"Tue, 17 Nov 2020 15:55:40 +0100","From":"Jacopo Mondi ","To":"Paul Elder ","Message-ID":"<20201117145540.2zxf4cefodqcck22@uno.localdomain>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201106103707.49660-5-paul.elder@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13798,"web_url":"https://patchwork.libcamera.org/comment/13798/","msgid":"<20201119123917.GA4563@pendragon.ideasonboard.com>","date":"2020-11-19T12:39:17","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nThank you for the patch.\n\nOn Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> Add templates to mojo to generate code for the IPC mechanism. These\n> templates generate:\n> - module header\n> - module serializer\n> - IPA proxy cpp, header, and worker\n> \n> Given an input data definition mojom file for a pipeline.\n> \n> Signed-off-by: Paul Elder \n> \n> ---\n> Changes in v4:\n> For the non-template files:\n> - rename IPA{pipeline_name}CallbackInterface to\n> IPA{pipeline_name}EventInterface\n> - to avoid the notion of \"callback\" and emphasize that it's an event\n> - add support for strings in custom structs\n> - add validation, that async methods must not have return values\n> - it throws exception and isn't very clear though...?\n> - rename controls to libcamera::{pipeline_name}::controls (controls is\n> now lowercase)\n> - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> - same for their corresponding template files\n> For the template files:\n> - fix spacing (now it's all {{var}} instead of some {{ var }})\n> - except if it's code, so code is still {{ code }}\n> - move inclusion of corresponding header to first in the inclusion list\n> - fix copy&paste errors\n> - change snake_case to camelCase in the generated code\n> - template code still uses snake_case\n> - change the generated command enums to an enum class, and make it\n> capitalized (instead of allcaps)\n> - add length checks to recvIPC (in proxy)\n> - fix some template spacing\n> - don't use const for PODs in function/signal parameters\n> - add the proper length checks to readPOD/appendPOD\n> - the helper functions for reading and writing PODs to and from\n> serialized data\n> - rename readUInt/appendUInt to readPOD/appendPOD\n> - add support for strings in custom structs\n> \n> Changes in v3:\n> - add support for namespaces\n> - fix enum assignment (used to have +1 for CMD applied to all enums)\n> - use readHeader, writeHeader, and eraseHeader as static class functions\n> of IPAIPCUnixSocket (in the proxy worker)\n> - add requirement that base controls *must* be defined in\n> libcamera::{pipeline_name}::Controls\n> \n> Changes in v2:\n> - mandate the main and callback interfaces, and init(), start(), stop()\n> and their parameters\n> - fix returning single pod value from IPC-called function\n> - add licenses\n> - improve auto-generated message\n> - other fixes related to serdes\n> ---\n> .../module_ipa_interface.h.tmpl | 113 ++++\n> .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> .../module_ipa_proxy.h.tmpl | 118 +++++\n> .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> .../module_ipa_serializer.h.tmpl | 44 ++\n> .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> 8 files changed, 1673 insertions(+)\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n\nLet's start with the generator.\n\n[snip]\n\n> diff --git a/utils/ipc/generators/mojom_libcamera_generator.py b/utils/ipc/generators/mojom_libcamera_generator.py\n> new file mode 100644\n> index 00000000..c4fbf9b3\n> --- /dev/null\n> +++ b/utils/ipc/generators/mojom_libcamera_generator.py\n> @@ -0,0 +1,488 @@\n\nMissing SPDX header and copyright notice.\n\n> +'''Generates libcamera files from a mojom.Module.'''\n\nComments in Python use # :-)\n\n> +\n> +import argparse\n> +import ast\n> +import datetime\n> +import contextlib\n> +import os\n> +import re\n> +import shutil\n> +import sys\n> +import tempfile\n\nast, contextlib, shutil, sys and tempfile don't seem to be used.\n\n> +\n> +from jinja2 import contextfilter\n\ncontextfilter isn't used either.\n\n> +\n> +import mojom.fileutil as fileutil\n> +import mojom.generate.generator as generator\n> +import mojom.generate.module as mojom\n> +from mojom.generate.template_expander import UseJinja\n> +\n> +\n> +GENERATOR_PREFIX = 'libcamera'\n> +\n> +_kind_to_cpp_type = {\n> + mojom.BOOL: 'bool',\n> + mojom.INT8: 'int8_t',\n> + mojom.UINT8: 'uint8_t',\n> + mojom.INT16: 'int16_t',\n> + mojom.UINT16: 'uint16_t',\n> + mojom.INT32: 'int32_t',\n> + mojom.UINT32: 'uint32_t',\n> + mojom.FLOAT: 'float',\n> + mojom.INT64: 'int64_t',\n> + mojom.UINT64: 'uint64_t',\n> + mojom.DOUBLE: 'double',\n> +}\n> +\n> +_bit_widths = {\n> + mojom.BOOL: '8',\n> + mojom.INT8: '8',\n> + mojom.UINT8: '8',\n> + mojom.INT16: '16',\n> + mojom.UINT16: '16',\n> + mojom.INT32: '32',\n> + mojom.UINT32: '32',\n> + mojom.FLOAT: '32',\n> + mojom.INT64: '64',\n> + mojom.UINT64: '64',\n> + mojom.DOUBLE: '64',\n> +}\n> +\n> +def ModuleName(path):\n> + return path.split('/')[-1].split('.')[0]\n> +\n> +def ModuleClassName(module):\n> + s = re.sub(r'^IPA', '', module.interfaces[0].mojom_name)\n> + return re.sub(r'Interface$', '', s)\n\nYou could combine the two with\n\n return re.sub(r'^IPA(.*)Interface$', lambda match: match.group(1), module)\n\nand possibly optimize it further by pre-compiling the regular\nexpression. I'll let you decide if it's worth it.\n\n> +\n> +def UpperCamelCase(name):\n> + return ''.join([x.capitalize() for x in generator.SplitCamelCase(name)])\n> +\n> +def CamelCase(name):\n> + uccc = UpperCamelCase(name)\n> + return uccc[0].lower() + uccc[1:]\n\nThese two functions don't seem to be used.\n\n> +\n> +def Capitalize(name):\n> + return name[0].upper() + name[1:]\n> +\n> +def ConstantStyle(name):\n> + return generator.ToUpperSnakeCase(name)\n> +\n> +def Choose(cond, t, f):\n> + return t if cond else f\n> +\n> +def CommaSep(l):\n> + return ', '.join([m for m in l])\n> +\n> +def ParamsCommaSep(l):\n> + return ', '.join([m.mojom_name for m in l])\n> +\n> +def GetDefaultValue(element):\n> + if element.default is not None:\n> + return element.default\n> + if type(element.kind) == mojom.Kind:\n> + return '0'\n> + if mojom.IsEnumKind(element.kind):\n> + return f'static_cast<{element.kind.mojom_name}>(0)'\n> + if (isinstance(element.kind, mojom.Struct) and\n> + element.kind.mojom_name == 'FileDescriptor'):\n\nIs there a need for the outer parentheses ? Or is this dictated by the\nmojom coding style ? Same comment for several locations below.\n\n> + return '-1'\n> + return ''\n> +\n> +def WithDefaultValues(element):\n> + return [x for x in element if HasDefaultValue(x)]\n> +\n> +def WithFds(element):\n> + return [x for x in element if HasFd(x)]\n\nShould these be moved a little bit further down, after the functions\nthey call ? The code works fine, but it would create an easier to read\nflow.\n\n> +\n> +def HasDefaultValue(element):\n> + return GetDefaultValue(element) != ''\n> +\n> +def HasDefaultFields(element):\n> + return True in [HasDefaultValue(x) for x in element.fields]\n> +\n> +def GetAllTypes(element):\n> + if mojom.IsArrayKind(element):\n> + return GetAllTypes(element.kind)\n> + if mojom.IsMapKind(element):\n> + return GetAllTypes(element.key_kind) + GetAllTypes(element.value_kind)\n> + if isinstance(element, mojom.Parameter):\n> + return GetAllTypes(element.kind)\n> + if mojom.IsEnumKind(element):\n> + return [element.mojom_name]\n> + if not mojom.IsStructKind(element):\n> + return [element.spec]\n> + if len(element.fields) == 0:\n> + return [element.mojom_name]\n> + ret = [GetAllTypes(x.kind) for x in element.fields]\n> + ret = [x for sublist in ret for x in sublist]\n> + return list(set(ret))\n> +\n> +def GetAllAttrs(element):\n> + if mojom.IsArrayKind(element):\n> + return GetAllAttrs(element.kind)\n> + if mojom.IsMapKind(element):\n> + return {**GetAllAttrs(element.key_kind), **GetAllAttrs(element.value_kind)}\n> + if isinstance(element, mojom.Parameter):\n> + return GetAllAttrs(element.kind)\n> + if mojom.IsEnumKind(element):\n> + return element.attributes if element.attributes is not None else {}\n> + if mojom.IsStructKind(element) and len(element.fields) == 0:\n> + return element.attributes if element.attributes is not None else {}\n> + if not mojom.IsStructKind(element):\n> + return {}\n> + attrs = [(x.attributes) for x in element.fields]\n> + ret = {}\n> + for d in attrs:\n> + if d is not None:\n> + ret = {**ret, **d}\n\nMaybe\n\n ret.update(d)\n\n> + return ret\n> +\n> +def NeedsControlSerializer(element):\n> + types = GetAllTypes(element)\n> + return \"ControlList\" in types or \"ControlInfoMap\" in types\n> +\n> +def HasFd(element):\n> + attrs = GetAllAttrs(element)\n> + if isinstance(element, mojom.Kind):\n> + types = GetAllTypes(element)\n> + else:\n> + types = GetAllTypes(element.kind)\n> + return \"FileDescriptor\" in types or (attrs is not None and \"hasFd\" in attrs)\n> +\n> +def MethodInputHasFd(method):\n> + if len([x for x in method.parameters if HasFd(x)]) > 0:\n\nYou can drop the len() call, [] evaluates to False.\n\n> + return True\n> + return False\n> +\n> +def MethodOutputHasFd(method):\n> + if (MethodReturnValue(method) != 'void' or\n> + method.response_parameters is None):\n> + return False\n> + if len([x for x in method.parameters if HasFd(x)]) > 0:\n\nSame here.\n\nShouldn't it be method.response_parameters instead of method.parameters\n?\n\n> + return True\n> + return False\n> +\n> +def MethodParamInputs(method):\n> + return method.parameters\n> +\n> +def MethodParamOutputs(method):\n> + if (MethodReturnValue(method) != 'void' or\n> + method.response_parameters is None):\n> + return []\n> + return method.response_parameters\n\nThe increase code reuse, you could implement these functions as follows.\n\n\ndef MethodParamInputs(method):\n return method.parameters\n\ndef MethodParamOutputs(method):\n if (MethodReturnValue(method) != 'void' or\n method.response_parameters is None):\n return []\n return method.response_parameters\n\ndef MethodParamHasFd(parameters):\n if [x for x in parameters if HasFd(x)]:\n return True\n return False\n\ndef MethodInputHasFd(method):\n return MethodParamHasFd(MethodParamInputs(method))\n\ndef MethodOutputHasFd(method):\n return MethodParamHasFd(MethodParamOutputs(method))\n\n> +\n> +def MethodParamNames(method):\n> + params = []\n> + for param in method.parameters:\n> + params.append(param.mojom_name)\n> + if MethodReturnValue(method) == 'void':\n> + if method.response_parameters is None:\n> + return params\n> + for param in method.response_parameters:\n> + params.append(param.mojom_name)\n> + return params\n> +\n> +def MethodParameters(method):\n> + params = []\n> + for param in method.parameters:\n> + params.append('const %s %s%s' % (GetNameForElement(param),\n> + ('&' if not IsPod(param) else ''),\n\nNo need for the outer parentheses.\n\n> + param.mojom_name))\n> + if MethodReturnValue(method) == 'void':\n> + if method.response_parameters is None:\n> + return params\n> + for param in method.response_parameters:\n> + params.append(f'{GetNameForElement(param)} *{param.mojom_name}')\n> + return params\n> +\n> +def MethodReturnValue(method):\n> + if method.response_parameters is None:\n> + return 'void'\n> + if len(method.response_parameters) == 1 and IsPod(method.response_parameters[0]):\n> + return GetNameForElement(method.response_parameters[0])\n> + return 'void'\n> +\n> +def IsAsync(method):\n> + # callbacks are always async\n\ns/callbacks/Events/ ?\n\n> + if re.match(\"^IPA.*EventInterface$\", method.interface.mojom_name):\n> + return True\n> + elif re.match(\"^IPA.*Interface$\", method.interface.mojom_name):\n> + if method.attributes is None:\n> + return False\n> + elif 'async' in method.attributes and method.attributes['async']:\n> + return True\n> + return False\n> +\n> +def IsArray(element):\n> + return mojom.IsArrayKind(element.kind)\n> +\n> +def IsControls(element):\n> + return mojom.IsStructKind(element.kind) and (element.kind.mojom_name == \"ControlList\" or\n> + element.kind.mojom_name == \"ControlInfoMap\")\n> +\n> +def IsEnum(element):\n> + return mojom.IsEnumKind(element.kind)\n> +\n> +def IsFd(element):\n> + return mojom.IsStructKind(element.kind) and element.kind.mojom_name == \"FileDescriptor\"\n> +\n> +def IsMap(element):\n> + return mojom.IsMapKind(element.kind)\n> +\n> +def IsPlainStruct(element):\n> + return mojom.IsStructKind(element.kind) and not IsControls(element) and not IsFd(element)\n> +\n> +def IsPod(element):\n> + return element.kind in _kind_to_cpp_type\n> +\n> +def IsStr(element):\n> + return element.kind.spec == 's'\n> +\n> +def BitWidth(element):\n> + if element.kind in _bit_widths:\n> + return _bit_widths[element.kind]\n> + if mojom.IsEnumKind(element.kind):\n> + return '32'\n> + return ''\n> +\n> +def GetNameForElement(element):\n> + if (mojom.IsEnumKind(element) or\n> + mojom.IsInterfaceKind(element) or\n> + mojom.IsStructKind(element)):\n> + return element.mojom_name\n> + if (mojom.IsArrayKind(element)):\n> + elem_name = GetNameForElement(element.kind)\n> + return f'std::vector<{elem_name}>'\n> + if (mojom.IsMapKind(element)):\n> + key_name = GetNameForElement(element.key_kind)\n> + value_name = GetNameForElement(element.value_kind)\n> + return f'std::map<{key_name}, {value_name}>'\n> + if isinstance(element, (mojom.Field, mojom.Method, mojom.Parameter)):\n> + if (mojom.IsArrayKind(element.kind) or mojom.IsMapKind(element.kind)):\n> + return GetNameForElement(element.kind)\n> + if (mojom.IsReferenceKind(element.kind) and element.kind.spec == 's'):\n> + return 'std::string'\n> + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n\nIs the hasattr() check needed ? If element has no kind attribute, I\nwould expect the previous two if's to raise an exception.\n\n> + return _kind_to_cpp_type[element.kind]\n> + return element.kind.mojom_name\n> + if isinstance(element, mojom.EnumValue):\n\nExtra space after comma.\n\n> + return (GetNameForElement(element.enum) + '.' +\n> + ConstantStyle(element.name))\n> + if isinstance(element, (mojom.NamedValue,\n> + mojom.Constant,\n> + mojom.EnumField)):\n> + return ConstantStyle(element.name)\n> + if (hasattr(element, '__hash__') and element in _kind_to_cpp_type):\n> + return _kind_to_cpp_type[element]\n\nFor my own education, what type of elements does this cover ?\n\n> + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> + return _kind_to_cpp_type[element.kind]\n> + if (hasattr(element, 'spec')):\n> + if (element.spec == 's'):\n> + return 'std::string'\n> + return element.spec.split(':')[-1]\n> + if (mojom.IsInterfaceRequestKind(element) or\n> + mojom.IsAssociatedKind(element) or\n> + mojom.IsPendingRemoteKind(element) or\n> + mojom.IsPendingReceiverKind(element) or\n> + mojom.IsUnionKind(element)):\n> + raise Exception('Unsupported element: %s' % element)\n> + raise Exception('Unexpected element: %s' % element)\n> +\n> +def GetFullNameForElement(element, namespace_str):\n> + name = GetNameForElement(element)\n> + if namespace_str == '':\n> + return name\n> + return f'{namespace_str}::{name}'\n> +\n> +def ValidateZeroLength(l, s, cap=True):\n> + if l is None:\n> + return\n> + if len(l) > 0:\n> + raise Exception(f'{s.capitalize() if cap else s} should be empty')\n> +\n> +def ValidateSingleLength(l, s, cap=True):\n> + if len(l) > 1:\n> + raise Exception(f'Only one {s} allowed')\n> + if len(l) < 1:\n> + raise Exception(f'{s.capitalize() if cap else s} is required')\n> +\n> +def ValidateInterfaces(interfaces):\n> + # Validate presence of main interface\n> + intf = [x for x in interfaces\n> + if re.match(\"^IPA.*Interface\", x.mojom_name) and\n> + not re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> + ValidateSingleLength(intf, 'main interface')\n> + intf = intf[0]\n> +\n> + # Validate presence of callback interface\n\ns/callback/event/\n\n> + cb = [x for x in interfaces if re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n\ns/cb/event/ ?\n\n> + ValidateSingleLength(cb, 'event interface')\n> + cb = cb[0]\n> +\n> + # Validate required main interface functions\n> + f_init = [x for x in intf.methods if x.mojom_name == 'init']\n> + f_start = [x for x in intf.methods if x.mojom_name == 'start']\n> + f_stop = [x for x in intf.methods if x.mojom_name == 'stop']\n> +\n> + ValidateSingleLength(f_init, 'init()', False)\n> + ValidateSingleLength(f_start, 'start()', False)\n> + ValidateSingleLength(f_stop, 'stop()', False)\n> +\n> + f_init = f_init[0]\n> + f_start = f_start[0]\n> + f_stop = f_stop[0]\n> +\n> + # Validate parameters to init()\n> + ValidateSingleLength(f_init.parameters, 'input parameter to init()')\n> + ValidateSingleLength(f_init.response_parameters, 'output parameter from init()')\n> + if f_init.parameters[0].kind.mojom_name != 'IPASettings':\n> + raise Exception('init() must have single IPASettings input parameter')\n> + if f_init.response_parameters[0].kind.spec != 'i32':\n> + raise Exception('init() must have single int32 output parameter')\n> +\n> + # Validate parameters to start()\n> + ValidateZeroLength(f_start.parameters, 'input parameter to start()')\n> + ValidateSingleLength(f_start.response_parameters, 'output parameter from start()')\n> + if f_start.response_parameters[0].kind.spec != 'i32':\n> + raise Exception('start() must have single int32 output parameter')\n> +\n> + # Validate parameters to stop()\n> + ValidateZeroLength(f_stop.parameters, 'input parameter to stop()')\n> + ValidateZeroLength(f_stop.parameters, 'output parameter from stop()')\n> +\n> + # Validate that all async methods don't have return values\n> + intf_methods_async = [x for x in intf.methods if IsAsync(x)]\n> + for method in intf_methods_async:\n> + ValidateZeroLength(method.response_parameters,\n> + f'{method.mojom_name} response parameters', False)\n> +\n> + cb_methods_async = [x for x in cb.methods if IsAsync(x)]\n\ns/cb_method_async/event_methods_async/\n\n> + for method in cb_methods_async:\n> + ValidateZeroLength(method.response_parameters,\n> + f'{method.mojom_name} response parameters', False)\n> +\n> +class Generator(generator.Generator):\n> +\n> + @staticmethod\n> + def GetTemplatePrefix():\n> + return 'libcamera_templates'\n> +\n> + def GetFilters(self):\n> + libcamera_filters = {\n> + 'all_types': GetAllTypes,\n> + 'bit_width': BitWidth,\n> + 'cap': Capitalize,\n\n 'cap': str.capitalize,\n\nand drop the custom Capitalize function.\n\n> + 'choose': Choose,\n> + 'comma_sep': CommaSep,\n> + 'default_value': GetDefaultValue,\n> + 'has_default_fields': HasDefaultFields,\n> + 'has_fd': HasFd,\n> + 'is_async': IsAsync,\n> + 'is_array': IsArray,\n> + 'is_controls': IsControls,\n> + 'is_enum': IsEnum,\n> + 'is_fd': IsFd,\n> + 'is_map': IsMap,\n> + 'is_plain_struct': IsPlainStruct,\n> + 'is_pod': IsPod,\n> + 'is_str': IsStr,\n> + 'method_input_has_fd': MethodInputHasFd,\n> + 'method_output_has_fd': MethodOutputHasFd,\n> + 'method_param_names': MethodParamNames,\n> + 'method_param_inputs': MethodParamInputs,\n> + 'method_param_outputs': MethodParamOutputs,\n> + 'method_parameters': MethodParameters,\n> + 'method_return_value': MethodReturnValue,\n> + 'name': GetNameForElement,\n> + 'name_full': GetFullNameForElement,\n> + 'needs_control_serializer': NeedsControlSerializer,\n> + 'params_comma_sep': ParamsCommaSep,\n> + 'with_default_values': WithDefaultValues,\n> + 'with_fds': WithFds,\n> + }\n> + return libcamera_filters\n> +\n> + def _GetJinjaExports(self):\n\nWould it make sense to store the return value of\nModuleClassName(self.module) in a local variable instead of calling the\nfunction several times below ? If self.module is constant during the\nlifetime of the Generator (I think it is), you could even compute it in\nthe constructor and store it in a member variable.\n\n> + return {\n> + 'base_controls': '%s::controls' % ModuleClassName(self.module),\n> + 'cmd_enum_name': '_%sCmd' % ModuleClassName(self.module),\n> + 'cmd_event_enum_name': '_%sEventCmd' % ModuleClassName(self.module),\n> + 'enums': self.module.enums,\n> + 'has_array': len([x for x in self.module.kinds.keys() if x[0] == 'a']) > 0,\n> + 'has_map': len([x for x in self.module.kinds.keys() if x[0] == 'm']) > 0,\n> + 'has_namespace': self.module.mojom_namespace != '',\n> + 'imports': self.module.imports,\n\nI don't see any mention of 'imports' in the templates. Is this used\ninternally by mojom and/or jinja, or is it unused ?\n\n> + 'interface_cb': self.module.interfaces[1],\n\ninterface_event ?\n\n> + 'interface_main': self.module.interfaces[0],\n\nIs there a guarantee the interfaces will always be specified in this\norder ? The code in ValidateInterfaces() finds interfaces based on their\nname.\n\n> + 'interface_name': 'IPA%sInterface' % ModuleClassName(self.module),\n> + 'ipc_name': 'IPAIPCUnixSocket',\n\nThis seems unused.\n\n> + 'kinds': self.module.kinds,\n> + 'module': self.module,\n\nSame for those two (unless they are used elsewhere than in the\ntemplates).\n\n> + 'module_name': ModuleName(self.module.path),\n> + 'module_class_name': ModuleClassName(self.module),\n\nDitto.\n\n> + 'namespace': self.module.mojom_namespace.split('.'),\n> + 'namespace_str': self.module.mojom_namespace.replace('.', '::') if\n> + self.module.mojom_namespace is not None else '',\n> + 'proxy_name': 'IPAProxy%s' % ModuleClassName(self.module),\n> + 'proxy_worker_name': 'IPAProxy%sWorker' % ModuleClassName(self.module),\n\nSame here.\n\n> + 'structs_nonempty': [x for x in self.module.structs if len(x.fields) > 0],\n> + 'year': datetime.datetime.now().year,\n> + }\n> +\n> +\n> + @UseJinja('module_ipa_interface.h.tmpl')\n> + def _GenerateDataHeader(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_serializer.h.tmpl')\n> + def _GenerateSerializer(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy.cpp.tmpl')\n> + def _GenerateProxyCpp(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy.h.tmpl')\n> + def _GenerateProxyHeader(self):\n> + return self._GetJinjaExports()\n> +\n> + @UseJinja('module_ipa_proxy_worker.cpp.tmpl')\n> + def _GenerateProxyWorker(self):\n> + return self._GetJinjaExports()\n> +\n> + def GenerateFiles(self, unparsed_args):\n> + parser = argparse.ArgumentParser()\n> + parser.add_argument('--libcamera_generate_header', action='store_true')\n> + parser.add_argument('--libcamera_generate_serializer', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_cpp', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_h', action='store_true')\n> + parser.add_argument('--libcamera_generate_proxy_worker', action='store_true')\n> + parser.add_argument('--libcamera_output_path')\n> + args = parser.parse_args(unparsed_args)\n> +\n> + ValidateInterfaces(self.module.interfaces)\n> +\n> + fileutil.EnsureDirectoryExists(os.path.dirname(args.libcamera_output_path))\n> +\n> + module_name = ModuleName(self.module.path)\n\nThis seems unused.\n\n> +\n> + if args.libcamera_generate_header:\n> + self.Write(self._GenerateDataHeader(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_serializer:\n> + self.Write(self._GenerateSerializer(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_cpp:\n> + self.Write(self._GenerateProxyCpp(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_h:\n> + self.Write(self._GenerateProxyHeader(),\n> + args.libcamera_output_path)\n> +\n> + if args.libcamera_generate_proxy_worker:\n> + self.Write(self._GenerateProxyWorker(),\n> + args.libcamera_output_path)\n\nI'm sure we could avoid the manual unrolling of all this, using a\ndictionary to store the template files and iterating over it to add\narguments to the parser and replacing the above code, but it's probably\nnot worth it as I don't expect this will change a lot.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 3ED49BE176\n\tfor ;\n\tThu, 19 Nov 2020 12:39:26 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 9E98C615A8;\n\tThu, 19 Nov 2020 13:39:25 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 391C2615A1\n\tfor ;\n\tThu, 19 Nov 2020 13:39:24 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 9A701292;\n\tThu, 19 Nov 2020 13:39:23 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"brfg0+Ly\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1605789563;\n\tbh=kG/UMXdhYgZZDjsIK/hxC7D95ThUAlquoCLMGnYpd18=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=brfg0+LyVRtyY9ZH1ZiNITg1Plk/fvdD56Pl779Sd1X6naVDBpo2xVzTmfrmPOKRE\n\tKz36j+b3LRLosureD2p8mYSR0/gJ6zRQDeH4s2zjZhScujrHHNBX8FiqeP9bRPt3+C\n\tRJ5RNBAgHWS8KoXaQx7/0NO7DtefzaeXe87Bm/DU=","Date":"Thu, 19 Nov 2020 14:39:17 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Message-ID":"<20201119123917.GA4563@pendragon.ideasonboard.com>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201106103707.49660-5-paul.elder@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13808,"web_url":"https://patchwork.libcamera.org/comment/13808/","msgid":"<20201119172100.GC4563@pendragon.ideasonboard.com>","date":"2020-11-19T17:21:00","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nThank you for the patch.\n\nOn Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> Add templates to mojo to generate code for the IPC mechanism. These\n> templates generate:\n> - module header\n> - module serializer\n> - IPA proxy cpp, header, and worker\n> \n> Given an input data definition mojom file for a pipeline.\n> \n> Signed-off-by: Paul Elder \n> \n> ---\n> Changes in v4:\n> For the non-template files:\n> - rename IPA{pipeline_name}CallbackInterface to\n> IPA{pipeline_name}EventInterface\n> - to avoid the notion of \"callback\" and emphasize that it's an event\n> - add support for strings in custom structs\n> - add validation, that async methods must not have return values\n> - it throws exception and isn't very clear though...?\n> - rename controls to libcamera::{pipeline_name}::controls (controls is\n> now lowercase)\n> - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> - same for their corresponding template files\n> For the template files:\n> - fix spacing (now it's all {{var}} instead of some {{ var }})\n> - except if it's code, so code is still {{ code }}\n> - move inclusion of corresponding header to first in the inclusion list\n> - fix copy&paste errors\n> - change snake_case to camelCase in the generated code\n> - template code still uses snake_case\n> - change the generated command enums to an enum class, and make it\n> capitalized (instead of allcaps)\n> - add length checks to recvIPC (in proxy)\n> - fix some template spacing\n> - don't use const for PODs in function/signal parameters\n> - add the proper length checks to readPOD/appendPOD\n> - the helper functions for reading and writing PODs to and from\n> serialized data\n> - rename readUInt/appendUInt to readPOD/appendPOD\n> - add support for strings in custom structs\n> \n> Changes in v3:\n> - add support for namespaces\n> - fix enum assignment (used to have +1 for CMD applied to all enums)\n> - use readHeader, writeHeader, and eraseHeader as static class functions\n> of IPAIPCUnixSocket (in the proxy worker)\n> - add requirement that base controls *must* be defined in\n> libcamera::{pipeline_name}::Controls\n> \n> Changes in v2:\n> - mandate the main and callback interfaces, and init(), start(), stop()\n> and their parameters\n> - fix returning single pod value from IPC-called function\n> - add licenses\n> - improve auto-generated message\n> - other fixes related to serdes\n> ---\n> .../module_ipa_interface.h.tmpl | 113 ++++\n> .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> .../module_ipa_proxy.h.tmpl | 118 +++++\n> .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> .../module_ipa_serializer.h.tmpl | 44 ++\n> .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> 8 files changed, 1673 insertions(+)\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n\nThe proxy .h and .cpp files are named ipa_proxy_{pipeline}.{h,cpp},\nwhile the other generated files are named {pipeline}_ipa_interface.h and\n{pipeline}_ipa_serializer.h. Could you pick one naming convention, and\napply it to all files ? The .tmpl files should then follow the same\nconvention.\n\n> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n\nAnd now, for the templates.\n\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> new file mode 100644\n> index 00000000..a470b99e\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> @@ -0,0 +1,113 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n\nComing back to the question of copyright on generated code.\n\nFirst of all, the template itself is a creative work, which should have\na copyright notice, with a fixed date. This should go to the comment at\nthe beginning of the file.\n\nThe generated file is the result of a non-creative process that takes\ncreative work as input. As such, it is covered by the copyright of the\ntemplate, and the mojom file. Would it be difficult to extract the\ncopyright data of the .mojom file and add it here ? I'm thnking about\nthe following at the beginning of the template:\n\n{#-\n # SPDX-License-Identifier: LGPL-2.1-or-later\n # Copyright (C) 2020, Google Inc.\n-#}\n/* SPDX-License-Identifier: LGPL-2.1-or-later */\n/*\n * Copyright (C) 2020, Google Inc.\n * {{copyright}}\n *\n * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n *\n * This file is auto-generated. Do not edit.\n */\n\nwith {{copyright}} being replaced with the copyright statement from the\n.mojom file. Mojo won't give you the information, but if you pass the\ntemplate name to _GetJinjaExport(), it may not be too difficult to\nextract the copyright line.\n\nThis may not be worth it, so feel free to just have a fixed copyright\nhere:\n\n/* SPDX-License-Identifier: LGPL-2.1-or-later */\n/*\n * Copyright (C) 2020, Google Inc.\n *\n * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n *\n * This file is auto-generated. Do not edit.\n */\n\n> + *\n> + * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> +#define __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> +\n> +#include \n> +#include \n> +\n> +{% if has_map %}#include {% endif %}\n> +{% if has_array %}#include {% endif %}\n> +\n> +namespace libcamera {\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +enum class {{cmd_enum_name}} {\n> +\tExit = 0,\n> +{%- for method in interface_main.methods %}\n> +\t{{method.mojom_name|cap}} = {{loop.index}},\n> +{%- endfor %}\n> +};\n> +\n> +enum class {{cmd_event_enum_name}} {\n> +{%- for method in interface_cb.methods %}\n> +\t{{method.mojom_name|cap}} = {{loop.index}},\n> +{%- endfor %}\n> +};\n> +\n> +{% for enum in enums %}\n> +enum {{enum.mojom_name}} {\n> +{%- for field in enum.fields %}\n> +\t{{field.mojom_name}} = {{field.numeric_value}},\n> +{%- endfor %}\n> +};\n> +{% endfor %}\n> +\n> +{%- for struct in structs_nonempty %}\n> +struct {{struct.mojom_name}}\n> +{\n> +public:\n> +\t{{struct.mojom_name}}() {%- if struct|has_default_fields %}\n> +\t\t:{% endif %}\n> +{%- for field in struct.fields|with_default_values -%}\n> +{{\" \" if loop.first}}{{field.mojom_name}}_({{field|default_value}}){{\", \" if not loop.last}}\n> +{%- endfor %}\n> +\t{\n> +\t}\n> +\n> +\t~{{struct.mojom_name}}() {}\n\nI think you can skip empty destructors.\n\n> +\n> +\t{{struct.mojom_name}}(\n> +{%- for field in struct.fields -%}\n> +{{field|name}} {{field.mojom_name}}{{\", \" if not loop.last}}\n\nShould non-trivial parameters be passed by const reference instead of\nvalue ?\n\n> +{%- endfor -%}\n> +)\n> +\t\t:\n> +{%- for field in struct.fields -%}\n> +{{\" \" if loop.first}}{{field.mojom_name}}_({{field.mojom_name}}){{\", \" if not loop.last}}\n> +{%- endfor %}\n> +\t{\n> +\t}\n> +{% for field in struct.fields %}\n> +\t{{field|name}} {{field.mojom_name}}_;\n> +{%- endfor %}\n> +};\n> +{% endfor %}\n> +\n> +{#-\n> +Any consts or #defines should be moved to the mojom file when possible.\n> +If anything needs to be #included, then {{module_name}}.h needs to have the\n> +#include.\n> +#}\n> +class {{interface_name}} : public IPAInterface\n> +{\n> +public:\n> +\tvirtual ~{{interface_name}}() {}\n\nThe base class has a virtual destructor already, so you can drop this\none.\n\n> +{% for method in interface_main.methods %}\n> +\tvirtual {{method|method_return_value}} {{method.mojom_name}}(\n> +{%- for param in method|method_parameters %}\n> +\t\t{{param}}{{- \",\" if not loop.last}}\n> +{%- endfor -%}\n> +) = 0;\n> +{% endfor %}\n> +\n> +{%- for method in interface_cb.methods %}\n> +\tSignal<\n> +{%- for param in method.parameters -%}\n> +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> +\t\t{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +> {{method.mojom_name}};\n> +{% endfor -%}\n> +};\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n\nThis should be\n\n} /* namespace {{ns}} */\n\nSame in a few locations below.\n\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> new file mode 100644\n> index 00000000..9328c7ca\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> @@ -0,0 +1,238 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}.cpp - Image Processing Algorithm proxy for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#include \n> +\n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +#include \"libcamera/internal/ipa_module.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/log.h\"\n> +#include \"libcamera/internal/process.h\"\n> +#include \"libcamera/internal/thread.h\"\n\nYOu got the alphabetical order nearly right :-)\n\n> +\n> +namespace libcamera {\n> +\n> +LOG_DECLARE_CATEGORY(IPAProxy)\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +{{proxy_name}}::{{proxy_name}}(IPAModule *ipam, bool isolate)\n> +\t: IPAProxy(ipam), running_(false),\n> +\t isolate_(isolate)\n> +{\n> +\tLOG(IPAProxy, Debug)\n> +\t\t<< \"initializing {{module_name}} proxy: loading IPA from \"\n> +\t\t<< ipam->path();\n> +\n> +\tif (isolate_) {\n> +\t\tconst std::string proxyWorkerPath = resolvePath(\"ipa_proxy_{{module_name}}\");\n> +\t\tif (proxyWorkerPath.empty()) {\n> +\t\t\tLOG(IPAProxy, Error)\n> +\t\t\t\t<< \"Failed to get proxy worker path\";\n> +\t\t\treturn;\n> +\t\t}\n> +\n> +\t\tipc_ = std::make_unique(ipam->path().c_str(), proxyWorkerPath.c_str());\n\nLine wrap maybe ?\n\n> +\t\tif (!ipc_->isValid()) {\n> +\t\t\tLOG(IPAProxy, Error) << \"Failed to create IPAIPC\";\n> +\t\t\treturn;\n> +\t\t}\n> +\n> +{% if interface_cb.methods|length > 0 %}\n> +\t\tipc_->recvIPC.connect(this, &{{proxy_name}}::recvIPC);\n> +{% endif %}\n\nI think we can assume there will always be events. An IPA module that\nonly consumes data would be a bit useless, wouldn't it ? :-)\n\n> +\n> +\t\tvalid_ = true;\n> +\t\treturn;\n> +\t}\n> +\n> +\tif (!ipam->load())\n> +\t\treturn;\n> +\n> +\tIPAInterface *ipai = ipam->createInterface();\n> +\tif (!ipai) {\n> +\t\tLOG(IPAProxy, Error)\n> +\t\t\t<< \"Failed to create IPA context for \" << ipam->path();\n> +\t\treturn;\n> +\t}\n> +\n> +\tipa_ = std::unique_ptr<{{interface_name}}>(dynamic_cast<{{interface_name}} *>(ipai));\n\nDo we need a dynamic cast ?\n\n> +\tproxy_.setIPA(ipa_.get());\n> +\n> +{% for method in interface_cb.methods %}\n> +\tipa_->{{method.mojom_name}}.connect(this, &{{proxy_name}}::{{method.mojom_name}}Thread);\n> +{%- endfor %}\n> +\n> +\tvalid_ = true;\n> +}\n> +\n> +{{proxy_name}}::~{{proxy_name}}()\n> +{\n> +\tif (isolate_)\n> +\t\tipc_->sendAsync(static_cast({{cmd_enum_name}}::Exit), {}, {});\n> +}\n> +\n> +{% if interface_cb.methods|length > 0 %}\n> +void {{proxy_name}}::recvIPC(std::vector &data, std::vector &fds)\n\nHow about calling this receiveMessage() ?\n\n> +{\n> +\tif (data.size() < 8) {\n> +\t\tLOG(IPAProxy, Error)\n> +\t\t\t<< \"Didn't receive enough bytes to parse event\";\n> +\t\treturn;\n> +\t}\n> +\n> +\t{{cmd_event_enum_name}} cmd = static_cast<{{cmd_event_enum_name}}>((\n> +\t\tdata[0]) | (data[1] << 8) | (data[2] << 16) | (data[3] << 24));\n> +\n> +\t/* Need to skip another 4 bytes for the sequence number. */\n\nIs the protocol documented somewhere ?\n\nThere's a bit of an imbalance in the API, with recvIPC() having to deal\nwith unpacking the command and skipping the sequence number, and\nsendSync() and sendAsync() dealing with those internally. Could\nrecvIPC() be called with the command and two spans instead ?\n\n> +\tstd::vector::iterator vec = data.begin() + 8;\n> +\tsize_t dataSize = data.size() - 8;\n> +\tswitch (cmd) {\n> +{%- for method in interface_cb.methods %}\n> +\tcase {{cmd_event_enum_name}}::{{method.mojom_name|cap}}: {\n> +\t\t{{method.mojom_name}}IPC(vec, dataSize, fds);\n> +\t\tbreak;\n> +\t}\n> +{%- endfor %}\n> +\tdefault:\n> +\t\tLOG(IPAProxy, Error) << \"Unknown command \" << static_cast(cmd);\n> +\t}\n> +}\n> +{%- endif %}\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method)}}\n> +{\n> +\tif (isolate_)\n> +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}IPC(\n> +{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +\telse\n> +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}Thread(\n> +{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +}\n> +\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> +{\n> +{%- if method.mojom_name == \"init\" %}\n> +\t{{proxy_funcs.init_thread_body()}}\n> +{%- elif method.mojom_name == \"start\" %}\n> +\t{{proxy_funcs.start_thread_body()}}\n> +{%- elif method.mojom_name == \"stop\" %}\n> +\t{{proxy_funcs.stop_thread_body()}}\n> +{%- elif not method|is_async %}\n> +\tipa_->{{method.mojom_name}}(\n> +\t{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +\t{%- endfor -%}\n> +);\n> +{% elif method|is_async %}\n> +\tproxy_.invokeMethod(&ThreadProxy::{{method.mojom_name}}, ConnectionTypeQueued,\n> +\t{%- for param in method|method_param_names -%}\n> +\t\t{{param}}{{- \", \" if not loop.last}}\n> +\t{%- endfor -%}\n> +);\n> +{%- endif %}\n> +}\n> +\n> +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\")}}\n> +{\n> +{%- set has_input = true if method|method_param_inputs|length > 0 %}\n> +{%- set has_output = true if method|method_param_outputs|length > 0 or method|method_return_value != \"void\" %}\n> +{%- if has_input %}\n> +\tstd::vector _ipcInputBuf;\n> +{%- if method|method_input_has_fd %}\n> +\tstd::vector _ipcInputFds;\n> +{%- endif %}\n> +{%- endif %}\n> +{%- if has_output %}\n> +\tstd::vector _ipcOutputBuf;\n> +{%- if method|method_output_has_fd %}\n> +\tstd::vector _ipcOutputFds;\n> +{%- endif %}\n> +{%- endif %}\n> +\n> +{{proxy_funcs.serialize_call(method|method_param_inputs, '_ipcInputBuf', '_ipcInputFds', base_controls)}}\n> +\n> +{%- set input_buf = \"_ipcInputBuf\" if has_input else \"{}\" %}\n> +{%- set fds_buf = \"_ipcInputFds\" if method|method_input_has_fd else \"{}\" %}\n> +{%- set cmd = cmd_enum_name + \"::\" + method.mojom_name|cap %}\n> +{% if method|is_async %}\n> +\tint ret = ipc_->sendAsync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}});\n> +{%- else %}\n> +\tint ret = ipc_->sendSync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}}\n> +{{- \", &_ipcOutputBuf\" if has_output -}}\n> +{{- \", &_ipcOutputFds\" if has_output and method|method_output_has_fd -}}\n> +);\n> +{%- endif %}\n> +\tif (ret < 0) {\n> +\t\tLOG(IPAProxy, Error) << \"Failed to call {{method.mojom_name}}\";\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\treturn static_cast<{{method|method_return_value}}>(ret);\n> +{%- else %}\n> +\t\treturn;\n> +{%- endif %}\n> +\t}\n> +{% if method|method_return_value != \"void\" %}\n> +\treturn IPADataSerializer<{{method.response_parameters|first|name}}>::deserialize(_ipcOutputBuf, 0);\n> +{% elif method|method_param_outputs|length > 0 %}\n> +{{proxy_funcs.deserialize_call(method|method_param_outputs, '_ipcOutputBuf', '_ipcOutputFds')}}\n> +{% endif -%}\n> +}\n> +\n> +{% endfor %}\n> +\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> +{\n> +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> +}\n> +\n> +void {{proxy_name}}::{{method.mojom_name}}IPC(\n> +\tstd::vector::iterator data,\n> +\tsize_t dataSize,\n> +\t[[maybe_unused]] std::vector &fds)\n> +{ \n> +{%- for param in method.parameters %}\n> +\t{{param|name}} {{param.mojom_name}};\n> +{%- endfor %}\n> +{{proxy_funcs.deserialize_call(method.parameters, 'data', 'fds', false, false, true, 'dataSize')}}\n> +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> +}\n> +{% endfor %}\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> new file mode 100644\n> index 00000000..3fb7192f\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> @@ -0,0 +1,118 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}.h - Image Processing Algorithm proxy for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> +#define __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/thread.h\"\n> +\n> +namespace libcamera {\n> +{%- if has_namespace %}\n> +{% for ns in namespace %}\n> +namespace {{ns}} {\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +class {{proxy_name}} : public IPAProxy, public {{interface_name}}, public Object\n> +{\n> +public:\n> +\t{{proxy_name}}(IPAModule *ipam, bool isolate);\n> +\t~{{proxy_name}}();\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"\", false, true)|indent(8, true)}};\n> +{% endfor %}\n> +\n> +{%- for method in interface_cb.methods %}\n> +\tSignal<\n> +{%- for param in method.parameters -%}\n> +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> +\t\t{{- \", \" if not loop.last}}\n> +{%- endfor -%}\n> +> {{method.mojom_name}};\n> +{% endfor %}\n> +\n> +private:\n> +\tvoid recvIPC(std::vector &data, std::vector &fds);\n\ndata and fds shouldn't be modified, let's make them const. I'd even go\none step further, and make them Span and Span.\n\n> +\n> +{% for method in interface_main.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\", false)|indent(8, true)}};\n> +{% endfor %}\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> +\tvoid {{method.mojom_name}}IPC(\n> +\t\tstd::vector::iterator data,\n> +\t\tsize_t dataSize,\n> +\t\tstd::vector &fds);\n> +{% endfor %}\n> +\n> +\t/* Helper class to invoke async functions in another thread. */\n> +\tclass ThreadProxy : public Object\n> +\t{\n> +\tpublic:\n> +\t\tvoid setIPA({{interface_name}} *ipa)\n> +\t\t{\n> +\t\t\tipa_ = ipa;\n> +\t\t}\n> +\n> +\t\tint start()\n> +\t\t{\n> +\t\t\treturn ipa_->start();\n> +\t\t}\n> +\n> +\t\tvoid stop()\n> +\t\t{\n> +\t\t\tipa_->stop();\n> +\t\t}\n> +{% for method in interface_main.methods %}\n> +{%- if method|is_async %}\n> +\t\t{{proxy_funcs.func_sig(proxy_name, method, \"\", false)|indent(16)}}\n> +\t\t{\n> +\t\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> +\t\t}\n> +{%- endif %}\n> +{%- endfor %}\n> +\n> +\tprivate:\n> +\t\t{{interface_name}} *ipa_;\n> +\t};\n> +\n> +\tbool running_;\n> +\tThread thread_;\n> +\tThreadProxy proxy_;\n> +\tstd::unique_ptr<{{interface_name}}> ipa_;\n> +\n> +\tconst bool isolate_;\n> +\n> +\tstd::unique_ptr ipc_;\n> +\n> +\tControlSerializer controlSerializer_;\n\nI wonder if it would make sense to move some of the fields to the\nIPAProxy class (and possible some of the functions too). That can be\ndone on top in a separate series.\n\n> +};\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace|reverse %}\n> +} /* {{ns}} */\n> +{% endfor %}\n> +{%- endif %}\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> new file mode 100644\n> index 00000000..dca4f99d\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> @@ -0,0 +1,187 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * ipa_proxy_{{module_name}}_worker.cpp - Image Processing Algorithm proxy worker for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/camera_sensor.h\"\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> +#include \"libcamera/internal/ipa_module.h\"\n> +#include \"libcamera/internal/ipa_proxy.h\"\n> +#include \"libcamera/internal/ipc_unixsocket.h\"\n> +#include \"libcamera/internal/log.h\"\n> +#include \"libcamera/internal/thread.h\"\n> +\n> +using namespace libcamera;\n> +\n> +LOG_DEFINE_CATEGORY({{proxy_name}}Worker)\n> +\n> +{%- if has_namespace %}\n> +{% for ns in namespace -%}\n> +using namespace {{ns}};\n> +{% endfor %}\n> +{%- endif %}\n> +\n> +struct CallData {\n> +\tIPCUnixSocket::Payload *response;\n> +\tbool done;\n> +};\n\nThis doesn't seem to be used.\n\n> +\n> +{{interface_name}} *ipa_;\n> +IPCUnixSocket socket_;\n> +\n> +ControlSerializer controlSerializer_;\n> +\n> +bool exit_ = false;\n\nI still think it would be nicer to move the data and functions to a\nclass, with a single instance created in the main() function.\n\n> +\n> +void readyRead(IPCUnixSocket *socket)\n> +{\n> +\tIPCUnixSocket::Payload _message, _response;\n> +\tint _retRecv = socket->receive(&_message);\n> +\tif (_retRecv) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"Receive message failed\" << _retRecv;\n> +\t\treturn;\n> +\t}\n> +\n> +\tuint32_t _cmdUint, _seq;\n> +\tstd::tie(_cmdUint, _seq) = IPAIPCUnixSocket::readHeader(_message);\n\nI think there's some room to refactor the IPC-related classes. The\npayload and header should probably not be tied to the IPAIPCUnixSocket\nclass for instance, as they should be independent of the underlying\nmechanism. I'll try to comment more on this topic when reviewing the\ncorresponding patches.\n\n> +\tIPAIPCUnixSocket::eraseHeader(_message);\n> +\n> +\t{{cmd_enum_name}} _cmd = static_cast<{{cmd_enum_name}}>(_cmdUint);\n> +\n> +\tswitch (_cmd) {\n> +\tcase {{cmd_enum_name}}::Exit: {\n> +\t\texit_ = true;\n> +\t\tbreak;\n> +\t}\n> +\n> +{% for method in interface_main.methods %}\n> +\tcase {{cmd_enum_name}}::{{method.mojom_name|cap}}: {\n> +\t{{proxy_funcs.deserialize_call(method|method_param_inputs, '_message.data', '_message.fds', false, true)|indent(8, true)}}\n> +{% for param in method|method_param_outputs %}\n> +\t\t{{param|name}} {{param.mojom_name}};\n> +{% endfor %}\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\t{{method|method_return_value}} _callRet = ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> +{%- else %}\n> +\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}}\n> +{{- \", \" if method|method_param_outputs|params_comma_sep -}}\n> +{%- for param in method|method_param_outputs -%}\n> +&{{param.mojom_name}}{{\", \" if not loop.last}}\n> +{%- endfor -%}\n> +);\n> +{%- endif %}\n> +{% if not method|is_async %}\n> +\t\tIPAIPCUnixSocket::writeHeader(_response, _cmdUint, _seq);\n> +{%- if method|method_return_value != \"void\" %}\n> +\t\tstd::vector _callRetBuf;\n> +\t\tstd::tie(_callRetBuf, std::ignore) =\n> +\t\t\tIPADataSerializer<{{method|method_return_value}}>::serialize(_callRet);\n> +\t\t_response.data.insert(_response.data.end(), _callRetBuf.begin(), _callRetBuf.end());\n> +{%- else %}\n> +\t{{proxy_funcs.serialize_call(method|method_param_outputs, \"_response.data\", \"_response.fds\", base_controls)|indent(8, true)}}\n> +{%- endif %}\n> +\t\tint _ret = socket_.send(_response);\n> +\t\tif (_ret < 0) {\n> +\t\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t\t<< \"Reply to {{method.mojom_name}}() failed\" << _ret;\n> +\t\t}\n> +\t\tLOG({{proxy_name}}Worker, Debug) << \"Done replying to {{method.mojom_name}}()\";\n> +{%- endif %}\n> +\t\tbreak;\n> +\t}\n> +{% endfor %}\n> +\tdefault:\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"Unknown command \" << _cmdUint;\n> +\t}\n> +}\n> +\n> +{% for method in interface_cb.methods %}\n> +{{proxy_funcs.func_sig(proxy_name, method, \"\", false)}}\n> +{\n> +\tIPCUnixSocket::Payload _message;\n> +\n> +\tIPAIPCUnixSocket::writeHeader(_message, static_cast({{cmd_event_enum_name}}::{{method.mojom_name|cap}}), 0);\n> +\t{{proxy_funcs.serialize_call(method|method_param_inputs, \"_message.data\", \"_message.fds\", base_controls)}}\n> +\n> +\tsocket_.send(_message);\n> +\n> +\tLOG({{proxy_name}}Worker, Debug) << \"{{method.mojom_name}} done\";\n> +}\n> +{% endfor %}\n> +\n> +int main(int argc, char **argv)\n> +{\n> +\t/* Uncomment this for debugging. */\n> +#if 0\n> +\tstd::string logPath = \"/tmp/libcamera.worker.\" +\n> +\t\t\t std::to_string(getpid()) + \".log\";\n> +\tlogSetFile(logPath.c_str());\n> +#endif\n\nLikely something we'll have to handle more dynamically, but it doesn't\nhave to be done now. A \\todo comment would be good though.\n\n> +\n> +\tif (argc < 3) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"Tried to start worker with no args\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\n> +\tint fd = std::stoi(argv[2]);\n> +\tLOG({{proxy_name}}Worker, Info)\n> +\t\t<< \"Starting worker for IPA module \" << argv[1]\n> +\t\t<< \" with IPC fd = \" << fd;\n> +\n> +\tstd::unique_ptr ipam = std::make_unique(argv[1]);\n> +\tif (!ipam->isValid() || !ipam->load()) {\n> +\t\tLOG({{proxy_name}}Worker, Error)\n> +\t\t\t<< \"IPAModule \" << argv[1] << \" isn't valid\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\n> +\tif (socket_.bind(fd) < 0) {\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"IPC socket binding failed\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +\tsocket_.readyRead.connect(&readyRead);\n> +\n> +\tipa_ = dynamic_cast<{{interface_name}} *>(ipam->createInterface());\n> +\tif (!ipa_) {\n> +\t\tLOG({{proxy_name}}Worker, Error) << \"Failed to create IPA interface instance\";\n> +\t\treturn EXIT_FAILURE;\n> +\t}\n> +{% for method in interface_cb.methods %}\n> +\tipa_->{{method.mojom_name}}.connect(&{{method.mojom_name}});\n> +{%- endfor %}\n> +\n> +\tLOG({{proxy_name}}Worker, Debug) << \"Proxy worker successfully started\";\n> +\n> +\t/* \\todo upgrade listening loop */\n\nUpgrade to what ? :-)\n\n> +\tEventDispatcher *dispatcher = Thread::current()->eventDispatcher();\n> +\twhile (!exit_)\n> +\t\tdispatcher->processEvents();\n> +\n> +\tdelete ipa_;\n> +\tsocket_.close();\n> +\n> +\treturn 0;\n> +}\n> diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> new file mode 100644\n> index 00000000..675f9adf\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> @@ -0,0 +1,44 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{%- import \"serializer.tmpl\" as serializer -%}\n> +\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) {{year}}, Google Inc.\n> + *\n> + * {{module_name}}_serializer.h - Image Processing Algorithm data serializer for {{module_name}}\n> + *\n> + * This file is auto-generated. Do not edit.\n> + */\n> +\n> +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> +\n> +#include \n> +#include \n> +\n> +#include \"libcamera/internal/control_serializer.h\"\n> +#include \"libcamera/internal/ipa_data_serializer.h\"\n> +\n> +#include \n> +#include \n\nThese should go first.\n\n> +\n> +namespace libcamera {\n> +\n> +LOG_DECLARE_CATEGORY(IPADataSerializer)\n> +{% for struct in structs_nonempty %}\n> +template<>\n> +class IPADataSerializer<{{struct|name_full(namespace_str)}}>\n> +{\n> +public:\n> +{{- serializer.serializer(struct, base_controls, namespace_str)}}\n> +{%- if struct|has_fd %}\n> +{{serializer.deserializer_fd(struct, namespace_str)}}\n> +{%- else %}\n> +{{serializer.deserializer_no_fd(struct, namespace_str)}}\n> +{%- endif %}\n> +};\n> +{% endfor %}\n> +\n> +} /* namespace libcamera */\n> +\n> +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__ */\n> diff --git a/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> new file mode 100644\n> index 00000000..f6836034\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> @@ -0,0 +1,205 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{#\n> + # \\brief Generate fuction prototype\n> + #\n> + # \\param class Class name\n> + # \\param method mojom Method object\n> + # \\param suffix Suffix to append to \\a method function name\n> + # \\param need_class_name True to generate class name with function\n> + # \\param override True to generate override tag after the function prototype\n> + #}\n> +{%- macro func_sig(class, method, suffix, need_class_name = true, override = false) -%}\n\nThere's one occurrence of a call to this macro with two arguments only.\nShould suffix have a default of \"\" ?\n\n> +{{method|method_return_value}} {{class + \"::\" if need_class_name}}{{method.mojom_name}}{{suffix}}(\n> +{%- for param in method|method_parameters %}\n> +\t{{param}}{{- \",\" if not loop.last}}\n> +{%- endfor -%}\n> +){{\" override\" if override}}\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA init() function for thread\n> + #}\n> +{%- macro init_thread_body() -%}\n> +\tint ret = ipa_->init(settings);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\tproxy_.moveToThread(&thread_);\n> +\n> +\treturn 0;\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA start() function for thread\n> + #}\n> +{%- macro start_thread_body() -%}\n> +\trunning_ = true;\n> +\tthread_.start();\n> +\n> +\treturn proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking);\n> +{%- endmacro -%}\n> +\n> +{#\n> + # \\brief Generate function body for IPA stop() function for thread\n> + #}\n> +{%- macro stop_thread_body() -%}\n> +\tif (!running_)\n> +\t\treturn;\n> +\n> +\trunning_ = false;\n> +\n> +\tproxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking);\n> +\n> +\tthread_.exit();\n> +\tthread_.wait();\n> +{%- endmacro -%}\n> +\n> +\n> +{#\n> + # \\brief Serialize multiple objects into data buffer and fd vector\n> + #\n> + # Generate code to serialize multiple objects, as specified in \\a params\n> + # (which are the parameters to some function), into \\a buf data buffer and\n> + # \\a fds fd vector.\n> + # This code is meant to be used by the proxy, for serializing prior to IPC calls.\n> + #}\n> +{%- macro serialize_call(params, buf, fds, base_controls) %}\n> +{%- for param in params %}\n> +\tstd::vector {{param.mojom_name}}Buf;\n> +{%- if param|has_fd %}\n> +\tstd::vector {{param.mojom_name}}Fds;\n> +\tstd::tie({{param.mojom_name}}Buf, {{param.mojom_name}}Fds) =\n> +{%- else %}\n> +\tstd::tie({{param.mojom_name}}Buf, std::ignore) =\n> +{%- endif %}\n> +{%- if param|is_controls %}\n> +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}, {{param.mojom_name}}.infoMap() ? *{{param.mojom_name}}.infoMap() : {{base_controls}}\n> +{%- else %}\n> +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}\n> +{%- endif %}\n> +{{- \", &controlSerializer_\" if param|needs_control_serializer -}}\n> +);\n> +{%- endfor %}\n> +\n> +{%- if params|length > 1 %}\n> +{%- for param in params %}\n> +\tappendPOD({{buf}}, {{param.mojom_name}}Buf.size());\n> +{%- if param|has_fd %}\n> +\tappendPOD({{buf}}, {{param.mojom_name}}Fds.size());\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endif %}\n> +\n> +{%- for param in params %}\n> +\t{{buf}}.insert({{buf}}.end(), {{param.mojom_name}}Buf.begin(), {{param.mojom_name}}Buf.end());\n> +{%- endfor %}\n> +\n> +{%- for param in params %}\n> +{%- if param|has_fd %}\n> +\t{{fds}}.insert({{fds}}.end(), {{param.mojom_name}}Fds.begin(), {{param.mojom_name}}Fds.end());\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endmacro -%}\n\nI wonder if we could optimize this by avoiding the intermediate vectors,\nbut that's a candidate for a later optimization.\n\n> +\n> +\n> +{#\n> + # \\brief Deserialize a single object from data buffer and fd vector\n> + #\n> + # \\param pointer True deserializes the object into a dereferenced pointer\n\ns/True deserializes/If true, deserialize/ ?\n\nSame in other locations below.\n\n> + # \\param iter True treats \\a buf as an iterator instead of a vector\n> + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> + #\n> + # Generate code to deserialize a single object, as specified in \\a param,\n> + # from \\a buf data buffer and \\a fds fd vector.\n> + # This code is meant to be used by macro deserialize_call.\n> + #}\n> +{%- macro deserialize_param(param, pointer, loop, buf, fds, iter, data_size) -%}\n> +{{\"*\" if pointer}}{{param.mojom_name}} = IPADataSerializer<{{param|name}}>::deserialize(\n> +{%- if not iter %}\n> +\t{{buf}}.begin() + {{param.mojom_name}}Start,\n> +{%- else %}\n> +\t{{buf}} + {{param.mojom_name}}Start,\n> +{%- endif %}\n\nMaybe\n\n\t{{buf}}{{- \".begin()\" if not iter}} + {{param.mojom_name}}Start,\n\n?\n\n> +{%- if loop.last and not iter %}\n> +\t{{buf}}.end()\n> +{%- elif not iter %}\n> +\t{{buf}}.begin() + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> +{%- elif iter and loop.length == 1 %}\n> +\t{{buf}} + {{data_size}}\n> +{%- else %}\n> +\t{{buf}} + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> +{%- endif -%}\n\nI wonder if it would simplify the generated code if we standardized on\nSpan. Passing both the vector iterator and the size in separate\narguments can more easily result in them becoming out of sync. I was\nthinking the Span<>::subspan() function could be quite helpful here, but\nit doesn't handle overflows (requesting a subspan with offset > size is\nill-defined) :-S\n\nNot necessarily something to be addressed now, but do you think it would\nmake sense ? And of course if it helps fixing some of the issues\noutlined elsewhere in this review, we could already switch to span.\n\n> +{{- \",\" if param|has_fd}}\n> +{%- if param|has_fd %}\n> +\t{{fds}}.begin() + {{param.mojom_name}}FdStart,\n> +{%- if loop.last %}\n> +\t{{fds}}.end()\n> +{%- else %}\n> +\t{{fds}}.begin() + {{param.mojom_name}}FdStart + {{param.mojom_name}}FdsSize\n> +{%- endif -%}\n> +{%- endif -%}\n> +{{- \",\" if param|needs_control_serializer}}\n> +{%- if param|needs_control_serializer %}\n> +\t&controlSerializer_\n> +{%- endif -%}\n> +);\n> +{%- endmacro -%}\n> +\n> +\n> +{#\n> + # \\brief Deserialize multiple objects from data buffer and fd vector\n> + #\n> + # \\param pointer True deserializes objects into pointers, and adds a null check.\n> + # \\param declare True declares the objects in addition to deserialization.\n> + # \\param iter True treats \\a buf as an iterator instead of a vector\n> + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> + #\n> + # Generate code to deserialize multiple objects, as specified in \\a params\n> + # (which are the parameters to some function), from \\a buf data buffer and\n> + # \\a fds fd vector.\n> + # This code is meant to be used by the proxy, for deserializing after IPC calls.\n> + #}\n> +{%- macro deserialize_call(params, buf, fds, pointer = true, declare = false, iter = false, data_size = '') -%}\n> +{% set ns = namespace(size_offset = 0) %}\n> +{%- if params|length > 1 %}\n> +{%- for param in params %}\n> +\t[[maybe_unused]] size_t {{param.mojom_name}}BufSize = readPOD({{buf}}, {{ns.size_offset}}\n\nconst ?\n\n> +{%- if iter -%}\n> +, {{buf}} + {{data_size}}\n> +{%- endif -%}\n> +);\n> +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> +{%- if param|has_fd %}\n> +\t[[maybe_unused]] size_t {{param.mojom_name}}FdsSize = readPOD({{buf}}, {{ns.size_offset}}\n\nconst ?\n\n> +{%- if iter -%}\n> +, {{buf}} + {{data_size}}\n> +{%- endif -%}\n> +);\n> +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> +{%- endif %}\n> +{%- endfor %}\n> +{%- endif %}\n> +{% for param in params %}\n> +{%- if loop.first %}\n> +\tsize_t {{param.mojom_name}}Start = {{ns.size_offset}};\n\nconst ?\n\nSame in a few locations below.\n\n> +{%- else %}\n> +\tsize_t {{param.mojom_name}}Start = {{loop.previtem.mojom_name}}Start + {{loop.previtem.mojom_name}}BufSize;\n> +{%- endif %}\n> +{%- endfor %}\n> +{% for param in params|with_fds %}\n> +{%- if loop.first %}\n> +\tsize_t {{param.mojom_name}}FdStart = 0;\n> +{%- elif not loop.last %}\n> +\tsize_t {{param.mojom_name}}FdStart = {{loop.previtem.mojom_name}}FdStart + {{loop.previtem.mojom_name}}FdsSize;\n> +{%- endif %}\n> +{%- endfor %}\n> +{% for param in params %}\n> +\t{%- if pointer %}\n> +\tif ({{param.mojom_name}}) {\n> +{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(16, True)}}\n> +\t}\n> +\t{%- else %}\n> +\t{{param|name + \" \" if declare}}{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(8)}}\n> +\t{%- endif %}\n> +{% endfor %}\n> +{%- endmacro -%}\n> diff --git a/utils/ipc/generators/libcamera_templates/serializer.tmpl b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> new file mode 100644\n> index 00000000..51dbeb0e\n> --- /dev/null\n> +++ b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> @@ -0,0 +1,280 @@\n> +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> +{# Turn this into a C macro? #}\n> +{#\n> + # \\brief Verify that there is enough bytes to deserialize\n> + #\n> + # Generate code that verifies that \\a size is not greater than \\a dataSize.\n> + # Otherwise log an error with \\a name and \\a typename.\n> + #}\n> +{%- macro check_data_size(size, dataSize, name, typename) %}\n> +\t\tif ({{size}} > {{dataSize}}) {\n\n\nMaybe\n\t\tif ({{dataSize}} < {{size}}) {\n\n?\n\n> +\t\t\tLOG(IPADataSerializer, Error)\n> +\t\t\t\t<< \"Failed to deserialize {{name}}: not enough {{typename}}, expected \"\n\n\t\t\t\t<< \"Failed to deserialize \" << \"{{name}}\"\n\t\t\t\t<< \": not enough {{typename}}, expected \"\n\nThis should allow the compiler to deduplicate strings.\n\n> +\t\t\t\t<< ({{size}}) << \", got \" << ({{dataSize}});\n> +\t\t\treturn ret;\n> +\t\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Serialize some field into return vector\n> + #\n> + # Generate code to serialize \\a field into retData, including size of the\n> + # field and fds (where appropriate). \\a base_controls indicates the\n> + # default ControlInfoMap in the event that the ControlList does not have one.\n> + # This code is meant to be used by the IPADataSerializer specialization.\n> + #}\n> +{%- macro serializer_field(field, base_controls, namespace, loop) %}\n> +{%- if field|is_pod or field|is_enum %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t{%- if field|is_pod %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t{%- elif field|is_enum %}\n> +\t\t\tIPADataSerializer::serialize(data.{{field.mojom_name}}_);\n> +\t{%- endif %}\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +{%- elif field|is_fd %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t\tstd::vector {{field.mojom_name}}Fds;\n> +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> +{%- elif field|is_controls %}\n> +\t\tif (data.{{field.mojom_name}}_.size() > 0) {\n> +\t\t\tstd::vector {{field.mojom_name}};\n> +\t\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_,\n> +\t\t\t\t\tdata.{{field.mojom_name}}_.infoMap() ? *data.{{field.mojom_name}}_.infoMap() : {{base_controls}},\n> +\t\t\t\t\tcs);\n> +\t\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> +\t\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t\t} else {\n> +\t\t\tappendPOD(retData, 0);\n> +\t\t}\n> +{%- elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> +\t\tstd::vector {{field.mojom_name}};\n> +\t{%- if field|has_fd %}\n> +\t\tstd::vector {{field.mojom_name}}Fds;\n> +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> +\t{%- else %}\n> +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> +\t{%- endif %}\n> +\t{%- if field|is_array or field|is_map %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, cs);\n> +\t{%- elif field|is_str %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> +\t{%- else %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::serialize(data.{{field.mojom_name}}_, cs);\n> +\t{%- endif %}\n> +\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> +\t{%- if field|has_fd %}\n> +\t\tappendPOD(retData, {{field.mojom_name}}Fds.size());\n> +\t{%- endif %}\n> +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> +\t{%- if field|has_fd %}\n> +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> +\t{%- endif %}\n> +{%- else %}\n> +\t\t/* Unknown serialization for {{field.mojom_name}}. */\n> +{%- endif %}\n> +{%- endmacro %}\n\nSame here, I think there's room for optimization if we could avoid the\nintermediate vectors.\n\n> +\n> +\n> +{#\n> + # \\brief Deserialize some field into return struct\n> + #\n> + # Generate code to deserialize \\a field into object ret.\n> + # This code is meant to be used by the IPADataSerializer specialization.\n> + #}\n> +{%- macro deserializer_field(field, namespace, loop) %}\n> +{% if field|is_pod or field|is_enum %}\n> +\t{%- set field_size = (field|bit_width|int / 8)|int %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\t{%- if field|is_pod %}\n> +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer<{{field|name}}>::deserialize(m, m + {{field_size}}));\n\nDo we need the static_cast<>, doesn't\nIPADataSerializer<{{field|name}}>::deserialize() return a {{field|name}}\n?\n\n> +\t\t{%- else %}\n> +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer::deserialize(m, m + {{field_size}}));\n> +\t\t{%- endif %}\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- endif %}\n> +{% elif field|is_fd %}\n> +\t{%- set field_size = 1 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tret.{{field.mojom_name}}_ = IPADataSerializer<{{field|name}}>::deserialize(m, m + 1, n, n + 1, cs);\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t\tn += ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> +\t\tfdsSize -= ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> +\t{%- endif %}\n> +{% elif field|is_controls %}\n> +\t{%- set field_size = 4 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n\nconst ?\n\nYou call IPADataSerializer<{{field|name}}>::deserialize() above for POD\ntypes, and readPOD() here. Do we need the IPADataSerializer<>\nspecializations for all the PODs, can't we call readPOD() directly ?\n\n> +\t{%- set field_size = '4 + ' + field.mojom_name + 'Size' -%}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tif ({{field.mojom_name}}Size > 0)\n> +\t\t\tret.{{field.mojom_name}}_ =\n> +\t\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- endif %}\n> +{% elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> +\t{%- set field_size = 4 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n\nconst ?\n\n> +\t{%- if field|has_fd %}\n> +\t{%- set field_size = 8 %}\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'FdsSize', 'data')}}\n> +\t\tsize_t {{field.mojom_name}}FdsSize = readPOD(m, 4, data.end());\n\nconst ?\n\n> +\t\t{{- check_data_size(field.mojom_name + 'FdsSize', 'fdsSize', field.mojom_name, 'fds')}}\n> +\t{%- endif %}\n> +\t{%- set field_size = field|has_fd|choose('8 + ', '4 + ') + field.mojom_name + 'Size' -%}\n\nI'm afraid you've got quite a few integer overflow issues here :-S\n\n> +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> +\t\tret.{{field.mojom_name}}_ =\n> +\t{%- if field|is_str %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size);\n> +\t{%- elif field|has_fd and (field|is_array or field|is_map) %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> +\t{%- elif field|has_fd and (not (field|is_array or field|is_map)) %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> +\t{%- elif (not field|has_fd) and (field|is_array or field|is_map) %}\n> +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- else %}\n> +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> +\t{%- endif %}\n\nCan't we advance m after reading the size fields ? That would avoid all\nthe error-prone calculation, and get rid of at least some of the integer\noverflow issues.\n\n> +\t{%- if not loop.last %}\n> +\t\tm += {{field_size}};\n> +\t\tdataSize -= {{field_size}};\n> +\t{%- if field|has_fd %}\n> +\t\tn += {{field.mojom_name}}FdsSize;\n> +\t\tfdsSize -= {{field.mojom_name}}FdsSize;\n> +\t{%- endif %}\n> +\t{%- endif %}\n> +{% else %}\n> +\t\t/* Unknown deserialization for {{field.mojom_name}}. */\n> +{%- endif %}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Serialize a struct\n> + #\n> + # Generate code for IPADataSerializer specialization, for serializing\n> + # \\a struct. \\a base_controls indicates the default ControlInfoMap\n> + # in the event that the ControlList does not have one.\n> + #}\n> +{%- macro serializer(struct, base_controls, namespace) %}\n> +\tstatic std::tuple, std::vector>\n> +\tserialize(const {{struct|name_full(namespace)}} &data,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector retData;\n> +{%- if struct|has_fd %}\n> +\t\tstd::vector retFds;\n> +{%- endif %}\n> +{%- for field in struct.fields %}\n> +{{serializer_field(field, base_controls, namespace, loop)}}\n> +{%- endfor %}\n> +{% if struct|has_fd %}\n> +\t\treturn {retData, retFds};\n> +{%- else %}\n> +\t\treturn {retData, {}};\n> +{%- endif %}\n> +\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Deserialize a struct that has fds\n> + #\n> + # Generate code for IPADataSerializer specialization, for deserializing\n> + # \\a struct, in the case that \\a struct has file descriptors.\n> + #}\n> +{%- macro deserializer_fd(struct, namespace) %}\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector &data,\n> +\t\t std::vector &fds,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\t{{struct|name_full(namespace)}} ret;\n> +\t\tstd::vector::iterator m = data.begin();\n> +\t\tstd::vector::iterator n = fds.begin();\n> +\n> +\t\tsize_t dataSize = data.size();\n> +\t\tsize_t fdsSize = fds.size();\n> +{%- for field in struct.fields -%}\n> +{{deserializer_field(field, namespace, loop)}}\n> +{%- endfor %}\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector::iterator dataBegin,\n> +\t\t std::vector::iterator dataEnd,\n> +\t\t std::vector::iterator fdsBegin,\n> +\t\t std::vector::iterator fdsEnd,\n\nAll these should const const_iterator. There are quite a few locations\nin the deserializer where the data and fds vectors or iterators are not\nconst. Could you fix that ?\n\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector data(dataBegin, dataEnd);\n\nThis creates a copy of the data, do we really need that ? I think you\nshould turn this around, implement the deserialization in this function,\nand implement the previous function as a wrapper around this one.\n\n> +\t\tstd::vector fds(fdsBegin, fdsEnd);\n> +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, fds, cs);\n> +\t}\n> +{%- endmacro %}\n> +\n> +\n> +{#\n> + # \\brief Deserialize a struct that has no fds\n> + #\n> + # Generate code for IPADataSerializer specialization, for deserializing\n> + # \\a struct, in the case that \\a struct does not have file descriptors.\n> + #}\n> +{%- macro deserializer_no_fd(struct, namespace) %}\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector &data,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\t{{struct|name_full(namespace)}} ret;\n> +\t\tstd::vector::iterator m = data.begin();\n> +\n> +\t\tsize_t dataSize = data.size();\n> +{%- for field in struct.fields -%}\n> +{{deserializer_field(field, namespace, loop)}}\n> +{%- endfor %}\n> +\t\treturn ret;\n> +\t}\n> +\n> +\tstatic {{struct|name_full(namespace)}}\n> +\tdeserialize(std::vector::iterator dataBegin,\n> +\t\t std::vector::iterator dataEnd,\n> +{%- if struct|needs_control_serializer %}\n> +\t\t ControlSerializer *cs)\n> +{%- else %}\n> +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> +{%- endif %}\n> +\t{\n> +\t\tstd::vector data(dataBegin, dataEnd);\n> +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, cs);\n> +\t}\n> +{%- endmacro %}","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id EBDC1BE08A\n\tfor ;\n\tThu, 19 Nov 2020 17:21:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 611FC615B4;\n\tThu, 19 Nov 2020 18:21:09 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AB39560D4B\n\tfor ;\n\tThu, 19 Nov 2020 18:21:07 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id EAB1D24D;\n\tThu, 19 Nov 2020 18:21:06 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"bXxHXozl\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1605806467;\n\tbh=5HWoWfREaiGuwyB1GnbAqQFxfhY56O+9DuLJKvdWkzs=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=bXxHXozlXq9FDeHCmOtCQq7wfRJl411gwLyeLJ0H9Hmr2DN6OE1c151lB3drWfD26\n\tm0LyA8B96Ny8mQmN/+W4XLS1E13Y4NbcenvAhmAZ+89L+sjxJG2NdgcuLHGeg0ttrW\n\tW7aGWlDt+0coj0I2mDvIgv0j8klrwk/wM/Qr65Ps=","Date":"Thu, 19 Nov 2020 19:21:00 +0200","From":"Laurent Pinchart ","To":"Paul Elder ","Message-ID":"<20201119172100.GC4563@pendragon.ideasonboard.com>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201106103707.49660-5-paul.elder@ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13824,"web_url":"https://patchwork.libcamera.org/comment/13824/","msgid":"<20201123081156.GA2050@pyrite.rasen.tech>","date":"2020-11-23T08:11:56","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Hi Laurent,\n\nThank you for the review.\n\nOn Thu, Nov 19, 2020 at 02:39:17PM +0200, Laurent Pinchart wrote:\n> Hi Paul,\n> \n> Thank you for the patch.\n> \n> On Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> > Add templates to mojo to generate code for the IPC mechanism. These\n> > templates generate:\n> > - module header\n> > - module serializer\n> > - IPA proxy cpp, header, and worker\n> > \n> > Given an input data definition mojom file for a pipeline.\n> > \n> > Signed-off-by: Paul Elder \n> > \n> > ---\n> > Changes in v4:\n> > For the non-template files:\n> > - rename IPA{pipeline_name}CallbackInterface to\n> > IPA{pipeline_name}EventInterface\n> > - to avoid the notion of \"callback\" and emphasize that it's an event\n> > - add support for strings in custom structs\n> > - add validation, that async methods must not have return values\n> > - it throws exception and isn't very clear though...?\n> > - rename controls to libcamera::{pipeline_name}::controls (controls is\n> > now lowercase)\n> > - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> > and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> > - same for their corresponding template files\n> > For the template files:\n> > - fix spacing (now it's all {{var}} instead of some {{ var }})\n> > - except if it's code, so code is still {{ code }}\n> > - move inclusion of corresponding header to first in the inclusion list\n> > - fix copy&paste errors\n> > - change snake_case to camelCase in the generated code\n> > - template code still uses snake_case\n> > - change the generated command enums to an enum class, and make it\n> > capitalized (instead of allcaps)\n> > - add length checks to recvIPC (in proxy)\n> > - fix some template spacing\n> > - don't use const for PODs in function/signal parameters\n> > - add the proper length checks to readPOD/appendPOD\n> > - the helper functions for reading and writing PODs to and from\n> > serialized data\n> > - rename readUInt/appendUInt to readPOD/appendPOD\n> > - add support for strings in custom structs\n> > \n> > Changes in v3:\n> > - add support for namespaces\n> > - fix enum assignment (used to have +1 for CMD applied to all enums)\n> > - use readHeader, writeHeader, and eraseHeader as static class functions\n> > of IPAIPCUnixSocket (in the proxy worker)\n> > - add requirement that base controls *must* be defined in\n> > libcamera::{pipeline_name}::Controls\n> > \n> > Changes in v2:\n> > - mandate the main and callback interfaces, and init(), start(), stop()\n> > and their parameters\n> > - fix returning single pod value from IPC-called function\n> > - add licenses\n> > - improve auto-generated message\n> > - other fixes related to serdes\n> > ---\n> > .../module_ipa_interface.h.tmpl | 113 ++++\n> > .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> > .../module_ipa_proxy.h.tmpl | 118 +++++\n> > .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> > .../module_ipa_serializer.h.tmpl | 44 ++\n> > .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> > .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> > .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> > 8 files changed, 1673 insertions(+)\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n> \n> Let's start with the generator.\n> \n> [snip]\n> \n> > diff --git a/utils/ipc/generators/mojom_libcamera_generator.py b/utils/ipc/generators/mojom_libcamera_generator.py\n> > new file mode 100644\n> > index 00000000..c4fbf9b3\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/mojom_libcamera_generator.py\n> > @@ -0,0 +1,488 @@\n> \n> Missing SPDX header and copyright notice.\n> \n> > +'''Generates libcamera files from a mojom.Module.'''\n> \n> Comments in Python use # :-)\n> \n> > +\n> > +import argparse\n> > +import ast\n> > +import datetime\n> > +import contextlib\n> > +import os\n> > +import re\n> > +import shutil\n> > +import sys\n> > +import tempfile\n> \n> ast, contextlib, shutil, sys and tempfile don't seem to be used.\n> \n> > +\n> > +from jinja2 import contextfilter\n> \n> contextfilter isn't used either.\n> \n> > +\n> > +import mojom.fileutil as fileutil\n> > +import mojom.generate.generator as generator\n> > +import mojom.generate.module as mojom\n> > +from mojom.generate.template_expander import UseJinja\n> > +\n> > +\n> > +GENERATOR_PREFIX = 'libcamera'\n> > +\n> > +_kind_to_cpp_type = {\n> > + mojom.BOOL: 'bool',\n> > + mojom.INT8: 'int8_t',\n> > + mojom.UINT8: 'uint8_t',\n> > + mojom.INT16: 'int16_t',\n> > + mojom.UINT16: 'uint16_t',\n> > + mojom.INT32: 'int32_t',\n> > + mojom.UINT32: 'uint32_t',\n> > + mojom.FLOAT: 'float',\n> > + mojom.INT64: 'int64_t',\n> > + mojom.UINT64: 'uint64_t',\n> > + mojom.DOUBLE: 'double',\n> > +}\n> > +\n> > +_bit_widths = {\n> > + mojom.BOOL: '8',\n> > + mojom.INT8: '8',\n> > + mojom.UINT8: '8',\n> > + mojom.INT16: '16',\n> > + mojom.UINT16: '16',\n> > + mojom.INT32: '32',\n> > + mojom.UINT32: '32',\n> > + mojom.FLOAT: '32',\n> > + mojom.INT64: '64',\n> > + mojom.UINT64: '64',\n> > + mojom.DOUBLE: '64',\n> > +}\n> > +\n> > +def ModuleName(path):\n> > + return path.split('/')[-1].split('.')[0]\n> > +\n> > +def ModuleClassName(module):\n> > + s = re.sub(r'^IPA', '', module.interfaces[0].mojom_name)\n> > + return re.sub(r'Interface$', '', s)\n> \n> You could combine the two with\n> \n> return re.sub(r'^IPA(.*)Interface$', lambda match: match.group(1), module)\n> \n> and possibly optimize it further by pre-compiling the regular\n> expression. I'll let you decide if it's worth it.\n> \n> > +\n> > +def UpperCamelCase(name):\n> > + return ''.join([x.capitalize() for x in generator.SplitCamelCase(name)])\n> > +\n> > +def CamelCase(name):\n> > + uccc = UpperCamelCase(name)\n> > + return uccc[0].lower() + uccc[1:]\n> \n> These two functions don't seem to be used.\n> \n> > +\n> > +def Capitalize(name):\n> > + return name[0].upper() + name[1:]\n> > +\n> > +def ConstantStyle(name):\n> > + return generator.ToUpperSnakeCase(name)\n> > +\n> > +def Choose(cond, t, f):\n> > + return t if cond else f\n> > +\n> > +def CommaSep(l):\n> > + return ', '.join([m for m in l])\n> > +\n> > +def ParamsCommaSep(l):\n> > + return ', '.join([m.mojom_name for m in l])\n> > +\n> > +def GetDefaultValue(element):\n> > + if element.default is not None:\n> > + return element.default\n> > + if type(element.kind) == mojom.Kind:\n> > + return '0'\n> > + if mojom.IsEnumKind(element.kind):\n> > + return f'static_cast<{element.kind.mojom_name}>(0)'\n> > + if (isinstance(element.kind, mojom.Struct) and\n> > + element.kind.mojom_name == 'FileDescriptor'):\n> \n> Is there a need for the outer parentheses ? Or is this dictated by the\n> mojom coding style ? Same comment for several locations below.\n> \n> > + return '-1'\n> > + return ''\n> > +\n> > +def WithDefaultValues(element):\n> > + return [x for x in element if HasDefaultValue(x)]\n> > +\n> > +def WithFds(element):\n> > + return [x for x in element if HasFd(x)]\n> \n> Should these be moved a little bit further down, after the functions\n> they call ? The code works fine, but it would create an easier to read\n> flow.\n> \n> > +\n> > +def HasDefaultValue(element):\n> > + return GetDefaultValue(element) != ''\n> > +\n> > +def HasDefaultFields(element):\n> > + return True in [HasDefaultValue(x) for x in element.fields]\n> > +\n> > +def GetAllTypes(element):\n> > + if mojom.IsArrayKind(element):\n> > + return GetAllTypes(element.kind)\n> > + if mojom.IsMapKind(element):\n> > + return GetAllTypes(element.key_kind) + GetAllTypes(element.value_kind)\n> > + if isinstance(element, mojom.Parameter):\n> > + return GetAllTypes(element.kind)\n> > + if mojom.IsEnumKind(element):\n> > + return [element.mojom_name]\n> > + if not mojom.IsStructKind(element):\n> > + return [element.spec]\n> > + if len(element.fields) == 0:\n> > + return [element.mojom_name]\n> > + ret = [GetAllTypes(x.kind) for x in element.fields]\n> > + ret = [x for sublist in ret for x in sublist]\n> > + return list(set(ret))\n> > +\n> > +def GetAllAttrs(element):\n> > + if mojom.IsArrayKind(element):\n> > + return GetAllAttrs(element.kind)\n> > + if mojom.IsMapKind(element):\n> > + return {**GetAllAttrs(element.key_kind), **GetAllAttrs(element.value_kind)}\n> > + if isinstance(element, mojom.Parameter):\n> > + return GetAllAttrs(element.kind)\n> > + if mojom.IsEnumKind(element):\n> > + return element.attributes if element.attributes is not None else {}\n> > + if mojom.IsStructKind(element) and len(element.fields) == 0:\n> > + return element.attributes if element.attributes is not None else {}\n> > + if not mojom.IsStructKind(element):\n> > + return {}\n> > + attrs = [(x.attributes) for x in element.fields]\n> > + ret = {}\n> > + for d in attrs:\n> > + if d is not None:\n> > + ret = {**ret, **d}\n> \n> Maybe\n> \n> ret.update(d)\n> \n> > + return ret\n> > +\n> > +def NeedsControlSerializer(element):\n> > + types = GetAllTypes(element)\n> > + return \"ControlList\" in types or \"ControlInfoMap\" in types\n> > +\n> > +def HasFd(element):\n> > + attrs = GetAllAttrs(element)\n> > + if isinstance(element, mojom.Kind):\n> > + types = GetAllTypes(element)\n> > + else:\n> > + types = GetAllTypes(element.kind)\n> > + return \"FileDescriptor\" in types or (attrs is not None and \"hasFd\" in attrs)\n> > +\n> > +def MethodInputHasFd(method):\n> > + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> \n> You can drop the len() call, [] evaluates to False.\n\nWhat do you mean? It evaluates to something like this:\n\n[]\n\nSo it checks how many parameters have fd.\n\n> > + return True\n> > + return False\n> > +\n> > +def MethodOutputHasFd(method):\n> > + if (MethodReturnValue(method) != 'void' or\n> > + method.response_parameters is None):\n> > + return False\n> > + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> \n> Same here.\n> \n> Shouldn't it be method.response_parameters instead of method.parameters\n> ?\n\nYeah it should be.\n\n> > + return True\n> > + return False\n> > +\n> > +def MethodParamInputs(method):\n> > + return method.parameters\n> > +\n> > +def MethodParamOutputs(method):\n> > + if (MethodReturnValue(method) != 'void' or\n> > + method.response_parameters is None):\n> > + return []\n> > + return method.response_parameters\n> \n> The increase code reuse, you could implement these functions as follows.\n> \n> \n> def MethodParamInputs(method):\n> return method.parameters\n> \n> def MethodParamOutputs(method):\n> if (MethodReturnValue(method) != 'void' or\n> method.response_parameters is None):\n> return []\n> return method.response_parameters\n> \n> def MethodParamHasFd(parameters):\n> if [x for x in parameters if HasFd(x)]:\n> return True\n> return False\n> \n> def MethodInputHasFd(method):\n> return MethodParamHasFd(MethodParamInputs(method))\n> \n> def MethodOutputHasFd(method):\n> return MethodParamHasFd(MethodParamOutputs(method))\n> \n> > +\n> > +def MethodParamNames(method):\n> > + params = []\n> > + for param in method.parameters:\n> > + params.append(param.mojom_name)\n> > + if MethodReturnValue(method) == 'void':\n> > + if method.response_parameters is None:\n> > + return params\n> > + for param in method.response_parameters:\n> > + params.append(param.mojom_name)\n> > + return params\n> > +\n> > +def MethodParameters(method):\n> > + params = []\n> > + for param in method.parameters:\n> > + params.append('const %s %s%s' % (GetNameForElement(param),\n> > + ('&' if not IsPod(param) else ''),\n> \n> No need for the outer parentheses.\n> \n> > + param.mojom_name))\n> > + if MethodReturnValue(method) == 'void':\n> > + if method.response_parameters is None:\n> > + return params\n> > + for param in method.response_parameters:\n> > + params.append(f'{GetNameForElement(param)} *{param.mojom_name}')\n> > + return params\n> > +\n> > +def MethodReturnValue(method):\n> > + if method.response_parameters is None:\n> > + return 'void'\n> > + if len(method.response_parameters) == 1 and IsPod(method.response_parameters[0]):\n> > + return GetNameForElement(method.response_parameters[0])\n> > + return 'void'\n> > +\n> > +def IsAsync(method):\n> > + # callbacks are always async\n> \n> s/callbacks/Events/ ?\n> \n> > + if re.match(\"^IPA.*EventInterface$\", method.interface.mojom_name):\n> > + return True\n> > + elif re.match(\"^IPA.*Interface$\", method.interface.mojom_name):\n> > + if method.attributes is None:\n> > + return False\n> > + elif 'async' in method.attributes and method.attributes['async']:\n> > + return True\n> > + return False\n> > +\n> > +def IsArray(element):\n> > + return mojom.IsArrayKind(element.kind)\n> > +\n> > +def IsControls(element):\n> > + return mojom.IsStructKind(element.kind) and (element.kind.mojom_name == \"ControlList\" or\n> > + element.kind.mojom_name == \"ControlInfoMap\")\n> > +\n> > +def IsEnum(element):\n> > + return mojom.IsEnumKind(element.kind)\n> > +\n> > +def IsFd(element):\n> > + return mojom.IsStructKind(element.kind) and element.kind.mojom_name == \"FileDescriptor\"\n> > +\n> > +def IsMap(element):\n> > + return mojom.IsMapKind(element.kind)\n> > +\n> > +def IsPlainStruct(element):\n> > + return mojom.IsStructKind(element.kind) and not IsControls(element) and not IsFd(element)\n> > +\n> > +def IsPod(element):\n> > + return element.kind in _kind_to_cpp_type\n> > +\n> > +def IsStr(element):\n> > + return element.kind.spec == 's'\n> > +\n> > +def BitWidth(element):\n> > + if element.kind in _bit_widths:\n> > + return _bit_widths[element.kind]\n> > + if mojom.IsEnumKind(element.kind):\n> > + return '32'\n> > + return ''\n> > +\n> > +def GetNameForElement(element):\n> > + if (mojom.IsEnumKind(element) or\n> > + mojom.IsInterfaceKind(element) or\n> > + mojom.IsStructKind(element)):\n> > + return element.mojom_name\n> > + if (mojom.IsArrayKind(element)):\n> > + elem_name = GetNameForElement(element.kind)\n> > + return f'std::vector<{elem_name}>'\n> > + if (mojom.IsMapKind(element)):\n> > + key_name = GetNameForElement(element.key_kind)\n> > + value_name = GetNameForElement(element.value_kind)\n> > + return f'std::map<{key_name}, {value_name}>'\n> > + if isinstance(element, (mojom.Field, mojom.Method, mojom.Parameter)):\n> > + if (mojom.IsArrayKind(element.kind) or mojom.IsMapKind(element.kind)):\n> > + return GetNameForElement(element.kind)\n> > + if (mojom.IsReferenceKind(element.kind) and element.kind.spec == 's'):\n> > + return 'std::string'\n> > + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> \n> Is the hasattr() check needed ? If element has no kind attribute, I\n> would expect the previous two if's to raise an exception.\n\nI've added comments for each section, because this confuses even me...\n\nSo the upper-level if is to catch struct fields and function parameters.\nThis if catches PODs.\n\nAnd you're right, the hasattr() check isn't needed. I think it was a\nremnant of fiddling with it and python complaining that element has no\nattribute kind.\n\n> > + return _kind_to_cpp_type[element.kind]\n> > + return element.kind.mojom_name\n> > + if isinstance(element, mojom.EnumValue):\n> \n> Extra space after comma.\n> \n> > + return (GetNameForElement(element.enum) + '.' +\n> > + ConstantStyle(element.name))\n> > + if isinstance(element, (mojom.NamedValue,\n> > + mojom.Constant,\n> > + mojom.EnumField)):\n> > + return ConstantStyle(element.name)\n> > + if (hasattr(element, '__hash__') and element in _kind_to_cpp_type):\n> > + return _kind_to_cpp_type[element]\n> \n> For my own education, what type of elements does this cover ?\n\nThis covers PODs that are members of vectors/maps.\n\nI've tested each case, and removed the ones that I can't remember what\nthey do.\n\n> > + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> > + return _kind_to_cpp_type[element.kind]\n> > + if (hasattr(element, 'spec')):\n> > + if (element.spec == 's'):\n> > + return 'std::string'\n> > + return element.spec.split(':')[-1]\n> > + if (mojom.IsInterfaceRequestKind(element) or\n> > + mojom.IsAssociatedKind(element) or\n> > + mojom.IsPendingRemoteKind(element) or\n> > + mojom.IsPendingReceiverKind(element) or\n> > + mojom.IsUnionKind(element)):\n> > + raise Exception('Unsupported element: %s' % element)\n> > + raise Exception('Unexpected element: %s' % element)\n> > +\n> > +def GetFullNameForElement(element, namespace_str):\n> > + name = GetNameForElement(element)\n> > + if namespace_str == '':\n> > + return name\n> > + return f'{namespace_str}::{name}'\n> > +\n> > +def ValidateZeroLength(l, s, cap=True):\n> > + if l is None:\n> > + return\n> > + if len(l) > 0:\n> > + raise Exception(f'{s.capitalize() if cap else s} should be empty')\n> > +\n> > +def ValidateSingleLength(l, s, cap=True):\n> > + if len(l) > 1:\n> > + raise Exception(f'Only one {s} allowed')\n> > + if len(l) < 1:\n> > + raise Exception(f'{s.capitalize() if cap else s} is required')\n> > +\n> > +def ValidateInterfaces(interfaces):\n> > + # Validate presence of main interface\n> > + intf = [x for x in interfaces\n> > + if re.match(\"^IPA.*Interface\", x.mojom_name) and\n> > + not re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> > + ValidateSingleLength(intf, 'main interface')\n> > + intf = intf[0]\n> > +\n> > + # Validate presence of callback interface\n> \n> s/callback/event/\n> \n> > + cb = [x for x in interfaces if re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> \n> s/cb/event/ ?\n> \n> > + ValidateSingleLength(cb, 'event interface')\n> > + cb = cb[0]\n> > +\n> > + # Validate required main interface functions\n> > + f_init = [x for x in intf.methods if x.mojom_name == 'init']\n> > + f_start = [x for x in intf.methods if x.mojom_name == 'start']\n> > + f_stop = [x for x in intf.methods if x.mojom_name == 'stop']\n> > +\n> > + ValidateSingleLength(f_init, 'init()', False)\n> > + ValidateSingleLength(f_start, 'start()', False)\n> > + ValidateSingleLength(f_stop, 'stop()', False)\n> > +\n> > + f_init = f_init[0]\n> > + f_start = f_start[0]\n> > + f_stop = f_stop[0]\n> > +\n> > + # Validate parameters to init()\n> > + ValidateSingleLength(f_init.parameters, 'input parameter to init()')\n> > + ValidateSingleLength(f_init.response_parameters, 'output parameter from init()')\n> > + if f_init.parameters[0].kind.mojom_name != 'IPASettings':\n> > + raise Exception('init() must have single IPASettings input parameter')\n> > + if f_init.response_parameters[0].kind.spec != 'i32':\n> > + raise Exception('init() must have single int32 output parameter')\n> > +\n> > + # Validate parameters to start()\n> > + ValidateZeroLength(f_start.parameters, 'input parameter to start()')\n> > + ValidateSingleLength(f_start.response_parameters, 'output parameter from start()')\n> > + if f_start.response_parameters[0].kind.spec != 'i32':\n> > + raise Exception('start() must have single int32 output parameter')\n> > +\n> > + # Validate parameters to stop()\n> > + ValidateZeroLength(f_stop.parameters, 'input parameter to stop()')\n> > + ValidateZeroLength(f_stop.parameters, 'output parameter from stop()')\n> > +\n> > + # Validate that all async methods don't have return values\n> > + intf_methods_async = [x for x in intf.methods if IsAsync(x)]\n> > + for method in intf_methods_async:\n> > + ValidateZeroLength(method.response_parameters,\n> > + f'{method.mojom_name} response parameters', False)\n> > +\n> > + cb_methods_async = [x for x in cb.methods if IsAsync(x)]\n> \n> s/cb_method_async/event_methods_async/\n> \n> > + for method in cb_methods_async:\n> > + ValidateZeroLength(method.response_parameters,\n> > + f'{method.mojom_name} response parameters', False)\n> > +\n> > +class Generator(generator.Generator):\n> > +\n> > + @staticmethod\n> > + def GetTemplatePrefix():\n> > + return 'libcamera_templates'\n> > +\n> > + def GetFilters(self):\n> > + libcamera_filters = {\n> > + 'all_types': GetAllTypes,\n> > + 'bit_width': BitWidth,\n> > + 'cap': Capitalize,\n> \n> 'cap': str.capitalize,\n> \n> and drop the custom Capitalize function.\n\nThe built-in capitalize function doesn't preserve camel-casing so it's\nnot sufficient:\n\n>>> \"camelCaseClass\".capitalize()\n'Camelcaseclass'\n>>> Capitalize(\"camelCaseClass\")\n'CamelCaseClass'\n\n> > + 'choose': Choose,\n> > + 'comma_sep': CommaSep,\n> > + 'default_value': GetDefaultValue,\n> > + 'has_default_fields': HasDefaultFields,\n> > + 'has_fd': HasFd,\n> > + 'is_async': IsAsync,\n> > + 'is_array': IsArray,\n> > + 'is_controls': IsControls,\n> > + 'is_enum': IsEnum,\n> > + 'is_fd': IsFd,\n> > + 'is_map': IsMap,\n> > + 'is_plain_struct': IsPlainStruct,\n> > + 'is_pod': IsPod,\n> > + 'is_str': IsStr,\n> > + 'method_input_has_fd': MethodInputHasFd,\n> > + 'method_output_has_fd': MethodOutputHasFd,\n> > + 'method_param_names': MethodParamNames,\n> > + 'method_param_inputs': MethodParamInputs,\n> > + 'method_param_outputs': MethodParamOutputs,\n> > + 'method_parameters': MethodParameters,\n> > + 'method_return_value': MethodReturnValue,\n> > + 'name': GetNameForElement,\n> > + 'name_full': GetFullNameForElement,\n> > + 'needs_control_serializer': NeedsControlSerializer,\n> > + 'params_comma_sep': ParamsCommaSep,\n> > + 'with_default_values': WithDefaultValues,\n> > + 'with_fds': WithFds,\n> > + }\n> > + return libcamera_filters\n> > +\n> > + def _GetJinjaExports(self):\n> \n> Would it make sense to store the return value of\n> ModuleClassName(self.module) in a local variable instead of calling the\n> function several times below ? If self.module is constant during the\n> lifetime of the Generator (I think it is), you could even compute it in\n> the constructor and store it in a member variable.\n\nYeah I did think that was a good idea.\n\nHm, it doesn't work in the constructor, but it works in GenerateFiles.\n\n> > + return {\n> > + 'base_controls': '%s::controls' % ModuleClassName(self.module),\n> > + 'cmd_enum_name': '_%sCmd' % ModuleClassName(self.module),\n> > + 'cmd_event_enum_name': '_%sEventCmd' % ModuleClassName(self.module),\n> > + 'enums': self.module.enums,\n> > + 'has_array': len([x for x in self.module.kinds.keys() if x[0] == 'a']) > 0,\n> > + 'has_map': len([x for x in self.module.kinds.keys() if x[0] == 'm']) > 0,\n> > + 'has_namespace': self.module.mojom_namespace != '',\n> > + 'imports': self.module.imports,\n> \n> I don't see any mention of 'imports' in the templates. Is this used\n> internally by mojom and/or jinja, or is it unused ?\n\nIt was probably copied from the mojom C++ generator. I think I was going\nto use it for something... but then importing ended up working in such a\nway where this is no longer needed.\n\n> > + 'interface_cb': self.module.interfaces[1],\n> \n> interface_event ?\n> \n> > + 'interface_main': self.module.interfaces[0],\n> \n> Is there a guarantee the interfaces will always be specified in this\n> order ? The code in ValidateInterfaces() finds interfaces based on their\n> name.\n\nAh, there is indeed no such guarantee, just a undocumented convention.\n\n> > + 'interface_name': 'IPA%sInterface' % ModuleClassName(self.module),\n> > + 'ipc_name': 'IPAIPCUnixSocket',\n> \n> This seems unused.\n\nIt will be used when we support different IPC mechanisms :)\n\nWait nvm, it won't. I'll remove it.\n\n> > + 'kinds': self.module.kinds,\n> > + 'module': self.module,\n> \n> Same for those two (unless they are used elsewhere than in the\n> templates).\n> \n> > + 'module_name': ModuleName(self.module.path),\n> > + 'module_class_name': ModuleClassName(self.module),\n> \n> Ditto.\n> \n> > + 'namespace': self.module.mojom_namespace.split('.'),\n> > + 'namespace_str': self.module.mojom_namespace.replace('.', '::') if\n> > + self.module.mojom_namespace is not None else '',\n> > + 'proxy_name': 'IPAProxy%s' % ModuleClassName(self.module),\n> > + 'proxy_worker_name': 'IPAProxy%sWorker' % ModuleClassName(self.module),\n> \n> Same here.\n> \n> > + 'structs_nonempty': [x for x in self.module.structs if len(x.fields) > 0],\n> > + 'year': datetime.datetime.now().year,\n> > + }\n> > +\n> > +\n> > + @UseJinja('module_ipa_interface.h.tmpl')\n> > + def _GenerateDataHeader(self):\n> > + return self._GetJinjaExports()\n> > +\n> > + @UseJinja('module_ipa_serializer.h.tmpl')\n> > + def _GenerateSerializer(self):\n> > + return self._GetJinjaExports()\n> > +\n> > + @UseJinja('module_ipa_proxy.cpp.tmpl')\n> > + def _GenerateProxyCpp(self):\n> > + return self._GetJinjaExports()\n> > +\n> > + @UseJinja('module_ipa_proxy.h.tmpl')\n> > + def _GenerateProxyHeader(self):\n> > + return self._GetJinjaExports()\n> > +\n> > + @UseJinja('module_ipa_proxy_worker.cpp.tmpl')\n> > + def _GenerateProxyWorker(self):\n> > + return self._GetJinjaExports()\n> > +\n> > + def GenerateFiles(self, unparsed_args):\n> > + parser = argparse.ArgumentParser()\n> > + parser.add_argument('--libcamera_generate_header', action='store_true')\n> > + parser.add_argument('--libcamera_generate_serializer', action='store_true')\n> > + parser.add_argument('--libcamera_generate_proxy_cpp', action='store_true')\n> > + parser.add_argument('--libcamera_generate_proxy_h', action='store_true')\n> > + parser.add_argument('--libcamera_generate_proxy_worker', action='store_true')\n> > + parser.add_argument('--libcamera_output_path')\n> > + args = parser.parse_args(unparsed_args)\n> > +\n> > + ValidateInterfaces(self.module.interfaces)\n> > +\n> > + fileutil.EnsureDirectoryExists(os.path.dirname(args.libcamera_output_path))\n> > +\n> > + module_name = ModuleName(self.module.path)\n> \n> This seems unused.\n> \n> > +\n> > + if args.libcamera_generate_header:\n> > + self.Write(self._GenerateDataHeader(),\n> > + args.libcamera_output_path)\n> > +\n> > + if args.libcamera_generate_serializer:\n> > + self.Write(self._GenerateSerializer(),\n> > + args.libcamera_output_path)\n> > +\n> > + if args.libcamera_generate_proxy_cpp:\n> > + self.Write(self._GenerateProxyCpp(),\n> > + args.libcamera_output_path)\n> > +\n> > + if args.libcamera_generate_proxy_h:\n> > + self.Write(self._GenerateProxyHeader(),\n> > + args.libcamera_output_path)\n> > +\n> > + if args.libcamera_generate_proxy_worker:\n> > + self.Write(self._GenerateProxyWorker(),\n> > + args.libcamera_output_path)\n> \n> I'm sure we could avoid the manual unrolling of all this, using a\n> dictionary to store the template files and iterating over it to add\n> arguments to the parser and replacing the above code, but it's probably\n> not worth it as I don't expect this will change a lot.\n\n\nThanks,\n\nPaul","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7A653BE176\n\tfor ;\n\tMon, 23 Nov 2020 08:12:09 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C960563328;\n\tMon, 23 Nov 2020 09:12:08 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 9E08360334\n\tfor ;\n\tMon, 23 Nov 2020 09:12:06 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2400:4051:61:600:2c71:1b79:d06d:5032])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id E5D882A4;\n\tMon, 23 Nov 2020 09:12:03 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"b0OI30cR\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1606119126;\n\tbh=RNyisuKnswZSVay+tkJsx/6pjYWaNLG8Jh+b/Z/iT50=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=b0OI30cREzuPMDNPKl11Sl2S3E6kvMi4VeGvuCrXRAPoI2qMsOb8fS5jy9F/I1jDE\n\txT0FeiAp0Gzfly6tbdWQYu2KDez6/5pKeaOoK+4gNTw3BlogSK9Z9hQ1TJ2GqrYyTq\n\tUSU6Do5lHmx/YE0nM069Qlf2YGp0vDjTa7s5v+ls=","Date":"Mon, 23 Nov 2020 17:11:56 +0900","From":"paul.elder@ideasonboard.com","To":"Laurent Pinchart ","Message-ID":"<20201123081156.GA2050@pyrite.rasen.tech>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>\n\t<20201119123917.GA4563@pendragon.ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201119123917.GA4563@pendragon.ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13890,"web_url":"https://patchwork.libcamera.org/comment/13890/","msgid":"<20201126064918.GB2050@pyrite.rasen.tech>","date":"2020-11-26T06:49:18","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Hi Laurent,\n\nThank you for the review.\n\nOn Thu, Nov 19, 2020 at 07:21:00PM +0200, Laurent Pinchart wrote:\n> Hi Paul,\n> \n> Thank you for the patch.\n> \n> On Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> > Add templates to mojo to generate code for the IPC mechanism. These\n> > templates generate:\n> > - module header\n> > - module serializer\n> > - IPA proxy cpp, header, and worker\n> > \n> > Given an input data definition mojom file for a pipeline.\n> > \n> > Signed-off-by: Paul Elder \n> > \n> > ---\n> > Changes in v4:\n> > For the non-template files:\n> > - rename IPA{pipeline_name}CallbackInterface to\n> > IPA{pipeline_name}EventInterface\n> > - to avoid the notion of \"callback\" and emphasize that it's an event\n> > - add support for strings in custom structs\n> > - add validation, that async methods must not have return values\n> > - it throws exception and isn't very clear though...?\n> > - rename controls to libcamera::{pipeline_name}::controls (controls is\n> > now lowercase)\n> > - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> > and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> > - same for their corresponding template files\n> > For the template files:\n> > - fix spacing (now it's all {{var}} instead of some {{ var }})\n> > - except if it's code, so code is still {{ code }}\n> > - move inclusion of corresponding header to first in the inclusion list\n> > - fix copy&paste errors\n> > - change snake_case to camelCase in the generated code\n> > - template code still uses snake_case\n> > - change the generated command enums to an enum class, and make it\n> > capitalized (instead of allcaps)\n> > - add length checks to recvIPC (in proxy)\n> > - fix some template spacing\n> > - don't use const for PODs in function/signal parameters\n> > - add the proper length checks to readPOD/appendPOD\n> > - the helper functions for reading and writing PODs to and from\n> > serialized data\n> > - rename readUInt/appendUInt to readPOD/appendPOD\n> > - add support for strings in custom structs\n> > \n> > Changes in v3:\n> > - add support for namespaces\n> > - fix enum assignment (used to have +1 for CMD applied to all enums)\n> > - use readHeader, writeHeader, and eraseHeader as static class functions\n> > of IPAIPCUnixSocket (in the proxy worker)\n> > - add requirement that base controls *must* be defined in\n> > libcamera::{pipeline_name}::Controls\n> > \n> > Changes in v2:\n> > - mandate the main and callback interfaces, and init(), start(), stop()\n> > and their parameters\n> > - fix returning single pod value from IPC-called function\n> > - add licenses\n> > - improve auto-generated message\n> > - other fixes related to serdes\n> > ---\n> > .../module_ipa_interface.h.tmpl | 113 ++++\n> > .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> > .../module_ipa_proxy.h.tmpl | 118 +++++\n> > .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> > .../module_ipa_serializer.h.tmpl | 44 ++\n> > .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> > .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> > .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> > 8 files changed, 1673 insertions(+)\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> \n> The proxy .h and .cpp files are named ipa_proxy_{pipeline}.{h,cpp},\n> while the other generated files are named {pipeline}_ipa_interface.h and\n> {pipeline}_ipa_serializer.h. Could you pick one naming convention, and\n> apply it to all files ? The .tmpl files should then follow the same\n> convention.\n\nAh, yeah. I'll make them all {pipeline}_ipa_{proxy,proxy_worker,interface,serializer}.{h,cpp}\n\n> > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n> \n> And now, for the templates.\n> \n> > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > new file mode 100644\n> > index 00000000..a470b99e\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > @@ -0,0 +1,113 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) {{year}}, Google Inc.\n> \n> Coming back to the question of copyright on generated code.\n> \n> First of all, the template itself is a creative work, which should have\n> a copyright notice, with a fixed date. This should go to the comment at\n> the beginning of the file.\n> \n> The generated file is the result of a non-creative process that takes\n> creative work as input. As such, it is covered by the copyright of the\n> template, and the mojom file. Would it be difficult to extract the\n> copyright data of the .mojom file and add it here ? I'm thnking about\n> the following at the beginning of the template:\n> \n> {#-\n> # SPDX-License-Identifier: LGPL-2.1-or-later\n> # Copyright (C) 2020, Google Inc.\n> -#}\n> /* SPDX-License-Identifier: LGPL-2.1-or-later */\n> /*\n> * Copyright (C) 2020, Google Inc.\n> * {{copyright}}\n> *\n> * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> *\n> * This file is auto-generated. Do not edit.\n> */\n> \n> with {{copyright}} being replaced with the copyright statement from the\n> .mojom file. Mojo won't give you the information, but if you pass the\n> template name to _GetJinjaExport(), it may not be too difficult to\n> extract the copyright line.\n> \n> This may not be worth it, so feel free to just have a fixed copyright\n> here:\n> \n> /* SPDX-License-Identifier: LGPL-2.1-or-later */\n> /*\n> * Copyright (C) 2020, Google Inc.\n> *\n> * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> *\n> * This file is auto-generated. Do not edit.\n> */\n\nOkay. I'll take the latter.\n\n> > + *\n> > + * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> > + *\n> > + * This file is auto-generated. Do not edit.\n> > + */\n> > +\n> > +#ifndef __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> > +#define __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> > +\n> > +#include \n> > +#include \n> > +\n> > +{% if has_map %}#include {% endif %}\n> > +{% if has_array %}#include {% endif %}\n> > +\n> > +namespace libcamera {\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace %}\n> > +namespace {{ns}} {\n> > +{% endfor %}\n> > +{%- endif %}\n> > +\n> > +enum class {{cmd_enum_name}} {\n> > +\tExit = 0,\n> > +{%- for method in interface_main.methods %}\n> > +\t{{method.mojom_name|cap}} = {{loop.index}},\n> > +{%- endfor %}\n> > +};\n> > +\n> > +enum class {{cmd_event_enum_name}} {\n> > +{%- for method in interface_cb.methods %}\n> > +\t{{method.mojom_name|cap}} = {{loop.index}},\n> > +{%- endfor %}\n> > +};\n> > +\n> > +{% for enum in enums %}\n> > +enum {{enum.mojom_name}} {\n> > +{%- for field in enum.fields %}\n> > +\t{{field.mojom_name}} = {{field.numeric_value}},\n> > +{%- endfor %}\n> > +};\n> > +{% endfor %}\n> > +\n> > +{%- for struct in structs_nonempty %}\n> > +struct {{struct.mojom_name}}\n> > +{\n> > +public:\n> > +\t{{struct.mojom_name}}() {%- if struct|has_default_fields %}\n> > +\t\t:{% endif %}\n> > +{%- for field in struct.fields|with_default_values -%}\n> > +{{\" \" if loop.first}}{{field.mojom_name}}_({{field|default_value}}){{\", \" if not loop.last}}\n> > +{%- endfor %}\n> > +\t{\n> > +\t}\n> > +\n> > +\t~{{struct.mojom_name}}() {}\n> \n> I think you can skip empty destructors.\n> \n> > +\n> > +\t{{struct.mojom_name}}(\n> > +{%- for field in struct.fields -%}\n> > +{{field|name}} {{field.mojom_name}}{{\", \" if not loop.last}}\n> \n> Should non-trivial parameters be passed by const reference instead of\n> value ?\n\nThey should.\n\n> > +{%- endfor -%}\n> > +)\n> > +\t\t:\n> > +{%- for field in struct.fields -%}\n> > +{{\" \" if loop.first}}{{field.mojom_name}}_({{field.mojom_name}}){{\", \" if not loop.last}}\n> > +{%- endfor %}\n> > +\t{\n> > +\t}\n> > +{% for field in struct.fields %}\n> > +\t{{field|name}} {{field.mojom_name}}_;\n> > +{%- endfor %}\n> > +};\n> > +{% endfor %}\n> > +\n> > +{#-\n> > +Any consts or #defines should be moved to the mojom file when possible.\n> > +If anything needs to be #included, then {{module_name}}.h needs to have the\n> > +#include.\n> > +#}\n> > +class {{interface_name}} : public IPAInterface\n> > +{\n> > +public:\n> > +\tvirtual ~{{interface_name}}() {}\n> \n> The base class has a virtual destructor already, so you can drop this\n> one.\n> \n> > +{% for method in interface_main.methods %}\n> > +\tvirtual {{method|method_return_value}} {{method.mojom_name}}(\n> > +{%- for param in method|method_parameters %}\n> > +\t\t{{param}}{{- \",\" if not loop.last}}\n> > +{%- endfor -%}\n> > +) = 0;\n> > +{% endfor %}\n> > +\n> > +{%- for method in interface_cb.methods %}\n> > +\tSignal<\n> > +{%- for param in method.parameters -%}\n> > +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> > +\t\t{{- \", \" if not loop.last}}\n> > +{%- endfor -%}\n> > +> {{method.mojom_name}};\n> > +{% endfor -%}\n> > +};\n> > +\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace|reverse %}\n> > +} /* {{ns}} */\n> \n> This should be\n> \n> } /* namespace {{ns}} */\n> \n> Same in a few locations below.\n> \n> > +{% endfor %}\n> > +{%- endif %}\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__ */\n> > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > new file mode 100644\n> > index 00000000..9328c7ca\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > @@ -0,0 +1,238 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > +\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) {{year}}, Google Inc.\n> > + *\n> > + * ipa_proxy_{{module_name}}.cpp - Image Processing Algorithm proxy for {{module_name}}\n> > + *\n> > + * This file is auto-generated. Do not edit.\n> > + */\n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \"libcamera/internal/control_serializer.h\"\n> > +#include \"libcamera/internal/ipa_ipc.h\"\n> > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > +#include \"libcamera/internal/ipa_module.h\"\n> > +#include \"libcamera/internal/ipa_proxy.h\"\n> > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/log.h\"\n> > +#include \"libcamera/internal/process.h\"\n> > +#include \"libcamera/internal/thread.h\"\n> \n> YOu got the alphabetical order nearly right :-)\n> \n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DECLARE_CATEGORY(IPAProxy)\n> > +\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace %}\n> > +namespace {{ns}} {\n> > +{% endfor %}\n> > +{%- endif %}\n> > +\n> > +{{proxy_name}}::{{proxy_name}}(IPAModule *ipam, bool isolate)\n> > +\t: IPAProxy(ipam), running_(false),\n> > +\t isolate_(isolate)\n> > +{\n> > +\tLOG(IPAProxy, Debug)\n> > +\t\t<< \"initializing {{module_name}} proxy: loading IPA from \"\n> > +\t\t<< ipam->path();\n> > +\n> > +\tif (isolate_) {\n> > +\t\tconst std::string proxyWorkerPath = resolvePath(\"ipa_proxy_{{module_name}}\");\n> > +\t\tif (proxyWorkerPath.empty()) {\n> > +\t\t\tLOG(IPAProxy, Error)\n> > +\t\t\t\t<< \"Failed to get proxy worker path\";\n> > +\t\t\treturn;\n> > +\t\t}\n> > +\n> > +\t\tipc_ = std::make_unique(ipam->path().c_str(), proxyWorkerPath.c_str());\n> \n> Line wrap maybe ?\n> \n> > +\t\tif (!ipc_->isValid()) {\n> > +\t\t\tLOG(IPAProxy, Error) << \"Failed to create IPAIPC\";\n> > +\t\t\treturn;\n> > +\t\t}\n> > +\n> > +{% if interface_cb.methods|length > 0 %}\n> > +\t\tipc_->recvIPC.connect(this, &{{proxy_name}}::recvIPC);\n> > +{% endif %}\n> \n> I think we can assume there will always be events. An IPA module that\n> only consumes data would be a bit useless, wouldn't it ? :-)\n> \n> > +\n> > +\t\tvalid_ = true;\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\tif (!ipam->load())\n> > +\t\treturn;\n> > +\n> > +\tIPAInterface *ipai = ipam->createInterface();\n> > +\tif (!ipai) {\n> > +\t\tLOG(IPAProxy, Error)\n> > +\t\t\t<< \"Failed to create IPA context for \" << ipam->path();\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\tipa_ = std::unique_ptr<{{interface_name}}>(dynamic_cast<{{interface_name}} *>(ipai));\n> \n> Do we need a dynamic cast ?\n> \n\nYeah.\n\nsrc/libcamera/proxy/raspberrypi_ipa_proxy.cpp:78:46: error: no matching function for call to\n‘std::unique_ptr::unique_ptr(libcamera::IPAInterface*&)’\n78 | ipa_ = std::unique_ptr(ipai);\n\nipai is pointer to IPAInterface, but we need unique pointer to IPA{pipeline}Interface.\n\n> > +\tproxy_.setIPA(ipa_.get());\n> > +\n> > +{% for method in interface_cb.methods %}\n> > +\tipa_->{{method.mojom_name}}.connect(this, &{{proxy_name}}::{{method.mojom_name}}Thread);\n> > +{%- endfor %}\n> > +\n> > +\tvalid_ = true;\n> > +}\n> > +\n> > +{{proxy_name}}::~{{proxy_name}}()\n> > +{\n> > +\tif (isolate_)\n> > +\t\tipc_->sendAsync(static_cast({{cmd_enum_name}}::Exit), {}, {});\n> > +}\n> > +\n> > +{% if interface_cb.methods|length > 0 %}\n> > +void {{proxy_name}}::recvIPC(std::vector &data, std::vector &fds)\n> \n> How about calling this receiveMessage() ?\n\nYeah. Is recvMessage fine?\n\n> > +{\n> > +\tif (data.size() < 8) {\n> > +\t\tLOG(IPAProxy, Error)\n> > +\t\t\t<< \"Didn't receive enough bytes to parse event\";\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\t{{cmd_event_enum_name}} cmd = static_cast<{{cmd_event_enum_name}}>((\n> > +\t\tdata[0]) | (data[1] << 8) | (data[2] << 16) | (data[3] << 24));\n> > +\n> > +\t/* Need to skip another 4 bytes for the sequence number. */\n> \n> Is the protocol documented somewhere ?\n\nNo...\n\n> There's a bit of an imbalance in the API, with recvIPC() having to deal\n> with unpacking the command and skipping the sequence number, and\n> sendSync() and sendAsync() dealing with those internally. Could\n> recvIPC() be called with the command and two spans instead ?\n\nOh shoot, yeah that is imbalanced.\n\n> > +\tstd::vector::iterator vec = data.begin() + 8;\n> > +\tsize_t dataSize = data.size() - 8;\n> > +\tswitch (cmd) {\n> > +{%- for method in interface_cb.methods %}\n> > +\tcase {{cmd_event_enum_name}}::{{method.mojom_name|cap}}: {\n> > +\t\t{{method.mojom_name}}IPC(vec, dataSize, fds);\n> > +\t\tbreak;\n> > +\t}\n> > +{%- endfor %}\n> > +\tdefault:\n> > +\t\tLOG(IPAProxy, Error) << \"Unknown command \" << static_cast(cmd);\n> > +\t}\n> > +}\n> > +{%- endif %}\n> > +\n> > +{% for method in interface_main.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method)}}\n> > +{\n> > +\tif (isolate_)\n> > +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}IPC(\n> > +{%- for param in method|method_param_names -%}\n> > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > +{%- endfor -%}\n> > +);\n> > +\telse\n> > +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}Thread(\n> > +{%- for param in method|method_param_names -%}\n> > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > +{%- endfor -%}\n> > +);\n> > +}\n> > +\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> > +{\n> > +{%- if method.mojom_name == \"init\" %}\n> > +\t{{proxy_funcs.init_thread_body()}}\n> > +{%- elif method.mojom_name == \"start\" %}\n> > +\t{{proxy_funcs.start_thread_body()}}\n> > +{%- elif method.mojom_name == \"stop\" %}\n> > +\t{{proxy_funcs.stop_thread_body()}}\n> > +{%- elif not method|is_async %}\n> > +\tipa_->{{method.mojom_name}}(\n> > +\t{%- for param in method|method_param_names -%}\n> > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > +\t{%- endfor -%}\n> > +);\n> > +{% elif method|is_async %}\n> > +\tproxy_.invokeMethod(&ThreadProxy::{{method.mojom_name}}, ConnectionTypeQueued,\n> > +\t{%- for param in method|method_param_names -%}\n> > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > +\t{%- endfor -%}\n> > +);\n> > +{%- endif %}\n> > +}\n> > +\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\")}}\n> > +{\n> > +{%- set has_input = true if method|method_param_inputs|length > 0 %}\n> > +{%- set has_output = true if method|method_param_outputs|length > 0 or method|method_return_value != \"void\" %}\n> > +{%- if has_input %}\n> > +\tstd::vector _ipcInputBuf;\n> > +{%- if method|method_input_has_fd %}\n> > +\tstd::vector _ipcInputFds;\n> > +{%- endif %}\n> > +{%- endif %}\n> > +{%- if has_output %}\n> > +\tstd::vector _ipcOutputBuf;\n> > +{%- if method|method_output_has_fd %}\n> > +\tstd::vector _ipcOutputFds;\n> > +{%- endif %}\n> > +{%- endif %}\n> > +\n> > +{{proxy_funcs.serialize_call(method|method_param_inputs, '_ipcInputBuf', '_ipcInputFds', base_controls)}}\n> > +\n> > +{%- set input_buf = \"_ipcInputBuf\" if has_input else \"{}\" %}\n> > +{%- set fds_buf = \"_ipcInputFds\" if method|method_input_has_fd else \"{}\" %}\n> > +{%- set cmd = cmd_enum_name + \"::\" + method.mojom_name|cap %}\n> > +{% if method|is_async %}\n> > +\tint ret = ipc_->sendAsync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}});\n> > +{%- else %}\n> > +\tint ret = ipc_->sendSync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}}\n> > +{{- \", &_ipcOutputBuf\" if has_output -}}\n> > +{{- \", &_ipcOutputFds\" if has_output and method|method_output_has_fd -}}\n> > +);\n> > +{%- endif %}\n> > +\tif (ret < 0) {\n> > +\t\tLOG(IPAProxy, Error) << \"Failed to call {{method.mojom_name}}\";\n> > +{%- if method|method_return_value != \"void\" %}\n> > +\t\treturn static_cast<{{method|method_return_value}}>(ret);\n> > +{%- else %}\n> > +\t\treturn;\n> > +{%- endif %}\n> > +\t}\n> > +{% if method|method_return_value != \"void\" %}\n> > +\treturn IPADataSerializer<{{method.response_parameters|first|name}}>::deserialize(_ipcOutputBuf, 0);\n> > +{% elif method|method_param_outputs|length > 0 %}\n> > +{{proxy_funcs.deserialize_call(method|method_param_outputs, '_ipcOutputBuf', '_ipcOutputFds')}}\n> > +{% endif -%}\n> > +}\n> > +\n> > +{% endfor %}\n> > +\n> > +{% for method in interface_cb.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> > +{\n> > +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> > +}\n> > +\n> > +void {{proxy_name}}::{{method.mojom_name}}IPC(\n> > +\tstd::vector::iterator data,\n> > +\tsize_t dataSize,\n> > +\t[[maybe_unused]] std::vector &fds)\n> > +{ \n> > +{%- for param in method.parameters %}\n> > +\t{{param|name}} {{param.mojom_name}};\n> > +{%- endfor %}\n> > +{{proxy_funcs.deserialize_call(method.parameters, 'data', 'fds', false, false, true, 'dataSize')}}\n> > +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> > +}\n> > +{% endfor %}\n> > +\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace|reverse %}\n> > +} /* {{ns}} */\n> > +{% endfor %}\n> > +{%- endif %}\n> > +} /* namespace libcamera */\n> > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > new file mode 100644\n> > index 00000000..3fb7192f\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > @@ -0,0 +1,118 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > +\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) {{year}}, Google Inc.\n> > + *\n> > + * ipa_proxy_{{module_name}}.h - Image Processing Algorithm proxy for {{module_name}}\n> > + *\n> > + * This file is auto-generated. Do not edit.\n> > + */\n> > +\n> > +#ifndef __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> > +#define __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \"libcamera/internal/control_serializer.h\"\n> > +#include \"libcamera/internal/ipa_ipc.h\"\n> > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/ipa_proxy.h\"\n> > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/thread.h\"\n> > +\n> > +namespace libcamera {\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace %}\n> > +namespace {{ns}} {\n> > +{% endfor %}\n> > +{%- endif %}\n> > +\n> > +class {{proxy_name}} : public IPAProxy, public {{interface_name}}, public Object\n> > +{\n> > +public:\n> > +\t{{proxy_name}}(IPAModule *ipam, bool isolate);\n> > +\t~{{proxy_name}}();\n> > +\n> > +{% for method in interface_main.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"\", false, true)|indent(8, true)}};\n> > +{% endfor %}\n> > +\n> > +{%- for method in interface_cb.methods %}\n> > +\tSignal<\n> > +{%- for param in method.parameters -%}\n> > +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> > +\t\t{{- \", \" if not loop.last}}\n> > +{%- endfor -%}\n> > +> {{method.mojom_name}};\n> > +{% endfor %}\n> > +\n> > +private:\n> > +\tvoid recvIPC(std::vector &data, std::vector &fds);\n> \n> data and fds shouldn't be modified, let's make them const. I'd even go\n> one step further, and make them Span and Span.\n\nI'll make them const for now.\n\n> > +\n> > +{% for method in interface_main.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\", false)|indent(8, true)}};\n> > +{% endfor %}\n> > +{% for method in interface_cb.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> > +\tvoid {{method.mojom_name}}IPC(\n> > +\t\tstd::vector::iterator data,\n> > +\t\tsize_t dataSize,\n> > +\t\tstd::vector &fds);\n> > +{% endfor %}\n> > +\n> > +\t/* Helper class to invoke async functions in another thread. */\n> > +\tclass ThreadProxy : public Object\n> > +\t{\n> > +\tpublic:\n> > +\t\tvoid setIPA({{interface_name}} *ipa)\n> > +\t\t{\n> > +\t\t\tipa_ = ipa;\n> > +\t\t}\n> > +\n> > +\t\tint start()\n> > +\t\t{\n> > +\t\t\treturn ipa_->start();\n> > +\t\t}\n> > +\n> > +\t\tvoid stop()\n> > +\t\t{\n> > +\t\t\tipa_->stop();\n> > +\t\t}\n> > +{% for method in interface_main.methods %}\n> > +{%- if method|is_async %}\n> > +\t\t{{proxy_funcs.func_sig(proxy_name, method, \"\", false)|indent(16)}}\n> > +\t\t{\n> > +\t\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> > +\t\t}\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +\n> > +\tprivate:\n> > +\t\t{{interface_name}} *ipa_;\n> > +\t};\n> > +\n> > +\tbool running_;\n> > +\tThread thread_;\n> > +\tThreadProxy proxy_;\n> > +\tstd::unique_ptr<{{interface_name}}> ipa_;\n> > +\n> > +\tconst bool isolate_;\n> > +\n> > +\tstd::unique_ptr ipc_;\n> > +\n> > +\tControlSerializer controlSerializer_;\n> \n> I wonder if it would make sense to move some of the fields to the\n> IPAProxy class (and possible some of the functions too). That can be\n> done on top in a separate series.\n\nYeah, probably.\n\n> > +};\n> > +\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace|reverse %}\n> > +} /* {{ns}} */\n> > +{% endfor %}\n> > +{%- endif %}\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__ */\n> > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > new file mode 100644\n> > index 00000000..dca4f99d\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > @@ -0,0 +1,187 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > +\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) {{year}}, Google Inc.\n> > + *\n> > + * ipa_proxy_{{module_name}}_worker.cpp - Image Processing Algorithm proxy worker for {{module_name}}\n> > + *\n> > + * This file is auto-generated. Do not edit.\n> > + */\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \"libcamera/internal/camera_sensor.h\"\n> > +#include \"libcamera/internal/control_serializer.h\"\n> > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/ipa_module.h\"\n> > +#include \"libcamera/internal/ipa_proxy.h\"\n> > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > +#include \"libcamera/internal/log.h\"\n> > +#include \"libcamera/internal/thread.h\"\n> > +\n> > +using namespace libcamera;\n> > +\n> > +LOG_DEFINE_CATEGORY({{proxy_name}}Worker)\n> > +\n> > +{%- if has_namespace %}\n> > +{% for ns in namespace -%}\n> > +using namespace {{ns}};\n> > +{% endfor %}\n> > +{%- endif %}\n> > +\n> > +struct CallData {\n> > +\tIPCUnixSocket::Payload *response;\n> > +\tbool done;\n> > +};\n> \n> This doesn't seem to be used.\n\nAh indeed it's not.\n\n> > +\n> > +{{interface_name}} *ipa_;\n> > +IPCUnixSocket socket_;\n> > +\n> > +ControlSerializer controlSerializer_;\n> > +\n> > +bool exit_ = false;\n> \n> I still think it would be nicer to move the data and functions to a\n> class, with a single instance created in the main() function.\n\nHm, okay.\n\n> > +\n> > +void readyRead(IPCUnixSocket *socket)\n> > +{\n> > +\tIPCUnixSocket::Payload _message, _response;\n> > +\tint _retRecv = socket->receive(&_message);\n> > +\tif (_retRecv) {\n> > +\t\tLOG({{proxy_name}}Worker, Error)\n> > +\t\t\t<< \"Receive message failed\" << _retRecv;\n> > +\t\treturn;\n> > +\t}\n> > +\n> > +\tuint32_t _cmdUint, _seq;\n> > +\tstd::tie(_cmdUint, _seq) = IPAIPCUnixSocket::readHeader(_message);\n> \n> I think there's some room to refactor the IPC-related classes. The\n> payload and header should probably not be tied to the IPAIPCUnixSocket\n> class for instance, as they should be independent of the underlying\n> mechanism. I'll try to comment more on this topic when reviewing the\n> corresponding patches.\n> \n> > +\tIPAIPCUnixSocket::eraseHeader(_message);\n> > +\n> > +\t{{cmd_enum_name}} _cmd = static_cast<{{cmd_enum_name}}>(_cmdUint);\n> > +\n> > +\tswitch (_cmd) {\n> > +\tcase {{cmd_enum_name}}::Exit: {\n> > +\t\texit_ = true;\n> > +\t\tbreak;\n> > +\t}\n> > +\n> > +{% for method in interface_main.methods %}\n> > +\tcase {{cmd_enum_name}}::{{method.mojom_name|cap}}: {\n> > +\t{{proxy_funcs.deserialize_call(method|method_param_inputs, '_message.data', '_message.fds', false, true)|indent(8, true)}}\n> > +{% for param in method|method_param_outputs %}\n> > +\t\t{{param|name}} {{param.mojom_name}};\n> > +{% endfor %}\n> > +{%- if method|method_return_value != \"void\" %}\n> > +\t\t{{method|method_return_value}} _callRet = ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> > +{%- else %}\n> > +\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}}\n> > +{{- \", \" if method|method_param_outputs|params_comma_sep -}}\n> > +{%- for param in method|method_param_outputs -%}\n> > +&{{param.mojom_name}}{{\", \" if not loop.last}}\n> > +{%- endfor -%}\n> > +);\n> > +{%- endif %}\n> > +{% if not method|is_async %}\n> > +\t\tIPAIPCUnixSocket::writeHeader(_response, _cmdUint, _seq);\n> > +{%- if method|method_return_value != \"void\" %}\n> > +\t\tstd::vector _callRetBuf;\n> > +\t\tstd::tie(_callRetBuf, std::ignore) =\n> > +\t\t\tIPADataSerializer<{{method|method_return_value}}>::serialize(_callRet);\n> > +\t\t_response.data.insert(_response.data.end(), _callRetBuf.begin(), _callRetBuf.end());\n> > +{%- else %}\n> > +\t{{proxy_funcs.serialize_call(method|method_param_outputs, \"_response.data\", \"_response.fds\", base_controls)|indent(8, true)}}\n> > +{%- endif %}\n> > +\t\tint _ret = socket_.send(_response);\n> > +\t\tif (_ret < 0) {\n> > +\t\t\tLOG({{proxy_name}}Worker, Error)\n> > +\t\t\t\t<< \"Reply to {{method.mojom_name}}() failed\" << _ret;\n> > +\t\t}\n> > +\t\tLOG({{proxy_name}}Worker, Debug) << \"Done replying to {{method.mojom_name}}()\";\n> > +{%- endif %}\n> > +\t\tbreak;\n> > +\t}\n> > +{% endfor %}\n> > +\tdefault:\n> > +\t\tLOG({{proxy_name}}Worker, Error) << \"Unknown command \" << _cmdUint;\n> > +\t}\n> > +}\n> > +\n> > +{% for method in interface_cb.methods %}\n> > +{{proxy_funcs.func_sig(proxy_name, method, \"\", false)}}\n> > +{\n> > +\tIPCUnixSocket::Payload _message;\n> > +\n> > +\tIPAIPCUnixSocket::writeHeader(_message, static_cast({{cmd_event_enum_name}}::{{method.mojom_name|cap}}), 0);\n> > +\t{{proxy_funcs.serialize_call(method|method_param_inputs, \"_message.data\", \"_message.fds\", base_controls)}}\n> > +\n> > +\tsocket_.send(_message);\n> > +\n> > +\tLOG({{proxy_name}}Worker, Debug) << \"{{method.mojom_name}} done\";\n> > +}\n> > +{% endfor %}\n> > +\n> > +int main(int argc, char **argv)\n> > +{\n> > +\t/* Uncomment this for debugging. */\n> > +#if 0\n> > +\tstd::string logPath = \"/tmp/libcamera.worker.\" +\n> > +\t\t\t std::to_string(getpid()) + \".log\";\n> > +\tlogSetFile(logPath.c_str());\n> > +#endif\n> \n> Likely something we'll have to handle more dynamically, but it doesn't\n> have to be done now. A \\todo comment would be good though.\n\nack\n\n> > +\n> > +\tif (argc < 3) {\n> > +\t\tLOG({{proxy_name}}Worker, Error)\n> > +\t\t\t<< \"Tried to start worker with no args\";\n> > +\t\treturn EXIT_FAILURE;\n> > +\t}\n> > +\n> > +\tint fd = std::stoi(argv[2]);\n> > +\tLOG({{proxy_name}}Worker, Info)\n> > +\t\t<< \"Starting worker for IPA module \" << argv[1]\n> > +\t\t<< \" with IPC fd = \" << fd;\n> > +\n> > +\tstd::unique_ptr ipam = std::make_unique(argv[1]);\n> > +\tif (!ipam->isValid() || !ipam->load()) {\n> > +\t\tLOG({{proxy_name}}Worker, Error)\n> > +\t\t\t<< \"IPAModule \" << argv[1] << \" isn't valid\";\n> > +\t\treturn EXIT_FAILURE;\n> > +\t}\n> > +\n> > +\tif (socket_.bind(fd) < 0) {\n> > +\t\tLOG({{proxy_name}}Worker, Error) << \"IPC socket binding failed\";\n> > +\t\treturn EXIT_FAILURE;\n> > +\t}\n> > +\tsocket_.readyRead.connect(&readyRead);\n> > +\n> > +\tipa_ = dynamic_cast<{{interface_name}} *>(ipam->createInterface());\n> > +\tif (!ipa_) {\n> > +\t\tLOG({{proxy_name}}Worker, Error) << \"Failed to create IPA interface instance\";\n> > +\t\treturn EXIT_FAILURE;\n> > +\t}\n> > +{% for method in interface_cb.methods %}\n> > +\tipa_->{{method.mojom_name}}.connect(&{{method.mojom_name}});\n> > +{%- endfor %}\n> > +\n> > +\tLOG({{proxy_name}}Worker, Debug) << \"Proxy worker successfully started\";\n> > +\n> > +\t/* \\todo upgrade listening loop */\n> \n> Upgrade to what ? :-)\n\nI copied this from the old IPAProxyLinuxWorker :)\n\n> > +\tEventDispatcher *dispatcher = Thread::current()->eventDispatcher();\n> > +\twhile (!exit_)\n> > +\t\tdispatcher->processEvents();\n> > +\n> > +\tdelete ipa_;\n> > +\tsocket_.close();\n> > +\n> > +\treturn 0;\n> > +}\n> > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > new file mode 100644\n> > index 00000000..675f9adf\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > @@ -0,0 +1,44 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{%- import \"serializer.tmpl\" as serializer -%}\n> > +\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) {{year}}, Google Inc.\n> > + *\n> > + * {{module_name}}_serializer.h - Image Processing Algorithm data serializer for {{module_name}}\n> > + *\n> > + * This file is auto-generated. Do not edit.\n> > + */\n> > +\n> > +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> > +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> > +\n> > +#include \n> > +#include \n> > +\n> > +#include \"libcamera/internal/control_serializer.h\"\n> > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > +\n> > +#include \n> > +#include \n> \n> These should go first.\n> \n> > +\n> > +namespace libcamera {\n> > +\n> > +LOG_DECLARE_CATEGORY(IPADataSerializer)\n> > +{% for struct in structs_nonempty %}\n> > +template<>\n> > +class IPADataSerializer<{{struct|name_full(namespace_str)}}>\n> > +{\n> > +public:\n> > +{{- serializer.serializer(struct, base_controls, namespace_str)}}\n> > +{%- if struct|has_fd %}\n> > +{{serializer.deserializer_fd(struct, namespace_str)}}\n> > +{%- else %}\n> > +{{serializer.deserializer_no_fd(struct, namespace_str)}}\n> > +{%- endif %}\n> > +};\n> > +{% endfor %}\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__ */\n> > diff --git a/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > new file mode 100644\n> > index 00000000..f6836034\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > @@ -0,0 +1,205 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{#\n> > + # \\brief Generate fuction prototype\n> > + #\n> > + # \\param class Class name\n> > + # \\param method mojom Method object\n> > + # \\param suffix Suffix to append to \\a method function name\n> > + # \\param need_class_name True to generate class name with function\n> > + # \\param override True to generate override tag after the function prototype\n> > + #}\n> > +{%- macro func_sig(class, method, suffix, need_class_name = true, override = false) -%}\n> \n> There's one occurrence of a call to this macro with two arguments only.\n> Should suffix have a default of \"\" ?\n\nThat's the default default value of macro parameters in jinja, that's why\nI left it blank.\n\n> > +{{method|method_return_value}} {{class + \"::\" if need_class_name}}{{method.mojom_name}}{{suffix}}(\n> > +{%- for param in method|method_parameters %}\n> > +\t{{param}}{{- \",\" if not loop.last}}\n> > +{%- endfor -%}\n> > +){{\" override\" if override}}\n> > +{%- endmacro -%}\n> > +\n> > +{#\n> > + # \\brief Generate function body for IPA init() function for thread\n> > + #}\n> > +{%- macro init_thread_body() -%}\n> > +\tint ret = ipa_->init(settings);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> > +\tproxy_.moveToThread(&thread_);\n> > +\n> > +\treturn 0;\n> > +{%- endmacro -%}\n> > +\n> > +{#\n> > + # \\brief Generate function body for IPA start() function for thread\n> > + #}\n> > +{%- macro start_thread_body() -%}\n> > +\trunning_ = true;\n> > +\tthread_.start();\n> > +\n> > +\treturn proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking);\n> > +{%- endmacro -%}\n> > +\n> > +{#\n> > + # \\brief Generate function body for IPA stop() function for thread\n> > + #}\n> > +{%- macro stop_thread_body() -%}\n> > +\tif (!running_)\n> > +\t\treturn;\n> > +\n> > +\trunning_ = false;\n> > +\n> > +\tproxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking);\n> > +\n> > +\tthread_.exit();\n> > +\tthread_.wait();\n> > +{%- endmacro -%}\n> > +\n> > +\n> > +{#\n> > + # \\brief Serialize multiple objects into data buffer and fd vector\n> > + #\n> > + # Generate code to serialize multiple objects, as specified in \\a params\n> > + # (which are the parameters to some function), into \\a buf data buffer and\n> > + # \\a fds fd vector.\n> > + # This code is meant to be used by the proxy, for serializing prior to IPC calls.\n> > + #}\n> > +{%- macro serialize_call(params, buf, fds, base_controls) %}\n> > +{%- for param in params %}\n> > +\tstd::vector {{param.mojom_name}}Buf;\n> > +{%- if param|has_fd %}\n> > +\tstd::vector {{param.mojom_name}}Fds;\n> > +\tstd::tie({{param.mojom_name}}Buf, {{param.mojom_name}}Fds) =\n> > +{%- else %}\n> > +\tstd::tie({{param.mojom_name}}Buf, std::ignore) =\n> > +{%- endif %}\n> > +{%- if param|is_controls %}\n> > +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}, {{param.mojom_name}}.infoMap() ? *{{param.mojom_name}}.infoMap() : {{base_controls}}\n> > +{%- else %}\n> > +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}\n> > +{%- endif %}\n> > +{{- \", &controlSerializer_\" if param|needs_control_serializer -}}\n> > +);\n> > +{%- endfor %}\n> > +\n> > +{%- if params|length > 1 %}\n> > +{%- for param in params %}\n> > +\tappendPOD({{buf}}, {{param.mojom_name}}Buf.size());\n> > +{%- if param|has_fd %}\n> > +\tappendPOD({{buf}}, {{param.mojom_name}}Fds.size());\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +{%- endif %}\n> > +\n> > +{%- for param in params %}\n> > +\t{{buf}}.insert({{buf}}.end(), {{param.mojom_name}}Buf.begin(), {{param.mojom_name}}Buf.end());\n> > +{%- endfor %}\n> > +\n> > +{%- for param in params %}\n> > +{%- if param|has_fd %}\n> > +\t{{fds}}.insert({{fds}}.end(), {{param.mojom_name}}Fds.begin(), {{param.mojom_name}}Fds.end());\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +{%- endmacro -%}\n> \n> I wonder if we could optimize this by avoiding the intermediate vectors,\n> but that's a candidate for a later optimization.\n\nI'll add it to the todo list.\n\n> > +\n> > +\n> > +{#\n> > + # \\brief Deserialize a single object from data buffer and fd vector\n> > + #\n> > + # \\param pointer True deserializes the object into a dereferenced pointer\n> \n> s/True deserializes/If true, deserialize/ ?\n> \n> Same in other locations below.\n\nYes.\n\n> > + # \\param iter True treats \\a buf as an iterator instead of a vector\n> > + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> > + #\n> > + # Generate code to deserialize a single object, as specified in \\a param,\n> > + # from \\a buf data buffer and \\a fds fd vector.\n> > + # This code is meant to be used by macro deserialize_call.\n> > + #}\n> > +{%- macro deserialize_param(param, pointer, loop, buf, fds, iter, data_size) -%}\n> > +{{\"*\" if pointer}}{{param.mojom_name}} = IPADataSerializer<{{param|name}}>::deserialize(\n> > +{%- if not iter %}\n> > +\t{{buf}}.begin() + {{param.mojom_name}}Start,\n> > +{%- else %}\n> > +\t{{buf}} + {{param.mojom_name}}Start,\n> > +{%- endif %}\n> \n> Maybe\n> \n> \t{{buf}}{{- \".begin()\" if not iter}} + {{param.mojom_name}}Start,\n> \n> ?\n\nYeah that's better.\n\n> > +{%- if loop.last and not iter %}\n> > +\t{{buf}}.end()\n> > +{%- elif not iter %}\n> > +\t{{buf}}.begin() + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> > +{%- elif iter and loop.length == 1 %}\n> > +\t{{buf}} + {{data_size}}\n> > +{%- else %}\n> > +\t{{buf}} + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> > +{%- endif -%}\n> \n> I wonder if it would simplify the generated code if we standardized on\n> Span. Passing both the vector iterator and the size in separate\n> arguments can more easily result in them becoming out of sync. I was\n> thinking the Span<>::subspan() function could be quite helpful here, but\n> it doesn't handle overflows (requesting a subspan with offset > size is\n> ill-defined) :-S\n> \n> Not necessarily something to be addressed now, but do you think it would\n> make sense ? And of course if it helps fixing some of the issues\n> outlined elsewhere in this review, we could already switch to span.\n\nIt probably would make sense... but it'll be a pain to convert :/\n\n> > +{{- \",\" if param|has_fd}}\n> > +{%- if param|has_fd %}\n> > +\t{{fds}}.begin() + {{param.mojom_name}}FdStart,\n> > +{%- if loop.last %}\n> > +\t{{fds}}.end()\n> > +{%- else %}\n> > +\t{{fds}}.begin() + {{param.mojom_name}}FdStart + {{param.mojom_name}}FdsSize\n> > +{%- endif -%}\n> > +{%- endif -%}\n> > +{{- \",\" if param|needs_control_serializer}}\n> > +{%- if param|needs_control_serializer %}\n> > +\t&controlSerializer_\n> > +{%- endif -%}\n> > +);\n> > +{%- endmacro -%}\n> > +\n> > +\n> > +{#\n> > + # \\brief Deserialize multiple objects from data buffer and fd vector\n> > + #\n> > + # \\param pointer True deserializes objects into pointers, and adds a null check.\n> > + # \\param declare True declares the objects in addition to deserialization.\n> > + # \\param iter True treats \\a buf as an iterator instead of a vector\n> > + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> > + #\n> > + # Generate code to deserialize multiple objects, as specified in \\a params\n> > + # (which are the parameters to some function), from \\a buf data buffer and\n> > + # \\a fds fd vector.\n> > + # This code is meant to be used by the proxy, for deserializing after IPC calls.\n> > + #}\n> > +{%- macro deserialize_call(params, buf, fds, pointer = true, declare = false, iter = false, data_size = '') -%}\n> > +{% set ns = namespace(size_offset = 0) %}\n> > +{%- if params|length > 1 %}\n> > +{%- for param in params %}\n> > +\t[[maybe_unused]] size_t {{param.mojom_name}}BufSize = readPOD({{buf}}, {{ns.size_offset}}\n> \n> const ?\n> \n> > +{%- if iter -%}\n> > +, {{buf}} + {{data_size}}\n> > +{%- endif -%}\n> > +);\n> > +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> > +{%- if param|has_fd %}\n> > +\t[[maybe_unused]] size_t {{param.mojom_name}}FdsSize = readPOD({{buf}}, {{ns.size_offset}}\n> \n> const ?\n> \n> > +{%- if iter -%}\n> > +, {{buf}} + {{data_size}}\n> > +{%- endif -%}\n> > +);\n> > +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +{%- endif %}\n> > +{% for param in params %}\n> > +{%- if loop.first %}\n> > +\tsize_t {{param.mojom_name}}Start = {{ns.size_offset}};\n> \n> const ?\n> \n> Same in a few locations below.\n\nYes.\n\n> > +{%- else %}\n> > +\tsize_t {{param.mojom_name}}Start = {{loop.previtem.mojom_name}}Start + {{loop.previtem.mojom_name}}BufSize;\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +{% for param in params|with_fds %}\n> > +{%- if loop.first %}\n> > +\tsize_t {{param.mojom_name}}FdStart = 0;\n> > +{%- elif not loop.last %}\n> > +\tsize_t {{param.mojom_name}}FdStart = {{loop.previtem.mojom_name}}FdStart + {{loop.previtem.mojom_name}}FdsSize;\n> > +{%- endif %}\n> > +{%- endfor %}\n> > +{% for param in params %}\n> > +\t{%- if pointer %}\n> > +\tif ({{param.mojom_name}}) {\n> > +{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(16, True)}}\n> > +\t}\n> > +\t{%- else %}\n> > +\t{{param|name + \" \" if declare}}{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(8)}}\n> > +\t{%- endif %}\n> > +{% endfor %}\n> > +{%- endmacro -%}\n> > diff --git a/utils/ipc/generators/libcamera_templates/serializer.tmpl b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > new file mode 100644\n> > index 00000000..51dbeb0e\n> > --- /dev/null\n> > +++ b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > @@ -0,0 +1,280 @@\n> > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > +{# Turn this into a C macro? #}\n> > +{#\n> > + # \\brief Verify that there is enough bytes to deserialize\n> > + #\n> > + # Generate code that verifies that \\a size is not greater than \\a dataSize.\n> > + # Otherwise log an error with \\a name and \\a typename.\n> > + #}\n> > +{%- macro check_data_size(size, dataSize, name, typename) %}\n> > +\t\tif ({{size}} > {{dataSize}}) {\n> \n> \n> Maybe\n> \t\tif ({{dataSize}} < {{size}}) {\n> \n> ?\n> \n> > +\t\t\tLOG(IPADataSerializer, Error)\n> > +\t\t\t\t<< \"Failed to deserialize {{name}}: not enough {{typename}}, expected \"\n> \n> \t\t\t\t<< \"Failed to deserialize \" << \"{{name}}\"\n> \t\t\t\t<< \": not enough {{typename}}, expected \"\n> \n> This should allow the compiler to deduplicate strings.\n> \n> > +\t\t\t\t<< ({{size}}) << \", got \" << ({{dataSize}});\n> > +\t\t\treturn ret;\n> > +\t\t}\n> > +{%- endmacro %}\n> > +\n> > +\n> > +{#\n> > + # \\brief Serialize some field into return vector\n> > + #\n> > + # Generate code to serialize \\a field into retData, including size of the\n> > + # field and fds (where appropriate). \\a base_controls indicates the\n> > + # default ControlInfoMap in the event that the ControlList does not have one.\n> > + # This code is meant to be used by the IPADataSerializer specialization.\n> > + #}\n> > +{%- macro serializer_field(field, base_controls, namespace, loop) %}\n> > +{%- if field|is_pod or field|is_enum %}\n> > +\t\tstd::vector {{field.mojom_name}};\n> > +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > +\t{%- if field|is_pod %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > +\t{%- elif field|is_enum %}\n> > +\t\t\tIPADataSerializer::serialize(data.{{field.mojom_name}}_);\n> > +\t{%- endif %}\n> > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > +{%- elif field|is_fd %}\n> > +\t\tstd::vector {{field.mojom_name}};\n> > +\t\tstd::vector {{field.mojom_name}}Fds;\n> > +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> > +{%- elif field|is_controls %}\n> > +\t\tif (data.{{field.mojom_name}}_.size() > 0) {\n> > +\t\t\tstd::vector {{field.mojom_name}};\n> > +\t\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > +\t\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_,\n> > +\t\t\t\t\tdata.{{field.mojom_name}}_.infoMap() ? *data.{{field.mojom_name}}_.infoMap() : {{base_controls}},\n> > +\t\t\t\t\tcs);\n> > +\t\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> > +\t\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > +\t\t} else {\n> > +\t\t\tappendPOD(retData, 0);\n> > +\t\t}\n> > +{%- elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> > +\t\tstd::vector {{field.mojom_name}};\n> > +\t{%- if field|has_fd %}\n> > +\t\tstd::vector {{field.mojom_name}}Fds;\n> > +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> > +\t{%- else %}\n> > +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > +\t{%- endif %}\n> > +\t{%- if field|is_array or field|is_map %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, cs);\n> > +\t{%- elif field|is_str %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > +\t{%- else %}\n> > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::serialize(data.{{field.mojom_name}}_, cs);\n> > +\t{%- endif %}\n> > +\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> > +\t{%- if field|has_fd %}\n> > +\t\tappendPOD(retData, {{field.mojom_name}}Fds.size());\n> > +\t{%- endif %}\n> > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > +\t{%- if field|has_fd %}\n> > +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> > +\t{%- endif %}\n> > +{%- else %}\n> > +\t\t/* Unknown serialization for {{field.mojom_name}}. */\n> > +{%- endif %}\n> > +{%- endmacro %}\n> \n> Same here, I think there's room for optimization if we could avoid the\n> intermediate vectors.\n> \n> > +\n> > +\n> > +{#\n> > + # \\brief Deserialize some field into return struct\n> > + #\n> > + # Generate code to deserialize \\a field into object ret.\n> > + # This code is meant to be used by the IPADataSerializer specialization.\n> > + #}\n> > +{%- macro deserializer_field(field, namespace, loop) %}\n> > +{% if field|is_pod or field|is_enum %}\n> > +\t{%- set field_size = (field|bit_width|int / 8)|int %}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > +\t\t{%- if field|is_pod %}\n> > +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer<{{field|name}}>::deserialize(m, m + {{field_size}}));\n> \n> Do we need the static_cast<>, doesn't\n> IPADataSerializer<{{field|name}}>::deserialize() return a {{field|name}}\n> ?\n\nAh yes, this one doesn't need the static_cast. The next one does due to\nenums though.\n\n> > +\t\t{%- else %}\n> > +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer::deserialize(m, m + {{field_size}}));\n> > +\t\t{%- endif %}\n> > +\t{%- if not loop.last %}\n> > +\t\tm += {{field_size}};\n> > +\t\tdataSize -= {{field_size}};\n> > +\t{%- endif %}\n> > +{% elif field|is_fd %}\n> > +\t{%- set field_size = 1 %}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > +\t\tret.{{field.mojom_name}}_ = IPADataSerializer<{{field|name}}>::deserialize(m, m + 1, n, n + 1, cs);\n> > +\t{%- if not loop.last %}\n> > +\t\tm += {{field_size}};\n> > +\t\tdataSize -= {{field_size}};\n> > +\t\tn += ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> > +\t\tfdsSize -= ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> > +\t{%- endif %}\n> > +{% elif field|is_controls %}\n> > +\t{%- set field_size = 4 %}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> > +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> \n> const ?\n> \n> You call IPADataSerializer<{{field|name}}>::deserialize() above for POD\n> types, and readPOD() here. Do we need the IPADataSerializer<>\n> specializations for all the PODs, can't we call readPOD() directly ?\n\nNo we can't get rid of the POD specializations of IPADataSerializer,\nbecause the IPADataSerializer for vectors and maps needs them. So for\nserdesing fields that are PODs we use IPADataSerializer, and for\nserdesing metadata like field size, we use readPOD.\n\n> > +\t{%- set field_size = '4 + ' + field.mojom_name + 'Size' -%}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > +\t\tif ({{field.mojom_name}}Size > 0)\n> > +\t\t\tret.{{field.mojom_name}}_ =\n> > +\t\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > +\t{%- if not loop.last %}\n> > +\t\tm += {{field_size}};\n> > +\t\tdataSize -= {{field_size}};\n> > +\t{%- endif %}\n> > +{% elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> > +\t{%- set field_size = 4 %}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> > +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> \n> const ?\n> \n> > +\t{%- if field|has_fd %}\n> > +\t{%- set field_size = 8 %}\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'FdsSize', 'data')}}\n> > +\t\tsize_t {{field.mojom_name}}FdsSize = readPOD(m, 4, data.end());\n> \n> const ?\n> \n> > +\t\t{{- check_data_size(field.mojom_name + 'FdsSize', 'fdsSize', field.mojom_name, 'fds')}}\n> > +\t{%- endif %}\n> > +\t{%- set field_size = field|has_fd|choose('8 + ', '4 + ') + field.mojom_name + 'Size' -%}\n> \n> I'm afraid you've got quite a few integer overflow issues here :-S\n\n:S\n\n> > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > +\t\tret.{{field.mojom_name}}_ =\n> > +\t{%- if field|is_str %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size);\n> > +\t{%- elif field|has_fd and (field|is_array or field|is_map) %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> > +\t{%- elif field|has_fd and (not (field|is_array or field|is_map)) %}\n> > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> > +\t{%- elif (not field|has_fd) and (field|is_array or field|is_map) %}\n> > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > +\t{%- else %}\n> > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > +\t{%- endif %}\n> \n> Can't we advance m after reading the size fields ? That would avoid all\n> the error-prone calculation, and get rid of at least some of the integer\n> overflow issues.\n\nOh, right.\n\n> > +\t{%- if not loop.last %}\n> > +\t\tm += {{field_size}};\n> > +\t\tdataSize -= {{field_size}};\n> > +\t{%- if field|has_fd %}\n> > +\t\tn += {{field.mojom_name}}FdsSize;\n> > +\t\tfdsSize -= {{field.mojom_name}}FdsSize;\n> > +\t{%- endif %}\n> > +\t{%- endif %}\n> > +{% else %}\n> > +\t\t/* Unknown deserialization for {{field.mojom_name}}. */\n> > +{%- endif %}\n> > +{%- endmacro %}\n> > +\n> > +\n> > +{#\n> > + # \\brief Serialize a struct\n> > + #\n> > + # Generate code for IPADataSerializer specialization, for serializing\n> > + # \\a struct. \\a base_controls indicates the default ControlInfoMap\n> > + # in the event that the ControlList does not have one.\n> > + #}\n> > +{%- macro serializer(struct, base_controls, namespace) %}\n> > +\tstatic std::tuple, std::vector>\n> > +\tserialize(const {{struct|name_full(namespace)}} &data,\n> > +{%- if struct|needs_control_serializer %}\n> > +\t\t ControlSerializer *cs)\n> > +{%- else %}\n> > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > +{%- endif %}\n> > +\t{\n> > +\t\tstd::vector retData;\n> > +{%- if struct|has_fd %}\n> > +\t\tstd::vector retFds;\n> > +{%- endif %}\n> > +{%- for field in struct.fields %}\n> > +{{serializer_field(field, base_controls, namespace, loop)}}\n> > +{%- endfor %}\n> > +{% if struct|has_fd %}\n> > +\t\treturn {retData, retFds};\n> > +{%- else %}\n> > +\t\treturn {retData, {}};\n> > +{%- endif %}\n> > +\t}\n> > +{%- endmacro %}\n> > +\n> > +\n> > +{#\n> > + # \\brief Deserialize a struct that has fds\n> > + #\n> > + # Generate code for IPADataSerializer specialization, for deserializing\n> > + # \\a struct, in the case that \\a struct has file descriptors.\n> > + #}\n> > +{%- macro deserializer_fd(struct, namespace) %}\n> > +\tstatic {{struct|name_full(namespace)}}\n> > +\tdeserialize(std::vector &data,\n> > +\t\t std::vector &fds,\n> > +{%- if struct|needs_control_serializer %}\n> > +\t\t ControlSerializer *cs)\n> > +{%- else %}\n> > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > +{%- endif %}\n> > +\t{\n> > +\t\t{{struct|name_full(namespace)}} ret;\n> > +\t\tstd::vector::iterator m = data.begin();\n> > +\t\tstd::vector::iterator n = fds.begin();\n> > +\n> > +\t\tsize_t dataSize = data.size();\n> > +\t\tsize_t fdsSize = fds.size();\n> > +{%- for field in struct.fields -%}\n> > +{{deserializer_field(field, namespace, loop)}}\n> > +{%- endfor %}\n> > +\t\treturn ret;\n> > +\t}\n> > +\n> > +\tstatic {{struct|name_full(namespace)}}\n> > +\tdeserialize(std::vector::iterator dataBegin,\n> > +\t\t std::vector::iterator dataEnd,\n> > +\t\t std::vector::iterator fdsBegin,\n> > +\t\t std::vector::iterator fdsEnd,\n> \n> All these should const const_iterator. There are quite a few locations\n> in the deserializer where the data and fds vectors or iterators are not\n> const. Could you fix that ?\n\nYeah, I'll fix all of that.\n\n> > +{%- if struct|needs_control_serializer %}\n> > +\t\t ControlSerializer *cs)\n> > +{%- else %}\n> > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > +{%- endif %}\n> > +\t{\n> > +\t\tstd::vector data(dataBegin, dataEnd);\n> \n> This creates a copy of the data, do we really need that ? I think you\n> should turn this around, implement the deserialization in this function,\n> and implement the previous function as a wrapper around this one.\n\nAh, okay.\n\n> > +\t\tstd::vector fds(fdsBegin, fdsEnd);\n> > +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, fds, cs);\n> > +\t}\n> > +{%- endmacro %}\n> > +\n> > +\n> > +{#\n> > + # \\brief Deserialize a struct that has no fds\n> > + #\n> > + # Generate code for IPADataSerializer specialization, for deserializing\n> > + # \\a struct, in the case that \\a struct does not have file descriptors.\n> > + #}\n> > +{%- macro deserializer_no_fd(struct, namespace) %}\n> > +\tstatic {{struct|name_full(namespace)}}\n> > +\tdeserialize(std::vector &data,\n> > +{%- if struct|needs_control_serializer %}\n> > +\t\t ControlSerializer *cs)\n> > +{%- else %}\n> > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > +{%- endif %}\n> > +\t{\n> > +\t\t{{struct|name_full(namespace)}} ret;\n> > +\t\tstd::vector::iterator m = data.begin();\n> > +\n> > +\t\tsize_t dataSize = data.size();\n> > +{%- for field in struct.fields -%}\n> > +{{deserializer_field(field, namespace, loop)}}\n> > +{%- endfor %}\n> > +\t\treturn ret;\n> > +\t}\n> > +\n> > +\tstatic {{struct|name_full(namespace)}}\n> > +\tdeserialize(std::vector::iterator dataBegin,\n> > +\t\t std::vector::iterator dataEnd,\n> > +{%- if struct|needs_control_serializer %}\n> > +\t\t ControlSerializer *cs)\n> > +{%- else %}\n> > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > +{%- endif %}\n> > +\t{\n> > +\t\tstd::vector data(dataBegin, dataEnd);\n> > +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, cs);\n> > +\t}\n> > +{%- endmacro %}\n\n\nThanks,\n\nPaul","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 97B0CBE08A\n\tfor ;\n\tThu, 26 Nov 2020 06:49:33 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 13B8563416;\n\tThu, 26 Nov 2020 07:49:29 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 41C356032F\n\tfor ;\n\tThu, 26 Nov 2020 07:49:27 +0100 (CET)","from pyrite.rasen.tech (unknown\n\t[IPv6:2400:4051:61:600:2c71:1b79:d06d:5032])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 0819F5A6;\n\tThu, 26 Nov 2020 07:49:24 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"K3OjBQYb\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1606373366;\n\tbh=7yCqeE7FwDXfe3x1y2D3S0WSr0M37WC6tlzsTlbGiNc=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=K3OjBQYbQCHMe/uPp1I9IKidz+UQ6rAhdMqQfOoBGKGtk5RBwFnMykDlID04UxlcE\n\toiDm3Aeesl2hFamr/tmJV3vPQ5cNyGy2WbiyUXlF6RrzT8f+ZjIHERBM4tqNWNVF48\n\tCqpTlQ+0Vej7YgpOqtpF6TaKCtbgKNRw7rbAiNNw=","Date":"Thu, 26 Nov 2020 15:49:18 +0900","From":"paul.elder@ideasonboard.com","To":"Laurent Pinchart ","Message-ID":"<20201126064918.GB2050@pyrite.rasen.tech>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>\n\t<20201119172100.GC4563@pendragon.ideasonboard.com>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201119172100.GC4563@pendragon.ideasonboard.com>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13933,"web_url":"https://patchwork.libcamera.org/comment/13933/","msgid":"<20201126160047.GW3905@pendragon.ideasonboard.com>","date":"2020-11-26T16:00:47","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nOn Mon, Nov 23, 2020 at 05:11:56PM +0900, paul.elder@ideasonboard.com wrote:\n> On Thu, Nov 19, 2020 at 02:39:17PM +0200, Laurent Pinchart wrote:\n> > On Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> > > Add templates to mojo to generate code for the IPC mechanism. These\n> > > templates generate:\n> > > - module header\n> > > - module serializer\n> > > - IPA proxy cpp, header, and worker\n> > > \n> > > Given an input data definition mojom file for a pipeline.\n> > > \n> > > Signed-off-by: Paul Elder \n> > > \n> > > ---\n> > > Changes in v4:\n> > > For the non-template files:\n> > > - rename IPA{pipeline_name}CallbackInterface to\n> > > IPA{pipeline_name}EventInterface\n> > > - to avoid the notion of \"callback\" and emphasize that it's an event\n> > > - add support for strings in custom structs\n> > > - add validation, that async methods must not have return values\n> > > - it throws exception and isn't very clear though...?\n> > > - rename controls to libcamera::{pipeline_name}::controls (controls is\n> > > now lowercase)\n> > > - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> > > and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> > > - same for their corresponding template files\n> > > For the template files:\n> > > - fix spacing (now it's all {{var}} instead of some {{ var }})\n> > > - except if it's code, so code is still {{ code }}\n> > > - move inclusion of corresponding header to first in the inclusion list\n> > > - fix copy&paste errors\n> > > - change snake_case to camelCase in the generated code\n> > > - template code still uses snake_case\n> > > - change the generated command enums to an enum class, and make it\n> > > capitalized (instead of allcaps)\n> > > - add length checks to recvIPC (in proxy)\n> > > - fix some template spacing\n> > > - don't use const for PODs in function/signal parameters\n> > > - add the proper length checks to readPOD/appendPOD\n> > > - the helper functions for reading and writing PODs to and from\n> > > serialized data\n> > > - rename readUInt/appendUInt to readPOD/appendPOD\n> > > - add support for strings in custom structs\n> > > \n> > > Changes in v3:\n> > > - add support for namespaces\n> > > - fix enum assignment (used to have +1 for CMD applied to all enums)\n> > > - use readHeader, writeHeader, and eraseHeader as static class functions\n> > > of IPAIPCUnixSocket (in the proxy worker)\n> > > - add requirement that base controls *must* be defined in\n> > > libcamera::{pipeline_name}::Controls\n> > > \n> > > Changes in v2:\n> > > - mandate the main and callback interfaces, and init(), start(), stop()\n> > > and their parameters\n> > > - fix returning single pod value from IPC-called function\n> > > - add licenses\n> > > - improve auto-generated message\n> > > - other fixes related to serdes\n> > > ---\n> > > .../module_ipa_interface.h.tmpl | 113 ++++\n> > > .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> > > .../module_ipa_proxy.h.tmpl | 118 +++++\n> > > .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> > > .../module_ipa_serializer.h.tmpl | 44 ++\n> > > .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> > > .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> > > .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> > > 8 files changed, 1673 insertions(+)\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > > create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n> > \n> > Let's start with the generator.\n> > \n> > [snip]\n> > \n> > > diff --git a/utils/ipc/generators/mojom_libcamera_generator.py b/utils/ipc/generators/mojom_libcamera_generator.py\n> > > new file mode 100644\n> > > index 00000000..c4fbf9b3\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/mojom_libcamera_generator.py\n> > > @@ -0,0 +1,488 @@\n> > \n> > Missing SPDX header and copyright notice.\n> > \n> > > +'''Generates libcamera files from a mojom.Module.'''\n> > \n> > Comments in Python use # :-)\n> > \n> > > +\n> > > +import argparse\n> > > +import ast\n> > > +import datetime\n> > > +import contextlib\n> > > +import os\n> > > +import re\n> > > +import shutil\n> > > +import sys\n> > > +import tempfile\n> > \n> > ast, contextlib, shutil, sys and tempfile don't seem to be used.\n> > \n> > > +\n> > > +from jinja2 import contextfilter\n> > \n> > contextfilter isn't used either.\n> > \n> > > +\n> > > +import mojom.fileutil as fileutil\n> > > +import mojom.generate.generator as generator\n> > > +import mojom.generate.module as mojom\n> > > +from mojom.generate.template_expander import UseJinja\n> > > +\n> > > +\n> > > +GENERATOR_PREFIX = 'libcamera'\n> > > +\n> > > +_kind_to_cpp_type = {\n> > > + mojom.BOOL: 'bool',\n> > > + mojom.INT8: 'int8_t',\n> > > + mojom.UINT8: 'uint8_t',\n> > > + mojom.INT16: 'int16_t',\n> > > + mojom.UINT16: 'uint16_t',\n> > > + mojom.INT32: 'int32_t',\n> > > + mojom.UINT32: 'uint32_t',\n> > > + mojom.FLOAT: 'float',\n> > > + mojom.INT64: 'int64_t',\n> > > + mojom.UINT64: 'uint64_t',\n> > > + mojom.DOUBLE: 'double',\n> > > +}\n> > > +\n> > > +_bit_widths = {\n> > > + mojom.BOOL: '8',\n> > > + mojom.INT8: '8',\n> > > + mojom.UINT8: '8',\n> > > + mojom.INT16: '16',\n> > > + mojom.UINT16: '16',\n> > > + mojom.INT32: '32',\n> > > + mojom.UINT32: '32',\n> > > + mojom.FLOAT: '32',\n> > > + mojom.INT64: '64',\n> > > + mojom.UINT64: '64',\n> > > + mojom.DOUBLE: '64',\n> > > +}\n> > > +\n> > > +def ModuleName(path):\n> > > + return path.split('/')[-1].split('.')[0]\n> > > +\n> > > +def ModuleClassName(module):\n> > > + s = re.sub(r'^IPA', '', module.interfaces[0].mojom_name)\n> > > + return re.sub(r'Interface$', '', s)\n> > \n> > You could combine the two with\n> > \n> > return re.sub(r'^IPA(.*)Interface$', lambda match: match.group(1), module)\n> > \n> > and possibly optimize it further by pre-compiling the regular\n> > expression. I'll let you decide if it's worth it.\n> > \n> > > +\n> > > +def UpperCamelCase(name):\n> > > + return ''.join([x.capitalize() for x in generator.SplitCamelCase(name)])\n> > > +\n> > > +def CamelCase(name):\n> > > + uccc = UpperCamelCase(name)\n> > > + return uccc[0].lower() + uccc[1:]\n> > \n> > These two functions don't seem to be used.\n> > \n> > > +\n> > > +def Capitalize(name):\n> > > + return name[0].upper() + name[1:]\n> > > +\n> > > +def ConstantStyle(name):\n> > > + return generator.ToUpperSnakeCase(name)\n> > > +\n> > > +def Choose(cond, t, f):\n> > > + return t if cond else f\n> > > +\n> > > +def CommaSep(l):\n> > > + return ', '.join([m for m in l])\n> > > +\n> > > +def ParamsCommaSep(l):\n> > > + return ', '.join([m.mojom_name for m in l])\n> > > +\n> > > +def GetDefaultValue(element):\n> > > + if element.default is not None:\n> > > + return element.default\n> > > + if type(element.kind) == mojom.Kind:\n> > > + return '0'\n> > > + if mojom.IsEnumKind(element.kind):\n> > > + return f'static_cast<{element.kind.mojom_name}>(0)'\n> > > + if (isinstance(element.kind, mojom.Struct) and\n> > > + element.kind.mojom_name == 'FileDescriptor'):\n> > \n> > Is there a need for the outer parentheses ? Or is this dictated by the\n> > mojom coding style ? Same comment for several locations below.\n> > \n> > > + return '-1'\n> > > + return ''\n> > > +\n> > > +def WithDefaultValues(element):\n> > > + return [x for x in element if HasDefaultValue(x)]\n> > > +\n> > > +def WithFds(element):\n> > > + return [x for x in element if HasFd(x)]\n> > \n> > Should these be moved a little bit further down, after the functions\n> > they call ? The code works fine, but it would create an easier to read\n> > flow.\n> > \n> > > +\n> > > +def HasDefaultValue(element):\n> > > + return GetDefaultValue(element) != ''\n> > > +\n> > > +def HasDefaultFields(element):\n> > > + return True in [HasDefaultValue(x) for x in element.fields]\n> > > +\n> > > +def GetAllTypes(element):\n> > > + if mojom.IsArrayKind(element):\n> > > + return GetAllTypes(element.kind)\n> > > + if mojom.IsMapKind(element):\n> > > + return GetAllTypes(element.key_kind) + GetAllTypes(element.value_kind)\n> > > + if isinstance(element, mojom.Parameter):\n> > > + return GetAllTypes(element.kind)\n> > > + if mojom.IsEnumKind(element):\n> > > + return [element.mojom_name]\n> > > + if not mojom.IsStructKind(element):\n> > > + return [element.spec]\n> > > + if len(element.fields) == 0:\n> > > + return [element.mojom_name]\n> > > + ret = [GetAllTypes(x.kind) for x in element.fields]\n> > > + ret = [x for sublist in ret for x in sublist]\n> > > + return list(set(ret))\n> > > +\n> > > +def GetAllAttrs(element):\n> > > + if mojom.IsArrayKind(element):\n> > > + return GetAllAttrs(element.kind)\n> > > + if mojom.IsMapKind(element):\n> > > + return {**GetAllAttrs(element.key_kind), **GetAllAttrs(element.value_kind)}\n> > > + if isinstance(element, mojom.Parameter):\n> > > + return GetAllAttrs(element.kind)\n> > > + if mojom.IsEnumKind(element):\n> > > + return element.attributes if element.attributes is not None else {}\n> > > + if mojom.IsStructKind(element) and len(element.fields) == 0:\n> > > + return element.attributes if element.attributes is not None else {}\n> > > + if not mojom.IsStructKind(element):\n> > > + return {}\n> > > + attrs = [(x.attributes) for x in element.fields]\n> > > + ret = {}\n> > > + for d in attrs:\n> > > + if d is not None:\n> > > + ret = {**ret, **d}\n> > \n> > Maybe\n> > \n> > ret.update(d)\n> > \n> > > + return ret\n> > > +\n> > > +def NeedsControlSerializer(element):\n> > > + types = GetAllTypes(element)\n> > > + return \"ControlList\" in types or \"ControlInfoMap\" in types\n> > > +\n> > > +def HasFd(element):\n> > > + attrs = GetAllAttrs(element)\n> > > + if isinstance(element, mojom.Kind):\n> > > + types = GetAllTypes(element)\n> > > + else:\n> > > + types = GetAllTypes(element.kind)\n> > > + return \"FileDescriptor\" in types or (attrs is not None and \"hasFd\" in attrs)\n> > > +\n> > > +def MethodInputHasFd(method):\n> > > + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> > \n> > You can drop the len() call, [] evaluates to False.\n> \n> What do you mean? It evaluates to something like this:\n> \n> []\n> \n> So it checks how many parameters have fd.\n\nMy bad. I thought python would make it a list, not a generator, and\nempty lists evaluate to False. Please ignore this comment.\n\nIt would be nice if python had a syntax similar to\n\n\tif one of HasFd(x) for x in method.parameters:\n\n:-)\n\n> > > + return True\n> > > + return False\n> > > +\n> > > +def MethodOutputHasFd(method):\n> > > + if (MethodReturnValue(method) != 'void' or\n> > > + method.response_parameters is None):\n> > > + return False\n> > > + if len([x for x in method.parameters if HasFd(x)]) > 0:\n> > \n> > Same here.\n> > \n> > Shouldn't it be method.response_parameters instead of method.parameters\n> > ?\n> \n> Yeah it should be.\n> \n> > > + return True\n> > > + return False\n> > > +\n> > > +def MethodParamInputs(method):\n> > > + return method.parameters\n> > > +\n> > > +def MethodParamOutputs(method):\n> > > + if (MethodReturnValue(method) != 'void' or\n> > > + method.response_parameters is None):\n> > > + return []\n> > > + return method.response_parameters\n> > \n> > The increase code reuse, you could implement these functions as follows.\n> > \n> > \n> > def MethodParamInputs(method):\n> > return method.parameters\n> > \n> > def MethodParamOutputs(method):\n> > if (MethodReturnValue(method) != 'void' or\n> > method.response_parameters is None):\n> > return []\n> > return method.response_parameters\n> > \n> > def MethodParamHasFd(parameters):\n> > if [x for x in parameters if HasFd(x)]:\n> > return True\n> > return False\n> > \n> > def MethodInputHasFd(method):\n> > return MethodParamHasFd(MethodParamInputs(method))\n> > \n> > def MethodOutputHasFd(method):\n> > return MethodParamHasFd(MethodParamOutputs(method))\n> > \n> > > +\n> > > +def MethodParamNames(method):\n> > > + params = []\n> > > + for param in method.parameters:\n> > > + params.append(param.mojom_name)\n> > > + if MethodReturnValue(method) == 'void':\n> > > + if method.response_parameters is None:\n> > > + return params\n> > > + for param in method.response_parameters:\n> > > + params.append(param.mojom_name)\n> > > + return params\n> > > +\n> > > +def MethodParameters(method):\n> > > + params = []\n> > > + for param in method.parameters:\n> > > + params.append('const %s %s%s' % (GetNameForElement(param),\n> > > + ('&' if not IsPod(param) else ''),\n> > \n> > No need for the outer parentheses.\n> > \n> > > + param.mojom_name))\n> > > + if MethodReturnValue(method) == 'void':\n> > > + if method.response_parameters is None:\n> > > + return params\n> > > + for param in method.response_parameters:\n> > > + params.append(f'{GetNameForElement(param)} *{param.mojom_name}')\n> > > + return params\n> > > +\n> > > +def MethodReturnValue(method):\n> > > + if method.response_parameters is None:\n> > > + return 'void'\n> > > + if len(method.response_parameters) == 1 and IsPod(method.response_parameters[0]):\n> > > + return GetNameForElement(method.response_parameters[0])\n> > > + return 'void'\n> > > +\n> > > +def IsAsync(method):\n> > > + # callbacks are always async\n> > \n> > s/callbacks/Events/ ?\n> > \n> > > + if re.match(\"^IPA.*EventInterface$\", method.interface.mojom_name):\n> > > + return True\n> > > + elif re.match(\"^IPA.*Interface$\", method.interface.mojom_name):\n> > > + if method.attributes is None:\n> > > + return False\n> > > + elif 'async' in method.attributes and method.attributes['async']:\n> > > + return True\n> > > + return False\n> > > +\n> > > +def IsArray(element):\n> > > + return mojom.IsArrayKind(element.kind)\n> > > +\n> > > +def IsControls(element):\n> > > + return mojom.IsStructKind(element.kind) and (element.kind.mojom_name == \"ControlList\" or\n> > > + element.kind.mojom_name == \"ControlInfoMap\")\n> > > +\n> > > +def IsEnum(element):\n> > > + return mojom.IsEnumKind(element.kind)\n> > > +\n> > > +def IsFd(element):\n> > > + return mojom.IsStructKind(element.kind) and element.kind.mojom_name == \"FileDescriptor\"\n> > > +\n> > > +def IsMap(element):\n> > > + return mojom.IsMapKind(element.kind)\n> > > +\n> > > +def IsPlainStruct(element):\n> > > + return mojom.IsStructKind(element.kind) and not IsControls(element) and not IsFd(element)\n> > > +\n> > > +def IsPod(element):\n> > > + return element.kind in _kind_to_cpp_type\n> > > +\n> > > +def IsStr(element):\n> > > + return element.kind.spec == 's'\n> > > +\n> > > +def BitWidth(element):\n> > > + if element.kind in _bit_widths:\n> > > + return _bit_widths[element.kind]\n> > > + if mojom.IsEnumKind(element.kind):\n> > > + return '32'\n> > > + return ''\n> > > +\n> > > +def GetNameForElement(element):\n> > > + if (mojom.IsEnumKind(element) or\n> > > + mojom.IsInterfaceKind(element) or\n> > > + mojom.IsStructKind(element)):\n> > > + return element.mojom_name\n> > > + if (mojom.IsArrayKind(element)):\n> > > + elem_name = GetNameForElement(element.kind)\n> > > + return f'std::vector<{elem_name}>'\n> > > + if (mojom.IsMapKind(element)):\n> > > + key_name = GetNameForElement(element.key_kind)\n> > > + value_name = GetNameForElement(element.value_kind)\n> > > + return f'std::map<{key_name}, {value_name}>'\n> > > + if isinstance(element, (mojom.Field, mojom.Method, mojom.Parameter)):\n> > > + if (mojom.IsArrayKind(element.kind) or mojom.IsMapKind(element.kind)):\n> > > + return GetNameForElement(element.kind)\n> > > + if (mojom.IsReferenceKind(element.kind) and element.kind.spec == 's'):\n> > > + return 'std::string'\n> > > + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> > \n> > Is the hasattr() check needed ? If element has no kind attribute, I\n> > would expect the previous two if's to raise an exception.\n> \n> I've added comments for each section, because this confuses even me...\n> \n> So the upper-level if is to catch struct fields and function parameters.\n> This if catches PODs.\n> \n> And you're right, the hasattr() check isn't needed. I think it was a\n> remnant of fiddling with it and python complaining that element has no\n> attribute kind.\n> \n> > > + return _kind_to_cpp_type[element.kind]\n> > > + return element.kind.mojom_name\n> > > + if isinstance(element, mojom.EnumValue):\n> > \n> > Extra space after comma.\n> > \n> > > + return (GetNameForElement(element.enum) + '.' +\n> > > + ConstantStyle(element.name))\n> > > + if isinstance(element, (mojom.NamedValue,\n> > > + mojom.Constant,\n> > > + mojom.EnumField)):\n> > > + return ConstantStyle(element.name)\n> > > + if (hasattr(element, '__hash__') and element in _kind_to_cpp_type):\n> > > + return _kind_to_cpp_type[element]\n> > \n> > For my own education, what type of elements does this cover ?\n> \n> This covers PODs that are members of vectors/maps.\n> \n> I've tested each case, and removed the ones that I can't remember what\n> they do.\n\nThank you, that will make the code easier to maintain.\n\n> > > + if (hasattr(element, 'kind') and element.kind in _kind_to_cpp_type):\n> > > + return _kind_to_cpp_type[element.kind]\n> > > + if (hasattr(element, 'spec')):\n> > > + if (element.spec == 's'):\n> > > + return 'std::string'\n> > > + return element.spec.split(':')[-1]\n> > > + if (mojom.IsInterfaceRequestKind(element) or\n> > > + mojom.IsAssociatedKind(element) or\n> > > + mojom.IsPendingRemoteKind(element) or\n> > > + mojom.IsPendingReceiverKind(element) or\n> > > + mojom.IsUnionKind(element)):\n> > > + raise Exception('Unsupported element: %s' % element)\n> > > + raise Exception('Unexpected element: %s' % element)\n> > > +\n> > > +def GetFullNameForElement(element, namespace_str):\n> > > + name = GetNameForElement(element)\n> > > + if namespace_str == '':\n> > > + return name\n> > > + return f'{namespace_str}::{name}'\n> > > +\n> > > +def ValidateZeroLength(l, s, cap=True):\n> > > + if l is None:\n> > > + return\n> > > + if len(l) > 0:\n> > > + raise Exception(f'{s.capitalize() if cap else s} should be empty')\n> > > +\n> > > +def ValidateSingleLength(l, s, cap=True):\n> > > + if len(l) > 1:\n> > > + raise Exception(f'Only one {s} allowed')\n> > > + if len(l) < 1:\n> > > + raise Exception(f'{s.capitalize() if cap else s} is required')\n> > > +\n> > > +def ValidateInterfaces(interfaces):\n> > > + # Validate presence of main interface\n> > > + intf = [x for x in interfaces\n> > > + if re.match(\"^IPA.*Interface\", x.mojom_name) and\n> > > + not re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> > > + ValidateSingleLength(intf, 'main interface')\n> > > + intf = intf[0]\n> > > +\n> > > + # Validate presence of callback interface\n> > \n> > s/callback/event/\n> > \n> > > + cb = [x for x in interfaces if re.match(\"^IPA.*EventInterface\", x.mojom_name)]\n> > \n> > s/cb/event/ ?\n> > \n> > > + ValidateSingleLength(cb, 'event interface')\n> > > + cb = cb[0]\n> > > +\n> > > + # Validate required main interface functions\n> > > + f_init = [x for x in intf.methods if x.mojom_name == 'init']\n> > > + f_start = [x for x in intf.methods if x.mojom_name == 'start']\n> > > + f_stop = [x for x in intf.methods if x.mojom_name == 'stop']\n> > > +\n> > > + ValidateSingleLength(f_init, 'init()', False)\n> > > + ValidateSingleLength(f_start, 'start()', False)\n> > > + ValidateSingleLength(f_stop, 'stop()', False)\n> > > +\n> > > + f_init = f_init[0]\n> > > + f_start = f_start[0]\n> > > + f_stop = f_stop[0]\n> > > +\n> > > + # Validate parameters to init()\n> > > + ValidateSingleLength(f_init.parameters, 'input parameter to init()')\n> > > + ValidateSingleLength(f_init.response_parameters, 'output parameter from init()')\n> > > + if f_init.parameters[0].kind.mojom_name != 'IPASettings':\n> > > + raise Exception('init() must have single IPASettings input parameter')\n> > > + if f_init.response_parameters[0].kind.spec != 'i32':\n> > > + raise Exception('init() must have single int32 output parameter')\n> > > +\n> > > + # Validate parameters to start()\n> > > + ValidateZeroLength(f_start.parameters, 'input parameter to start()')\n> > > + ValidateSingleLength(f_start.response_parameters, 'output parameter from start()')\n> > > + if f_start.response_parameters[0].kind.spec != 'i32':\n> > > + raise Exception('start() must have single int32 output parameter')\n> > > +\n> > > + # Validate parameters to stop()\n> > > + ValidateZeroLength(f_stop.parameters, 'input parameter to stop()')\n> > > + ValidateZeroLength(f_stop.parameters, 'output parameter from stop()')\n> > > +\n> > > + # Validate that all async methods don't have return values\n> > > + intf_methods_async = [x for x in intf.methods if IsAsync(x)]\n> > > + for method in intf_methods_async:\n> > > + ValidateZeroLength(method.response_parameters,\n> > > + f'{method.mojom_name} response parameters', False)\n> > > +\n> > > + cb_methods_async = [x for x in cb.methods if IsAsync(x)]\n> > \n> > s/cb_method_async/event_methods_async/\n> > \n> > > + for method in cb_methods_async:\n> > > + ValidateZeroLength(method.response_parameters,\n> > > + f'{method.mojom_name} response parameters', False)\n> > > +\n> > > +class Generator(generator.Generator):\n> > > +\n> > > + @staticmethod\n> > > + def GetTemplatePrefix():\n> > > + return 'libcamera_templates'\n> > > +\n> > > + def GetFilters(self):\n> > > + libcamera_filters = {\n> > > + 'all_types': GetAllTypes,\n> > > + 'bit_width': BitWidth,\n> > > + 'cap': Capitalize,\n> > \n> > 'cap': str.capitalize,\n> > \n> > and drop the custom Capitalize function.\n> \n> The built-in capitalize function doesn't preserve camel-casing so it's\n> not sufficient:\n> \n> >>> \"camelCaseClass\".capitalize()\n> 'Camelcaseclass'\n> >>> Capitalize(\"camelCaseClass\")\n> 'CamelCaseClass'\n\nOops.\n\n> > > + 'choose': Choose,\n> > > + 'comma_sep': CommaSep,\n> > > + 'default_value': GetDefaultValue,\n> > > + 'has_default_fields': HasDefaultFields,\n> > > + 'has_fd': HasFd,\n> > > + 'is_async': IsAsync,\n> > > + 'is_array': IsArray,\n> > > + 'is_controls': IsControls,\n> > > + 'is_enum': IsEnum,\n> > > + 'is_fd': IsFd,\n> > > + 'is_map': IsMap,\n> > > + 'is_plain_struct': IsPlainStruct,\n> > > + 'is_pod': IsPod,\n> > > + 'is_str': IsStr,\n> > > + 'method_input_has_fd': MethodInputHasFd,\n> > > + 'method_output_has_fd': MethodOutputHasFd,\n> > > + 'method_param_names': MethodParamNames,\n> > > + 'method_param_inputs': MethodParamInputs,\n> > > + 'method_param_outputs': MethodParamOutputs,\n> > > + 'method_parameters': MethodParameters,\n> > > + 'method_return_value': MethodReturnValue,\n> > > + 'name': GetNameForElement,\n> > > + 'name_full': GetFullNameForElement,\n> > > + 'needs_control_serializer': NeedsControlSerializer,\n> > > + 'params_comma_sep': ParamsCommaSep,\n> > > + 'with_default_values': WithDefaultValues,\n> > > + 'with_fds': WithFds,\n> > > + }\n> > > + return libcamera_filters\n> > > +\n> > > + def _GetJinjaExports(self):\n> > \n> > Would it make sense to store the return value of\n> > ModuleClassName(self.module) in a local variable instead of calling the\n> > function several times below ? If self.module is constant during the\n> > lifetime of the Generator (I think it is), you could even compute it in\n> > the constructor and store it in a member variable.\n> \n> Yeah I did think that was a good idea.\n> \n> Hm, it doesn't work in the constructor, but it works in GenerateFiles.\n> \n> > > + return {\n> > > + 'base_controls': '%s::controls' % ModuleClassName(self.module),\n> > > + 'cmd_enum_name': '_%sCmd' % ModuleClassName(self.module),\n> > > + 'cmd_event_enum_name': '_%sEventCmd' % ModuleClassName(self.module),\n> > > + 'enums': self.module.enums,\n> > > + 'has_array': len([x for x in self.module.kinds.keys() if x[0] == 'a']) > 0,\n> > > + 'has_map': len([x for x in self.module.kinds.keys() if x[0] == 'm']) > 0,\n> > > + 'has_namespace': self.module.mojom_namespace != '',\n> > > + 'imports': self.module.imports,\n> > \n> > I don't see any mention of 'imports' in the templates. Is this used\n> > internally by mojom and/or jinja, or is it unused ?\n> \n> It was probably copied from the mojom C++ generator. I think I was going\n> to use it for something... but then importing ended up working in such a\n> way where this is no longer needed.\n> \n> > > + 'interface_cb': self.module.interfaces[1],\n> > \n> > interface_event ?\n> > \n> > > + 'interface_main': self.module.interfaces[0],\n> > \n> > Is there a guarantee the interfaces will always be specified in this\n> > order ? The code in ValidateInterfaces() finds interfaces based on their\n> > name.\n> \n> Ah, there is indeed no such guarantee, just a undocumented convention.\n\nCould we then either avoid relying on this, or document the convention ?\n:-)\n\n> > > + 'interface_name': 'IPA%sInterface' % ModuleClassName(self.module),\n> > > + 'ipc_name': 'IPAIPCUnixSocket',\n> > \n> > This seems unused.\n> \n> It will be used when we support different IPC mechanisms :)\n> \n> Wait nvm, it won't. I'll remove it.\n> \n> > > + 'kinds': self.module.kinds,\n> > > + 'module': self.module,\n> > \n> > Same for those two (unless they are used elsewhere than in the\n> > templates).\n> > \n> > > + 'module_name': ModuleName(self.module.path),\n> > > + 'module_class_name': ModuleClassName(self.module),\n> > \n> > Ditto.\n> > \n> > > + 'namespace': self.module.mojom_namespace.split('.'),\n> > > + 'namespace_str': self.module.mojom_namespace.replace('.', '::') if\n> > > + self.module.mojom_namespace is not None else '',\n> > > + 'proxy_name': 'IPAProxy%s' % ModuleClassName(self.module),\n> > > + 'proxy_worker_name': 'IPAProxy%sWorker' % ModuleClassName(self.module),\n> > \n> > Same here.\n> > \n> > > + 'structs_nonempty': [x for x in self.module.structs if len(x.fields) > 0],\n> > > + 'year': datetime.datetime.now().year,\n> > > + }\n> > > +\n> > > +\n> > > + @UseJinja('module_ipa_interface.h.tmpl')\n> > > + def _GenerateDataHeader(self):\n> > > + return self._GetJinjaExports()\n> > > +\n> > > + @UseJinja('module_ipa_serializer.h.tmpl')\n> > > + def _GenerateSerializer(self):\n> > > + return self._GetJinjaExports()\n> > > +\n> > > + @UseJinja('module_ipa_proxy.cpp.tmpl')\n> > > + def _GenerateProxyCpp(self):\n> > > + return self._GetJinjaExports()\n> > > +\n> > > + @UseJinja('module_ipa_proxy.h.tmpl')\n> > > + def _GenerateProxyHeader(self):\n> > > + return self._GetJinjaExports()\n> > > +\n> > > + @UseJinja('module_ipa_proxy_worker.cpp.tmpl')\n> > > + def _GenerateProxyWorker(self):\n> > > + return self._GetJinjaExports()\n> > > +\n> > > + def GenerateFiles(self, unparsed_args):\n> > > + parser = argparse.ArgumentParser()\n> > > + parser.add_argument('--libcamera_generate_header', action='store_true')\n> > > + parser.add_argument('--libcamera_generate_serializer', action='store_true')\n> > > + parser.add_argument('--libcamera_generate_proxy_cpp', action='store_true')\n> > > + parser.add_argument('--libcamera_generate_proxy_h', action='store_true')\n> > > + parser.add_argument('--libcamera_generate_proxy_worker', action='store_true')\n> > > + parser.add_argument('--libcamera_output_path')\n> > > + args = parser.parse_args(unparsed_args)\n> > > +\n> > > + ValidateInterfaces(self.module.interfaces)\n> > > +\n> > > + fileutil.EnsureDirectoryExists(os.path.dirname(args.libcamera_output_path))\n> > > +\n> > > + module_name = ModuleName(self.module.path)\n> > \n> > This seems unused.\n> > \n> > > +\n> > > + if args.libcamera_generate_header:\n> > > + self.Write(self._GenerateDataHeader(),\n> > > + args.libcamera_output_path)\n> > > +\n> > > + if args.libcamera_generate_serializer:\n> > > + self.Write(self._GenerateSerializer(),\n> > > + args.libcamera_output_path)\n> > > +\n> > > + if args.libcamera_generate_proxy_cpp:\n> > > + self.Write(self._GenerateProxyCpp(),\n> > > + args.libcamera_output_path)\n> > > +\n> > > + if args.libcamera_generate_proxy_h:\n> > > + self.Write(self._GenerateProxyHeader(),\n> > > + args.libcamera_output_path)\n> > > +\n> > > + if args.libcamera_generate_proxy_worker:\n> > > + self.Write(self._GenerateProxyWorker(),\n> > > + args.libcamera_output_path)\n> > \n> > I'm sure we could avoid the manual unrolling of all this, using a\n> > dictionary to store the template files and iterating over it to add\n> > arguments to the parser and replacing the above code, but it's probably\n> > not worth it as I don't expect this will change a lot.","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 07B4ABE08A\n\tfor ;\n\tThu, 26 Nov 2020 16:00:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 5A0E76347E;\n\tThu, 26 Nov 2020 17:00:58 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 7C6F363469\n\tfor ;\n\tThu, 26 Nov 2020 17:00:57 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 32186A1B;\n\tThu, 26 Nov 2020 17:00:56 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ozaoOQ6s\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1606406456;\n\tbh=CCQoLTWExnDRsqCUYH4/ezeiUNxLsxRKQ3e3WQHBq5g=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=ozaoOQ6sZ6tAd4GQDfzKmv2PGl0scnOhAXesqkwCaAAOZBFNUXLiT2rt9lovDxWL1\n\t/9/qmldNwvpYQdQ3LU3tbASLW1Yq81wAZiZun8EjC8GvRiSZI8lIeZpGGoZpNnnrw3\n\t6wp4eDshXh5vZxbm6V5Ht5kgu68NTWupg5NpocBI=","Date":"Thu, 26 Nov 2020 18:00:47 +0200","From":"Laurent Pinchart ","To":"paul.elder@ideasonboard.com","Message-ID":"<20201126160047.GW3905@pendragon.ideasonboard.com>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>\n\t<20201119123917.GA4563@pendragon.ideasonboard.com>\n\t<20201123081156.GA2050@pyrite.rasen.tech>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201123081156.GA2050@pyrite.rasen.tech>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":13934,"web_url":"https://patchwork.libcamera.org/comment/13934/","msgid":"<20201126160754.GX3905@pendragon.ideasonboard.com>","date":"2020-11-26T16:07:54","subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","submitter":{"id":2,"url":"https://patchwork.libcamera.org/api/people/2/","name":"Laurent Pinchart","email":"laurent.pinchart@ideasonboard.com"},"content":"Hi Paul,\n\nOn Thu, Nov 26, 2020 at 03:49:18PM +0900, paul.elder@ideasonboard.com wrote:\n> On Thu, Nov 19, 2020 at 07:21:00PM +0200, Laurent Pinchart wrote:\n> > On Fri, Nov 06, 2020 at 07:36:34PM +0900, Paul Elder wrote:\n> > > Add templates to mojo to generate code for the IPC mechanism. These\n> > > templates generate:\n> > > - module header\n> > > - module serializer\n> > > - IPA proxy cpp, header, and worker\n> > > \n> > > Given an input data definition mojom file for a pipeline.\n> > > \n> > > Signed-off-by: Paul Elder \n> > > \n> > > ---\n> > > Changes in v4:\n> > > For the non-template files:\n> > > - rename IPA{pipeline_name}CallbackInterface to\n> > > IPA{pipeline_name}EventInterface\n> > > - to avoid the notion of \"callback\" and emphasize that it's an event\n> > > - add support for strings in custom structs\n> > > - add validation, that async methods must not have return values\n> > > - it throws exception and isn't very clear though...?\n> > > - rename controls to libcamera::{pipeline_name}::controls (controls is\n> > > now lowercase)\n> > > - rename {pipeline_name}_generated.h to {pipeline_name}_ipa_interface.h,\n> > > and {pipeline_name}_serializer.h to {pipeline_name}_ipa_serializer.h\n> > > - same for their corresponding template files\n> > > For the template files:\n> > > - fix spacing (now it's all {{var}} instead of some {{ var }})\n> > > - except if it's code, so code is still {{ code }}\n> > > - move inclusion of corresponding header to first in the inclusion list\n> > > - fix copy&paste errors\n> > > - change snake_case to camelCase in the generated code\n> > > - template code still uses snake_case\n> > > - change the generated command enums to an enum class, and make it\n> > > capitalized (instead of allcaps)\n> > > - add length checks to recvIPC (in proxy)\n> > > - fix some template spacing\n> > > - don't use const for PODs in function/signal parameters\n> > > - add the proper length checks to readPOD/appendPOD\n> > > - the helper functions for reading and writing PODs to and from\n> > > serialized data\n> > > - rename readUInt/appendUInt to readPOD/appendPOD\n> > > - add support for strings in custom structs\n> > > \n> > > Changes in v3:\n> > > - add support for namespaces\n> > > - fix enum assignment (used to have +1 for CMD applied to all enums)\n> > > - use readHeader, writeHeader, and eraseHeader as static class functions\n> > > of IPAIPCUnixSocket (in the proxy worker)\n> > > - add requirement that base controls *must* be defined in\n> > > libcamera::{pipeline_name}::Controls\n> > > \n> > > Changes in v2:\n> > > - mandate the main and callback interfaces, and init(), start(), stop()\n> > > and their parameters\n> > > - fix returning single pod value from IPC-called function\n> > > - add licenses\n> > > - improve auto-generated message\n> > > - other fixes related to serdes\n> > > ---\n> > > .../module_ipa_interface.h.tmpl | 113 ++++\n> > > .../module_ipa_proxy.cpp.tmpl | 238 +++++++++\n> > > .../module_ipa_proxy.h.tmpl | 118 +++++\n> > > .../module_ipa_proxy_worker.cpp.tmpl | 187 +++++++\n> > > .../module_ipa_serializer.h.tmpl | 44 ++\n> > > .../libcamera_templates/proxy_functions.tmpl | 205 ++++++++\n> > > .../libcamera_templates/serializer.tmpl | 280 ++++++++++\n> > > .../generators/mojom_libcamera_generator.py | 488 ++++++++++++++++++\n> > > 8 files changed, 1673 insertions(+)\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > \n> > The proxy .h and .cpp files are named ipa_proxy_{pipeline}.{h,cpp},\n> > while the other generated files are named {pipeline}_ipa_interface.h and\n> > {pipeline}_ipa_serializer.h. Could you pick one naming convention, and\n> > apply it to all files ? The .tmpl files should then follow the same\n> > convention.\n> \n> Ah, yeah. I'll make them all {pipeline}_ipa_{proxy,proxy_worker,interface,serializer}.{h,cpp}\n> \n> > > create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > > create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > > create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py\n> > \n> > And now, for the templates.\n> > \n> > > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > > new file mode 100644\n> > > index 00000000..a470b99e\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_interface.h.tmpl\n> > > @@ -0,0 +1,113 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) {{year}}, Google Inc.\n> > \n> > Coming back to the question of copyright on generated code.\n> > \n> > First of all, the template itself is a creative work, which should have\n> > a copyright notice, with a fixed date. This should go to the comment at\n> > the beginning of the file.\n> > \n> > The generated file is the result of a non-creative process that takes\n> > creative work as input. As such, it is covered by the copyright of the\n> > template, and the mojom file. Would it be difficult to extract the\n> > copyright data of the .mojom file and add it here ? I'm thnking about\n> > the following at the beginning of the template:\n> > \n> > {#-\n> > # SPDX-License-Identifier: LGPL-2.1-or-later\n> > # Copyright (C) 2020, Google Inc.\n> > -#}\n> > /* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > /*\n> > * Copyright (C) 2020, Google Inc.\n> > * {{copyright}}\n> > *\n> > * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> > *\n> > * This file is auto-generated. Do not edit.\n> > */\n> > \n> > with {{copyright}} being replaced with the copyright statement from the\n> > .mojom file. Mojo won't give you the information, but if you pass the\n> > template name to _GetJinjaExport(), it may not be too difficult to\n> > extract the copyright line.\n> > \n> > This may not be worth it, so feel free to just have a fixed copyright\n> > here:\n> > \n> > /* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > /*\n> > * Copyright (C) 2020, Google Inc.\n> > *\n> > * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> > *\n> > * This file is auto-generated. Do not edit.\n> > */\n> \n> Okay. I'll take the latter.\n> \n> > > + *\n> > > + * {{module_name}}_ipa_interface.h - Image Processing Algorithm interface for {{module_name}}\n> > > + *\n> > > + * This file is auto-generated. Do not edit.\n> > > + */\n> > > +\n> > > +#ifndef __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> > > +#define __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +\n> > > +{% if has_map %}#include {% endif %}\n> > > +{% if has_array %}#include {% endif %}\n> > > +\n> > > +namespace libcamera {\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace %}\n> > > +namespace {{ns}} {\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +\n> > > +enum class {{cmd_enum_name}} {\n> > > +\tExit = 0,\n> > > +{%- for method in interface_main.methods %}\n> > > +\t{{method.mojom_name|cap}} = {{loop.index}},\n> > > +{%- endfor %}\n> > > +};\n> > > +\n> > > +enum class {{cmd_event_enum_name}} {\n> > > +{%- for method in interface_cb.methods %}\n> > > +\t{{method.mojom_name|cap}} = {{loop.index}},\n> > > +{%- endfor %}\n> > > +};\n> > > +\n> > > +{% for enum in enums %}\n> > > +enum {{enum.mojom_name}} {\n> > > +{%- for field in enum.fields %}\n> > > +\t{{field.mojom_name}} = {{field.numeric_value}},\n> > > +{%- endfor %}\n> > > +};\n> > > +{% endfor %}\n> > > +\n> > > +{%- for struct in structs_nonempty %}\n> > > +struct {{struct.mojom_name}}\n> > > +{\n> > > +public:\n> > > +\t{{struct.mojom_name}}() {%- if struct|has_default_fields %}\n> > > +\t\t:{% endif %}\n> > > +{%- for field in struct.fields|with_default_values -%}\n> > > +{{\" \" if loop.first}}{{field.mojom_name}}_({{field|default_value}}){{\", \" if not loop.last}}\n> > > +{%- endfor %}\n> > > +\t{\n> > > +\t}\n> > > +\n> > > +\t~{{struct.mojom_name}}() {}\n> > \n> > I think you can skip empty destructors.\n> > \n> > > +\n> > > +\t{{struct.mojom_name}}(\n> > > +{%- for field in struct.fields -%}\n> > > +{{field|name}} {{field.mojom_name}}{{\", \" if not loop.last}}\n> > \n> > Should non-trivial parameters be passed by const reference instead of\n> > value ?\n> \n> They should.\n> \n> > > +{%- endfor -%}\n> > > +)\n> > > +\t\t:\n> > > +{%- for field in struct.fields -%}\n> > > +{{\" \" if loop.first}}{{field.mojom_name}}_({{field.mojom_name}}){{\", \" if not loop.last}}\n> > > +{%- endfor %}\n> > > +\t{\n> > > +\t}\n> > > +{% for field in struct.fields %}\n> > > +\t{{field|name}} {{field.mojom_name}}_;\n> > > +{%- endfor %}\n> > > +};\n> > > +{% endfor %}\n> > > +\n> > > +{#-\n> > > +Any consts or #defines should be moved to the mojom file when possible.\n> > > +If anything needs to be #included, then {{module_name}}.h needs to have the\n> > > +#include.\n> > > +#}\n> > > +class {{interface_name}} : public IPAInterface\n> > > +{\n> > > +public:\n> > > +\tvirtual ~{{interface_name}}() {}\n> > \n> > The base class has a virtual destructor already, so you can drop this\n> > one.\n> > \n> > > +{% for method in interface_main.methods %}\n> > > +\tvirtual {{method|method_return_value}} {{method.mojom_name}}(\n> > > +{%- for param in method|method_parameters %}\n> > > +\t\t{{param}}{{- \",\" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +) = 0;\n> > > +{% endfor %}\n> > > +\n> > > +{%- for method in interface_cb.methods %}\n> > > +\tSignal<\n> > > +{%- for param in method.parameters -%}\n> > > +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> > > +\t\t{{- \", \" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +> {{method.mojom_name}};\n> > > +{% endfor -%}\n> > > +};\n> > > +\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace|reverse %}\n> > > +} /* {{ns}} */\n> > \n> > This should be\n> > \n> > } /* namespace {{ns}} */\n> > \n> > Same in a few locations below.\n> > \n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_IPA_INTERFACE_{{module_name|upper}}_GENERATED_H__ */\n> > > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > > new file mode 100644\n> > > index 00000000..9328c7ca\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl\n> > > @@ -0,0 +1,238 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > > +\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) {{year}}, Google Inc.\n> > > + *\n> > > + * ipa_proxy_{{module_name}}.cpp - Image Processing Algorithm proxy for {{module_name}}\n> > > + *\n> > > + * This file is auto-generated. Do not edit.\n> > > + */\n> > > +\n> > > +#include \n> > > +\n> > > +#include \n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/control_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_ipc.h\"\n> > > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_module.h\"\n> > > +#include \"libcamera/internal/ipa_proxy.h\"\n> > > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/log.h\"\n> > > +#include \"libcamera/internal/process.h\"\n> > > +#include \"libcamera/internal/thread.h\"\n> > \n> > YOu got the alphabetical order nearly right :-)\n> > \n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DECLARE_CATEGORY(IPAProxy)\n> > > +\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace %}\n> > > +namespace {{ns}} {\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +\n> > > +{{proxy_name}}::{{proxy_name}}(IPAModule *ipam, bool isolate)\n> > > +\t: IPAProxy(ipam), running_(false),\n> > > +\t isolate_(isolate)\n> > > +{\n> > > +\tLOG(IPAProxy, Debug)\n> > > +\t\t<< \"initializing {{module_name}} proxy: loading IPA from \"\n> > > +\t\t<< ipam->path();\n> > > +\n> > > +\tif (isolate_) {\n> > > +\t\tconst std::string proxyWorkerPath = resolvePath(\"ipa_proxy_{{module_name}}\");\n> > > +\t\tif (proxyWorkerPath.empty()) {\n> > > +\t\t\tLOG(IPAProxy, Error)\n> > > +\t\t\t\t<< \"Failed to get proxy worker path\";\n> > > +\t\t\treturn;\n> > > +\t\t}\n> > > +\n> > > +\t\tipc_ = std::make_unique(ipam->path().c_str(), proxyWorkerPath.c_str());\n> > \n> > Line wrap maybe ?\n> > \n> > > +\t\tif (!ipc_->isValid()) {\n> > > +\t\t\tLOG(IPAProxy, Error) << \"Failed to create IPAIPC\";\n> > > +\t\t\treturn;\n> > > +\t\t}\n> > > +\n> > > +{% if interface_cb.methods|length > 0 %}\n> > > +\t\tipc_->recvIPC.connect(this, &{{proxy_name}}::recvIPC);\n> > > +{% endif %}\n> > \n> > I think we can assume there will always be events. An IPA module that\n> > only consumes data would be a bit useless, wouldn't it ? :-)\n> > \n> > > +\n> > > +\t\tvalid_ = true;\n> > > +\t\treturn;\n> > > +\t}\n> > > +\n> > > +\tif (!ipam->load())\n> > > +\t\treturn;\n> > > +\n> > > +\tIPAInterface *ipai = ipam->createInterface();\n> > > +\tif (!ipai) {\n> > > +\t\tLOG(IPAProxy, Error)\n> > > +\t\t\t<< \"Failed to create IPA context for \" << ipam->path();\n> > > +\t\treturn;\n> > > +\t}\n> > > +\n> > > +\tipa_ = std::unique_ptr<{{interface_name}}>(dynamic_cast<{{interface_name}} *>(ipai));\n> > \n> > Do we need a dynamic cast ?\n> \n> Yeah.\n> \n> src/libcamera/proxy/raspberrypi_ipa_proxy.cpp:78:46: error: no matching function for call to\n> ‘std::unique_ptr::unique_ptr(libcamera::IPAInterface*&)’\n> 78 | ipa_ = std::unique_ptr(ipai);\n> \n> ipai is pointer to IPAInterface, but we need unique pointer to IPA{pipeline}Interface.\n\nI mean, can't you use a static_cast ?\n\n> > > +\tproxy_.setIPA(ipa_.get());\n> > > +\n> > > +{% for method in interface_cb.methods %}\n> > > +\tipa_->{{method.mojom_name}}.connect(this, &{{proxy_name}}::{{method.mojom_name}}Thread);\n> > > +{%- endfor %}\n> > > +\n> > > +\tvalid_ = true;\n> > > +}\n> > > +\n> > > +{{proxy_name}}::~{{proxy_name}}()\n> > > +{\n> > > +\tif (isolate_)\n> > > +\t\tipc_->sendAsync(static_cast({{cmd_enum_name}}::Exit), {}, {});\n> > > +}\n> > > +\n> > > +{% if interface_cb.methods|length > 0 %}\n> > > +void {{proxy_name}}::recvIPC(std::vector &data, std::vector &fds)\n> > \n> > How about calling this receiveMessage() ?\n> \n> Yeah. Is recvMessage fine?\n\nWe usually try to avoid abbreviations when possible, but if\nreceiveMessage() is too long, exceptions are allowed.\n\n> > > +{\n> > > +\tif (data.size() < 8) {\n> > > +\t\tLOG(IPAProxy, Error)\n> > > +\t\t\t<< \"Didn't receive enough bytes to parse event\";\n> > > +\t\treturn;\n> > > +\t}\n> > > +\n> > > +\t{{cmd_event_enum_name}} cmd = static_cast<{{cmd_event_enum_name}}>((\n> > > +\t\tdata[0]) | (data[1] << 8) | (data[2] << 16) | (data[3] << 24));\n> > > +\n> > > +\t/* Need to skip another 4 bytes for the sequence number. */\n> > \n> > Is the protocol documented somewhere ?\n> \n> No...\n\nI think this should be fixed, but I also wonder if we shouldn't use the\nsame protocol as mojo. Documentation would mostly be free then :-) It's\nsomething that can be done on top, can you record it in a \\todo ?\n\n> > There's a bit of an imbalance in the API, with recvIPC() having to deal\n> > with unpacking the command and skipping the sequence number, and\n> > sendSync() and sendAsync() dealing with those internally. Could\n> > recvIPC() be called with the command and two spans instead ?\n> \n> Oh shoot, yeah that is imbalanced.\n> \n> > > +\tstd::vector::iterator vec = data.begin() + 8;\n> > > +\tsize_t dataSize = data.size() - 8;\n> > > +\tswitch (cmd) {\n> > > +{%- for method in interface_cb.methods %}\n> > > +\tcase {{cmd_event_enum_name}}::{{method.mojom_name|cap}}: {\n> > > +\t\t{{method.mojom_name}}IPC(vec, dataSize, fds);\n> > > +\t\tbreak;\n> > > +\t}\n> > > +{%- endfor %}\n> > > +\tdefault:\n> > > +\t\tLOG(IPAProxy, Error) << \"Unknown command \" << static_cast(cmd);\n> > > +\t}\n> > > +}\n> > > +{%- endif %}\n> > > +\n> > > +{% for method in interface_main.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method)}}\n> > > +{\n> > > +\tif (isolate_)\n> > > +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}IPC(\n> > > +{%- for param in method|method_param_names -%}\n> > > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +);\n> > > +\telse\n> > > +\t\t{{\"return \" if method|method_return_value != \"void\"}}{{method.mojom_name}}Thread(\n> > > +{%- for param in method|method_param_names -%}\n> > > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +);\n> > > +}\n> > > +\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> > > +{\n> > > +{%- if method.mojom_name == \"init\" %}\n> > > +\t{{proxy_funcs.init_thread_body()}}\n> > > +{%- elif method.mojom_name == \"start\" %}\n> > > +\t{{proxy_funcs.start_thread_body()}}\n> > > +{%- elif method.mojom_name == \"stop\" %}\n> > > +\t{{proxy_funcs.stop_thread_body()}}\n> > > +{%- elif not method|is_async %}\n> > > +\tipa_->{{method.mojom_name}}(\n> > > +\t{%- for param in method|method_param_names -%}\n> > > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > > +\t{%- endfor -%}\n> > > +);\n> > > +{% elif method|is_async %}\n> > > +\tproxy_.invokeMethod(&ThreadProxy::{{method.mojom_name}}, ConnectionTypeQueued,\n> > > +\t{%- for param in method|method_param_names -%}\n> > > +\t\t{{param}}{{- \", \" if not loop.last}}\n> > > +\t{%- endfor -%}\n> > > +);\n> > > +{%- endif %}\n> > > +}\n> > > +\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\")}}\n> > > +{\n> > > +{%- set has_input = true if method|method_param_inputs|length > 0 %}\n> > > +{%- set has_output = true if method|method_param_outputs|length > 0 or method|method_return_value != \"void\" %}\n> > > +{%- if has_input %}\n> > > +\tstd::vector _ipcInputBuf;\n> > > +{%- if method|method_input_has_fd %}\n> > > +\tstd::vector _ipcInputFds;\n> > > +{%- endif %}\n> > > +{%- endif %}\n> > > +{%- if has_output %}\n> > > +\tstd::vector _ipcOutputBuf;\n> > > +{%- if method|method_output_has_fd %}\n> > > +\tstd::vector _ipcOutputFds;\n> > > +{%- endif %}\n> > > +{%- endif %}\n> > > +\n> > > +{{proxy_funcs.serialize_call(method|method_param_inputs, '_ipcInputBuf', '_ipcInputFds', base_controls)}}\n> > > +\n> > > +{%- set input_buf = \"_ipcInputBuf\" if has_input else \"{}\" %}\n> > > +{%- set fds_buf = \"_ipcInputFds\" if method|method_input_has_fd else \"{}\" %}\n> > > +{%- set cmd = cmd_enum_name + \"::\" + method.mojom_name|cap %}\n> > > +{% if method|is_async %}\n> > > +\tint ret = ipc_->sendAsync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}});\n> > > +{%- else %}\n> > > +\tint ret = ipc_->sendSync(static_cast({{cmd}}), {{input_buf}}, {{fds_buf}}\n> > > +{{- \", &_ipcOutputBuf\" if has_output -}}\n> > > +{{- \", &_ipcOutputFds\" if has_output and method|method_output_has_fd -}}\n> > > +);\n> > > +{%- endif %}\n> > > +\tif (ret < 0) {\n> > > +\t\tLOG(IPAProxy, Error) << \"Failed to call {{method.mojom_name}}\";\n> > > +{%- if method|method_return_value != \"void\" %}\n> > > +\t\treturn static_cast<{{method|method_return_value}}>(ret);\n> > > +{%- else %}\n> > > +\t\treturn;\n> > > +{%- endif %}\n> > > +\t}\n> > > +{% if method|method_return_value != \"void\" %}\n> > > +\treturn IPADataSerializer<{{method.response_parameters|first|name}}>::deserialize(_ipcOutputBuf, 0);\n> > > +{% elif method|method_param_outputs|length > 0 %}\n> > > +{{proxy_funcs.deserialize_call(method|method_param_outputs, '_ipcOutputBuf', '_ipcOutputFds')}}\n> > > +{% endif -%}\n> > > +}\n> > > +\n> > > +{% endfor %}\n> > > +\n> > > +{% for method in interface_cb.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\")}}\n> > > +{\n> > > +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> > > +}\n> > > +\n> > > +void {{proxy_name}}::{{method.mojom_name}}IPC(\n> > > +\tstd::vector::iterator data,\n> > > +\tsize_t dataSize,\n> > > +\t[[maybe_unused]] std::vector &fds)\n> > > +{ \n> > > +{%- for param in method.parameters %}\n> > > +\t{{param|name}} {{param.mojom_name}};\n> > > +{%- endfor %}\n> > > +{{proxy_funcs.deserialize_call(method.parameters, 'data', 'fds', false, false, true, 'dataSize')}}\n> > > +\t{{method.mojom_name}}.emit({{method.parameters|params_comma_sep}});\n> > > +}\n> > > +{% endfor %}\n> > > +\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace|reverse %}\n> > > +} /* {{ns}} */\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +} /* namespace libcamera */\n> > > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > > new file mode 100644\n> > > index 00000000..3fb7192f\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl\n> > > @@ -0,0 +1,118 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > > +\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) {{year}}, Google Inc.\n> > > + *\n> > > + * ipa_proxy_{{module_name}}.h - Image Processing Algorithm proxy for {{module_name}}\n> > > + *\n> > > + * This file is auto-generated. Do not edit.\n> > > + */\n> > > +\n> > > +#ifndef __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> > > +#define __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/control_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_ipc.h\"\n> > > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/ipa_proxy.h\"\n> > > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/thread.h\"\n> > > +\n> > > +namespace libcamera {\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace %}\n> > > +namespace {{ns}} {\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +\n> > > +class {{proxy_name}} : public IPAProxy, public {{interface_name}}, public Object\n> > > +{\n> > > +public:\n> > > +\t{{proxy_name}}(IPAModule *ipam, bool isolate);\n> > > +\t~{{proxy_name}}();\n> > > +\n> > > +{% for method in interface_main.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"\", false, true)|indent(8, true)}};\n> > > +{% endfor %}\n> > > +\n> > > +{%- for method in interface_cb.methods %}\n> > > +\tSignal<\n> > > +{%- for param in method.parameters -%}\n> > > +\t\t{{\"const \" if not param|is_pod}}{{param|name}}{{\" &\" if not param|is_pod}}\n> > > +\t\t{{- \", \" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +> {{method.mojom_name}};\n> > > +{% endfor %}\n> > > +\n> > > +private:\n> > > +\tvoid recvIPC(std::vector &data, std::vector &fds);\n> > \n> > data and fds shouldn't be modified, let's make them const. I'd even go\n> > one step further, and make them Span and Span.\n> \n> I'll make them const for now.\n> \n> > > +\n> > > +{% for method in interface_main.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"IPC\", false)|indent(8, true)}};\n> > > +{% endfor %}\n> > > +{% for method in interface_cb.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"Thread\", false)|indent(8, true)}};\n> > > +\tvoid {{method.mojom_name}}IPC(\n> > > +\t\tstd::vector::iterator data,\n> > > +\t\tsize_t dataSize,\n> > > +\t\tstd::vector &fds);\n> > > +{% endfor %}\n> > > +\n> > > +\t/* Helper class to invoke async functions in another thread. */\n> > > +\tclass ThreadProxy : public Object\n> > > +\t{\n> > > +\tpublic:\n> > > +\t\tvoid setIPA({{interface_name}} *ipa)\n> > > +\t\t{\n> > > +\t\t\tipa_ = ipa;\n> > > +\t\t}\n> > > +\n> > > +\t\tint start()\n> > > +\t\t{\n> > > +\t\t\treturn ipa_->start();\n> > > +\t\t}\n> > > +\n> > > +\t\tvoid stop()\n> > > +\t\t{\n> > > +\t\t\tipa_->stop();\n> > > +\t\t}\n> > > +{% for method in interface_main.methods %}\n> > > +{%- if method|is_async %}\n> > > +\t\t{{proxy_funcs.func_sig(proxy_name, method, \"\", false)|indent(16)}}\n> > > +\t\t{\n> > > +\t\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> > > +\t\t}\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +\n> > > +\tprivate:\n> > > +\t\t{{interface_name}} *ipa_;\n> > > +\t};\n> > > +\n> > > +\tbool running_;\n> > > +\tThread thread_;\n> > > +\tThreadProxy proxy_;\n> > > +\tstd::unique_ptr<{{interface_name}}> ipa_;\n> > > +\n> > > +\tconst bool isolate_;\n> > > +\n> > > +\tstd::unique_ptr ipc_;\n> > > +\n> > > +\tControlSerializer controlSerializer_;\n> > \n> > I wonder if it would make sense to move some of the fields to the\n> > IPAProxy class (and possible some of the functions too). That can be\n> > done on top in a separate series.\n> \n> Yeah, probably.\n> \n> > > +};\n> > > +\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace|reverse %}\n> > > +} /* {{ns}} */\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_INTERNAL_IPA_PROXY_{{module_name|upper}}_H__ */\n> > > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > > new file mode 100644\n> > > index 00000000..dca4f99d\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl\n> > > @@ -0,0 +1,187 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{%- import \"proxy_functions.tmpl\" as proxy_funcs -%}\n> > > +\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) {{year}}, Google Inc.\n> > > + *\n> > > + * ipa_proxy_{{module_name}}_worker.cpp - Image Processing Algorithm proxy worker for {{module_name}}\n> > > + *\n> > > + * This file is auto-generated. Do not edit.\n> > > + */\n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/camera_sensor.h\"\n> > > +#include \"libcamera/internal/control_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/ipa_module.h\"\n> > > +#include \"libcamera/internal/ipa_proxy.h\"\n> > > +#include \"libcamera/internal/ipc_unixsocket.h\"\n> > > +#include \"libcamera/internal/log.h\"\n> > > +#include \"libcamera/internal/thread.h\"\n> > > +\n> > > +using namespace libcamera;\n> > > +\n> > > +LOG_DEFINE_CATEGORY({{proxy_name}}Worker)\n> > > +\n> > > +{%- if has_namespace %}\n> > > +{% for ns in namespace -%}\n> > > +using namespace {{ns}};\n> > > +{% endfor %}\n> > > +{%- endif %}\n> > > +\n> > > +struct CallData {\n> > > +\tIPCUnixSocket::Payload *response;\n> > > +\tbool done;\n> > > +};\n> > \n> > This doesn't seem to be used.\n> \n> Ah indeed it's not.\n> \n> > > +\n> > > +{{interface_name}} *ipa_;\n> > > +IPCUnixSocket socket_;\n> > > +\n> > > +ControlSerializer controlSerializer_;\n> > > +\n> > > +bool exit_ = false;\n> > \n> > I still think it would be nicer to move the data and functions to a\n> > class, with a single instance created in the main() function.\n> \n> Hm, okay.\n> \n> > > +\n> > > +void readyRead(IPCUnixSocket *socket)\n> > > +{\n> > > +\tIPCUnixSocket::Payload _message, _response;\n> > > +\tint _retRecv = socket->receive(&_message);\n> > > +\tif (_retRecv) {\n> > > +\t\tLOG({{proxy_name}}Worker, Error)\n> > > +\t\t\t<< \"Receive message failed\" << _retRecv;\n> > > +\t\treturn;\n> > > +\t}\n> > > +\n> > > +\tuint32_t _cmdUint, _seq;\n> > > +\tstd::tie(_cmdUint, _seq) = IPAIPCUnixSocket::readHeader(_message);\n> > \n> > I think there's some room to refactor the IPC-related classes. The\n> > payload and header should probably not be tied to the IPAIPCUnixSocket\n> > class for instance, as they should be independent of the underlying\n> > mechanism. I'll try to comment more on this topic when reviewing the\n> > corresponding patches.\n> > \n> > > +\tIPAIPCUnixSocket::eraseHeader(_message);\n> > > +\n> > > +\t{{cmd_enum_name}} _cmd = static_cast<{{cmd_enum_name}}>(_cmdUint);\n> > > +\n> > > +\tswitch (_cmd) {\n> > > +\tcase {{cmd_enum_name}}::Exit: {\n> > > +\t\texit_ = true;\n> > > +\t\tbreak;\n> > > +\t}\n> > > +\n> > > +{% for method in interface_main.methods %}\n> > > +\tcase {{cmd_enum_name}}::{{method.mojom_name|cap}}: {\n> > > +\t{{proxy_funcs.deserialize_call(method|method_param_inputs, '_message.data', '_message.fds', false, true)|indent(8, true)}}\n> > > +{% for param in method|method_param_outputs %}\n> > > +\t\t{{param|name}} {{param.mojom_name}};\n> > > +{% endfor %}\n> > > +{%- if method|method_return_value != \"void\" %}\n> > > +\t\t{{method|method_return_value}} _callRet = ipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}});\n> > > +{%- else %}\n> > > +\t\tipa_->{{method.mojom_name}}({{method.parameters|params_comma_sep}}\n> > > +{{- \", \" if method|method_param_outputs|params_comma_sep -}}\n> > > +{%- for param in method|method_param_outputs -%}\n> > > +&{{param.mojom_name}}{{\", \" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +);\n> > > +{%- endif %}\n> > > +{% if not method|is_async %}\n> > > +\t\tIPAIPCUnixSocket::writeHeader(_response, _cmdUint, _seq);\n> > > +{%- if method|method_return_value != \"void\" %}\n> > > +\t\tstd::vector _callRetBuf;\n> > > +\t\tstd::tie(_callRetBuf, std::ignore) =\n> > > +\t\t\tIPADataSerializer<{{method|method_return_value}}>::serialize(_callRet);\n> > > +\t\t_response.data.insert(_response.data.end(), _callRetBuf.begin(), _callRetBuf.end());\n> > > +{%- else %}\n> > > +\t{{proxy_funcs.serialize_call(method|method_param_outputs, \"_response.data\", \"_response.fds\", base_controls)|indent(8, true)}}\n> > > +{%- endif %}\n> > > +\t\tint _ret = socket_.send(_response);\n> > > +\t\tif (_ret < 0) {\n> > > +\t\t\tLOG({{proxy_name}}Worker, Error)\n> > > +\t\t\t\t<< \"Reply to {{method.mojom_name}}() failed\" << _ret;\n> > > +\t\t}\n> > > +\t\tLOG({{proxy_name}}Worker, Debug) << \"Done replying to {{method.mojom_name}}()\";\n> > > +{%- endif %}\n> > > +\t\tbreak;\n> > > +\t}\n> > > +{% endfor %}\n> > > +\tdefault:\n> > > +\t\tLOG({{proxy_name}}Worker, Error) << \"Unknown command \" << _cmdUint;\n> > > +\t}\n> > > +}\n> > > +\n> > > +{% for method in interface_cb.methods %}\n> > > +{{proxy_funcs.func_sig(proxy_name, method, \"\", false)}}\n> > > +{\n> > > +\tIPCUnixSocket::Payload _message;\n> > > +\n> > > +\tIPAIPCUnixSocket::writeHeader(_message, static_cast({{cmd_event_enum_name}}::{{method.mojom_name|cap}}), 0);\n> > > +\t{{proxy_funcs.serialize_call(method|method_param_inputs, \"_message.data\", \"_message.fds\", base_controls)}}\n> > > +\n> > > +\tsocket_.send(_message);\n> > > +\n> > > +\tLOG({{proxy_name}}Worker, Debug) << \"{{method.mojom_name}} done\";\n> > > +}\n> > > +{% endfor %}\n> > > +\n> > > +int main(int argc, char **argv)\n> > > +{\n> > > +\t/* Uncomment this for debugging. */\n> > > +#if 0\n> > > +\tstd::string logPath = \"/tmp/libcamera.worker.\" +\n> > > +\t\t\t std::to_string(getpid()) + \".log\";\n> > > +\tlogSetFile(logPath.c_str());\n> > > +#endif\n> > \n> > Likely something we'll have to handle more dynamically, but it doesn't\n> > have to be done now. A \\todo comment would be good though.\n> \n> ack\n> \n> > > +\n> > > +\tif (argc < 3) {\n> > > +\t\tLOG({{proxy_name}}Worker, Error)\n> > > +\t\t\t<< \"Tried to start worker with no args\";\n> > > +\t\treturn EXIT_FAILURE;\n> > > +\t}\n> > > +\n> > > +\tint fd = std::stoi(argv[2]);\n> > > +\tLOG({{proxy_name}}Worker, Info)\n> > > +\t\t<< \"Starting worker for IPA module \" << argv[1]\n> > > +\t\t<< \" with IPC fd = \" << fd;\n> > > +\n> > > +\tstd::unique_ptr ipam = std::make_unique(argv[1]);\n> > > +\tif (!ipam->isValid() || !ipam->load()) {\n> > > +\t\tLOG({{proxy_name}}Worker, Error)\n> > > +\t\t\t<< \"IPAModule \" << argv[1] << \" isn't valid\";\n> > > +\t\treturn EXIT_FAILURE;\n> > > +\t}\n> > > +\n> > > +\tif (socket_.bind(fd) < 0) {\n> > > +\t\tLOG({{proxy_name}}Worker, Error) << \"IPC socket binding failed\";\n> > > +\t\treturn EXIT_FAILURE;\n> > > +\t}\n> > > +\tsocket_.readyRead.connect(&readyRead);\n> > > +\n> > > +\tipa_ = dynamic_cast<{{interface_name}} *>(ipam->createInterface());\n> > > +\tif (!ipa_) {\n> > > +\t\tLOG({{proxy_name}}Worker, Error) << \"Failed to create IPA interface instance\";\n> > > +\t\treturn EXIT_FAILURE;\n> > > +\t}\n> > > +{% for method in interface_cb.methods %}\n> > > +\tipa_->{{method.mojom_name}}.connect(&{{method.mojom_name}});\n> > > +{%- endfor %}\n> > > +\n> > > +\tLOG({{proxy_name}}Worker, Debug) << \"Proxy worker successfully started\";\n> > > +\n> > > +\t/* \\todo upgrade listening loop */\n> > \n> > Upgrade to what ? :-)\n> \n> I copied this from the old IPAProxyLinuxWorker :)\n\nMaybe it's not needed anymore, if nobody remembers what it is ? :-)\n\n> > > +\tEventDispatcher *dispatcher = Thread::current()->eventDispatcher();\n> > > +\twhile (!exit_)\n> > > +\t\tdispatcher->processEvents();\n> > > +\n> > > +\tdelete ipa_;\n> > > +\tsocket_.close();\n> > > +\n> > > +\treturn 0;\n> > > +}\n> > > diff --git a/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > > new file mode 100644\n> > > index 00000000..675f9adf\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/module_ipa_serializer.h.tmpl\n> > > @@ -0,0 +1,44 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{%- import \"serializer.tmpl\" as serializer -%}\n> > > +\n> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > > +/*\n> > > + * Copyright (C) {{year}}, Google Inc.\n> > > + *\n> > > + * {{module_name}}_serializer.h - Image Processing Algorithm data serializer for {{module_name}}\n> > > + *\n> > > + * This file is auto-generated. Do not edit.\n> > > + */\n> > > +\n> > > +#ifndef __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> > > +#define __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__\n> > > +\n> > > +#include \n> > > +#include \n> > > +\n> > > +#include \"libcamera/internal/control_serializer.h\"\n> > > +#include \"libcamera/internal/ipa_data_serializer.h\"\n> > > +\n> > > +#include \n> > > +#include \n> > \n> > These should go first.\n> > \n> > > +\n> > > +namespace libcamera {\n> > > +\n> > > +LOG_DECLARE_CATEGORY(IPADataSerializer)\n> > > +{% for struct in structs_nonempty %}\n> > > +template<>\n> > > +class IPADataSerializer<{{struct|name_full(namespace_str)}}>\n> > > +{\n> > > +public:\n> > > +{{- serializer.serializer(struct, base_controls, namespace_str)}}\n> > > +{%- if struct|has_fd %}\n> > > +{{serializer.deserializer_fd(struct, namespace_str)}}\n> > > +{%- else %}\n> > > +{{serializer.deserializer_no_fd(struct, namespace_str)}}\n> > > +{%- endif %}\n> > > +};\n> > > +{% endfor %}\n> > > +\n> > > +} /* namespace libcamera */\n> > > +\n> > > +#endif /* __LIBCAMERA_INTERNAL_IPA_DATA_SERIALIZER_{{module_name|upper}}_H__ */\n> > > diff --git a/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > > new file mode 100644\n> > > index 00000000..f6836034\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/proxy_functions.tmpl\n> > > @@ -0,0 +1,205 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{#\n> > > + # \\brief Generate fuction prototype\n> > > + #\n> > > + # \\param class Class name\n> > > + # \\param method mojom Method object\n> > > + # \\param suffix Suffix to append to \\a method function name\n> > > + # \\param need_class_name True to generate class name with function\n> > > + # \\param override True to generate override tag after the function prototype\n> > > + #}\n> > > +{%- macro func_sig(class, method, suffix, need_class_name = true, override = false) -%}\n> > \n> > There's one occurrence of a call to this macro with two arguments only.\n> > Should suffix have a default of \"\" ?\n> \n> That's the default default value of macro parameters in jinja, that's why\n> I left it blank.\n\nI thought it would be something like that. Maybe making it explicit\nwould be more readable, or maybe that's just because I don't have\nexperience with jinja. Up to you.\n\n> > > +{{method|method_return_value}} {{class + \"::\" if need_class_name}}{{method.mojom_name}}{{suffix}}(\n> > > +{%- for param in method|method_parameters %}\n> > > +\t{{param}}{{- \",\" if not loop.last}}\n> > > +{%- endfor -%}\n> > > +){{\" override\" if override}}\n> > > +{%- endmacro -%}\n> > > +\n> > > +{#\n> > > + # \\brief Generate function body for IPA init() function for thread\n> > > + #}\n> > > +{%- macro init_thread_body() -%}\n> > > +\tint ret = ipa_->init(settings);\n> > > +\tif (ret)\n> > > +\t\treturn ret;\n> > > +\n> > > +\tproxy_.moveToThread(&thread_);\n> > > +\n> > > +\treturn 0;\n> > > +{%- endmacro -%}\n> > > +\n> > > +{#\n> > > + # \\brief Generate function body for IPA start() function for thread\n> > > + #}\n> > > +{%- macro start_thread_body() -%}\n> > > +\trunning_ = true;\n> > > +\tthread_.start();\n> > > +\n> > > +\treturn proxy_.invokeMethod(&ThreadProxy::start, ConnectionTypeBlocking);\n> > > +{%- endmacro -%}\n> > > +\n> > > +{#\n> > > + # \\brief Generate function body for IPA stop() function for thread\n> > > + #}\n> > > +{%- macro stop_thread_body() -%}\n> > > +\tif (!running_)\n> > > +\t\treturn;\n> > > +\n> > > +\trunning_ = false;\n> > > +\n> > > +\tproxy_.invokeMethod(&ThreadProxy::stop, ConnectionTypeBlocking);\n> > > +\n> > > +\tthread_.exit();\n> > > +\tthread_.wait();\n> > > +{%- endmacro -%}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Serialize multiple objects into data buffer and fd vector\n> > > + #\n> > > + # Generate code to serialize multiple objects, as specified in \\a params\n> > > + # (which are the parameters to some function), into \\a buf data buffer and\n> > > + # \\a fds fd vector.\n> > > + # This code is meant to be used by the proxy, for serializing prior to IPC calls.\n> > > + #}\n> > > +{%- macro serialize_call(params, buf, fds, base_controls) %}\n> > > +{%- for param in params %}\n> > > +\tstd::vector {{param.mojom_name}}Buf;\n> > > +{%- if param|has_fd %}\n> > > +\tstd::vector {{param.mojom_name}}Fds;\n> > > +\tstd::tie({{param.mojom_name}}Buf, {{param.mojom_name}}Fds) =\n> > > +{%- else %}\n> > > +\tstd::tie({{param.mojom_name}}Buf, std::ignore) =\n> > > +{%- endif %}\n> > > +{%- if param|is_controls %}\n> > > +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}, {{param.mojom_name}}.infoMap() ? *{{param.mojom_name}}.infoMap() : {{base_controls}}\n> > > +{%- else %}\n> > > +\t\tIPADataSerializer<{{param|name}}>::serialize({{param.mojom_name}}\n> > > +{%- endif %}\n> > > +{{- \", &controlSerializer_\" if param|needs_control_serializer -}}\n> > > +);\n> > > +{%- endfor %}\n> > > +\n> > > +{%- if params|length > 1 %}\n> > > +{%- for param in params %}\n> > > +\tappendPOD({{buf}}, {{param.mojom_name}}Buf.size());\n> > > +{%- if param|has_fd %}\n> > > +\tappendPOD({{buf}}, {{param.mojom_name}}Fds.size());\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +{%- endif %}\n> > > +\n> > > +{%- for param in params %}\n> > > +\t{{buf}}.insert({{buf}}.end(), {{param.mojom_name}}Buf.begin(), {{param.mojom_name}}Buf.end());\n> > > +{%- endfor %}\n> > > +\n> > > +{%- for param in params %}\n> > > +{%- if param|has_fd %}\n> > > +\t{{fds}}.insert({{fds}}.end(), {{param.mojom_name}}Fds.begin(), {{param.mojom_name}}Fds.end());\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +{%- endmacro -%}\n> > \n> > I wonder if we could optimize this by avoiding the intermediate vectors,\n> > but that's a candidate for a later optimization.\n> \n> I'll add it to the todo list.\n> \n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Deserialize a single object from data buffer and fd vector\n> > > + #\n> > > + # \\param pointer True deserializes the object into a dereferenced pointer\n> > \n> > s/True deserializes/If true, deserialize/ ?\n> > \n> > Same in other locations below.\n> \n> Yes.\n> \n> > > + # \\param iter True treats \\a buf as an iterator instead of a vector\n> > > + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> > > + #\n> > > + # Generate code to deserialize a single object, as specified in \\a param,\n> > > + # from \\a buf data buffer and \\a fds fd vector.\n> > > + # This code is meant to be used by macro deserialize_call.\n> > > + #}\n> > > +{%- macro deserialize_param(param, pointer, loop, buf, fds, iter, data_size) -%}\n> > > +{{\"*\" if pointer}}{{param.mojom_name}} = IPADataSerializer<{{param|name}}>::deserialize(\n> > > +{%- if not iter %}\n> > > +\t{{buf}}.begin() + {{param.mojom_name}}Start,\n> > > +{%- else %}\n> > > +\t{{buf}} + {{param.mojom_name}}Start,\n> > > +{%- endif %}\n> > \n> > Maybe\n> > \n> > \t{{buf}}{{- \".begin()\" if not iter}} + {{param.mojom_name}}Start,\n> > \n> > ?\n> \n> Yeah that's better.\n> \n> > > +{%- if loop.last and not iter %}\n> > > +\t{{buf}}.end()\n> > > +{%- elif not iter %}\n> > > +\t{{buf}}.begin() + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> > > +{%- elif iter and loop.length == 1 %}\n> > > +\t{{buf}} + {{data_size}}\n> > > +{%- else %}\n> > > +\t{{buf}} + {{param.mojom_name}}Start + {{param.mojom_name}}BufSize\n> > > +{%- endif -%}\n> > \n> > I wonder if it would simplify the generated code if we standardized on\n> > Span. Passing both the vector iterator and the size in separate\n> > arguments can more easily result in them becoming out of sync. I was\n> > thinking the Span<>::subspan() function could be quite helpful here, but\n> > it doesn't handle overflows (requesting a subspan with offset > size is\n> > ill-defined) :-S\n> > \n> > Not necessarily something to be addressed now, but do you think it would\n> > make sense ? And of course if it helps fixing some of the issues\n> > outlined elsewhere in this review, we could already switch to span.\n> \n> It probably would make sense... but it'll be a pain to convert :/\n\nLet's record it in a \\todo then.\n\n> > > +{{- \",\" if param|has_fd}}\n> > > +{%- if param|has_fd %}\n> > > +\t{{fds}}.begin() + {{param.mojom_name}}FdStart,\n> > > +{%- if loop.last %}\n> > > +\t{{fds}}.end()\n> > > +{%- else %}\n> > > +\t{{fds}}.begin() + {{param.mojom_name}}FdStart + {{param.mojom_name}}FdsSize\n> > > +{%- endif -%}\n> > > +{%- endif -%}\n> > > +{{- \",\" if param|needs_control_serializer}}\n> > > +{%- if param|needs_control_serializer %}\n> > > +\t&controlSerializer_\n> > > +{%- endif -%}\n> > > +);\n> > > +{%- endmacro -%}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Deserialize multiple objects from data buffer and fd vector\n> > > + #\n> > > + # \\param pointer True deserializes objects into pointers, and adds a null check.\n> > > + # \\param declare True declares the objects in addition to deserialization.\n> > > + # \\param iter True treats \\a buf as an iterator instead of a vector\n> > > + # \\param data_size Variable that holds the size of the vector referenced by \\a buf\n> > > + #\n> > > + # Generate code to deserialize multiple objects, as specified in \\a params\n> > > + # (which are the parameters to some function), from \\a buf data buffer and\n> > > + # \\a fds fd vector.\n> > > + # This code is meant to be used by the proxy, for deserializing after IPC calls.\n> > > + #}\n> > > +{%- macro deserialize_call(params, buf, fds, pointer = true, declare = false, iter = false, data_size = '') -%}\n> > > +{% set ns = namespace(size_offset = 0) %}\n> > > +{%- if params|length > 1 %}\n> > > +{%- for param in params %}\n> > > +\t[[maybe_unused]] size_t {{param.mojom_name}}BufSize = readPOD({{buf}}, {{ns.size_offset}}\n> > \n> > const ?\n> > \n> > > +{%- if iter -%}\n> > > +, {{buf}} + {{data_size}}\n> > > +{%- endif -%}\n> > > +);\n> > > +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> > > +{%- if param|has_fd %}\n> > > +\t[[maybe_unused]] size_t {{param.mojom_name}}FdsSize = readPOD({{buf}}, {{ns.size_offset}}\n> > \n> > const ?\n> > \n> > > +{%- if iter -%}\n> > > +, {{buf}} + {{data_size}}\n> > > +{%- endif -%}\n> > > +);\n> > > +\t{%- set ns.size_offset = ns.size_offset + 4 %}\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +{%- endif %}\n> > > +{% for param in params %}\n> > > +{%- if loop.first %}\n> > > +\tsize_t {{param.mojom_name}}Start = {{ns.size_offset}};\n> > \n> > const ?\n> > \n> > Same in a few locations below.\n> \n> Yes.\n> \n> > > +{%- else %}\n> > > +\tsize_t {{param.mojom_name}}Start = {{loop.previtem.mojom_name}}Start + {{loop.previtem.mojom_name}}BufSize;\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +{% for param in params|with_fds %}\n> > > +{%- if loop.first %}\n> > > +\tsize_t {{param.mojom_name}}FdStart = 0;\n> > > +{%- elif not loop.last %}\n> > > +\tsize_t {{param.mojom_name}}FdStart = {{loop.previtem.mojom_name}}FdStart + {{loop.previtem.mojom_name}}FdsSize;\n> > > +{%- endif %}\n> > > +{%- endfor %}\n> > > +{% for param in params %}\n> > > +\t{%- if pointer %}\n> > > +\tif ({{param.mojom_name}}) {\n> > > +{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(16, True)}}\n> > > +\t}\n> > > +\t{%- else %}\n> > > +\t{{param|name + \" \" if declare}}{{deserialize_param(param, pointer, loop, buf, fds, iter, data_size)|indent(8)}}\n> > > +\t{%- endif %}\n> > > +{% endfor %}\n> > > +{%- endmacro -%}\n> > > diff --git a/utils/ipc/generators/libcamera_templates/serializer.tmpl b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > > new file mode 100644\n> > > index 00000000..51dbeb0e\n> > > --- /dev/null\n> > > +++ b/utils/ipc/generators/libcamera_templates/serializer.tmpl\n> > > @@ -0,0 +1,280 @@\n> > > +{#- SPDX-License-Identifier: LGPL-2.1-or-later -#}\n> > > +{# Turn this into a C macro? #}\n> > > +{#\n> > > + # \\brief Verify that there is enough bytes to deserialize\n> > > + #\n> > > + # Generate code that verifies that \\a size is not greater than \\a dataSize.\n> > > + # Otherwise log an error with \\a name and \\a typename.\n> > > + #}\n> > > +{%- macro check_data_size(size, dataSize, name, typename) %}\n> > > +\t\tif ({{size}} > {{dataSize}}) {\n> > \n> > \n> > Maybe\n> > \t\tif ({{dataSize}} < {{size}}) {\n> > \n> > ?\n> > \n> > > +\t\t\tLOG(IPADataSerializer, Error)\n> > > +\t\t\t\t<< \"Failed to deserialize {{name}}: not enough {{typename}}, expected \"\n> > \n> > \t\t\t\t<< \"Failed to deserialize \" << \"{{name}}\"\n> > \t\t\t\t<< \": not enough {{typename}}, expected \"\n> > \n> > This should allow the compiler to deduplicate strings.\n> > \n> > > +\t\t\t\t<< ({{size}}) << \", got \" << ({{dataSize}});\n> > > +\t\t\treturn ret;\n> > > +\t\t}\n> > > +{%- endmacro %}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Serialize some field into return vector\n> > > + #\n> > > + # Generate code to serialize \\a field into retData, including size of the\n> > > + # field and fds (where appropriate). \\a base_controls indicates the\n> > > + # default ControlInfoMap in the event that the ControlList does not have one.\n> > > + # This code is meant to be used by the IPADataSerializer specialization.\n> > > + #}\n> > > +{%- macro serializer_field(field, base_controls, namespace, loop) %}\n> > > +{%- if field|is_pod or field|is_enum %}\n> > > +\t\tstd::vector {{field.mojom_name}};\n> > > +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > > +\t{%- if field|is_pod %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > > +\t{%- elif field|is_enum %}\n> > > +\t\t\tIPADataSerializer::serialize(data.{{field.mojom_name}}_);\n> > > +\t{%- endif %}\n> > > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > > +{%- elif field|is_fd %}\n> > > +\t\tstd::vector {{field.mojom_name}};\n> > > +\t\tstd::vector {{field.mojom_name}}Fds;\n> > > +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > > +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> > > +{%- elif field|is_controls %}\n> > > +\t\tif (data.{{field.mojom_name}}_.size() > 0) {\n> > > +\t\t\tstd::vector {{field.mojom_name}};\n> > > +\t\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > > +\t\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_,\n> > > +\t\t\t\t\tdata.{{field.mojom_name}}_.infoMap() ? *data.{{field.mojom_name}}_.infoMap() : {{base_controls}},\n> > > +\t\t\t\t\tcs);\n> > > +\t\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> > > +\t\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > > +\t\t} else {\n> > > +\t\t\tappendPOD(retData, 0);\n> > > +\t\t}\n> > > +{%- elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> > > +\t\tstd::vector {{field.mojom_name}};\n> > > +\t{%- if field|has_fd %}\n> > > +\t\tstd::vector {{field.mojom_name}}Fds;\n> > > +\t\tstd::tie({{field.mojom_name}}, {{field.mojom_name}}Fds) =\n> > > +\t{%- else %}\n> > > +\t\tstd::tie({{field.mojom_name}}, std::ignore) =\n> > > +\t{%- endif %}\n> > > +\t{%- if field|is_array or field|is_map %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_, cs);\n> > > +\t{%- elif field|is_str %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::serialize(data.{{field.mojom_name}}_);\n> > > +\t{%- else %}\n> > > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::serialize(data.{{field.mojom_name}}_, cs);\n> > > +\t{%- endif %}\n> > > +\t\tappendPOD(retData, {{field.mojom_name}}.size());\n> > > +\t{%- if field|has_fd %}\n> > > +\t\tappendPOD(retData, {{field.mojom_name}}Fds.size());\n> > > +\t{%- endif %}\n> > > +\t\tretData.insert(retData.end(), {{field.mojom_name}}.begin(), {{field.mojom_name}}.end());\n> > > +\t{%- if field|has_fd %}\n> > > +\t\tretFds.insert(retFds.end(), {{field.mojom_name}}Fds.begin(), {{field.mojom_name}}Fds.end());\n> > > +\t{%- endif %}\n> > > +{%- else %}\n> > > +\t\t/* Unknown serialization for {{field.mojom_name}}. */\n> > > +{%- endif %}\n> > > +{%- endmacro %}\n> > \n> > Same here, I think there's room for optimization if we could avoid the\n> > intermediate vectors.\n> > \n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Deserialize some field into return struct\n> > > + #\n> > > + # Generate code to deserialize \\a field into object ret.\n> > > + # This code is meant to be used by the IPADataSerializer specialization.\n> > > + #}\n> > > +{%- macro deserializer_field(field, namespace, loop) %}\n> > > +{% if field|is_pod or field|is_enum %}\n> > > +\t{%- set field_size = (field|bit_width|int / 8)|int %}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > > +\t\t{%- if field|is_pod %}\n> > > +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer<{{field|name}}>::deserialize(m, m + {{field_size}}));\n> > \n> > Do we need the static_cast<>, doesn't\n> > IPADataSerializer<{{field|name}}>::deserialize() return a {{field|name}}\n> > ?\n> \n> Ah yes, this one doesn't need the static_cast. The next one does due to\n> enums though.\n> \n> > > +\t\t{%- else %}\n> > > +\t\tret.{{field.mojom_name}}_ = static_cast<{{field|name}}>(IPADataSerializer::deserialize(m, m + {{field_size}}));\n> > > +\t\t{%- endif %}\n> > > +\t{%- if not loop.last %}\n> > > +\t\tm += {{field_size}};\n> > > +\t\tdataSize -= {{field_size}};\n> > > +\t{%- endif %}\n> > > +{% elif field|is_fd %}\n> > > +\t{%- set field_size = 1 %}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > > +\t\tret.{{field.mojom_name}}_ = IPADataSerializer<{{field|name}}>::deserialize(m, m + 1, n, n + 1, cs);\n> > > +\t{%- if not loop.last %}\n> > > +\t\tm += {{field_size}};\n> > > +\t\tdataSize -= {{field_size}};\n> > > +\t\tn += ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> > > +\t\tfdsSize -= ret.{{field.mojom_name}}_.isValid() ? 1 : 0;\n> > > +\t{%- endif %}\n> > > +{% elif field|is_controls %}\n> > > +\t{%- set field_size = 4 %}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> > > +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> > \n> > const ?\n> > \n> > You call IPADataSerializer<{{field|name}}>::deserialize() above for POD\n> > types, and readPOD() here. Do we need the IPADataSerializer<>\n> > specializations for all the PODs, can't we call readPOD() directly ?\n> \n> No we can't get rid of the POD specializations of IPADataSerializer,\n> because the IPADataSerializer for vectors and maps needs them. So for\n> serdesing fields that are PODs we use IPADataSerializer, and for\n> serdesing metadata like field size, we use readPOD.\n> \n> > > +\t{%- set field_size = '4 + ' + field.mojom_name + 'Size' -%}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > > +\t\tif ({{field.mojom_name}}Size > 0)\n> > > +\t\t\tret.{{field.mojom_name}}_ =\n> > > +\t\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > > +\t{%- if not loop.last %}\n> > > +\t\tm += {{field_size}};\n> > > +\t\tdataSize -= {{field_size}};\n> > > +\t{%- endif %}\n> > > +{% elif field|is_plain_struct or field|is_array or field|is_map or field|is_str %}\n> > > +\t{%- set field_size = 4 %}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'Size', 'data')}}\n> > > +\t\tsize_t {{field.mojom_name}}Size = readPOD(m, 0, data.end());\n> > \n> > const ?\n> > \n> > > +\t{%- if field|has_fd %}\n> > > +\t{%- set field_size = 8 %}\n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name + 'FdsSize', 'data')}}\n> > > +\t\tsize_t {{field.mojom_name}}FdsSize = readPOD(m, 4, data.end());\n> > \n> > const ?\n> > \n> > > +\t\t{{- check_data_size(field.mojom_name + 'FdsSize', 'fdsSize', field.mojom_name, 'fds')}}\n> > > +\t{%- endif %}\n> > > +\t{%- set field_size = field|has_fd|choose('8 + ', '4 + ') + field.mojom_name + 'Size' -%}\n> > \n> > I'm afraid you've got quite a few integer overflow issues here :-S\n> \n> :S\n> \n> > > +\t\t{{- check_data_size(field_size, 'dataSize', field.mojom_name, 'data')}}\n> > > +\t\tret.{{field.mojom_name}}_ =\n> > > +\t{%- if field|is_str %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size);\n> > > +\t{%- elif field|has_fd and (field|is_array or field|is_map) %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> > > +\t{%- elif field|has_fd and (not (field|is_array or field|is_map)) %}\n> > > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 8, m + 8 + {{field.mojom_name}}Size, n, n + {{field.mojom_name}}FdsSize, cs);\n> > > +\t{%- elif (not field|has_fd) and (field|is_array or field|is_map) %}\n> > > +\t\t\tIPADataSerializer<{{field|name}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > > +\t{%- else %}\n> > > +\t\t\tIPADataSerializer<{{field|name_full(namespace)}}>::deserialize(m + 4, m + 4 + {{field.mojom_name}}Size, cs);\n> > > +\t{%- endif %}\n> > \n> > Can't we advance m after reading the size fields ? That would avoid all\n> > the error-prone calculation, and get rid of at least some of the integer\n> > overflow issues.\n> \n> Oh, right.\n> \n> > > +\t{%- if not loop.last %}\n> > > +\t\tm += {{field_size}};\n> > > +\t\tdataSize -= {{field_size}};\n> > > +\t{%- if field|has_fd %}\n> > > +\t\tn += {{field.mojom_name}}FdsSize;\n> > > +\t\tfdsSize -= {{field.mojom_name}}FdsSize;\n> > > +\t{%- endif %}\n> > > +\t{%- endif %}\n> > > +{% else %}\n> > > +\t\t/* Unknown deserialization for {{field.mojom_name}}. */\n> > > +{%- endif %}\n> > > +{%- endmacro %}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Serialize a struct\n> > > + #\n> > > + # Generate code for IPADataSerializer specialization, for serializing\n> > > + # \\a struct. \\a base_controls indicates the default ControlInfoMap\n> > > + # in the event that the ControlList does not have one.\n> > > + #}\n> > > +{%- macro serializer(struct, base_controls, namespace) %}\n> > > +\tstatic std::tuple, std::vector>\n> > > +\tserialize(const {{struct|name_full(namespace)}} &data,\n> > > +{%- if struct|needs_control_serializer %}\n> > > +\t\t ControlSerializer *cs)\n> > > +{%- else %}\n> > > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > > +{%- endif %}\n> > > +\t{\n> > > +\t\tstd::vector retData;\n> > > +{%- if struct|has_fd %}\n> > > +\t\tstd::vector retFds;\n> > > +{%- endif %}\n> > > +{%- for field in struct.fields %}\n> > > +{{serializer_field(field, base_controls, namespace, loop)}}\n> > > +{%- endfor %}\n> > > +{% if struct|has_fd %}\n> > > +\t\treturn {retData, retFds};\n> > > +{%- else %}\n> > > +\t\treturn {retData, {}};\n> > > +{%- endif %}\n> > > +\t}\n> > > +{%- endmacro %}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Deserialize a struct that has fds\n> > > + #\n> > > + # Generate code for IPADataSerializer specialization, for deserializing\n> > > + # \\a struct, in the case that \\a struct has file descriptors.\n> > > + #}\n> > > +{%- macro deserializer_fd(struct, namespace) %}\n> > > +\tstatic {{struct|name_full(namespace)}}\n> > > +\tdeserialize(std::vector &data,\n> > > +\t\t std::vector &fds,\n> > > +{%- if struct|needs_control_serializer %}\n> > > +\t\t ControlSerializer *cs)\n> > > +{%- else %}\n> > > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > > +{%- endif %}\n> > > +\t{\n> > > +\t\t{{struct|name_full(namespace)}} ret;\n> > > +\t\tstd::vector::iterator m = data.begin();\n> > > +\t\tstd::vector::iterator n = fds.begin();\n> > > +\n> > > +\t\tsize_t dataSize = data.size();\n> > > +\t\tsize_t fdsSize = fds.size();\n> > > +{%- for field in struct.fields -%}\n> > > +{{deserializer_field(field, namespace, loop)}}\n> > > +{%- endfor %}\n> > > +\t\treturn ret;\n> > > +\t}\n> > > +\n> > > +\tstatic {{struct|name_full(namespace)}}\n> > > +\tdeserialize(std::vector::iterator dataBegin,\n> > > +\t\t std::vector::iterator dataEnd,\n> > > +\t\t std::vector::iterator fdsBegin,\n> > > +\t\t std::vector::iterator fdsEnd,\n> > \n> > All these should const const_iterator. There are quite a few locations\n> > in the deserializer where the data and fds vectors or iterators are not\n> > const. Could you fix that ?\n> \n> Yeah, I'll fix all of that.\n> \n> > > +{%- if struct|needs_control_serializer %}\n> > > +\t\t ControlSerializer *cs)\n> > > +{%- else %}\n> > > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > > +{%- endif %}\n> > > +\t{\n> > > +\t\tstd::vector data(dataBegin, dataEnd);\n> > \n> > This creates a copy of the data, do we really need that ? I think you\n> > should turn this around, implement the deserialization in this function,\n> > and implement the previous function as a wrapper around this one.\n> \n> Ah, okay.\n> \n> > > +\t\tstd::vector fds(fdsBegin, fdsEnd);\n> > > +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, fds, cs);\n> > > +\t}\n> > > +{%- endmacro %}\n> > > +\n> > > +\n> > > +{#\n> > > + # \\brief Deserialize a struct that has no fds\n> > > + #\n> > > + # Generate code for IPADataSerializer specialization, for deserializing\n> > > + # \\a struct, in the case that \\a struct does not have file descriptors.\n> > > + #}\n> > > +{%- macro deserializer_no_fd(struct, namespace) %}\n> > > +\tstatic {{struct|name_full(namespace)}}\n> > > +\tdeserialize(std::vector &data,\n> > > +{%- if struct|needs_control_serializer %}\n> > > +\t\t ControlSerializer *cs)\n> > > +{%- else %}\n> > > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > > +{%- endif %}\n> > > +\t{\n> > > +\t\t{{struct|name_full(namespace)}} ret;\n> > > +\t\tstd::vector::iterator m = data.begin();\n> > > +\n> > > +\t\tsize_t dataSize = data.size();\n> > > +{%- for field in struct.fields -%}\n> > > +{{deserializer_field(field, namespace, loop)}}\n> > > +{%- endfor %}\n> > > +\t\treturn ret;\n> > > +\t}\n> > > +\n> > > +\tstatic {{struct|name_full(namespace)}}\n> > > +\tdeserialize(std::vector::iterator dataBegin,\n> > > +\t\t std::vector::iterator dataEnd,\n> > > +{%- if struct|needs_control_serializer %}\n> > > +\t\t ControlSerializer *cs)\n> > > +{%- else %}\n> > > +\t\t [[maybe_unused]] ControlSerializer *cs = nullptr)\n> > > +{%- endif %}\n> > > +\t{\n> > > +\t\tstd::vector data(dataBegin, dataEnd);\n> > > +\t\treturn IPADataSerializer<{{struct|name_full(namespace)}}>::deserialize(data, cs);\n> > > +\t}\n> > > +{%- endmacro %}","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7631ABE176\n\tfor ;\n\tThu, 26 Nov 2020 16:08:05 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id DECCD63481;\n\tThu, 26 Nov 2020 17:08:04 +0100 (CET)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id AE2976346B\n\tfor ;\n\tThu, 26 Nov 2020 17:08:03 +0100 (CET)","from pendragon.ideasonboard.com (62-78-145-57.bb.dnainternet.fi\n\t[62.78.145.57])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id ED469A1B;\n\tThu, 26 Nov 2020 17:08:02 +0100 (CET)"],"Authentication-Results":"lancelot.ideasonboard.com;\n\tdkim=fail reason=\"signature verification failed\" (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"Z9uGK6ac\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1606406883;\n\tbh=ZIDns0qYgyBTuVccFUzMawZN5Wp4kCVGSbwNKbY8p3k=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=Z9uGK6acGX6yL73gVAT4x0Sntq8B0hDiTzx3duWW4vngDKsllXOPaAAuUesntkShH\n\tiq04zQUarF2YXwFi5TBKuYm4j7sHBTHLbZK79BjvjrftgiBQIWVE3PjRHpE3eCPfie\n\tuOGa+1ttRcH+ZsY1Vu7eCOCVoV97x2ofmoSSYeoo=","Date":"Thu, 26 Nov 2020 18:07:54 +0200","From":"Laurent Pinchart ","To":"paul.elder@ideasonboard.com","Message-ID":"<20201126160754.GX3905@pendragon.ideasonboard.com>","References":"<20201106103707.49660-1-paul.elder@ideasonboard.com>\n\t<20201106103707.49660-5-paul.elder@ideasonboard.com>\n\t<20201119172100.GC4563@pendragon.ideasonboard.com>\n\t<20201126064918.GB2050@pyrite.rasen.tech>","MIME-Version":"1.0","Content-Disposition":"inline","In-Reply-To":"<20201126064918.GB2050@pyrite.rasen.tech>","Subject":"Re: [libcamera-devel] [PATCH v4 04/37] utils: ipc: add templates\n\tfor code generation for IPC mechanism","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"libcamera-devel@lists.libcamera.org","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"base64","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]