[{"id":35150,"web_url":"https://patchwork.libcamera.org/comment/35150/","msgid":"<2lweaavht65vexto5246wcvmxjoxhwgtprz5hjvbrz7y4h4hrk@r3rwki5uoefi>","date":"2025-07-25T16:08:24","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Barnabás\n\nOn Mon, Jul 21, 2025 at 12:46:07PM +0200, Barnabás Pőcze wrote:\n> Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n> reported by a camera for a given request. Previously, a `ControlList` was\n> used for this purpose. The reason for introducing a separate type is to\n> simplify the access to the returned metadata during the entire lifetime\n> of a request.\n>\n> Specifically, for early metadata completion to be easily usable it should be\n> guaranteed that any completed metadata item can be accessed and looked up\n> at least until the associated requested is reused with `Request::reuse()`.\n>\n> However, when a metadata item is completed early, the pipeline handler\n> might still work on the request in the `CameraManager`'s private thread,\n> therefore there is an inherent synchronization issue when an application\n> accesses early metadata.\n>\n> Restricting the user to only access the metadata items of a not yet completed\n> request in the early metadata availability signal handler by ways of\n> documenting or enforcing it at runtime could be an option, but it is not\n> too convenient for the user.\n>\n> The current `ControlList` implementation employs an `std::unordered_map`,\n> so pointers remain stable when the container is modified, so an application\n> could keep accessing particular metadata items outside the signal handler,\n> but this fact is far from obvious, and the user would still not be able\n> to make a copy of all metadata or do lookups based on the numeric ids or\n> the usual `libcamera::Control<>` objects, thus some type safety is lost.\n>\n> The above also requires that each metadata item is only completed once for\n> a given request, but this does not appear to be serious limitation,\n> and in fact, this restriction is enforced by `MetadataList`.\n>\n> The introduced `MetadataList` supports single writer - multiple reader\n> scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n> without introducing data races or other synchronization issues. This is\n> achieved by requiring the possible set of metadata items to be known\n> (such set is stored in a `MetadataListPlan` object). Based on the this\n> plan, a single contiguous allocation is made to accommodate all potential\n> metadata items. Due to this single contiguous allocation that is not modified\n> during the lifetime of a `MetadataList` and atomic modifications, it is\n> possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n> there is only ever a single writer.\n>\n> Signed-off-by: Barnabás Pőcze \n> ---\n> changes in v2:\n> * remove multiple not strictly necessary functions\n> ---\n> include/libcamera/meson.build | 2 +\n> include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n> include/libcamera/metadata_list_plan.h | 130 ++++++\n> src/libcamera/meson.build | 1 +\n> src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n> test/controls/meson.build | 1 +\n> test/controls/metadata_list.cpp | 170 ++++++++\n> 7 files changed, 1195 insertions(+)\n> create mode 100644 include/libcamera/metadata_list.h\n> create mode 100644 include/libcamera/metadata_list_plan.h\n> create mode 100644 src/libcamera/metadata_list.cpp\n> create mode 100644 test/controls/metadata_list.cpp\n>\n> diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n> index 30ea76f94..410b548dd 100644\n> --- a/include/libcamera/meson.build\n> +++ b/include/libcamera/meson.build\n> @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n> 'framebuffer_allocator.h',\n> 'geometry.h',\n> 'logging.h',\n> + 'metadata_list.h',\n> + 'metadata_list_plan.h',\n> 'orientation.h',\n> 'pixel_format.h',\n> 'request.h',\n> diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n> new file mode 100644\n> index 000000000..7fe3dbbab\n> --- /dev/null\n> +++ b/include/libcamera/metadata_list.h\n> @@ -0,0 +1,547 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + *\n> + * Metadata list\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +\n> +// TODO: want this?\n> +#if __has_include()\n> +#if __SANITIZE_ADDRESS__ /* gcc */\n> +#include \n> +#define HAS_ASAN 1\n> +#elif defined(__has_feature)\n> +#if __has_feature(address_sanitizer) /* clang */\n> +#include \n> +#define HAS_ASAN 1\n> +#endif\n> +#endif\n> +#endif\n> +\n> +namespace libcamera {\n> +\n> +class MetadataList\n> +{\n> +private:\n> +\tstruct ValueParams {\n> +\t\tControlType type;\n> +\t\tbool isArray;\n> +\t\tstd::uint32_t numElements;\n> +\t};\n> +\n> +\tstruct Entry {\n> +\t\tconst std::uint32_t tag;\n> +\t\tconst std::uint32_t capacity;\n> +\t\tconst std::uint32_t alignment;\n> +\t\tconst ControlType type;\n> +\t\tbool isArray;\n> +\n> +\t\tstatic constexpr std::uint32_t invalidOffset = -1;\n> +\t\t/*\n> +\t\t * Offset from the beginning of the allocation, and\n> +\t\t * and _not_ relative to `contentOffset_`.\n> +\t\t */\n> +\t\tstd::atomic_uint32_t headerOffset = invalidOffset;\n> +\n> +\t\t[[nodiscard]] std::optional hasValue() const\n> +\t\t{\n> +\t\t\tauto offset = headerOffset.load(std::memory_order_relaxed);\n> +\t\t\tif (offset == invalidOffset)\n> +\t\t\t\treturn {};\n> +\n> +\t\t\treturn offset;\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] std::optional acquireData() const\n> +\t\t{\n> +\t\t\tauto offset = hasValue();\n> +\t\t\tif (offset) {\n> +\t\t\t\t/* sync with release-store on `headerOffset` in `MetadataList::set()` */\n> +\t\t\t\tstd::atomic_thread_fence(std::memory_order_acquire);\n> +\t\t\t}\n> +\n> +\t\t\treturn offset;\n> +\t\t}\n> +\t};\n> +\n> +\tstruct ValueHeader {\n> +\t\tstd::uint32_t tag;\n> +\t\tstd::uint32_t size;\n> +\t\tstd::uint32_t alignment;\n> +\t\tValueParams params;\n> +\t};\n> +\n> +\tstruct State {\n> +\t\tstd::uint32_t count;\n> +\t\tstd::uint32_t fill;\n> +\t};\n> +\n> +public:\n> +\texplicit MetadataList(const MetadataListPlan &plan)\n> +\t\t: capacity_(plan.size()),\n> +\t\t contentOffset_(MetadataList::contentOffset(capacity_)),\n> +\t\t alloc_(contentOffset_)\n> +\t{\n> +\t\tfor (const auto &[tag, e] : plan) {\n> +\t\t\talloc_ += sizeof(ValueHeader);\n> +\t\t\talloc_ += e.alignment - 1; // XXX: this is the maximum\n> +\t\t\talloc_ += e.size * e.numElements;\n> +\t\t\talloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n> +\t\t}\n> +\n> +\t\tp_ = static_cast(::operator new(alloc_));\n> +\n> +\t\tauto *entries = reinterpret_cast(p_ + entriesOffset());\n> +\t\tauto it = plan.begin();\n> +\n> +\t\tfor (std::size_t i = 0; i < capacity_; i++, ++it) {\n> +\t\t\tconst auto &[tag, e] = *it;\n> +\n> +\t\t\tnew (&entries[i]) Entry{\n> +\t\t\t\t.tag = tag,\n> +\t\t\t\t.capacity = e.size * e.numElements,\n> +\t\t\t\t.alignment = e.alignment,\n> +\t\t\t\t.type = e.type,\n> +\t\t\t\t.isArray = e.isArray,\n> +\t\t\t};\n\nI was already about to rant against C++'s \"do the same usual thing in\nsome different way\" style and wanted to suggest\n\n\t\t\tentries[i].tag = tag;\n\t\t\tentries[i].capacity = e.size * e.numElements,\n\t\t\tentries[i].alignment = e.alignment;\n\t\t\tentries[i].type = e.type;\n\t\t\tentries[i].isArray = e.isArray;\n\nBut then you loose the const-ness of entries[i]'s fields.\n\nLet's use \"placement new\" then...\n\n\n> +\t\t}\n> +\n> +#if HAS_ASAN\n> +\t\t::__sanitizer_annotate_contiguous_container(\n> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> +\t\t\tp_ + alloc_, p_ + contentOffset_\n> +\t\t);\n> +#endif\n> +\t}\n> +\n> +\tMetadataList(const MetadataList &) = delete;\n> +\tMetadataList(MetadataList &&) = delete;\n> +\n> +\tMetadataList &operator=(const MetadataList &) = delete;\n> +\tMetadataList &operator=(MetadataList &&) = delete;\n> +\n> +\t~MetadataList()\n> +\t{\n> +#if HAS_ASAN\n> +\t\t/*\n> +\t\t * The documentation says the range apparently has to be\n> +\t\t * restored to its initial state before it is deallocated.\n> +\t\t */\n> +\t\t::__sanitizer_annotate_contiguous_container(\n> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> +\t\t\tp_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n> +\t\t);\n> +#endif\n> +\n> +\t\t::operator delete(p_, alloc_);\n> +\t}\n> +\n> +\t// TODO: want these?\n\nWhy not ?\n\n> +\t[[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n> +\t[[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n> +\n> +\tenum class SetError {\n> +\t\tUnknownTag = 1,\n> +\t\tAlreadySet,\n> +\t\tSizeMismatch,\n> +\t\tTypeMismatch,\n> +\t};\n\nThe only user of \"set\" will be pipeline handler, so I wonder if coding\nthe errors as enum is really something that gives any value here or\nit's just overdesign. It's not like pipelines will do\n\n if (AlreadySet)\n doSomething\n\nIf they get an error when adding metadata it will be caught during\ndevelopment I presume.\n\n> +\n> +\t[[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n\nSince we end up copying the data into the MetadataList, why use a\nControlValueView and not a ControlValue ?\n\n> +\t{\n> +\t\tauto *e = find(tag);\n> +\t\tif (!e)\n> +\t\t\treturn SetError::UnknownTag;\n> +\n> +\t\treturn set(*e, v);\n> +\t}\n> +\n> +\ttemplate\n> +\t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n\nWhy TODO ?\n\nHere and everywhere else could you shorten lines length where is\ntrivial to do so ?\n\n\t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl,\n const details::cxx20::type_identity_t &value)\n\n\nWhat does type_identity_t<> gives us here ? Isn't T fully specified ?\n\n> +\t{\n> +\t\tusing TypeInfo = libcamera::details::control_type;\n> +\n> +\t\tif constexpr (TypeInfo::size > 0) {\n> +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n> +\n> +\t\t\treturn set(ctrl.id(), {\n> +\t\t\t\tTypeInfo::value,\n> +\t\t\t\ttrue,\n> +\t\t\t\tvalue.size(),\n> +\t\t\t\treinterpret_cast(value.data()),\n> +\t\t\t});\n> +\t\t} else {\n> +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n> +\n> +\t\t\treturn set(ctrl.id(), {\n> +\t\t\t\tTypeInfo::value,\n> +\t\t\t\tfalse,\n> +\t\t\t\t1,\n> +\t\t\t\treinterpret_cast(&value),\n> +\t\t\t});\n> +\t\t}\n\nThe private set() overload works with Views to avoid copies here, I\nget it, but as said before should the public set(tag, value) accept a\nControlValue & or do we want callers to go through a View ?\n\n> +\t}\n> +\n> +\ttemplate\n> +\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n\nisn't this simply an\n\t[[nodiscard]] std::optional get(const Control &ctrl) const\n\nwhy use decltype(auto) ?\n\n> +\t{\n> +\t\tControlValueView v = get(ctrl.id());\n> +\n> +\t\treturn v ? std::optional(v.get()) : std::nullopt;\n> +\t}\n> +\n> +\t// TODO: operator ControlListView() const ?\n> +\t// TODO: explicit operator ControlList() const ?\n\nDo these still apply ?\n\n> +\n> +\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> +\t{\n> +\t\tconst auto *e = find(tag);\n> +\t\tif (!e)\n> +\t\t\treturn {};\n> +\n> +\t\treturn data_of(*e);\n\nno snake_case but use camelCase please\n\n> +\t}\n> +\n> +\tvoid clear()\n> +\t{\n> +\t\tfor (auto &e : entries())\n> +\t\t\te.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n> +\n> +\t\t[[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n> +\n> +#if HAS_ASAN\n> +\t\t::__sanitizer_annotate_contiguous_container(\n> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> +\t\t\tp_ + contentOffset_ + s.fill, p_ + contentOffset_\n> +\t\t);\n> +#endif\n> +\t}\n> +\n> +\tclass iterator\n> +\t{\n> +\tpublic:\n> +\t\tusing difference_type = std::ptrdiff_t;\n> +\t\tusing value_type = std::pair;\n> +\t\tusing pointer = void;\n> +\t\tusing reference = value_type;\n\nWhy can't you use 'value_type' and need another indirection symbol ?\n\nAre those symbols required when defining a custom iterator ?\n\nAlso shouldn't this be \"value_type &\" ?\n\n> +\t\tusing iterator_category = std::forward_iterator_tag;\n> +\n> +\t\titerator() = default;\n> +\n> +\t\titerator& operator++()\n> +\t\t{\n> +\t\t\tconst auto &h = header();\n> +\n> +\t\t\tp_ += sizeof(h);\n> +\t\t\tp_ = details::align::up(p_, h.alignment);\n> +\t\t\tp_ += h.size;\n> +\t\t\tp_ = details::align::up(p_, alignof(decltype(h)));\n> +\n> +\t\t\treturn *this;\n> +\t\t}\n> +\n> +\t\titerator operator++(int)\n> +\t\t{\n> +\t\t\tauto copy = *this;\n> +\t\t\t++*this;\n> +\t\t\treturn copy;\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] reference operator*() const\n> +\t\t{\n> +\t\t\tconst auto &h = header();\n> +\t\t\tconst auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n> +\n> +\t\t\treturn { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] bool operator==(const iterator &other) const\n> +\t\t{\n> +\t\t\treturn p_ == other.p_;\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] bool operator!=(const iterator &other) const\n> +\t\t{\n> +\t\t\treturn !(*this == other);\n> +\t\t}\n> +\n> +\tprivate:\n> +\t\titerator(const std::byte *p)\n> +\t\t\t: p_(p)\n> +\t\t{\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] const ValueHeader &header() const\n> +\t\t{\n> +\t\t\treturn *reinterpret_cast(p_);\n> +\t\t}\n> +\n> +\t\tfriend MetadataList;\n> +\n> +\t\tconst std::byte *p_ = nullptr;\n> +\t};\n> +\n> +\t[[nodiscard]] iterator begin() const\n> +\t{\n> +\t\treturn { p_ + contentOffset_ };\n> +\t}\n> +\n> +\t[[nodiscard]] iterator end() const\n> +\t{\n> +\t\treturn { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n> +\t}\n> +\n> +\tclass Diff\n> +\t{\n> +\tpublic:\n> +\t\t// TODO: want these?\n\nWhy not you think ?\n\n> +\t\t[[nodiscard]] explicit operator bool() const { return !empty(); }\n> +\t\t[[nodiscard]] bool empty() const { return start_ == stop_; }\n> +\t\t[[nodiscard]] std::size_t size() const { return changed_; }\n> +\t\t[[nodiscard]] const MetadataList &list() const { return *l_; }\n> +\n> +\t\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> +\t\t{\n> +\t\t\tconst auto *e = l_->find(tag);\n> +\t\t\tif (!e)\n> +\t\t\t\treturn {};\n> +\n> +\t\t\tauto o = e->acquireData();\n> +\t\t\tif (!o)\n> +\t\t\t\treturn {};\n> +\n> +\t\t\tif (!(start_ <= *o && *o < stop_))\n> +\t\t\t\treturn {};\n> +\n> +\t\t\treturn l_->data_of(*o);\n> +\t\t}\n> +\n> +\t\ttemplate\n> +\t\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> +\t\t{\n> +\t\t\tControlValueView v = get(ctrl.id());\n> +\n> +\t\t\treturn v ? std::optional(v.get()) : std::nullopt;\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] iterator begin() const\n> +\t\t{\n> +\t\t\treturn { l_->p_ + start_ };\n> +\t\t}\n> +\n> +\t\t[[nodiscard]] iterator end() const\n> +\t\t{\n> +\t\t\treturn { l_->p_ + stop_ };\n> +\t\t}\n> +\n> +\tprivate:\n> +\t\tDiff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n> +\t\t\t: l_(&l),\n> +\t\t\t changed_(changed),\n> +\t\t\t start_(l.contentOffset_ + oldFill),\n> +\t\t\t stop_(l.contentOffset_ + newFill)\n> +\t\t{\n> +\t\t}\n> +\n> +\t\tfriend MetadataList;\n> +\t\tfriend struct Checkpoint;\n> +\n> +\t\tconst MetadataList *l_ = nullptr;\n> +\t\tstd::size_t changed_;\n> +\t\tstd::size_t start_;\n> +\t\tstd::size_t stop_;\n> +\t};\n> +\n> +\tDiff merge(const ControlList &other)\n\nSo the \"merge\" function is how pipeline handlers adds metadata to a\nMetadataList ?\n\n> +\t{\n> +\t\t// TODO: check id map of `other`?\n\nThis already is a proper /* \\todo */\n\n> +\n> +\t\tconst auto c = checkpoint();\n> +\n> +\t\tfor (const auto &[tag, value] : other) {\n> +\t\t\tauto *e = find(tag);\n> +\t\t\tif (e) {\n> +\t\t\t\t[[maybe_unused]] auto r = set(*e, value);\n> +\t\t\t\tassert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n\ndoes \"r == SetError()\" means r == 0 ? What's the TODO for ?\n\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\treturn c.diffSince();\n> +\t}\n> +\n> +\tclass Checkpoint\n> +\t{\n> +\tpublic:\n> +\t\t[[nodiscard]] Diff diffSince() const\n> +\t\t{\n> +\t\t\t/* sync with release-store on `state_` in `set()` */\n> +\t\t\tconst auto curr = l_->state_.load(std::memory_order_acquire);\n> +\n> +\t\t\tassert(s_.count <= curr.count);\n> +\t\t\tassert(s_.fill <= curr.fill);\n> +\n> +\t\t\treturn {\n> +\t\t\t\t*l_,\n> +\t\t\t\tcurr.count - s_.count,\n> +\t\t\t\ts_.fill,\n> +\t\t\t\tcurr.fill,\n> +\t\t\t};\n> +\t\t}\n> +\n> +\tprivate:\n> +\t\tCheckpoint(const MetadataList &l)\n> +\t\t\t: l_(&l),\n> +\t\t\t s_(l.state_.load(std::memory_order_relaxed))\n> +\t\t{\n> +\t\t}\n> +\n> +\t\tfriend MetadataList;\n> +\n> +\t\tconst MetadataList *l_ = nullptr;\n\nIs there in your opinion a case where a Checkpoint can overlive the\nMetadataList it refers to ?\n\n> +\t\tState s_ = {};\n> +\t};\n> +\n> +\t[[nodiscard]] Checkpoint checkpoint() const\n> +\t{\n> +\t\treturn { *this };\n> +\t}\n> +\n> +private:\n> +\t[[nodiscard]] static constexpr std::size_t entriesOffset()\n> +\t{\n> +\t\treturn 0;\n> +\t}\n> +\n> +\t[[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n> +\t{\n> +\t\treturn details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n> +\t}\n> +\n> +\t[[nodiscard]] Span entries() const\n> +\t{\n> +\t\treturn { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n> +\t}\n> +\n> +\t[[nodiscard]] Entry *find(std::uint32_t tag) const\n> +\t{\n> +\t\tconst auto entries = this->entries();\n> +\t\tauto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n\nhow is this different from find_if() ? Just out of curiosity\n\nI'll stop here for the time being. Just one more note, you moved\nMetadataListPlan out this header, so you probably need a dedicated\n.cpp file as well\n\n\n> +\t\t\treturn e.tag < tag;\n> +\t\t});\n> +\n> +\t\tif (it == entries.end() || it->tag != tag)\n> +\t\t\treturn nullptr;\n> +\n> +\t\treturn &*it;\n> +\t}\n> +\n> +\t[[nodiscard]] ControlValueView data_of(const Entry &e) const\n> +\t{\n> +\t\tconst auto o = e.acquireData();\n> +\t\treturn o ? data_of(*o) : ControlValueView{ };\n> +\t}\n> +\n> +\t[[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n> +\t{\n> +\t\tassert(headerOffset <= alloc_ - sizeof(ValueHeader));\n> +\t\tassert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n> +\n> +\t\tconst auto *vh = reinterpret_cast(p_ + headerOffset);\n> +\t\tconst auto *p = reinterpret_cast(vh) + sizeof(*vh);\n> +\t\tstd::size_t avail = p_ + alloc_ - p;\n> +\n> +\t\tconst auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n> +\t\tassert(data);\n> +\n> +\t\treturn { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n> +\t}\n> +\n> +\t[[nodiscard]] SetError set(Entry &e, ControlValueView v)\n> +\t{\n> +\t\tif (e.hasValue())\n> +\t\t\treturn SetError::AlreadySet;\n> +\t\tif (e.type != v.type() || e.isArray != v.isArray())\n> +\t\t\treturn SetError::TypeMismatch;\n> +\n> +\t\tconst auto src = v.data();\n> +\t\tif (e.isArray) {\n> +\t\t\tif (src.size_bytes() > e.capacity)\n> +\t\t\t\treturn SetError::SizeMismatch;\n> +\t\t} else {\n> +\t\t\tif (src.size_bytes() != e.capacity)\n> +\t\t\t\treturn SetError::SizeMismatch;\n> +\t\t}\n> +\n> +\t\tauto s = state_.load(std::memory_order_relaxed);\n> +\t\tstd::byte *oldEnd = p_ + contentOffset_ + s.fill;\n> +\t\tstd::byte *p = oldEnd;\n> +\n> +\t\tauto *headerPtr = details::align::up(p);\n> +\t\tauto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n> +\t\tdetails::align::up(0, alignof(ValueHeader), p);\n> +\n> +#if HAS_ASAN\n> +\t\t::__sanitizer_annotate_contiguous_container(\n> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> +\t\t\toldEnd, p\n> +\t\t);\n> +#endif\n> +\n> +\t\tnew (headerPtr) ValueHeader{\n> +\t\t\t.tag = e.tag,\n> +\t\t\t.size = std::uint32_t(src.size_bytes()),\n> +\t\t\t.alignment = e.alignment,\n> +\t\t\t.params = {\n> +\t\t\t\t.type = v.type(),\n> +\t\t\t\t.isArray = v.isArray(),\n> +\t\t\t\t.numElements = std::uint32_t(v.numElements()),\n> +\t\t\t},\n> +\t\t};\n> +\t\tstd::memcpy(dataPtr, src.data(), src.size_bytes());\n> +\t\te.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n> +\n> +\t\ts.fill += p - oldEnd;\n> +\t\ts.count += 1;\n> +\n> +\t\tstate_.store(s, std::memory_order_release);\n> +\n> +\t\treturn {};\n> +\t}\n> +\n> +\tstd::size_t capacity_ = 0;\n> +\tstd::size_t contentOffset_ = -1;\n> +\tstd::size_t alloc_ = 0;\n> +\tstd::atomic state_ = State{};\n> +\tstd::byte *p_ = nullptr;\n> +\t// TODO: ControlIdMap in any way shape or form?\n> +\n> +\t/*\n> +\t * If this is problematic on a 32-bit architecture, then\n> +\t * `count` can be stored in a separate atomic variable\n> +\t * but then `Diff::changed_` must be removed since the fill\n> +\t * level and item count cannot be retrieved atomically.\n> +\t */\n> +\tstatic_assert(decltype(state_)::is_always_lock_free);\n> +};\n> +\n> +} /* namespace libcamera */\n> +\n> +#undef HAS_ASAN\n> diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n> new file mode 100644\n> index 000000000..2ed35c54f\n> --- /dev/null\n> +++ b/include/libcamera/metadata_list_plan.h\n> @@ -0,0 +1,130 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +class MetadataListPlan\n> +{\n> +public:\n> +\tstruct Entry {\n> +\t\tstd::uint32_t size;\n> +\t\tstd::uint32_t alignment; // TODO: is this necessary?\n> +\t\tstd::uint32_t numElements;\n> +\t\tControlType type;\n> +\t\tbool isArray;\n> +\t};\n> +\n> +\t[[nodiscard]] bool empty() const { return items_.empty(); }\n> +\t[[nodiscard]] std::size_t size() const { return items_.size(); }\n> +\t[[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n> +\t[[nodiscard]] decltype(auto) end() const { return items_.end(); }\n> +\tvoid clear() { items_.clear(); }\n> +\n> +\ttemplate<\n> +\t\ttypename T,\n> +\t\tstd::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n> +\t>\n> +\tdecltype(auto) set(const Control &ctrl)\n> +\t{\n> +\t\tif constexpr (libcamera::details::control_type::size > 0) {\n> +\t\t\tstatic_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n> +\n> +\t\t\treturn set(\n> +\t\t\t\tctrl.id(),\n> +\t\t\t\tlibcamera::details::control_type::size,\n> +\t\t\t\ttrue\n> +\t\t\t);\n> +\t\t} else {\n> +\t\t\treturn set(ctrl.id(), 1, false);\n> +\t\t}\n> +\t}\n> +\n> +\ttemplate<\n> +\t\ttypename T,\n> +\t\tstd::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n> +\t>\n> +\tdecltype(auto) set(const Control &ctrl, std::size_t numElements)\n> +\t{\n> +\t\treturn set(ctrl.id(), numElements, true);\n> +\t}\n> +\n> +\t[[nodiscard]] bool set(std::uint32_t tag,\n> +\t\t\t std::size_t size, std::size_t alignment,\n> +\t\t\t std::size_t numElements, ControlType type, bool isArray)\n> +\t{\n> +\t\tif (size == 0 || size > std::numeric_limits::max())\n> +\t\t\treturn false;\n> +\t\tif (alignment > std::numeric_limits::max())\n> +\t\t\treturn false;\n> +\t\tif (!details::cxx20::has_single_bit(alignment))\n> +\t\t\treturn false;\n> +\t\tif (numElements > std::numeric_limits::max() / size)\n> +\t\t\treturn false;\n> +\t\tif (!isArray && numElements != 1)\n> +\t\t\treturn false;\n> +\n> +\t\titems_.insert_or_assign(tag, Entry{\n> +\t\t\t.size = std::uint32_t(size),\n> +\t\t\t.alignment = std::uint32_t(alignment),\n> +\t\t\t.numElements = std::uint32_t(numElements),\n> +\t\t\t.type = type,\n> +\t\t\t.isArray = isArray,\n> +\t\t});\n> +\n> +\t\treturn true;\n> +\t}\n> +\n> +\t[[nodiscard]] const Entry *get(std::uint32_t tag) const\n> +\t{\n> +\t\tauto it = items_.find(tag);\n> +\t\tif (it == items_.end())\n> +\t\t\treturn nullptr;\n> +\n> +\t\treturn &it->second;\n> +\t}\n> +\n> +\t[[nodiscard]] const Entry *get(const ControlId &cid) const\n> +\t{\n> +\t\tconst auto *e = get(cid.id());\n> +\t\tif (!e)\n> +\t\t\treturn nullptr;\n> +\n> +\t\tif (e->type != cid.type() || e->isArray != cid.isArray())\n> +\t\t\treturn nullptr;\n> +\n> +\t\treturn e;\n> +\t}\n> +\n> +private:\n> +\tstd::map items_;\n> +\n> +\ttemplate\n> +\tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> +\t{\n> +\t\tstatic_assert(std::is_trivially_copyable_v);\n> +\n> +\t\t[[maybe_unused]] bool ok = set(tag,\n> +\t\t\t\t\t sizeof(T), alignof(T),\n> +\t\t\t\t\t numElements, details::control_type::value, isArray);\n> +\t\tassert(ok);\n> +\n> +\t\treturn *this;\n> +\t}\n> +};\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> index de1eb99b2..8c5ce4503 100644\n> --- a/src/libcamera/meson.build\n> +++ b/src/libcamera/meson.build\n> @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n> 'framebuffer.cpp',\n> 'framebuffer_allocator.cpp',\n> 'geometry.cpp',\n> + 'metadata_list.cpp',\n> 'orientation.cpp',\n> 'pixel_format.cpp',\n> 'request.cpp',\n> diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n> new file mode 100644\n> index 000000000..ebefdfdad\n> --- /dev/null\n> +++ b/src/libcamera/metadata_list.cpp\n> @@ -0,0 +1,344 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + */\n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +/**\n> + * \\class MetadataListPlan\n> + * \\brief Class to hold the possible set of metadata items for a MetadataList\n> + */\n> +\n> +/**\n> + * \\class MetadataListPlan::Entry\n> + * \\brief Details of a metadata item\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\var MetadataListPlan::Entry::size\n> + * \\brief Number of bytes in a single element\n> + *\n> + * \\var MetadataListPlan::Entry::alignment\n> + * \\brief Required alignment of the elements\n> + * \\endinternal\n> + *\n> + * \\var MetadataListPlan::Entry::numElements\n> + * \\brief Number of elements in the value\n> + * \\sa ControlValueView::numElements()\n> + *\n> + * \\var MetadataListPlan::Entry::type\n> + * \\brief The type of the value\n> + * \\sa ControlValueView::type()\n> + *\n> + * \\var MetadataListPlan::Entry::isArray\n> + * \\brief Whether or not the value is array-like\n> + * \\sa ControlValueView::isArray()\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::begin() const\n> + * \\brief Retrieve the begin iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::end() const\n> + * \\brief Retrieve the end iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::size() const\n> + * \\brief Retrieve the number of entries\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::empty() const\n> + * \\brief Check if empty\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::clear()\n> + * \\brief Remove all controls\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(const Control &ctrl)\n> + * \\brief Add an entry for the given control to the metadata list plan\n> + * \\param[in] ctrl The control\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n> + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n> + * \\param[in] ctrl The control\n> + * \\param[in] count The maximum number of elements\n> + *\n> + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n> + * capacity of \\a count elements.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(std::uint32_t tag,\n> + *\t\t\t std::size_t size, std::size_t alignment,\n> + *\t\t\t std::size_t count, ControlType type, bool isArray)\n> + * \\brief Add an entry to the metadata list plan\n> + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n> + * would result in an invalid entry\n> + *\n> + * This functions adds an entry with essentially arbitrary parameters, without deriving\n> + * them from a given ControlId instance. This is mainly used when deserializing.\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n> + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::get(const ControlId &cid) const\n> + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n> + *\n> + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n> + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n> + */\n> +\n> +/**\n> + * \\class MetadataList\n> + * \\brief Class to hold metadata items\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n> + * \\brief Construct a metadata list according to \\a plan\n> + *\n> + * Construct a metadata list according to the provided \\a plan.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::size() const\n> + * \\brief Retrieve the number of controls\n> + * \\context This function is \\threadsafe.\n> + * \\note If the list is being modified, the return value may be out of\n> + * date by the time the function returns\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::empty() const\n> + * \\brief Check if empty\n> + * \\context This function is \\threadsafe.\n> + * \\note If the list is being modified, the return value may be out of\n> + * date by the time the function returns\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::clear()\n> + * \\brief Remove all items from the list\n> + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n> + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n> + * when this function is called.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::begin() const\n> + * \\brief Retrieve begin iterator\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::end() const\n> + * \\brief Retrieve end iterator\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::get(const Control &ctrl) const\n> + * \\brief Get the value of control \\a ctrl\n> + * \\return A std::optional containing the control value, or std::nullopt if\n> + * the control \\a ctrl is not present in the list\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::get(std::uint32_t tag) const\n> + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n> + * \\return A std::optional containing the control value, or std::nullopt if\n> + * the control is not present in the list\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> + * \\brief Set the value of control \\a ctrl to \\a value\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n> + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::merge(const ControlList &other)\n> + * \\brief Add all missing items from \\a other\n> + *\n> + * Add all items from \\a other that are not present in \\a this.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\enum MetadataList::SetError\n> + * \\brief Error code returned by a set operation\n> + *\n> + * \\var MetadataList::SetError::UnknownTag\n> + * \\brief The tag is not supported by the metadata list\n> + * \\var MetadataList::SetError::AlreadySet\n> + * \\brief A value has already been added with the given tag\n> + * \\var MetadataList::SetError::SizeMismatch\n> + * \\brief The size of the data is not appropriate for the given tag\n> + * \\var MetadataList::SetError::TypeMismatch\n> + * \\brief The type of the value does not match the expected type\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::checkpoint() const\n> + * \\brief Create a checkpoint\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\class MetadataList::iterator\n> + * \\brief Iterator\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::difference_type\n> + * \\brief iterator's difference type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::value_type\n> + * \\brief iterator's value type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::pointer\n> + * \\brief iterator's pointer type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::reference\n> + * \\brief iterator's reference type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::iterator_category\n> + * \\brief iterator's category\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator*()\n> + * \\brief Retrieve value at iterator\n> + * \\return A \\a ControlListView representing the value\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n> + * \\brief Check if two iterators are equal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n> + * \\brief Check if two iterators are not equal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator++(int)\n> + * \\brief Advance the iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator++()\n> + * \\brief Advance the iterator\n> + */\n> +\n> +/**\n> + * \\class MetadataList::Diff\n> + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n> + * \\sa Camera::metadataAvailable\n> + * \\internal\n> + * \\sa MetadataList::Checkpoint::diffSince()\n> + * \\endinternal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::list() const\n> + * \\brief Retrieve the associated MetadataList\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::size() const\n> + * \\brief Retrieve the number of metadata items designated\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::empty() const\n> + * \\brief Check if any metadata items are designated\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::operator bool() const\n> + * \\copydoc MetadataList::Diff::empty() const\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n> + * \\copydoc MetadataList::get(const Control &ctrl) const\n> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> + * even if it is otherwise present in the backing MetadataList.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n> + * \\copydoc MetadataList::get(std::uint32_t tag) const\n> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> + * even if it is otherwise present in the backing MetadataList.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::begin() const\n> + * \\brief Retrieve the begin iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::end() const\n> + * \\brief Retrieve the end iterator\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\class MetadataList::Checkpoint\n> + * \\brief Designates a point in the stream of metadata items\n> + *\n> + * A Checkpoint object designates a point in the stream of metadata items in the associated\n> + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n> + * added to the list after the designated point using diffSince().\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::Checkpoint::diffSince() const\n> + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n> + */\n> +\n> +} /* namespace libcamera */\n> diff --git a/test/controls/meson.build b/test/controls/meson.build\n> index 763f8905e..b68a4fc53 100644\n> --- a/test/controls/meson.build\n> +++ b/test/controls/meson.build\n> @@ -5,6 +5,7 @@ control_tests = [\n> {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n> {'name': 'control_list', 'sources': ['control_list.cpp']},\n> {'name': 'control_value', 'sources': ['control_value.cpp']},\n> + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n> ]\n>\n> foreach test : control_tests\n> diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n> new file mode 100644\n> index 000000000..f0872acd9\n> --- /dev/null\n> +++ b/test/controls/metadata_list.cpp\n> @@ -0,0 +1,170 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + *\n> + * MetadataList tests\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \"test.h\"\n> +\n> +using namespace std;\n> +using namespace libcamera;\n> +\n> +#define ASSERT(x) do { \\\n> +\tif (!static_cast(x)) { \\\n> +\t\tstd::cerr << '`' << #x << \"` failed\" << std::endl; \\\n> +\t\treturn TestFail; \\\n> +\t} \\\n> +} while (false)\n> +\n> +class MetadataListTest : public Test\n> +{\n> +public:\n> +\tMetadataListTest() = default;\n> +\n> +protected:\n> +\tint run() override\n> +\t{\n> +\t\tMetadataListPlan mlp;\n> +\t\tmlp.set(controls::ExposureTime);\n> +\t\tmlp.set(controls::ExposureValue);\n> +\t\tmlp.set(controls::ColourGains);\n> +\t\tmlp.set(controls::AfWindows, 10);\n> +\t\tmlp.set(controls::AeEnable);\n> +\t\tmlp.set(controls::SensorTimestamp);\n> +\n> +\t\tMetadataList ml(mlp);\n> +\n> +\t\tstatic_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n> +\t\tASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n> +\n> +\t\tASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n> +\t\tASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n> +\n> +\t\tauto f1 = std::async(std::launch::async, [&] {\n> +\t\t\tusing namespace std::chrono_literals;\n> +\n> +\t\t\tstd::this_thread::sleep_for(500ms);\n> +\t\t\tASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n> +\n> +\t\t\tstd::this_thread::sleep_for(500ms);\n> +\t\t\tASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n> +\n> +\t\t\tstd::this_thread::sleep_for(500ms);\n> +\t\t\tASSERT(ml.set(controls::ColourGains, std::array{\n> +\t\t\t\t123.f,\n> +\t\t\t\t456.f\n> +\t\t\t}) == MetadataList::SetError());\n> +\n> +\t\t\tstd::this_thread::sleep_for(500ms);\n> +\t\t\tASSERT(ml.set(controls::AfWindows, std::array{\n> +\t\t\t\tRectangle(),\n> +\t\t\t\tRectangle(1, 2, 3, 4),\n> +\t\t\t\tRectangle(0x1111, 0x2222, 0x3333, 0x4444),\n> +\t\t\t}) == MetadataList::SetError());\n> +\n> +\t\t\treturn TestPass;\n> +\t\t});\n> +\n> +\t\tauto f2 = std::async(std::launch::async, [&] {\n> +\t\t\tfor (;;) {\n> +\t\t\t\tconst auto x = ml.get(controls::ExposureTime);\n> +\t\t\t\tconst auto y = ml.get(controls::ExposureValue);\n> +\t\t\t\tconst auto z = ml.get(controls::ColourGains);\n> +\t\t\t\tconst auto w = ml.get(controls::AfWindows);\n> +\n> +\t\t\t\tif (x)\n> +\t\t\t\t\tASSERT(*x == 0x1111);\n> +\n> +\t\t\t\tif (y)\n> +\t\t\t\t\tASSERT(*y == 1.0f);\n> +\n> +\t\t\t\tif (z) {\n> +\t\t\t\t\tASSERT(z->size() == 2);\n> +\t\t\t\t\tASSERT((*z)[0] == 123.f);\n> +\t\t\t\t\tASSERT((*z)[1] == 456.f);\n> +\t\t\t\t}\n> +\n> +\t\t\t\tif (w) {\n> +\t\t\t\t\tASSERT(w->size() == 3);\n> +\t\t\t\t\tASSERT((*w)[0].isNull());\n> +\t\t\t\t\tASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n> +\t\t\t\t\tASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n> +\t\t\t\t}\n> +\n> +\t\t\t\tif (x && y && z && w)\n> +\t\t\t\t\tbreak;\n> +\t\t\t}\n> +\n> +\t\t\treturn TestPass;\n> +\t\t});\n> +\n> +\t\tASSERT(f1.get() == TestPass);\n> +\t\tASSERT(f2.get() == TestPass);\n> +\n> +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n> +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n> +\n> +\t\tASSERT(ml.get(controls::ExposureTime) == 0x1111);\n> +\t\tASSERT(ml.get(controls::ExposureValue) == 1);\n> +\n> +\t\tfor (auto &&[tag, v] : ml)\n> +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> +\n> +\t\tstd::cout << std::endl;\n> +\n> +\t\tml.clear();\n> +\t\tASSERT(ml.empty());\n> +\t\tASSERT(ml.size() == 0);\n> +\n> +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n> +\t\tASSERT(ml.get(controls::ExposureTime) == 0x2222);\n> +\n> +\t\tauto c = ml.checkpoint();\n> +\n> +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n> +\t\tASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n> +\n> +\t\tauto d = c.diffSince();\n> +\t\tASSERT(&d.list() == &ml);\n> +\n> +\t\tASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n> +\n> +\t\tASSERT(d);\n> +\t\tASSERT(!d.empty());\n> +\t\tASSERT(d.size() == 2);\n> +\t\tASSERT(!d.get(controls::ExposureTime));\n> +\t\tASSERT(!d.get(controls::ColourGains));\n> +\t\tASSERT(!d.get(controls::AfWindows));\n> +\t\tASSERT(d.get(controls::ExposureValue) == 2);\n> +\t\tASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n> +\n> +\t\tfor (auto &&[tag, v] : d)\n> +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> +\n> +\t\t/* Test if iterators work with algorithms. */\n> +\t\tstd::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n> +\t\t\treturn false;\n> +\t\t});\n> +\n> +#if 0\n> + {\n> + auto it = ml.begin();\n> + ml.clear();\n> + std::ignore = *it; /* Trigger ASAN. */\n> + }\n> +#endif\n> +\n> +\t\treturn TestPass;\n> +\t}\n> +};\n> +\n> +TEST_REGISTER(MetadataListTest)\n> --\n> 2.50.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id E813CBDCC1\n\tfor ;\n\tFri, 25 Jul 2025 16:08:31 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id CB61C69101;\n\tFri, 25 Jul 2025 18:08:30 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 95CB1690A6\n\tfor ;\n\tFri, 25 Jul 2025 18:08:28 +0200 (CEST)","from ideasonboard.com (mob-5-90-139-29.net.vodafone.it\n\t[5.90.139.29])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 0F44DB2B;\n\tFri, 25 Jul 2025 18:07:47 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"sU16ZQG4\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1753459668;\n\tbh=DhaazsFSXqXz42+mvGC+pgDaYMoxz6IttnVHADvJ1Vc=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=sU16ZQG4fXN6Z8qGY1R6q1o1hcTlOe4Kyrth5ZkAh1sJLugYYboMXkQ4j8TsjjHOa\n\t6u45It/fea825ue2aRECiUg7Fi1K3NctCCtsSFnSfcuYimfpwkjZIlSmKe/aXakgl5\n\tpJ6Vz0AM7smHCfs+WKAVp9jth4MxLbZqP62xepzU=","Date":"Fri, 25 Jul 2025 18:08:24 +0200","From":"Jacopo Mondi ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Cc":"libcamera-devel@lists.libcamera.org","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","Message-ID":"<2lweaavht65vexto5246wcvmxjoxhwgtprz5hjvbrz7y4h4hrk@r3rwki5uoefi>","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35181,"web_url":"https://patchwork.libcamera.org/comment/35181/","msgid":"","date":"2025-07-28T08:36:10","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":143,"url":"https://patchwork.libcamera.org/api/people/143/","name":"Jacopo Mondi","email":"jacopo.mondi@ideasonboard.com"},"content":"Hi Barnabás\n let me resume the review from where I stopped last time\n\nOn Fri, Jul 25, 2025 at 06:08:28PM +0200, Jacopo Mondi wrote:\n> Hi Barnabás\n>\n> On Mon, Jul 21, 2025 at 12:46:07PM +0200, Barnabás Pőcze wrote:\n> > Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n> > reported by a camera for a given request. Previously, a `ControlList` was\n> > used for this purpose. The reason for introducing a separate type is to\n> > simplify the access to the returned metadata during the entire lifetime\n> > of a request.\n> >\n> > Specifically, for early metadata completion to be easily usable it should be\n> > guaranteed that any completed metadata item can be accessed and looked up\n> > at least until the associated requested is reused with `Request::reuse()`.\n> >\n> > However, when a metadata item is completed early, the pipeline handler\n> > might still work on the request in the `CameraManager`'s private thread,\n> > therefore there is an inherent synchronization issue when an application\n> > accesses early metadata.\n> >\n> > Restricting the user to only access the metadata items of a not yet completed\n> > request in the early metadata availability signal handler by ways of\n> > documenting or enforcing it at runtime could be an option, but it is not\n> > too convenient for the user.\n> >\n> > The current `ControlList` implementation employs an `std::unordered_map`,\n> > so pointers remain stable when the container is modified, so an application\n> > could keep accessing particular metadata items outside the signal handler,\n> > but this fact is far from obvious, and the user would still not be able\n> > to make a copy of all metadata or do lookups based on the numeric ids or\n> > the usual `libcamera::Control<>` objects, thus some type safety is lost.\n> >\n> > The above also requires that each metadata item is only completed once for\n> > a given request, but this does not appear to be serious limitation,\n> > and in fact, this restriction is enforced by `MetadataList`.\n> >\n> > The introduced `MetadataList` supports single writer - multiple reader\n> > scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n> > without introducing data races or other synchronization issues. This is\n> > achieved by requiring the possible set of metadata items to be known\n> > (such set is stored in a `MetadataListPlan` object). Based on the this\n> > plan, a single contiguous allocation is made to accommodate all potential\n> > metadata items. Due to this single contiguous allocation that is not modified\n> > during the lifetime of a `MetadataList` and atomic modifications, it is\n> > possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n> > there is only ever a single writer.\n> >\n> > Signed-off-by: Barnabás Pőcze \n> > ---\n> > changes in v2:\n> > * remove multiple not strictly necessary functions\n> > ---\n> > include/libcamera/meson.build | 2 +\n> > include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n> > include/libcamera/metadata_list_plan.h | 130 ++++++\n> > src/libcamera/meson.build | 1 +\n> > src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n> > test/controls/meson.build | 1 +\n> > test/controls/metadata_list.cpp | 170 ++++++++\n> > 7 files changed, 1195 insertions(+)\n> > create mode 100644 include/libcamera/metadata_list.h\n> > create mode 100644 include/libcamera/metadata_list_plan.h\n> > create mode 100644 src/libcamera/metadata_list.cpp\n> > create mode 100644 test/controls/metadata_list.cpp\n> >\n> > diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n> > index 30ea76f94..410b548dd 100644\n> > --- a/include/libcamera/meson.build\n> > +++ b/include/libcamera/meson.build\n> > @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n> > 'framebuffer_allocator.h',\n> > 'geometry.h',\n> > 'logging.h',\n> > + 'metadata_list.h',\n> > + 'metadata_list_plan.h',\n> > 'orientation.h',\n> > 'pixel_format.h',\n> > 'request.h',\n> > diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n> > new file mode 100644\n> > index 000000000..7fe3dbbab\n> > --- /dev/null\n> > +++ b/include/libcamera/metadata_list.h\n> > @@ -0,0 +1,547 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2025, Ideas On Board Oy\n> > + *\n> > + * Metadata list\n> > + */\n> > +\n> > +#pragma once\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +\n> > +// TODO: want this?\n> > +#if __has_include()\n> > +#if __SANITIZE_ADDRESS__ /* gcc */\n> > +#include \n> > +#define HAS_ASAN 1\n> > +#elif defined(__has_feature)\n> > +#if __has_feature(address_sanitizer) /* clang */\n> > +#include \n> > +#define HAS_ASAN 1\n> > +#endif\n> > +#endif\n> > +#endif\n> > +\n> > +namespace libcamera {\n> > +\n> > +class MetadataList\n> > +{\n> > +private:\n> > +\tstruct ValueParams {\n> > +\t\tControlType type;\n> > +\t\tbool isArray;\n> > +\t\tstd::uint32_t numElements;\n> > +\t};\n> > +\n> > +\tstruct Entry {\n> > +\t\tconst std::uint32_t tag;\n> > +\t\tconst std::uint32_t capacity;\n> > +\t\tconst std::uint32_t alignment;\n> > +\t\tconst ControlType type;\n> > +\t\tbool isArray;\n> > +\n> > +\t\tstatic constexpr std::uint32_t invalidOffset = -1;\n> > +\t\t/*\n> > +\t\t * Offset from the beginning of the allocation, and\n> > +\t\t * and _not_ relative to `contentOffset_`.\n> > +\t\t */\n> > +\t\tstd::atomic_uint32_t headerOffset = invalidOffset;\n> > +\n> > +\t\t[[nodiscard]] std::optional hasValue() const\n> > +\t\t{\n> > +\t\t\tauto offset = headerOffset.load(std::memory_order_relaxed);\n> > +\t\t\tif (offset == invalidOffset)\n> > +\t\t\t\treturn {};\n> > +\n> > +\t\t\treturn offset;\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] std::optional acquireData() const\n> > +\t\t{\n> > +\t\t\tauto offset = hasValue();\n> > +\t\t\tif (offset) {\n> > +\t\t\t\t/* sync with release-store on `headerOffset` in `MetadataList::set()` */\n> > +\t\t\t\tstd::atomic_thread_fence(std::memory_order_acquire);\n> > +\t\t\t}\n> > +\n> > +\t\t\treturn offset;\n> > +\t\t}\n> > +\t};\n> > +\n> > +\tstruct ValueHeader {\n> > +\t\tstd::uint32_t tag;\n> > +\t\tstd::uint32_t size;\n> > +\t\tstd::uint32_t alignment;\n> > +\t\tValueParams params;\n> > +\t};\n> > +\n> > +\tstruct State {\n> > +\t\tstd::uint32_t count;\n> > +\t\tstd::uint32_t fill;\n> > +\t};\n> > +\n> > +public:\n> > +\texplicit MetadataList(const MetadataListPlan &plan)\n> > +\t\t: capacity_(plan.size()),\n> > +\t\t contentOffset_(MetadataList::contentOffset(capacity_)),\n> > +\t\t alloc_(contentOffset_)\n> > +\t{\n> > +\t\tfor (const auto &[tag, e] : plan) {\n> > +\t\t\talloc_ += sizeof(ValueHeader);\n> > +\t\t\talloc_ += e.alignment - 1; // XXX: this is the maximum\n> > +\t\t\talloc_ += e.size * e.numElements;\n> > +\t\t\talloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n> > +\t\t}\n> > +\n> > +\t\tp_ = static_cast(::operator new(alloc_));\n> > +\n> > +\t\tauto *entries = reinterpret_cast(p_ + entriesOffset());\n> > +\t\tauto it = plan.begin();\n> > +\n> > +\t\tfor (std::size_t i = 0; i < capacity_; i++, ++it) {\n> > +\t\t\tconst auto &[tag, e] = *it;\n> > +\n> > +\t\t\tnew (&entries[i]) Entry{\n> > +\t\t\t\t.tag = tag,\n> > +\t\t\t\t.capacity = e.size * e.numElements,\n> > +\t\t\t\t.alignment = e.alignment,\n> > +\t\t\t\t.type = e.type,\n> > +\t\t\t\t.isArray = e.isArray,\n> > +\t\t\t};\n>\n> I was already about to rant against C++'s \"do the same usual thing in\n> some different way\" style and wanted to suggest\n>\n> \t\t\tentries[i].tag = tag;\n> \t\t\tentries[i].capacity = e.size * e.numElements,\n> \t\t\tentries[i].alignment = e.alignment;\n> \t\t\tentries[i].type = e.type;\n> \t\t\tentries[i].isArray = e.isArray;\n>\n> But then you loose the const-ness of entries[i]'s fields.\n>\n> Let's use \"placement new\" then...\n>\n>\n> > +\t\t}\n> > +\n> > +#if HAS_ASAN\n> > +\t\t::__sanitizer_annotate_contiguous_container(\n> > +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> > +\t\t\tp_ + alloc_, p_ + contentOffset_\n> > +\t\t);\n> > +#endif\n> > +\t}\n> > +\n> > +\tMetadataList(const MetadataList &) = delete;\n> > +\tMetadataList(MetadataList &&) = delete;\n> > +\n> > +\tMetadataList &operator=(const MetadataList &) = delete;\n> > +\tMetadataList &operator=(MetadataList &&) = delete;\n> > +\n> > +\t~MetadataList()\n> > +\t{\n> > +#if HAS_ASAN\n> > +\t\t/*\n> > +\t\t * The documentation says the range apparently has to be\n> > +\t\t * restored to its initial state before it is deallocated.\n> > +\t\t */\n> > +\t\t::__sanitizer_annotate_contiguous_container(\n> > +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> > +\t\t\tp_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n> > +\t\t);\n> > +#endif\n> > +\n> > +\t\t::operator delete(p_, alloc_);\n> > +\t}\n> > +\n> > +\t// TODO: want these?\n>\n> Why not ?\n>\n> > +\t[[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n> > +\t[[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n> > +\n> > +\tenum class SetError {\n> > +\t\tUnknownTag = 1,\n> > +\t\tAlreadySet,\n> > +\t\tSizeMismatch,\n> > +\t\tTypeMismatch,\n> > +\t};\n>\n> The only user of \"set\" will be pipeline handler, so I wonder if coding\n> the errors as enum is really something that gives any value here or\n> it's just overdesign. It's not like pipelines will do\n>\n> if (AlreadySet)\n> doSomething\n>\n> If they get an error when adding metadata it will be caught during\n> development I presume.\n>\n> > +\n> > +\t[[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n>\n> Since we end up copying the data into the MetadataList, why use a\n> ControlValueView and not a ControlValue ?\n>\n> > +\t{\n> > +\t\tauto *e = find(tag);\n> > +\t\tif (!e)\n> > +\t\t\treturn SetError::UnknownTag;\n> > +\n> > +\t\treturn set(*e, v);\n> > +\t}\n> > +\n> > +\ttemplate\n> > +\t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n>\n> Why TODO ?\n>\n> Here and everywhere else could you shorten lines length where is\n> trivial to do so ?\n>\n> \t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl,\n> const details::cxx20::type_identity_t &value)\n>\n>\n> What does type_identity_t<> gives us here ? Isn't T fully specified ?\n>\n> > +\t{\n> > +\t\tusing TypeInfo = libcamera::details::control_type;\n> > +\n> > +\t\tif constexpr (TypeInfo::size > 0) {\n> > +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n> > +\n> > +\t\t\treturn set(ctrl.id(), {\n> > +\t\t\t\tTypeInfo::value,\n> > +\t\t\t\ttrue,\n> > +\t\t\t\tvalue.size(),\n> > +\t\t\t\treinterpret_cast(value.data()),\n> > +\t\t\t});\n> > +\t\t} else {\n> > +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n> > +\n> > +\t\t\treturn set(ctrl.id(), {\n> > +\t\t\t\tTypeInfo::value,\n> > +\t\t\t\tfalse,\n> > +\t\t\t\t1,\n> > +\t\t\t\treinterpret_cast(&value),\n> > +\t\t\t});\n> > +\t\t}\n>\n> The private set() overload works with Views to avoid copies here, I\n> get it, but as said before should the public set(tag, value) accept a\n> ControlValue & or do we want callers to go through a View ?\n>\n> > +\t}\n> > +\n> > +\ttemplate\n> > +\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n>\n> isn't this simply an\n> \t[[nodiscard]] std::optional get(const Control &ctrl) const\n>\n> why use decltype(auto) ?\n>\n> > +\t{\n> > +\t\tControlValueView v = get(ctrl.id());\n> > +\n> > +\t\treturn v ? std::optional(v.get()) : std::nullopt;\n> > +\t}\n> > +\n> > +\t// TODO: operator ControlListView() const ?\n> > +\t// TODO: explicit operator ControlList() const ?\n>\n> Do these still apply ?\n>\n> > +\n> > +\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> > +\t{\n> > +\t\tconst auto *e = find(tag);\n> > +\t\tif (!e)\n> > +\t\t\treturn {};\n> > +\n> > +\t\treturn data_of(*e);\n>\n> no snake_case but use camelCase please\n>\n> > +\t}\n> > +\n> > +\tvoid clear()\n> > +\t{\n> > +\t\tfor (auto &e : entries())\n> > +\t\t\te.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n> > +\n> > +\t\t[[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n> > +\n> > +#if HAS_ASAN\n> > +\t\t::__sanitizer_annotate_contiguous_container(\n> > +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> > +\t\t\tp_ + contentOffset_ + s.fill, p_ + contentOffset_\n> > +\t\t);\n> > +#endif\n> > +\t}\n> > +\n> > +\tclass iterator\n> > +\t{\n> > +\tpublic:\n> > +\t\tusing difference_type = std::ptrdiff_t;\n> > +\t\tusing value_type = std::pair;\n> > +\t\tusing pointer = void;\n> > +\t\tusing reference = value_type;\n>\n> Why can't you use 'value_type' and need another indirection symbol ?\n>\n> Are those symbols required when defining a custom iterator ?\n>\n> Also shouldn't this be \"value_type &\" ?\n>\n> > +\t\tusing iterator_category = std::forward_iterator_tag;\n> > +\n> > +\t\titerator() = default;\n> > +\n> > +\t\titerator& operator++()\n> > +\t\t{\n> > +\t\t\tconst auto &h = header();\n> > +\n> > +\t\t\tp_ += sizeof(h);\n> > +\t\t\tp_ = details::align::up(p_, h.alignment);\n> > +\t\t\tp_ += h.size;\n> > +\t\t\tp_ = details::align::up(p_, alignof(decltype(h)));\n> > +\n> > +\t\t\treturn *this;\n> > +\t\t}\n> > +\n> > +\t\titerator operator++(int)\n> > +\t\t{\n> > +\t\t\tauto copy = *this;\n> > +\t\t\t++*this;\n> > +\t\t\treturn copy;\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] reference operator*() const\n> > +\t\t{\n> > +\t\t\tconst auto &h = header();\n> > +\t\t\tconst auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n> > +\n> > +\t\t\treturn { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] bool operator==(const iterator &other) const\n> > +\t\t{\n> > +\t\t\treturn p_ == other.p_;\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] bool operator!=(const iterator &other) const\n> > +\t\t{\n> > +\t\t\treturn !(*this == other);\n> > +\t\t}\n> > +\n> > +\tprivate:\n> > +\t\titerator(const std::byte *p)\n> > +\t\t\t: p_(p)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] const ValueHeader &header() const\n> > +\t\t{\n> > +\t\t\treturn *reinterpret_cast(p_);\n> > +\t\t}\n> > +\n> > +\t\tfriend MetadataList;\n> > +\n> > +\t\tconst std::byte *p_ = nullptr;\n> > +\t};\n> > +\n> > +\t[[nodiscard]] iterator begin() const\n> > +\t{\n> > +\t\treturn { p_ + contentOffset_ };\n> > +\t}\n> > +\n> > +\t[[nodiscard]] iterator end() const\n> > +\t{\n> > +\t\treturn { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n> > +\t}\n> > +\n> > +\tclass Diff\n> > +\t{\n> > +\tpublic:\n> > +\t\t// TODO: want these?\n>\n> Why not you think ?\n>\n> > +\t\t[[nodiscard]] explicit operator bool() const { return !empty(); }\n> > +\t\t[[nodiscard]] bool empty() const { return start_ == stop_; }\n> > +\t\t[[nodiscard]] std::size_t size() const { return changed_; }\n> > +\t\t[[nodiscard]] const MetadataList &list() const { return *l_; }\n> > +\n> > +\t\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> > +\t\t{\n> > +\t\t\tconst auto *e = l_->find(tag);\n> > +\t\t\tif (!e)\n> > +\t\t\t\treturn {};\n> > +\n> > +\t\t\tauto o = e->acquireData();\n> > +\t\t\tif (!o)\n> > +\t\t\t\treturn {};\n> > +\n> > +\t\t\tif (!(start_ <= *o && *o < stop_))\n> > +\t\t\t\treturn {};\n> > +\n> > +\t\t\treturn l_->data_of(*o);\n> > +\t\t}\n> > +\n> > +\t\ttemplate\n> > +\t\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> > +\t\t{\n> > +\t\t\tControlValueView v = get(ctrl.id());\n> > +\n> > +\t\t\treturn v ? std::optional(v.get()) : std::nullopt;\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] iterator begin() const\n> > +\t\t{\n> > +\t\t\treturn { l_->p_ + start_ };\n> > +\t\t}\n> > +\n> > +\t\t[[nodiscard]] iterator end() const\n> > +\t\t{\n> > +\t\t\treturn { l_->p_ + stop_ };\n> > +\t\t}\n> > +\n> > +\tprivate:\n> > +\t\tDiff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n> > +\t\t\t: l_(&l),\n> > +\t\t\t changed_(changed),\n> > +\t\t\t start_(l.contentOffset_ + oldFill),\n> > +\t\t\t stop_(l.contentOffset_ + newFill)\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tfriend MetadataList;\n> > +\t\tfriend struct Checkpoint;\n> > +\n> > +\t\tconst MetadataList *l_ = nullptr;\n> > +\t\tstd::size_t changed_;\n> > +\t\tstd::size_t start_;\n> > +\t\tstd::size_t stop_;\n> > +\t};\n> > +\n> > +\tDiff merge(const ControlList &other)\n>\n> So the \"merge\" function is how pipeline handlers adds metadata to a\n> MetadataList ?\n>\n> > +\t{\n> > +\t\t// TODO: check id map of `other`?\n>\n> This already is a proper /* \\todo */\n>\n> > +\n> > +\t\tconst auto c = checkpoint();\n> > +\n> > +\t\tfor (const auto &[tag, value] : other) {\n> > +\t\t\tauto *e = find(tag);\n> > +\t\t\tif (e) {\n> > +\t\t\t\t[[maybe_unused]] auto r = set(*e, value);\n> > +\t\t\t\tassert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n>\n> does \"r == SetError()\" means r == 0 ? What's the TODO for ?\n>\n> > +\t\t\t}\n> > +\t\t}\n> > +\n> > +\t\treturn c.diffSince();\n> > +\t}\n> > +\n> > +\tclass Checkpoint\n> > +\t{\n> > +\tpublic:\n> > +\t\t[[nodiscard]] Diff diffSince() const\n> > +\t\t{\n> > +\t\t\t/* sync with release-store on `state_` in `set()` */\n> > +\t\t\tconst auto curr = l_->state_.load(std::memory_order_acquire);\n> > +\n> > +\t\t\tassert(s_.count <= curr.count);\n> > +\t\t\tassert(s_.fill <= curr.fill);\n> > +\n> > +\t\t\treturn {\n> > +\t\t\t\t*l_,\n> > +\t\t\t\tcurr.count - s_.count,\n> > +\t\t\t\ts_.fill,\n> > +\t\t\t\tcurr.fill,\n> > +\t\t\t};\n> > +\t\t}\n> > +\n> > +\tprivate:\n> > +\t\tCheckpoint(const MetadataList &l)\n> > +\t\t\t: l_(&l),\n> > +\t\t\t s_(l.state_.load(std::memory_order_relaxed))\n> > +\t\t{\n> > +\t\t}\n> > +\n> > +\t\tfriend MetadataList;\n> > +\n> > +\t\tconst MetadataList *l_ = nullptr;\n>\n> Is there in your opinion a case where a Checkpoint can overlive the\n> MetadataList it refers to ?\n>\n> > +\t\tState s_ = {};\n> > +\t};\n> > +\n> > +\t[[nodiscard]] Checkpoint checkpoint() const\n> > +\t{\n> > +\t\treturn { *this };\n> > +\t}\n> > +\n> > +private:\n> > +\t[[nodiscard]] static constexpr std::size_t entriesOffset()\n> > +\t{\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> > +\t[[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n> > +\t{\n> > +\t\treturn details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n> > +\t}\n> > +\n> > +\t[[nodiscard]] Span entries() const\n> > +\t{\n> > +\t\treturn { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n> > +\t}\n> > +\n> > +\t[[nodiscard]] Entry *find(std::uint32_t tag) const\n> > +\t{\n> > +\t\tconst auto entries = this->entries();\n> > +\t\tauto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n>\n> how is this different from find_if() ? Just out of curiosity\n>\n> I'll stop here for the time being. Just one more note, you moved\n> MetadataListPlan out this header, so you probably need a dedicated\n> .cpp file as well\n>\n>\n> > +\t\t\treturn e.tag < tag;\n> > +\t\t});\n> > +\n> > +\t\tif (it == entries.end() || it->tag != tag)\n> > +\t\t\treturn nullptr;\n> > +\n> > +\t\treturn &*it;\n> > +\t}\n> > +\n> > +\t[[nodiscard]] ControlValueView data_of(const Entry &e) const\n> > +\t{\n> > +\t\tconst auto o = e.acquireData();\n> > +\t\treturn o ? data_of(*o) : ControlValueView{ };\n> > +\t}\n> > +\n> > +\t[[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n> > +\t{\n> > +\t\tassert(headerOffset <= alloc_ - sizeof(ValueHeader));\n> > +\t\tassert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n> > +\n> > +\t\tconst auto *vh = reinterpret_cast(p_ + headerOffset);\n> > +\t\tconst auto *p = reinterpret_cast(vh) + sizeof(*vh);\n> > +\t\tstd::size_t avail = p_ + alloc_ - p;\n> > +\n> > +\t\tconst auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n> > +\t\tassert(data);\n> > +\n> > +\t\treturn { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n> > +\t}\n> > +\n> > +\t[[nodiscard]] SetError set(Entry &e, ControlValueView v)\n> > +\t{\n> > +\t\tif (e.hasValue())\n> > +\t\t\treturn SetError::AlreadySet;\n> > +\t\tif (e.type != v.type() || e.isArray != v.isArray())\n> > +\t\t\treturn SetError::TypeMismatch;\n\nIsn't this better expressed as an assertion ?\n\n> > +\n> > +\t\tconst auto src = v.data();\n> > +\t\tif (e.isArray) {\n> > +\t\t\tif (src.size_bytes() > e.capacity)\n> > +\t\t\t\treturn SetError::SizeMismatch;\n> > +\t\t} else {\n> > +\t\t\tif (src.size_bytes() != e.capacity)\n> > +\t\t\t\treturn SetError::SizeMismatch;\n> > +\t\t}\n\nIf any of these fails, it means the pipeline handler has wrongly sized\nthe MetadataPlan, should we assert here as well ?\n\n> > +\n> > +\t\tauto s = state_.load(std::memory_order_relaxed);\n> > +\t\tstd::byte *oldEnd = p_ + contentOffset_ + s.fill;\n> > +\t\tstd::byte *p = oldEnd;\n> > +\n> > +\t\tauto *headerPtr = details::align::up(p);\n\nThis aligns to the next ValueHeader\n\n> > +\t\tauto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n\nThis to size_bytes() + alignment\n\n> > +\t\tdetails::align::up(0, alignof(ValueHeader), p);\n\nWhat does this do ?\n\n> > +\n> > +#if HAS_ASAN\n> > +\t\t::__sanitizer_annotate_contiguous_container(\n> > +\t\t\tp_ + contentOffset_, p_ + alloc_,\n> > +\t\t\toldEnd, p\n> > +\t\t);\n> > +#endif\n> > +\n> > +\t\tnew (headerPtr) ValueHeader{\n> > +\t\t\t.tag = e.tag,\n> > +\t\t\t.size = std::uint32_t(src.size_bytes()),\n> > +\t\t\t.alignment = e.alignment,\n> > +\t\t\t.params = {\n> > +\t\t\t\t.type = v.type(),\n> > +\t\t\t\t.isArray = v.isArray(),\n> > +\t\t\t\t.numElements = std::uint32_t(v.numElements()),\n> > +\t\t\t},\n> > +\t\t};\n> > +\t\tstd::memcpy(dataPtr, src.data(), src.size_bytes());\n> > +\t\te.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n\nbreak these long lines when it's trivial to do so\n\n> > +\n> > +\t\ts.fill += p - oldEnd;\n> > +\t\ts.count += 1;\n> > +\n> > +\t\tstate_.store(s, std::memory_order_release);\n> > +\n> > +\t\treturn {};\n> > +\t}\n> > +\n> > +\tstd::size_t capacity_ = 0;\n> > +\tstd::size_t contentOffset_ = -1;\n> > +\tstd::size_t alloc_ = 0;\n> > +\tstd::atomic state_ = State{};\n> > +\tstd::byte *p_ = nullptr;\n> > +\t// TODO: ControlIdMap in any way shape or form?\n\nWhat do you mean here ?\n\n> > +\n> > +\t/*\n> > +\t * If this is problematic on a 32-bit architecture, then\n> > +\t * `count` can be stored in a separate atomic variable\n> > +\t * but then `Diff::changed_` must be removed since the fill\n> > +\t * level and item count cannot be retrieved atomically.\n\nor we can use a mutex or a different lock primitive ?\n\n> > +\t */\n> > +\tstatic_assert(decltype(state_)::is_always_lock_free);\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > +\n> > +#undef HAS_ASAN\n> > diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n> > new file mode 100644\n> > index 000000000..2ed35c54f\n> > --- /dev/null\n> > +++ b/include/libcamera/metadata_list_plan.h\n> > @@ -0,0 +1,130 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2025, Ideas On Board Oy\n> > + */\n> > +\n> > +#pragma once\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +class MetadataListPlan\n> > +{\n> > +public:\n> > +\tstruct Entry {\n> > +\t\tstd::uint32_t size;\n> > +\t\tstd::uint32_t alignment; // TODO: is this necessary?\n\nI wonder if the alignment shouldn't come from controls.h associated\nwith ControlType\n\nSomething like control_type::alignment\n\nI wonder if this would simplify the handling of alignment in\nMetadataList as well\n\n> > +\t\tstd::uint32_t numElements;\n> > +\t\tControlType type;\n> > +\t\tbool isArray;\n> > +\t};\n> > +\n> > +\t[[nodiscard]] bool empty() const { return items_.empty(); }\n> > +\t[[nodiscard]] std::size_t size() const { return items_.size(); }\n> > +\t[[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n> > +\t[[nodiscard]] decltype(auto) end() const { return items_.end(); }\n> > +\tvoid clear() { items_.clear(); }\n> > +\n> > +\ttemplate<\n> > +\t\ttypename T,\n> > +\t\tstd::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n> > +\t>\n> > +\tdecltype(auto) set(const Control &ctrl)\n\nThis calls the private overload\n\n\ttemplate\n\tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n\nWhich returns\n\n *this;\n\nwhich if I'm not mistaken means \"MetadataListPlan *\". Can this function\nreturns anything else ? If the return type is known and trivial to\ntype out why hide it behind decltype(auto) ?\n\n> > +\t{\n> > +\t\tif constexpr (libcamera::details::control_type::size > 0) {\n> > +\t\t\tstatic_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n> > +\n> > +\t\t\treturn set(\n> > +\t\t\t\tctrl.id(),\n> > +\t\t\t\tlibcamera::details::control_type::size,\n> > +\t\t\t\ttrue\n> > +\t\t\t);\n> > +\t\t} else {\n> > +\t\t\treturn set(ctrl.id(), 1, false);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\ttemplate<\n> > +\t\ttypename T,\n> > +\t\tstd::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n\nis this for strings only ? It seems to me that only std::string has\ncontrol_type::size == dynamic_extent\n\n> > +\t>\n> > +\tdecltype(auto) set(const Control &ctrl, std::size_t numElements)\n> > +\t{\n> > +\t\treturn set(ctrl.id(), numElements, true);\n> > +\t}\n> > +\n> > +\t[[nodiscard]] bool set(std::uint32_t tag,\n> > +\t\t\t std::size_t size, std::size_t alignment,\n> > +\t\t\t std::size_t numElements, ControlType type, bool isArray)\n\nIs this overload meant for the public interface of the class ?\n\nIt seems that the two previous overloads call the private one\n\n\ttemplate\n\tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n\nwhich then calls this public one. Is this intentional ?\n\n> > +\t{\n> > +\t\tif (size == 0 || size > std::numeric_limits::max())\n> > +\t\t\treturn false;\n> > +\t\tif (alignment > std::numeric_limits::max())\n> > +\t\t\treturn false;\n> > +\t\tif (!details::cxx20::has_single_bit(alignment))\n> > +\t\t\treturn false;\n> > +\t\tif (numElements > std::numeric_limits::max() / size)\n> > +\t\t\treturn false;\n> > +\t\tif (!isArray && numElements != 1)\n> > +\t\t\treturn false;\n\nShould these be assertions (where and if possible) ?\n\n> > +\n> > +\t\titems_.insert_or_assign(tag, Entry{\n\nDo we allow to re-write an entry in the MetadataListPlan ?\n\n> > +\t\t\t.size = std::uint32_t(size),\n> > +\t\t\t.alignment = std::uint32_t(alignment),\n> > +\t\t\t.numElements = std::uint32_t(numElements),\n> > +\t\t\t.type = type,\n> > +\t\t\t.isArray = isArray,\n> > +\t\t});\n> > +\n> > +\t\treturn true;\n> > +\t}\n> > +\n> > +\t[[nodiscard]] const Entry *get(std::uint32_t tag) const\n> > +\t{\n> > +\t\tauto it = items_.find(tag);\n> > +\t\tif (it == items_.end())\n> > +\t\t\treturn nullptr;\n> > +\n> > +\t\treturn &it->second;\n> > +\t}\n> > +\n> > +\t[[nodiscard]] const Entry *get(const ControlId &cid) const\n> > +\t{\n> > +\t\tconst auto *e = get(cid.id());\n> > +\t\tif (!e)\n> > +\t\t\treturn nullptr;\n> > +\n> > +\t\tif (e->type != cid.type() || e->isArray != cid.isArray())\n> > +\t\t\treturn nullptr;\n\nHow can this happen ?\n\n> > +\n> > +\t\treturn e;\n> > +\t}\n> > +\n> > +private:\n> > +\tstd::map items_;\n> > +\n> > +\ttemplate\n> > +\tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> > +\t{\n> > +\t\tstatic_assert(std::is_trivially_copyable_v);\n> > +\n> > +\t\t[[maybe_unused]] bool ok = set(tag,\n> > +\t\t\t\t\t sizeof(T), alignof(T),\n> > +\t\t\t\t\t numElements, details::control_type::value, isArray);\n> > +\t\tassert(ok);\n> > +\n> > +\t\treturn *this;\n> > +\t}\n> > +};\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> > index de1eb99b2..8c5ce4503 100644\n> > --- a/src/libcamera/meson.build\n> > +++ b/src/libcamera/meson.build\n> > @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n> > 'framebuffer.cpp',\n> > 'framebuffer_allocator.cpp',\n> > 'geometry.cpp',\n> > + 'metadata_list.cpp',\n> > 'orientation.cpp',\n> > 'pixel_format.cpp',\n> > 'request.cpp',\n> > diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n> > new file mode 100644\n> > index 000000000..ebefdfdad\n> > --- /dev/null\n> > +++ b/src/libcamera/metadata_list.cpp\n> > @@ -0,0 +1,344 @@\n> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> > +/*\n> > + * Copyright (C) 2025, Ideas On Board Oy\n> > + */\n> > +\n> > +#include \n> > +\n> > +namespace libcamera {\n> > +\n> > +/**\n> > + * \\class MetadataListPlan\n> > + * \\brief Class to hold the possible set of metadata items for a MetadataList\n> > + */\n> > +\n> > +/**\n> > + * \\class MetadataListPlan::Entry\n> > + * \\brief Details of a metadata item\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\var MetadataListPlan::Entry::size\n> > + * \\brief Number of bytes in a single element\n> > + *\n> > + * \\var MetadataListPlan::Entry::alignment\n> > + * \\brief Required alignment of the elements\n> > + * \\endinternal\n> > + *\n> > + * \\var MetadataListPlan::Entry::numElements\n> > + * \\brief Number of elements in the value\n> > + * \\sa ControlValueView::numElements()\n> > + *\n> > + * \\var MetadataListPlan::Entry::type\n> > + * \\brief The type of the value\n> > + * \\sa ControlValueView::type()\n> > + *\n> > + * \\var MetadataListPlan::Entry::isArray\n> > + * \\brief Whether or not the value is array-like\n> > + * \\sa ControlValueView::isArray()\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::begin() const\n> > + * \\brief Retrieve the begin iterator\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::end() const\n> > + * \\brief Retrieve the end iterator\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::size() const\n> > + * \\brief Retrieve the number of entries\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::empty() const\n> > + * \\brief Check if empty\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataListPlan::clear()\n> > + * \\brief Remove all controls\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataListPlan::set(const Control &ctrl)\n> > + * \\brief Add an entry for the given control to the metadata list plan\n> > + * \\param[in] ctrl The control\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n> > + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n> > + * \\param[in] ctrl The control\n> > + * \\param[in] count The maximum number of elements\n> > + *\n> > + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n> > + * capacity of \\a count elements.\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataListPlan::set(std::uint32_t tag,\n> > + *\t\t\t std::size_t size, std::size_t alignment,\n> > + *\t\t\t std::size_t count, ControlType type, bool isArray)\n> > + * \\brief Add an entry to the metadata list plan\n> > + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n> > + * would result in an invalid entry\n> > + *\n> > + * This functions adds an entry with essentially arbitrary parameters, without deriving\n> > + * them from a given ControlId instance. This is mainly used when deserializing.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n> > + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataListPlan::get(const ControlId &cid) const\n> > + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n> > + *\n> > + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n> > + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n> > + */\n> > +\n> > +/**\n> > + * \\class MetadataList\n> > + * \\brief Class to hold metadata items\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n> > + * \\brief Construct a metadata list according to \\a plan\n> > + *\n> > + * Construct a metadata list according to the provided \\a plan.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::size() const\n> > + * \\brief Retrieve the number of controls\n> > + * \\context This function is \\threadsafe.\n> > + * \\note If the list is being modified, the return value may be out of\n> > + * date by the time the function returns\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::empty() const\n> > + * \\brief Check if empty\n> > + * \\context This function is \\threadsafe.\n> > + * \\note If the list is being modified, the return value may be out of\n> > + * date by the time the function returns\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::clear()\n> > + * \\brief Remove all items from the list\n> > + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n> > + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n> > + * when this function is called.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::begin() const\n> > + * \\brief Retrieve begin iterator\n> > + * \\context This function is \\threadsafe.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::end() const\n> > + * \\brief Retrieve end iterator\n> > + * \\context This function is \\threadsafe.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::get(const Control &ctrl) const\n> > + * \\brief Get the value of control \\a ctrl\n> > + * \\return A std::optional containing the control value, or std::nullopt if\n> > + * the control \\a ctrl is not present in the list\n> > + * \\context This function is \\threadsafe.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::get(std::uint32_t tag) const\n> > + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n> > + * \\return A std::optional containing the control value, or std::nullopt if\n> > + * the control is not present in the list\n> > + * \\context This function is \\threadsafe.\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> > + * \\brief Set the value of control \\a ctrl to \\a value\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n> > + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::merge(const ControlList &other)\n> > + * \\brief Add all missing items from \\a other\n> > + *\n> > + * Add all items from \\a other that are not present in \\a this.\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\enum MetadataList::SetError\n> > + * \\brief Error code returned by a set operation\n> > + *\n> > + * \\var MetadataList::SetError::UnknownTag\n> > + * \\brief The tag is not supported by the metadata list\n> > + * \\var MetadataList::SetError::AlreadySet\n> > + * \\brief A value has already been added with the given tag\n> > + * \\var MetadataList::SetError::SizeMismatch\n> > + * \\brief The size of the data is not appropriate for the given tag\n> > + * \\var MetadataList::SetError::TypeMismatch\n> > + * \\brief The type of the value does not match the expected type\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::checkpoint() const\n> > + * \\brief Create a checkpoint\n> > + * \\context This function is \\threadsafe.\n> > + */\n> > +\n> > +/**\n> > + * \\class MetadataList::iterator\n> > + * \\brief Iterator\n> > + */\n> > +\n> > +/**\n> > + * \\typedef MetadataList::iterator::difference_type\n> > + * \\brief iterator's difference type\n> > + */\n> > +\n> > +/**\n> > + * \\typedef MetadataList::iterator::value_type\n> > + * \\brief iterator's value type\n> > + */\n> > +\n> > +/**\n> > + * \\typedef MetadataList::iterator::pointer\n> > + * \\brief iterator's pointer type\n> > + */\n> > +\n> > +/**\n> > + * \\typedef MetadataList::iterator::reference\n> > + * \\brief iterator's reference type\n> > + */\n> > +\n> > +/**\n> > + * \\typedef MetadataList::iterator::iterator_category\n> > + * \\brief iterator's category\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::iterator::operator*()\n> > + * \\brief Retrieve value at iterator\n> > + * \\return A \\a ControlListView representing the value\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n> > + * \\brief Check if two iterators are equal\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n> > + * \\brief Check if two iterators are not equal\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::iterator::operator++(int)\n> > + * \\brief Advance the iterator\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::iterator::operator++()\n> > + * \\brief Advance the iterator\n> > + */\n> > +\n> > +/**\n> > + * \\class MetadataList::Diff\n> > + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n> > + * \\sa Camera::metadataAvailable\n> > + * \\internal\n> > + * \\sa MetadataList::Checkpoint::diffSince()\n> > + * \\endinternal\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::list() const\n> > + * \\brief Retrieve the associated MetadataList\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::size() const\n> > + * \\brief Retrieve the number of metadata items designated\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::empty() const\n> > + * \\brief Check if any metadata items are designated\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::operator bool() const\n> > + * \\copydoc MetadataList::Diff::empty() const\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n> > + * \\copydoc MetadataList::get(const Control &ctrl) const\n> > + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> > + * even if it is otherwise present in the backing MetadataList.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n> > + * \\copydoc MetadataList::get(std::uint32_t tag) const\n> > + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> > + * even if it is otherwise present in the backing MetadataList.\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::begin() const\n> > + * \\brief Retrieve the begin iterator\n> > + */\n> > +\n> > +/**\n> > + * \\fn MetadataList::Diff::end() const\n> > + * \\brief Retrieve the end iterator\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\class MetadataList::Checkpoint\n> > + * \\brief Designates a point in the stream of metadata items\n> > + *\n> > + * A Checkpoint object designates a point in the stream of metadata items in the associated\n> > + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n> > + * added to the list after the designated point using diffSince().\n> > + */\n> > +\n> > +/**\n> > + * \\internal\n> > + * \\fn MetadataList::Checkpoint::diffSince() const\n> > + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n> > + */\n> > +\n> > +} /* namespace libcamera */\n> > diff --git a/test/controls/meson.build b/test/controls/meson.build\n> > index 763f8905e..b68a4fc53 100644\n> > --- a/test/controls/meson.build\n> > +++ b/test/controls/meson.build\n> > @@ -5,6 +5,7 @@ control_tests = [\n> > {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n> > {'name': 'control_list', 'sources': ['control_list.cpp']},\n> > {'name': 'control_value', 'sources': ['control_value.cpp']},\n> > + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n> > ]\n> >\n> > foreach test : control_tests\n> > diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n> > new file mode 100644\n> > index 000000000..f0872acd9\n> > --- /dev/null\n> > +++ b/test/controls/metadata_list.cpp\n> > @@ -0,0 +1,170 @@\n> > +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> > +/*\n> > + * Copyright (C) 2025, Ideas On Board Oy\n> > + *\n> > + * MetadataList tests\n> > + */\n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \n> > +#include \n> > +#include \n> > +\n> > +#include \"test.h\"\n> > +\n> > +using namespace std;\n> > +using namespace libcamera;\n> > +\n> > +#define ASSERT(x) do { \\\n> > +\tif (!static_cast(x)) { \\\n> > +\t\tstd::cerr << '`' << #x << \"` failed\" << std::endl; \\\n> > +\t\treturn TestFail; \\\n> > +\t} \\\n> > +} while (false)\n> > +\n> > +class MetadataListTest : public Test\n> > +{\n> > +public:\n> > +\tMetadataListTest() = default;\n> > +\n> > +protected:\n> > +\tint run() override\n> > +\t{\n> > +\t\tMetadataListPlan mlp;\n> > +\t\tmlp.set(controls::ExposureTime);\n> > +\t\tmlp.set(controls::ExposureValue);\n> > +\t\tmlp.set(controls::ColourGains);\n> > +\t\tmlp.set(controls::AfWindows, 10);\n> > +\t\tmlp.set(controls::AeEnable);\n> > +\t\tmlp.set(controls::SensorTimestamp);\n> > +\n> > +\t\tMetadataList ml(mlp);\n> > +\n> > +\t\tstatic_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n> > +\t\tASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n\nCan you add a comment to explain what this tests ?\n\n> > +\n> > +\t\tASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n> > +\t\tASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n> > +\n> > +\t\tauto f1 = std::async(std::launch::async, [&] {\n> > +\t\t\tusing namespace std::chrono_literals;\n> > +\n> > +\t\t\tstd::this_thread::sleep_for(500ms);\n> > +\t\t\tASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n> > +\n> > +\t\t\tstd::this_thread::sleep_for(500ms);\n> > +\t\t\tASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n> > +\n> > +\t\t\tstd::this_thread::sleep_for(500ms);\n> > +\t\t\tASSERT(ml.set(controls::ColourGains, std::array{\n> > +\t\t\t\t123.f,\n> > +\t\t\t\t456.f\n> > +\t\t\t}) == MetadataList::SetError());\n> > +\n> > +\t\t\tstd::this_thread::sleep_for(500ms);\n> > +\t\t\tASSERT(ml.set(controls::AfWindows, std::array{\n> > +\t\t\t\tRectangle(),\n> > +\t\t\t\tRectangle(1, 2, 3, 4),\n> > +\t\t\t\tRectangle(0x1111, 0x2222, 0x3333, 0x4444),\n> > +\t\t\t}) == MetadataList::SetError());\n\nI don't get why you expect these to fail..\n\n> > +\n> > +\t\t\treturn TestPass;\n> > +\t\t});\n> > +\n> > +\t\tauto f2 = std::async(std::launch::async, [&] {\n> > +\t\t\tfor (;;) {\n> > +\t\t\t\tconst auto x = ml.get(controls::ExposureTime);\n> > +\t\t\t\tconst auto y = ml.get(controls::ExposureValue);\n> > +\t\t\t\tconst auto z = ml.get(controls::ColourGains);\n> > +\t\t\t\tconst auto w = ml.get(controls::AfWindows);\n> > +\n> > +\t\t\t\tif (x)\n> > +\t\t\t\t\tASSERT(*x == 0x1111);\n> > +\n> > +\t\t\t\tif (y)\n> > +\t\t\t\t\tASSERT(*y == 1.0f);\n> > +\n> > +\t\t\t\tif (z) {\n> > +\t\t\t\t\tASSERT(z->size() == 2);\n> > +\t\t\t\t\tASSERT((*z)[0] == 123.f);\n> > +\t\t\t\t\tASSERT((*z)[1] == 456.f);\n> > +\t\t\t\t}\n> > +\n> > +\t\t\t\tif (w) {\n> > +\t\t\t\t\tASSERT(w->size() == 3);\n> > +\t\t\t\t\tASSERT((*w)[0].isNull());\n> > +\t\t\t\t\tASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n> > +\t\t\t\t\tASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n> > +\t\t\t\t}\n> > +\n> > +\t\t\t\tif (x && y && z && w)\n> > +\t\t\t\t\tbreak;\n> > +\t\t\t}\n> > +\n> > +\t\t\treturn TestPass;\n\nand if f1 is expected to fail how can f2 pass.\n\nI'm surely missing something\n\n> > +\t\t});\n> > +\n> > +\t\tASSERT(f1.get() == TestPass);\n> > +\t\tASSERT(f2.get() == TestPass);\n> > +\n> > +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n> > +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n> > +\n> > +\t\tASSERT(ml.get(controls::ExposureTime) == 0x1111);\n> > +\t\tASSERT(ml.get(controls::ExposureValue) == 1);\n> > +\n> > +\t\tfor (auto &&[tag, v] : ml)\n> > +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n\nAre we sure we want this in tests ? I mean, it doesn't hurt..\n\n> > +\n> > +\t\tstd::cout << std::endl;\n> > +\n> > +\t\tml.clear();\n> > +\t\tASSERT(ml.empty());\n> > +\t\tASSERT(ml.size() == 0);\n> > +\n> > +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n> > +\t\tASSERT(ml.get(controls::ExposureTime) == 0x2222);\n> > +\n> > +\t\tauto c = ml.checkpoint();\n> > +\n> > +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n> > +\t\tASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n> > +\n> > +\t\tauto d = c.diffSince();\n> > +\t\tASSERT(&d.list() == &ml);\n> > +\n> > +\t\tASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n> > +\n> > +\t\tASSERT(d);\n> > +\t\tASSERT(!d.empty());\n> > +\t\tASSERT(d.size() == 2);\n> > +\t\tASSERT(!d.get(controls::ExposureTime));\n> > +\t\tASSERT(!d.get(controls::ColourGains));\n> > +\t\tASSERT(!d.get(controls::AfWindows));\n> > +\t\tASSERT(d.get(controls::ExposureValue) == 2);\n> > +\t\tASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n> > +\n> > +\t\tfor (auto &&[tag, v] : d)\n> > +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> > +\n> > +\t\t/* Test if iterators work with algorithms. */\n> > +\t\tstd::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n> > +\t\t\treturn false;\n> > +\t\t});\n> > +\n> > +#if 0\n> > + {\n> > + auto it = ml.begin();\n> > + ml.clear();\n> > + std::ignore = *it; /* Trigger ASAN. */\n> > + }\n> > +#endif\n\nMaybe remove this ?\n\n> > +\n> > +\t\treturn TestPass;\n> > +\t}\n> > +};\n> > +\n> > +TEST_REGISTER(MetadataListTest)\n\nThanks again for your hard work on this!\n\n> > --\n> > 2.50.1\n> >","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 557E6C3237\n\tfor ;\n\tMon, 28 Jul 2025 08:36:23 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 1864C69141;\n\tMon, 28 Jul 2025 10:36:22 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id DECAB6146B\n\tfor ;\n\tMon, 28 Jul 2025 10:36:17 +0200 (CEST)","from ideasonboard.com (mob-5-90-139-29.net.vodafone.it\n\t[5.90.139.29])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 35374169;\n\tMon, 28 Jul 2025 10:35:35 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"LRE2t9Zq\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1753691735;\n\tbh=Se1u5OBv/AljIbqo+YYddiAn4qYPvH/oAjy/FSjne1M=;\n\th=Date:From:To:Cc:Subject:References:In-Reply-To:From;\n\tb=LRE2t9Zq/TbuuCO/z901IniIChQlMl9sbJWyBV4X2DqzxkBcDwS+XruwZad3rmGYu\n\tfMI4WcJ6NTDCLWRto9UqYs69UPaRuiNMZfwXAMQ0IujvE2wGyNInOAn1Gt4UjtyzAs\n\t3kOsW0ZLFhp0eyHi2M3ZoPG7FGBZHbTzBso21m0E=","Date":"Mon, 28 Jul 2025 10:36:10 +0200","From":"Jacopo Mondi ","To":"Jacopo Mondi ","Cc":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ,\n\tlibcamera-devel@lists.libcamera.org","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","Message-ID":"","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>\n\t<2lweaavht65vexto5246wcvmxjoxhwgtprz5hjvbrz7y4h4hrk@r3rwki5uoefi>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"<2lweaavht65vexto5246wcvmxjoxhwgtprz5hjvbrz7y4h4hrk@r3rwki5uoefi>","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35231,"web_url":"https://patchwork.libcamera.org/comment/35231/","msgid":"<1dce550f-c132-460c-8dc4-865df3cab60a@ideasonboard.com>","date":"2025-07-29T12:11:30","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":216,"url":"https://patchwork.libcamera.org/api/people/216/","name":"Barnabás Pőcze","email":"barnabas.pocze@ideasonboard.com"},"content":"Hi\n\n2025. 07. 28. 10:36 keltezéssel, Jacopo Mondi írta:\n> Hi Barnabás\n> let me resume the review from where I stopped last time\n> \n> On Fri, Jul 25, 2025 at 06:08:28PM +0200, Jacopo Mondi wrote:\n>> Hi Barnabás\n>>\n>> On Mon, Jul 21, 2025 at 12:46:07PM +0200, Barnabás Pőcze wrote:\n>>> Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n>>> reported by a camera for a given request. Previously, a `ControlList` was\n>>> used for this purpose. The reason for introducing a separate type is to\n>>> simplify the access to the returned metadata during the entire lifetime\n>>> of a request.\n>>>\n>>> Specifically, for early metadata completion to be easily usable it should be\n>>> guaranteed that any completed metadata item can be accessed and looked up\n>>> at least until the associated requested is reused with `Request::reuse()`.\n>>>\n>>> However, when a metadata item is completed early, the pipeline handler\n>>> might still work on the request in the `CameraManager`'s private thread,\n>>> therefore there is an inherent synchronization issue when an application\n>>> accesses early metadata.\n>>>\n>>> Restricting the user to only access the metadata items of a not yet completed\n>>> request in the early metadata availability signal handler by ways of\n>>> documenting or enforcing it at runtime could be an option, but it is not\n>>> too convenient for the user.\n>>>\n>>> The current `ControlList` implementation employs an `std::unordered_map`,\n>>> so pointers remain stable when the container is modified, so an application\n>>> could keep accessing particular metadata items outside the signal handler,\n>>> but this fact is far from obvious, and the user would still not be able\n>>> to make a copy of all metadata or do lookups based on the numeric ids or\n>>> the usual `libcamera::Control<>` objects, thus some type safety is lost.\n>>>\n>>> The above also requires that each metadata item is only completed once for\n>>> a given request, but this does not appear to be serious limitation,\n>>> and in fact, this restriction is enforced by `MetadataList`.\n>>>\n>>> The introduced `MetadataList` supports single writer - multiple reader\n>>> scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n>>> without introducing data races or other synchronization issues. This is\n>>> achieved by requiring the possible set of metadata items to be known\n>>> (such set is stored in a `MetadataListPlan` object). Based on the this\n>>> plan, a single contiguous allocation is made to accommodate all potential\n>>> metadata items. Due to this single contiguous allocation that is not modified\n>>> during the lifetime of a `MetadataList` and atomic modifications, it is\n>>> possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n>>> there is only ever a single writer.\n>>>\n>>> Signed-off-by: Barnabás Pőcze \n>>> ---\n>>> changes in v2:\n>>> * remove multiple not strictly necessary functions\n>>> ---\n>>> include/libcamera/meson.build | 2 +\n>>> include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n>>> include/libcamera/metadata_list_plan.h | 130 ++++++\n>>> src/libcamera/meson.build | 1 +\n>>> src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n>>> test/controls/meson.build | 1 +\n>>> test/controls/metadata_list.cpp | 170 ++++++++\n>>> 7 files changed, 1195 insertions(+)\n>>> create mode 100644 include/libcamera/metadata_list.h\n>>> create mode 100644 include/libcamera/metadata_list_plan.h\n>>> create mode 100644 src/libcamera/metadata_list.cpp\n>>> create mode 100644 test/controls/metadata_list.cpp\n>>>\n>>> diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n>>> index 30ea76f94..410b548dd 100644\n>>> --- a/include/libcamera/meson.build\n>>> +++ b/include/libcamera/meson.build\n>>> @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n>>> 'framebuffer_allocator.h',\n>>> 'geometry.h',\n>>> 'logging.h',\n>>> + 'metadata_list.h',\n>>> + 'metadata_list_plan.h',\n>>> 'orientation.h',\n>>> 'pixel_format.h',\n>>> 'request.h',\n>>> diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n>>> new file mode 100644\n>>> index 000000000..7fe3dbbab\n>>> --- /dev/null\n>>> +++ b/include/libcamera/metadata_list.h\n>>> @@ -0,0 +1,547 @@\n>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>> +/*\n>>> + * Copyright (C) 2025, Ideas On Board Oy\n>>> + *\n>>> + * Metadata list\n>>> + */\n>>> +\n>>> +#pragma once\n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \n>>> +#include \n>>> +\n>>> +// TODO: want this?\n>>> +#if __has_include()\n>>> +#if __SANITIZE_ADDRESS__ /* gcc */\n>>> +#include \n>>> +#define HAS_ASAN 1\n>>> +#elif defined(__has_feature)\n>>> +#if __has_feature(address_sanitizer) /* clang */\n>>> +#include \n>>> +#define HAS_ASAN 1\n>>> +#endif\n>>> +#endif\n>>> +#endif\n>>> +\n>>> +namespace libcamera {\n>>> +\n>>> +class MetadataList\n>>> +{\n>>> +private:\n>>> +\tstruct ValueParams {\n>>> +\t\tControlType type;\n>>> +\t\tbool isArray;\n>>> +\t\tstd::uint32_t numElements;\n>>> +\t};\n>>> +\n>>> +\tstruct Entry {\n>>> +\t\tconst std::uint32_t tag;\n>>> +\t\tconst std::uint32_t capacity;\n>>> +\t\tconst std::uint32_t alignment;\n>>> +\t\tconst ControlType type;\n>>> +\t\tbool isArray;\n>>> +\n>>> +\t\tstatic constexpr std::uint32_t invalidOffset = -1;\n>>> +\t\t/*\n>>> +\t\t * Offset from the beginning of the allocation, and\n>>> +\t\t * and _not_ relative to `contentOffset_`.\n>>> +\t\t */\n>>> +\t\tstd::atomic_uint32_t headerOffset = invalidOffset;\n>>> +\n>>> +\t\t[[nodiscard]] std::optional hasValue() const\n>>> +\t\t{\n>>> +\t\t\tauto offset = headerOffset.load(std::memory_order_relaxed);\n>>> +\t\t\tif (offset == invalidOffset)\n>>> +\t\t\t\treturn {};\n>>> +\n>>> +\t\t\treturn offset;\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] std::optional acquireData() const\n>>> +\t\t{\n>>> +\t\t\tauto offset = hasValue();\n>>> +\t\t\tif (offset) {\n>>> +\t\t\t\t/* sync with release-store on `headerOffset` in `MetadataList::set()` */\n>>> +\t\t\t\tstd::atomic_thread_fence(std::memory_order_acquire);\n>>> +\t\t\t}\n>>> +\n>>> +\t\t\treturn offset;\n>>> +\t\t}\n>>> +\t};\n>>> +\n>>> +\tstruct ValueHeader {\n>>> +\t\tstd::uint32_t tag;\n>>> +\t\tstd::uint32_t size;\n>>> +\t\tstd::uint32_t alignment;\n>>> +\t\tValueParams params;\n>>> +\t};\n>>> +\n>>> +\tstruct State {\n>>> +\t\tstd::uint32_t count;\n>>> +\t\tstd::uint32_t fill;\n>>> +\t};\n>>> +\n>>> +public:\n>>> +\texplicit MetadataList(const MetadataListPlan &plan)\n>>> +\t\t: capacity_(plan.size()),\n>>> +\t\t contentOffset_(MetadataList::contentOffset(capacity_)),\n>>> +\t\t alloc_(contentOffset_)\n>>> +\t{\n>>> +\t\tfor (const auto &[tag, e] : plan) {\n>>> +\t\t\talloc_ += sizeof(ValueHeader);\n>>> +\t\t\talloc_ += e.alignment - 1; // XXX: this is the maximum\n>>> +\t\t\talloc_ += e.size * e.numElements;\n>>> +\t\t\talloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n>>> +\t\t}\n>>> +\n>>> +\t\tp_ = static_cast(::operator new(alloc_));\n>>> +\n>>> +\t\tauto *entries = reinterpret_cast(p_ + entriesOffset());\n>>> +\t\tauto it = plan.begin();\n>>> +\n>>> +\t\tfor (std::size_t i = 0; i < capacity_; i++, ++it) {\n>>> +\t\t\tconst auto &[tag, e] = *it;\n>>> +\n>>> +\t\t\tnew (&entries[i]) Entry{\n>>> +\t\t\t\t.tag = tag,\n>>> +\t\t\t\t.capacity = e.size * e.numElements,\n>>> +\t\t\t\t.alignment = e.alignment,\n>>> +\t\t\t\t.type = e.type,\n>>> +\t\t\t\t.isArray = e.isArray,\n>>> +\t\t\t};\n>>\n>> I was already about to rant against C++'s \"do the same usual thing in\n>> some different way\" style and wanted to suggest\n>>\n>> \t\t\tentries[i].tag = tag;\n>> \t\t\tentries[i].capacity = e.size * e.numElements,\n>> \t\t\tentries[i].alignment = e.alignment;\n>> \t\t\tentries[i].type = e.type;\n>> \t\t\tentries[i].isArray = e.isArray;\n>>\n>> But then you loose the const-ness of entries[i]'s fields.\n>>\n>> Let's use \"placement new\" then...\n\nDue to the way everything is in one allocation, the objects have to be\nconstructed manually, so even without the `const` fields, something like\nthis would be required. Technically the `entries` could be move into a\nseparate allocation with e.g. an std::vector or std::unique_ptr.\n\n\n>>\n>>\n>>> +\t\t}\n>>> +\n>>> +#if HAS_ASAN\n>>> +\t\t::__sanitizer_annotate_contiguous_container(\n>>> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n>>> +\t\t\tp_ + alloc_, p_ + contentOffset_\n>>> +\t\t);\n>>> +#endif\n>>> +\t}\n>>> +\n>>> +\tMetadataList(const MetadataList &) = delete;\n>>> +\tMetadataList(MetadataList &&) = delete;\n>>> +\n>>> +\tMetadataList &operator=(const MetadataList &) = delete;\n>>> +\tMetadataList &operator=(MetadataList &&) = delete;\n>>> +\n>>> +\t~MetadataList()\n>>> +\t{\n>>> +#if HAS_ASAN\n>>> +\t\t/*\n>>> +\t\t * The documentation says the range apparently has to be\n>>> +\t\t * restored to its initial state before it is deallocated.\n>>> +\t\t */\n>>> +\t\t::__sanitizer_annotate_contiguous_container(\n>>> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n>>> +\t\t\tp_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n>>> +\t\t);\n>>> +#endif\n>>> +\n>>> +\t\t::operator delete(p_, alloc_);\n>>> +\t}\n>>> +\n>>> +\t// TODO: want these?\n>>\n>> Why not ?\n\nThis is related to the `is_always_lock_free` assertion below.\nDepending on what works on the target platforms, the interface\nmay need adjustments. I think `empty()` could always be provided,\nbut `size()` may not.\n\n\n>>\n>>> +\t[[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n>>> +\t[[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n>>> +\n>>> +\tenum class SetError {\n>>> +\t\tUnknownTag = 1,\n>>> +\t\tAlreadySet,\n>>> +\t\tSizeMismatch,\n>>> +\t\tTypeMismatch,\n>>> +\t};\n>>\n>> The only user of \"set\" will be pipeline handler, so I wonder if coding\n>> the errors as enum is really something that gives any value here or\n>> it's just overdesign. It's not like pipelines will do\n>>\n>> if (AlreadySet)\n>> doSomething\n>>\n>> If they get an error when adding metadata it will be caught during\n>> development I presume.\n\nSee `MetadataList::set(Entry &e, ControlValueView v)`.\n\n\n>>\n>>> +\n>>> +\t[[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n>>\n>> Since we end up copying the data into the MetadataList, why use a\n>> ControlValueView and not a ControlValue ?\n\nPlease see below the other `set()`, I believe it's answered there.\n\n\n>>\n>>> +\t{\n>>> +\t\tauto *e = find(tag);\n>>> +\t\tif (!e)\n>>> +\t\t\treturn SetError::UnknownTag;\n>>> +\n>>> +\t\treturn set(*e, v);\n>>> +\t}\n>>> +\n>>> +\ttemplate\n>>> +\t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n>>\n>> Why TODO ?\n>>\n>> Here and everywhere else could you shorten lines length where is\n>> trivial to do so ?\n>>\n>> \t/* TODO: [[nodiscard]] */ SetError set(const Control &ctrl,\n>> const details::cxx20::type_identity_t &value)\n>>\n>>\n>> What does type_identity_t<> gives us here ? Isn't T fully specified ?\n\nIt prevents potential deduction failures and e.g. allows the use of\ninitializer lists.\n\n\n>>\n>>> +\t{\n>>> +\t\tusing TypeInfo = libcamera::details::control_type;\n>>> +\n>>> +\t\tif constexpr (TypeInfo::size > 0) {\n>>> +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n>>> +\n>>> +\t\t\treturn set(ctrl.id(), {\n>>> +\t\t\t\tTypeInfo::value,\n>>> +\t\t\t\ttrue,\n>>> +\t\t\t\tvalue.size(),\n>>> +\t\t\t\treinterpret_cast(value.data()),\n>>> +\t\t\t});\n>>> +\t\t} else {\n>>> +\t\t\tstatic_assert(std::is_trivially_copyable_v);\n>>> +\n>>> +\t\t\treturn set(ctrl.id(), {\n>>> +\t\t\t\tTypeInfo::value,\n>>> +\t\t\t\tfalse,\n>>> +\t\t\t\t1,\n>>> +\t\t\t\treinterpret_cast(&value),\n>>> +\t\t\t});\n>>> +\t\t}\n>>\n>> The private set() overload works with Views to avoid copies here, I\n>> get it, but as said before should the public set(tag, value) accept a\n>> ControlValue & or do we want callers to go through a View ?\n\nA `ControlValue` is implicitly convertible to `ControlValueView`, so\nI think this works out fine.\n\n\n>>\n>>> +\t}\n>>> +\n>>> +\ttemplate\n>>> +\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n>>\n>> isn't this simply an\n>> \t[[nodiscard]] std::optional get(const Control &ctrl) const\n>>\n>> why use decltype(auto) ?\n\nIt is.\n\n\n>>\n>>> +\t{\n>>> +\t\tControlValueView v = get(ctrl.id());\n>>> +\n>>> +\t\treturn v ? std::optional(v.get()) : std::nullopt;\n>>> +\t}\n>>> +\n>>> +\t// TODO: operator ControlListView() const ?\n>>> +\t// TODO: explicit operator ControlList() const ?\n>>\n>> Do these still apply ?\n\nNo use case has come up so far, so I think these can be removed\nfor now.\n\n\n>>\n>>> +\n>>> +\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n>>> +\t{\n>>> +\t\tconst auto *e = find(tag);\n>>> +\t\tif (!e)\n>>> +\t\t\treturn {};\n>>> +\n>>> +\t\treturn data_of(*e);\n>>\n>> no snake_case but use camelCase please\n\ndone\n\n\n>>\n>>> +\t}\n>>> +\n>>> +\tvoid clear()\n>>> +\t{\n>>> +\t\tfor (auto &e : entries())\n>>> +\t\t\te.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n>>> +\n>>> +\t\t[[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n>>> +\n>>> +#if HAS_ASAN\n>>> +\t\t::__sanitizer_annotate_contiguous_container(\n>>> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n>>> +\t\t\tp_ + contentOffset_ + s.fill, p_ + contentOffset_\n>>> +\t\t);\n>>> +#endif\n>>> +\t}\n>>> +\n>>> +\tclass iterator\n>>> +\t{\n>>> +\tpublic:\n>>> +\t\tusing difference_type = std::ptrdiff_t;\n>>> +\t\tusing value_type = std::pair;\n>>> +\t\tusing pointer = void;\n>>> +\t\tusing reference = value_type;\n>>\n>> Why can't you use 'value_type' and need another indirection symbol ?\n>>\n>> Are those symbols required when defining a custom iterator ?\n\nYes.\n\n\n>>\n>> Also shouldn't this be \"value_type &\" ?\n\nThis iterator cannot provide references since it creates the values on the fly.\nSetting `reference = value_type` is probably not entirely correct, but it tends\nto work with most (STL) algorithms.\n\n\n\n>>\n>>> +\t\tusing iterator_category = std::forward_iterator_tag;\n>>> +\n>>> +\t\titerator() = default;\n>>> +\n>>> +\t\titerator& operator++()\n>>> +\t\t{\n>>> +\t\t\tconst auto &h = header();\n>>> +\n>>> +\t\t\tp_ += sizeof(h);\n>>> +\t\t\tp_ = details::align::up(p_, h.alignment);\n>>> +\t\t\tp_ += h.size;\n>>> +\t\t\tp_ = details::align::up(p_, alignof(decltype(h)));\n>>> +\n>>> +\t\t\treturn *this;\n>>> +\t\t}\n>>> +\n>>> +\t\titerator operator++(int)\n>>> +\t\t{\n>>> +\t\t\tauto copy = *this;\n>>> +\t\t\t++*this;\n>>> +\t\t\treturn copy;\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] reference operator*() const\n>>> +\t\t{\n>>> +\t\t\tconst auto &h = header();\n>>> +\t\t\tconst auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n>>> +\n>>> +\t\t\treturn { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] bool operator==(const iterator &other) const\n>>> +\t\t{\n>>> +\t\t\treturn p_ == other.p_;\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] bool operator!=(const iterator &other) const\n>>> +\t\t{\n>>> +\t\t\treturn !(*this == other);\n>>> +\t\t}\n>>> +\n>>> +\tprivate:\n>>> +\t\titerator(const std::byte *p)\n>>> +\t\t\t: p_(p)\n>>> +\t\t{\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] const ValueHeader &header() const\n>>> +\t\t{\n>>> +\t\t\treturn *reinterpret_cast(p_);\n>>> +\t\t}\n>>> +\n>>> +\t\tfriend MetadataList;\n>>> +\n>>> +\t\tconst std::byte *p_ = nullptr;\n>>> +\t};\n>>> +\n>>> +\t[[nodiscard]] iterator begin() const\n>>> +\t{\n>>> +\t\treturn { p_ + contentOffset_ };\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] iterator end() const\n>>> +\t{\n>>> +\t\treturn { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n>>> +\t}\n>>> +\n>>> +\tclass Diff\n>>> +\t{\n>>> +\tpublic:\n>>> +\t\t// TODO: want these?\n>>\n>> Why not you think ?\n\nI am not entirely sure how useful they are, e.g. whether or not\n`size()` can be provided is kind of depends on the `is_always_lock_free`\nassertion below.\n\n\n>>\n>>> +\t\t[[nodiscard]] explicit operator bool() const { return !empty(); }\n>>> +\t\t[[nodiscard]] bool empty() const { return start_ == stop_; }\n>>> +\t\t[[nodiscard]] std::size_t size() const { return changed_; }\n>>> +\t\t[[nodiscard]] const MetadataList &list() const { return *l_; }\n>>> +\n>>> +\t\t[[nodiscard]] ControlValueView get(std::uint32_t tag) const\n>>> +\t\t{\n>>> +\t\t\tconst auto *e = l_->find(tag);\n>>> +\t\t\tif (!e)\n>>> +\t\t\t\treturn {};\n>>> +\n>>> +\t\t\tauto o = e->acquireData();\n>>> +\t\t\tif (!o)\n>>> +\t\t\t\treturn {};\n>>> +\n>>> +\t\t\tif (!(start_ <= *o && *o < stop_))\n>>> +\t\t\t\treturn {};\n>>> +\n>>> +\t\t\treturn l_->data_of(*o);\n>>> +\t\t}\n>>> +\n>>> +\t\ttemplate\n>>> +\t\t[[nodiscard]] decltype(auto) get(const Control &ctrl) const\n>>> +\t\t{\n>>> +\t\t\tControlValueView v = get(ctrl.id());\n>>> +\n>>> +\t\t\treturn v ? std::optional(v.get()) : std::nullopt;\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] iterator begin() const\n>>> +\t\t{\n>>> +\t\t\treturn { l_->p_ + start_ };\n>>> +\t\t}\n>>> +\n>>> +\t\t[[nodiscard]] iterator end() const\n>>> +\t\t{\n>>> +\t\t\treturn { l_->p_ + stop_ };\n>>> +\t\t}\n>>> +\n>>> +\tprivate:\n>>> +\t\tDiff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n>>> +\t\t\t: l_(&l),\n>>> +\t\t\t changed_(changed),\n>>> +\t\t\t start_(l.contentOffset_ + oldFill),\n>>> +\t\t\t stop_(l.contentOffset_ + newFill)\n>>> +\t\t{\n>>> +\t\t}\n>>> +\n>>> +\t\tfriend MetadataList;\n>>> +\t\tfriend struct Checkpoint;\n>>> +\n>>> +\t\tconst MetadataList *l_ = nullptr;\n>>> +\t\tstd::size_t changed_;\n>>> +\t\tstd::size_t start_;\n>>> +\t\tstd::size_t stop_;\n>>> +\t};\n>>> +\n>>> +\tDiff merge(const ControlList &other)\n>>\n>> So the \"merge\" function is how pipeline handlers adds metadata to a\n>> MetadataList ?\n\nI think it's mainly `metadataAvailable()`, but yes, most pipeline handlers\nwill go through this merge() function when adding metadata from the IPA.\n\n\n>>\n>>> +\t{\n>>> +\t\t// TODO: check id map of `other`?\n>>\n>> This already is a proper /* \\todo */\n\nMy editor does not highlight \"\\todo\" :(\n\n>>\n>>> +\n>>> +\t\tconst auto c = checkpoint();\n>>> +\n>>> +\t\tfor (const auto &[tag, value] : other) {\n>>> +\t\t\tauto *e = find(tag);\n>>> +\t\t\tif (e) {\n>>> +\t\t\t\t[[maybe_unused]] auto r = set(*e, value);\n>>> +\t\t\t\tassert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n>>\n>> does \"r == SetError()\" means r == 0 ? What's the TODO for ?\n\nThe TODO is about how to handle (and which) errors here.\nSee `MetadataList::set()`.\n\n\n>>\n>>> +\t\t\t}\n>>> +\t\t}\n>>> +\n>>> +\t\treturn c.diffSince();\n>>> +\t}\n>>> +\n>>> +\tclass Checkpoint\n>>> +\t{\n>>> +\tpublic:\n>>> +\t\t[[nodiscard]] Diff diffSince() const\n>>> +\t\t{\n>>> +\t\t\t/* sync with release-store on `state_` in `set()` */\n>>> +\t\t\tconst auto curr = l_->state_.load(std::memory_order_acquire);\n>>> +\n>>> +\t\t\tassert(s_.count <= curr.count);\n>>> +\t\t\tassert(s_.fill <= curr.fill);\n>>> +\n>>> +\t\t\treturn {\n>>> +\t\t\t\t*l_,\n>>> +\t\t\t\tcurr.count - s_.count,\n>>> +\t\t\t\ts_.fill,\n>>> +\t\t\t\tcurr.fill,\n>>> +\t\t\t};\n>>> +\t\t}\n>>> +\n>>> +\tprivate:\n>>> +\t\tCheckpoint(const MetadataList &l)\n>>> +\t\t\t: l_(&l),\n>>> +\t\t\t s_(l.state_.load(std::memory_order_relaxed))\n>>> +\t\t{\n>>> +\t\t}\n>>> +\n>>> +\t\tfriend MetadataList;\n>>> +\n>>> +\t\tconst MetadataList *l_ = nullptr;\n>>\n>> Is there in your opinion a case where a Checkpoint can overlive the\n>> MetadataList it refers to ?\n\nI don't see how that could be useful. Technically we can add `diffSince(const Checkpoint&)`\nto `MetadataList` itself, and then this pointer is not needed. I originally\ndesigned it like this to prevent any confusion as to which checkpoint\nbelongs to which list. On the other hand dropping the pointer would\nprevent potential use after free issues.\n\n\n\n>>\n>>> +\t\tState s_ = {};\n>>> +\t};\n>>> +\n>>> +\t[[nodiscard]] Checkpoint checkpoint() const\n>>> +\t{\n>>> +\t\treturn { *this };\n>>> +\t}\n>>> +\n>>> +private:\n>>> +\t[[nodiscard]] static constexpr std::size_t entriesOffset()\n>>> +\t{\n>>> +\t\treturn 0;\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n>>> +\t{\n>>> +\t\treturn details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] Span entries() const\n>>> +\t{\n>>> +\t\treturn { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] Entry *find(std::uint32_t tag) const\n>>> +\t{\n>>> +\t\tconst auto entries = this->entries();\n>>> +\t\tauto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n>>\n>> how is this different from find_if() ? Just out of curiosity\n\n`std::partition_point` (and `std::{lower,upper}_bound`) implements binary search,\nwhile `std::find_if()` is linear.\n\n\n\n>>\n>> I'll stop here for the time being. Just one more note, you moved\n>> MetadataListPlan out this header, so you probably need a dedicated\n>> .cpp file as well\n>>\n>>\n>>> +\t\t\treturn e.tag < tag;\n>>> +\t\t});\n>>> +\n>>> +\t\tif (it == entries.end() || it->tag != tag)\n>>> +\t\t\treturn nullptr;\n>>> +\n>>> +\t\treturn &*it;\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] ControlValueView data_of(const Entry &e) const\n>>> +\t{\n>>> +\t\tconst auto o = e.acquireData();\n>>> +\t\treturn o ? data_of(*o) : ControlValueView{ };\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n>>> +\t{\n>>> +\t\tassert(headerOffset <= alloc_ - sizeof(ValueHeader));\n>>> +\t\tassert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n>>> +\n>>> +\t\tconst auto *vh = reinterpret_cast(p_ + headerOffset);\n>>> +\t\tconst auto *p = reinterpret_cast(vh) + sizeof(*vh);\n>>> +\t\tstd::size_t avail = p_ + alloc_ - p;\n>>> +\n>>> +\t\tconst auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n>>> +\t\tassert(data);\n>>> +\n>>> +\t\treturn { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] SetError set(Entry &e, ControlValueView v)\n>>> +\t{\n>>> +\t\tif (e.hasValue())\n>>> +\t\t\treturn SetError::AlreadySet;\n>>> +\t\tif (e.type != v.type() || e.isArray != v.isArray())\n>>> +\t\t\treturn SetError::TypeMismatch;\n> \n> Isn't this better expressed as an assertion ?\n\nSee below.\n\n\n> \n>>> +\n>>> +\t\tconst auto src = v.data();\n>>> +\t\tif (e.isArray) {\n>>> +\t\t\tif (src.size_bytes() > e.capacity)\n>>> +\t\t\t\treturn SetError::SizeMismatch;\n>>> +\t\t} else {\n>>> +\t\t\tif (src.size_bytes() != e.capacity)\n>>> +\t\t\t\treturn SetError::SizeMismatch;\n>>> +\t\t}\n> \n> If any of these fails, it means the pipeline handler has wrongly sized\n> the MetadataPlan, should we assert here as well ?\n\nI am not sure about what the best error handling strategy is. I feel like\nthis is good for testing. I imagine we can add the necessary assertions\nin `PipelineHandler::metadataAvailable()`.\n\nThe question of error handling in `MetadataList::merge()` still remains,\nbut I think we can make it so that it returns `std::optional` to\nsignal errors, and that can be checked in the other in `PipelineHandler::metadataAvailable()`\noverload.\n\nThoughts?\n\n\n> \n>>> +\n>>> +\t\tauto s = state_.load(std::memory_order_relaxed);\n>>> +\t\tstd::byte *oldEnd = p_ + contentOffset_ + s.fill;\n>>> +\t\tstd::byte *p = oldEnd;\n>>> +\n>>> +\t\tauto *headerPtr = details::align::up(p);\n> \n> This aligns to the next ValueHeader\n> \n>>> +\t\tauto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n> \n> This to size_bytes() + alignment\n> \n>>> +\t\tdetails::align::up(0, alignof(ValueHeader), p);\n> \n> What does this do ?\n\nThis aligns `p` to the beginning of the next value header. This is required\nbecause of how iteration works: every iterator is always aligned to the value\nheader, which means that the end iterator has to be aligned to that as well.\n(Otherwise `==` and `!=` on the iterators would not be as simple as they are now.)\n\nThis can be done by essentially adding tail padding after the data, which\nis what the above statement does. Technically `auto *headerPtr = details::align::up(p);`\ncould be rewritten as `auto *headerPtr = reinterpret_cast(p);`\nbecause `p` should already be aligned (due to the previous value's tail padding).\n\n\n> \n>>> +\n>>> +#if HAS_ASAN\n>>> +\t\t::__sanitizer_annotate_contiguous_container(\n>>> +\t\t\tp_ + contentOffset_, p_ + alloc_,\n>>> +\t\t\toldEnd, p\n>>> +\t\t);\n>>> +#endif\n>>> +\n>>> +\t\tnew (headerPtr) ValueHeader{\n>>> +\t\t\t.tag = e.tag,\n>>> +\t\t\t.size = std::uint32_t(src.size_bytes()),\n>>> +\t\t\t.alignment = e.alignment,\n>>> +\t\t\t.params = {\n>>> +\t\t\t\t.type = v.type(),\n>>> +\t\t\t\t.isArray = v.isArray(),\n>>> +\t\t\t\t.numElements = std::uint32_t(v.numElements()),\n>>> +\t\t\t},\n>>> +\t\t};\n>>> +\t\tstd::memcpy(dataPtr, src.data(), src.size_bytes());\n>>> +\t\te.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n> \n> break these long lines when it's trivial to do so\n\ndone\n\n\n> \n>>> +\n>>> +\t\ts.fill += p - oldEnd;\n>>> +\t\ts.count += 1;\n>>> +\n>>> +\t\tstate_.store(s, std::memory_order_release);\n>>> +\n>>> +\t\treturn {};\n>>> +\t}\n>>> +\n>>> +\tstd::size_t capacity_ = 0;\n>>> +\tstd::size_t contentOffset_ = -1;\n>>> +\tstd::size_t alloc_ = 0;\n>>> +\tstd::atomic state_ = State{};\n>>> +\tstd::byte *p_ = nullptr;\n>>> +\t// TODO: ControlIdMap in any way shape or form?\n> \n> What do you mean here ?\n\nA `ControlList` has an associated id map, info map, even validator.\nHere I am mainly wondering there should be anything like that.\nI personally don't see a convincing argument at the moment.\n\n\n> \n>>> +\n>>> +\t/*\n>>> +\t * If this is problematic on a 32-bit architecture, then\n>>> +\t * `count` can be stored in a separate atomic variable\n>>> +\t * but then `Diff::changed_` must be removed since the fill\n>>> +\t * level and item count cannot be retrieved atomically.\n> \n> or we can use a mutex or a different lock primitive ?\n\nThat's true, but I believe that would be a bit suboptimal.\n\n\n> \n>>> +\t */\n>>> +\tstatic_assert(decltype(state_)::is_always_lock_free);\n>>> +};\n>>> +\n>>> +} /* namespace libcamera */\n>>> +\n>>> +#undef HAS_ASAN\n>>> diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n>>> new file mode 100644\n>>> index 000000000..2ed35c54f\n>>> --- /dev/null\n>>> +++ b/include/libcamera/metadata_list_plan.h\n>>> @@ -0,0 +1,130 @@\n>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>> +/*\n>>> + * Copyright (C) 2025, Ideas On Board Oy\n>>> + */\n>>> +\n>>> +#pragma once\n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +#include \n>>> +\n>>> +namespace libcamera {\n>>> +\n>>> +class MetadataListPlan\n>>> +{\n>>> +public:\n>>> +\tstruct Entry {\n>>> +\t\tstd::uint32_t size;\n>>> +\t\tstd::uint32_t alignment; // TODO: is this necessary?\n> \n> I wonder if the alignment shouldn't come from controls.h associated\n> with ControlType\n> \n> Something like control_type::alignment\n\nIt could, and but that needs the addition of `alignment` to `control_type<>`\nand some minimal changes to `MetadataListPlan` to query it.\n\n\n> \n> I wonder if this would simplify the handling of alignment in\n> MetadataList as well\n\nIf we agree on a fixed static alignment, e.g. 8 or 16, then this\nfield will no longer be needed.\n\n\n> \n>>> +\t\tstd::uint32_t numElements;\n>>> +\t\tControlType type;\n>>> +\t\tbool isArray;\n>>> +\t};\n>>> +\n>>> +\t[[nodiscard]] bool empty() const { return items_.empty(); }\n>>> +\t[[nodiscard]] std::size_t size() const { return items_.size(); }\n>>> +\t[[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n>>> +\t[[nodiscard]] decltype(auto) end() const { return items_.end(); }\n>>> +\tvoid clear() { items_.clear(); }\n>>> +\n>>> +\ttemplate<\n>>> +\t\ttypename T,\n>>> +\t\tstd::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n>>> +\t>\n>>> +\tdecltype(auto) set(const Control &ctrl)\n> \n> This calls the private overload\n> \n> \ttemplate\n> \tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> \n> Which returns\n> \n> *this;\n> \n> which if I'm not mistaken means \"MetadataListPlan *\". Can this function\n> returns anything else ? If the return type is known and trivial to\n> type out why hide it behind decltype(auto) ?\n\nIt is `MetadataListPlan&`, my motivation in these cases is that development\nis easier if you don't have to change all the types up the call stack. Here\nI planned to replace it with the actual type when I would feel that the design\nis mostly fixed, but I will do so now.\n\n\n\n> \n>>> +\t{\n>>> +\t\tif constexpr (libcamera::details::control_type::size > 0) {\n>>> +\t\t\tstatic_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n>>> +\n>>> +\t\t\treturn set(\n>>> +\t\t\t\tctrl.id(),\n>>> +\t\t\t\tlibcamera::details::control_type::size,\n>>> +\t\t\t\ttrue\n>>> +\t\t\t);\n>>> +\t\t} else {\n>>> +\t\t\treturn set(ctrl.id(), 1, false);\n>>> +\t\t}\n>>> +\t}\n>>> +\n>>> +\ttemplate<\n>>> +\t\ttypename T,\n>>> +\t\tstd::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n> \n> is this for strings only ? It seems to me that only std::string has\n> control_type::size == dynamic_extent\n\nAlso for `libcamera::Span` (i.e. dynamically sized arrays).\n\n\n> \n>>> +\t>\n>>> +\tdecltype(auto) set(const Control &ctrl, std::size_t numElements)\n>>> +\t{\n>>> +\t\treturn set(ctrl.id(), numElements, true);\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] bool set(std::uint32_t tag,\n>>> +\t\t\t std::size_t size, std::size_t alignment,\n>>> +\t\t\t std::size_t numElements, ControlType type, bool isArray)\n> \n> Is this overload meant for the public interface of the class ?\n\nThis is public to allow calls from the IPC deserialization code.\n\n\n> \n> It seems that the two previous overloads call the private one\n> \n> \ttemplate\n> \tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> \n> which then calls this public one. Is this intentional ?\n\nYes, the order of overloads is intentional.\n\n\n> \n>>> +\t{\n>>> +\t\tif (size == 0 || size > std::numeric_limits::max())\n>>> +\t\t\treturn false;\n>>> +\t\tif (alignment > std::numeric_limits::max())\n>>> +\t\t\treturn false;\n>>> +\t\tif (!details::cxx20::has_single_bit(alignment))\n>>> +\t\t\treturn false;\n>>> +\t\tif (numElements > std::numeric_limits::max() / size)\n>>> +\t\t\treturn false;\n>>> +\t\tif (!isArray && numElements != 1)\n>>> +\t\t\treturn false;\n> \n> Should these be assertions (where and if possible) ?\n\nThis overload is used by the deserialization layer. I feel that these\nshould not be assertions, allowing the IPC deserialization to detect\nthe error, see https://patchwork.libcamera.org/project/libcamera/list/?series=5173\nNow, the IPC layer will most likely abort at the moment, but I would\nlike to keep this function without assertions.\n\n\n> \n>>> +\n>>> +\t\titems_.insert_or_assign(tag, Entry{\n> \n> Do we allow to re-write an entry in the MetadataListPlan ?\n\nYes, this is for easy updating. See the rpi5 pipeline handler configure().\n\n\n> \n>>> +\t\t\t.size = std::uint32_t(size),\n>>> +\t\t\t.alignment = std::uint32_t(alignment),\n>>> +\t\t\t.numElements = std::uint32_t(numElements),\n>>> +\t\t\t.type = type,\n>>> +\t\t\t.isArray = isArray,\n>>> +\t\t});\n>>> +\n>>> +\t\treturn true;\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] const Entry *get(std::uint32_t tag) const\n>>> +\t{\n>>> +\t\tauto it = items_.find(tag);\n>>> +\t\tif (it == items_.end())\n>>> +\t\t\treturn nullptr;\n>>> +\n>>> +\t\treturn &it->second;\n>>> +\t}\n>>> +\n>>> +\t[[nodiscard]] const Entry *get(const ControlId &cid) const\n>>> +\t{\n>>> +\t\tconst auto *e = get(cid.id());\n>>> +\t\tif (!e)\n>>> +\t\t\treturn nullptr;\n>>> +\n>>> +\t\tif (e->type != cid.type() || e->isArray != cid.isArray())\n>>> +\t\t\treturn nullptr;\n> \n> How can this happen ?\n\nThis could happen if someone does e.g. `get(properties::Location)` for some reason\nwhen the metadata plan contains `controls::AeEnable`. They have the same numeric\nid. I think it can be argued that this should be an assertion.\n\n\n> \n>>> +\n>>> +\t\treturn e;\n>>> +\t}\n>>> +\n>>> +private:\n>>> +\tstd::map items_;\n>>> +\n>>> +\ttemplate\n>>> +\tdecltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n>>> +\t{\n>>> +\t\tstatic_assert(std::is_trivially_copyable_v);\n>>> +\n>>> +\t\t[[maybe_unused]] bool ok = set(tag,\n>>> +\t\t\t\t\t sizeof(T), alignof(T),\n>>> +\t\t\t\t\t numElements, details::control_type::value, isArray);\n>>> +\t\tassert(ok);\n>>> +\n>>> +\t\treturn *this;\n>>> +\t}\n>>> +};\n>>> +\n>>> +} /* namespace libcamera */\n>>> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n>>> index de1eb99b2..8c5ce4503 100644\n>>> --- a/src/libcamera/meson.build\n>>> +++ b/src/libcamera/meson.build\n>>> @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n>>> 'framebuffer.cpp',\n>>> 'framebuffer_allocator.cpp',\n>>> 'geometry.cpp',\n>>> + 'metadata_list.cpp',\n>>> 'orientation.cpp',\n>>> 'pixel_format.cpp',\n>>> 'request.cpp',\n>>> diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n>>> new file mode 100644\n>>> index 000000000..ebefdfdad\n>>> --- /dev/null\n>>> +++ b/src/libcamera/metadata_list.cpp\n>>> @@ -0,0 +1,344 @@\n>>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>>> +/*\n>>> + * Copyright (C) 2025, Ideas On Board Oy\n>>> + */\n>>> +\n>>> +#include \n>>> +\n>>> +namespace libcamera {\n>>> +\n>>> +/**\n>>> + * \\class MetadataListPlan\n>>> + * \\brief Class to hold the possible set of metadata items for a MetadataList\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\class MetadataListPlan::Entry\n>>> + * \\brief Details of a metadata item\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\var MetadataListPlan::Entry::size\n>>> + * \\brief Number of bytes in a single element\n>>> + *\n>>> + * \\var MetadataListPlan::Entry::alignment\n>>> + * \\brief Required alignment of the elements\n>>> + * \\endinternal\n>>> + *\n>>> + * \\var MetadataListPlan::Entry::numElements\n>>> + * \\brief Number of elements in the value\n>>> + * \\sa ControlValueView::numElements()\n>>> + *\n>>> + * \\var MetadataListPlan::Entry::type\n>>> + * \\brief The type of the value\n>>> + * \\sa ControlValueView::type()\n>>> + *\n>>> + * \\var MetadataListPlan::Entry::isArray\n>>> + * \\brief Whether or not the value is array-like\n>>> + * \\sa ControlValueView::isArray()\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::begin() const\n>>> + * \\brief Retrieve the begin iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::end() const\n>>> + * \\brief Retrieve the end iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::size() const\n>>> + * \\brief Retrieve the number of entries\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::empty() const\n>>> + * \\brief Check if empty\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataListPlan::clear()\n>>> + * \\brief Remove all controls\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataListPlan::set(const Control &ctrl)\n>>> + * \\brief Add an entry for the given control to the metadata list plan\n>>> + * \\param[in] ctrl The control\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n>>> + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n>>> + * \\param[in] ctrl The control\n>>> + * \\param[in] count The maximum number of elements\n>>> + *\n>>> + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n>>> + * capacity of \\a count elements.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataListPlan::set(std::uint32_t tag,\n>>> + *\t\t\t std::size_t size, std::size_t alignment,\n>>> + *\t\t\t std::size_t count, ControlType type, bool isArray)\n>>> + * \\brief Add an entry to the metadata list plan\n>>> + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n>>> + * would result in an invalid entry\n>>> + *\n>>> + * This functions adds an entry with essentially arbitrary parameters, without deriving\n>>> + * them from a given ControlId instance. This is mainly used when deserializing.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n>>> + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataListPlan::get(const ControlId &cid) const\n>>> + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n>>> + *\n>>> + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n>>> + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\class MetadataList\n>>> + * \\brief Class to hold metadata items\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n>>> + * \\brief Construct a metadata list according to \\a plan\n>>> + *\n>>> + * Construct a metadata list according to the provided \\a plan.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::size() const\n>>> + * \\brief Retrieve the number of controls\n>>> + * \\context This function is \\threadsafe.\n>>> + * \\note If the list is being modified, the return value may be out of\n>>> + * date by the time the function returns\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::empty() const\n>>> + * \\brief Check if empty\n>>> + * \\context This function is \\threadsafe.\n>>> + * \\note If the list is being modified, the return value may be out of\n>>> + * date by the time the function returns\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::clear()\n>>> + * \\brief Remove all items from the list\n>>> + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n>>> + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n>>> + * when this function is called.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::begin() const\n>>> + * \\brief Retrieve begin iterator\n>>> + * \\context This function is \\threadsafe.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::end() const\n>>> + * \\brief Retrieve end iterator\n>>> + * \\context This function is \\threadsafe.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::get(const Control &ctrl) const\n>>> + * \\brief Get the value of control \\a ctrl\n>>> + * \\return A std::optional containing the control value, or std::nullopt if\n>>> + * the control \\a ctrl is not present in the list\n>>> + * \\context This function is \\threadsafe.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::get(std::uint32_t tag) const\n>>> + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n>>> + * \\return A std::optional containing the control value, or std::nullopt if\n>>> + * the control is not present in the list\n>>> + * \\context This function is \\threadsafe.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n>>> + * \\brief Set the value of control \\a ctrl to \\a value\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n>>> + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::merge(const ControlList &other)\n>>> + * \\brief Add all missing items from \\a other\n>>> + *\n>>> + * Add all items from \\a other that are not present in \\a this.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\enum MetadataList::SetError\n>>> + * \\brief Error code returned by a set operation\n>>> + *\n>>> + * \\var MetadataList::SetError::UnknownTag\n>>> + * \\brief The tag is not supported by the metadata list\n>>> + * \\var MetadataList::SetError::AlreadySet\n>>> + * \\brief A value has already been added with the given tag\n>>> + * \\var MetadataList::SetError::SizeMismatch\n>>> + * \\brief The size of the data is not appropriate for the given tag\n>>> + * \\var MetadataList::SetError::TypeMismatch\n>>> + * \\brief The type of the value does not match the expected type\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::checkpoint() const\n>>> + * \\brief Create a checkpoint\n>>> + * \\context This function is \\threadsafe.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\class MetadataList::iterator\n>>> + * \\brief Iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\typedef MetadataList::iterator::difference_type\n>>> + * \\brief iterator's difference type\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\typedef MetadataList::iterator::value_type\n>>> + * \\brief iterator's value type\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\typedef MetadataList::iterator::pointer\n>>> + * \\brief iterator's pointer type\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\typedef MetadataList::iterator::reference\n>>> + * \\brief iterator's reference type\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\typedef MetadataList::iterator::iterator_category\n>>> + * \\brief iterator's category\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::iterator::operator*()\n>>> + * \\brief Retrieve value at iterator\n>>> + * \\return A \\a ControlListView representing the value\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n>>> + * \\brief Check if two iterators are equal\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n>>> + * \\brief Check if two iterators are not equal\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::iterator::operator++(int)\n>>> + * \\brief Advance the iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::iterator::operator++()\n>>> + * \\brief Advance the iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\class MetadataList::Diff\n>>> + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n>>> + * \\sa Camera::metadataAvailable\n>>> + * \\internal\n>>> + * \\sa MetadataList::Checkpoint::diffSince()\n>>> + * \\endinternal\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::list() const\n>>> + * \\brief Retrieve the associated MetadataList\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::size() const\n>>> + * \\brief Retrieve the number of metadata items designated\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::empty() const\n>>> + * \\brief Check if any metadata items are designated\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::operator bool() const\n>>> + * \\copydoc MetadataList::Diff::empty() const\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n>>> + * \\copydoc MetadataList::get(const Control &ctrl) const\n>>> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n>>> + * even if it is otherwise present in the backing MetadataList.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n>>> + * \\copydoc MetadataList::get(std::uint32_t tag) const\n>>> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n>>> + * even if it is otherwise present in the backing MetadataList.\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::begin() const\n>>> + * \\brief Retrieve the begin iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\fn MetadataList::Diff::end() const\n>>> + * \\brief Retrieve the end iterator\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\class MetadataList::Checkpoint\n>>> + * \\brief Designates a point in the stream of metadata items\n>>> + *\n>>> + * A Checkpoint object designates a point in the stream of metadata items in the associated\n>>> + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n>>> + * added to the list after the designated point using diffSince().\n>>> + */\n>>> +\n>>> +/**\n>>> + * \\internal\n>>> + * \\fn MetadataList::Checkpoint::diffSince() const\n>>> + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n>>> + */\n>>> +\n>>> +} /* namespace libcamera */\n>>> diff --git a/test/controls/meson.build b/test/controls/meson.build\n>>> index 763f8905e..b68a4fc53 100644\n>>> --- a/test/controls/meson.build\n>>> +++ b/test/controls/meson.build\n>>> @@ -5,6 +5,7 @@ control_tests = [\n>>> {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n>>> {'name': 'control_list', 'sources': ['control_list.cpp']},\n>>> {'name': 'control_value', 'sources': ['control_value.cpp']},\n>>> + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n>>> ]\n>>>\n>>> foreach test : control_tests\n>>> diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n>>> new file mode 100644\n>>> index 000000000..f0872acd9\n>>> --- /dev/null\n>>> +++ b/test/controls/metadata_list.cpp\n>>> @@ -0,0 +1,170 @@\n>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>>> +/*\n>>> + * Copyright (C) 2025, Ideas On Board Oy\n>>> + *\n>>> + * MetadataList tests\n>>> + */\n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \n>>> +#include \n>>> +#include \n>>> +\n>>> +#include \"test.h\"\n>>> +\n>>> +using namespace std;\n>>> +using namespace libcamera;\n>>> +\n>>> +#define ASSERT(x) do { \\\n>>> +\tif (!static_cast(x)) { \\\n>>> +\t\tstd::cerr << '`' << #x << \"` failed\" << std::endl; \\\n>>> +\t\treturn TestFail; \\\n>>> +\t} \\\n>>> +} while (false)\n>>> +\n>>> +class MetadataListTest : public Test\n>>> +{\n>>> +public:\n>>> +\tMetadataListTest() = default;\n>>> +\n>>> +protected:\n>>> +\tint run() override\n>>> +\t{\n>>> +\t\tMetadataListPlan mlp;\n>>> +\t\tmlp.set(controls::ExposureTime);\n>>> +\t\tmlp.set(controls::ExposureValue);\n>>> +\t\tmlp.set(controls::ColourGains);\n>>> +\t\tmlp.set(controls::AfWindows, 10);\n>>> +\t\tmlp.set(controls::AeEnable);\n>>> +\t\tmlp.set(controls::SensorTimestamp);\n>>> +\n>>> +\t\tMetadataList ml(mlp);\n>>> +\n>>> +\t\tstatic_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n>>> +\t\tASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n> \n> Can you add a comment to explain what this tests ?\n\nPlease see\n\n\t\t/*\n\t\t *`properties::Location` has the same numeric id as `controls::AeEnable` (checked by the `static_assert`\n\t\t * below), but they have different types; check that this is detected.\n\t\t */\n\t\tstatic_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n\t\tASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n\n> \n>>> +\n>>> +\t\tASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n>>> +\t\tASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n>>> +\n>>> +\t\tauto f1 = std::async(std::launch::async, [&] {\n>>> +\t\t\tusing namespace std::chrono_literals;\n>>> +\n>>> +\t\t\tstd::this_thread::sleep_for(500ms);\n>>> +\t\t\tASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n>>> +\n>>> +\t\t\tstd::this_thread::sleep_for(500ms);\n>>> +\t\t\tASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n>>> +\n>>> +\t\t\tstd::this_thread::sleep_for(500ms);\n>>> +\t\t\tASSERT(ml.set(controls::ColourGains, std::array{\n>>> +\t\t\t\t123.f,\n>>> +\t\t\t\t456.f\n>>> +\t\t\t}) == MetadataList::SetError());\n>>> +\n>>> +\t\t\tstd::this_thread::sleep_for(500ms);\n>>> +\t\t\tASSERT(ml.set(controls::AfWindows, std::array{\n>>> +\t\t\t\tRectangle(),\n>>> +\t\t\t\tRectangle(1, 2, 3, 4),\n>>> +\t\t\t\tRectangle(0x1111, 0x2222, 0x3333, 0x4444),\n>>> +\t\t\t}) == MetadataList::SetError());\n> \n> I don't get why you expect these to fail..\n\nHmm... I think my intention is not clear. A default constructed `SetError` object,\ni.e. `MetadataList::SetError()` is supposed to denote success. This is modelled\nafter `std::errc`. I can rename it to `SetResult` and add another enumeration like so:\n\n enum class SetResult {\n Success,\n /* rest of the failure cases */\n };\n\ndo you think that would be clearer?\n\n\n> \n>>> +\n>>> +\t\t\treturn TestPass;\n>>> +\t\t});\n>>> +\n>>> +\t\tauto f2 = std::async(std::launch::async, [&] {\n>>> +\t\t\tfor (;;) {\n>>> +\t\t\t\tconst auto x = ml.get(controls::ExposureTime);\n>>> +\t\t\t\tconst auto y = ml.get(controls::ExposureValue);\n>>> +\t\t\t\tconst auto z = ml.get(controls::ColourGains);\n>>> +\t\t\t\tconst auto w = ml.get(controls::AfWindows);\n>>> +\n>>> +\t\t\t\tif (x)\n>>> +\t\t\t\t\tASSERT(*x == 0x1111);\n>>> +\n>>> +\t\t\t\tif (y)\n>>> +\t\t\t\t\tASSERT(*y == 1.0f);\n>>> +\n>>> +\t\t\t\tif (z) {\n>>> +\t\t\t\t\tASSERT(z->size() == 2);\n>>> +\t\t\t\t\tASSERT((*z)[0] == 123.f);\n>>> +\t\t\t\t\tASSERT((*z)[1] == 456.f);\n>>> +\t\t\t\t}\n>>> +\n>>> +\t\t\t\tif (w) {\n>>> +\t\t\t\t\tASSERT(w->size() == 3);\n>>> +\t\t\t\t\tASSERT((*w)[0].isNull());\n>>> +\t\t\t\t\tASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n>>> +\t\t\t\t\tASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n>>> +\t\t\t\t}\n>>> +\n>>> +\t\t\t\tif (x && y && z && w)\n>>> +\t\t\t\t\tbreak;\n>>> +\t\t\t}\n>>> +\n>>> +\t\t\treturn TestPass;\n> \n> and if f1 is expected to fail how can f2 pass.\n> \n> I'm surely missing something\n\nSee my comment above, both are expected to pass.\n\n\n> \n>>> +\t\t});\n>>> +\n>>> +\t\tASSERT(f1.get() == TestPass);\n>>> +\t\tASSERT(f2.get() == TestPass);\n>>> +\n>>> +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n>>> +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n>>> +\n>>> +\t\tASSERT(ml.get(controls::ExposureTime) == 0x1111);\n>>> +\t\tASSERT(ml.get(controls::ExposureValue) == 1);\n>>> +\n>>> +\t\tfor (auto &&[tag, v] : ml)\n>>> +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> \n> Are we sure we want this in tests ? I mean, it doesn't hurt..\n\nI don't know... I have been sorely missing prints when I was looking\nat other tests. You're essentially forced to use a debugger.\n\n\n> \n>>> +\n>>> +\t\tstd::cout << std::endl;\n>>> +\n>>> +\t\tml.clear();\n>>> +\t\tASSERT(ml.empty());\n>>> +\t\tASSERT(ml.size() == 0);\n>>> +\n>>> +\t\tASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n>>> +\t\tASSERT(ml.get(controls::ExposureTime) == 0x2222);\n>>> +\n>>> +\t\tauto c = ml.checkpoint();\n>>> +\n>>> +\t\tASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n>>> +\t\tASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n>>> +\n>>> +\t\tauto d = c.diffSince();\n>>> +\t\tASSERT(&d.list() == &ml);\n>>> +\n>>> +\t\tASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n>>> +\n>>> +\t\tASSERT(d);\n>>> +\t\tASSERT(!d.empty());\n>>> +\t\tASSERT(d.size() == 2);\n>>> +\t\tASSERT(!d.get(controls::ExposureTime));\n>>> +\t\tASSERT(!d.get(controls::ColourGains));\n>>> +\t\tASSERT(!d.get(controls::AfWindows));\n>>> +\t\tASSERT(d.get(controls::ExposureValue) == 2);\n>>> +\t\tASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n>>> +\n>>> +\t\tfor (auto &&[tag, v] : d)\n>>> +\t\t\tstd::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n>>> +\n>>> +\t\t/* Test if iterators work with algorithms. */\n>>> +\t\tstd::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n>>> +\t\t\treturn false;\n>>> +\t\t});\n>>> +\n>>> +#if 0\n>>> + {\n>>> + auto it = ml.begin();\n>>> + ml.clear();\n>>> + std::ignore = *it; /* Trigger ASAN. */\n>>> + }\n>>> +#endif\n> \n> Maybe remove this ?\n\nI think I will try to move it into a separate test that is only active\nwhen ASAN is used and expects failure.\n\n---\n\nI hope I haven't missed anything.\n\nRegards,\nBarnabás Pőcze\n\n> \n>>> +\n>>> +\t\treturn TestPass;\n>>> +\t}\n>>> +};\n>>> +\n>>> +TEST_REGISTER(MetadataListTest)\n> \n> Thanks again for your hard work on this!\n> \n>>> --\n>>> 2.50.1\n>>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id BA23DBDC71\n\tfor ;\n\tTue, 29 Jul 2025 12:11:59 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 6A643691DB;\n\tTue, 29 Jul 2025 14:11:58 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 691C469052\n\tfor ;\n\tTue, 29 Jul 2025 14:11:36 +0200 (CEST)","from [192.168.33.18] (185.221.140.39.nat.pool.zt.hu\n\t[185.221.140.39])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 5B1583A4;\n\tTue, 29 Jul 2025 14:10:52 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"rG7DPaqI\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1753791052;\n\tbh=04+A+YB3z5rviec/tMxI9AzEa3uP9Y1tQirwQvURi2s=;\n\th=Date:Subject:To:Cc:References:From:In-Reply-To:From;\n\tb=rG7DPaqI4ohzvQPy3gLxnyAOaHlIg/QvUYkxfDIT164oDCkET1p01exngYGTCAYav\n\tWUjdmQpjKbONP5yfwTs7aZxGwnOaASsou/hNkkt8SoWEe2uO0EoLQWd3DT/UWeKdi6\n\thMW5cCkBwxRBuPhKyDht3yZ389AbjUk7LBH07y48=","Message-ID":"<1dce550f-c132-460c-8dc4-865df3cab60a@ideasonboard.com>","Date":"Tue, 29 Jul 2025 14:11:30 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","To":"Jacopo Mondi ","Cc":"libcamera-devel@lists.libcamera.org","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>\n\t<2lweaavht65vexto5246wcvmxjoxhwgtprz5hjvbrz7y4h4hrk@r3rwki5uoefi>\n\t","From":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Content-Language":"en-US, hu-HU","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35853,"web_url":"https://patchwork.libcamera.org/comment/35853/","msgid":"<175811057476.2127323.6616992472481774741@neptunite.rasen.tech>","date":"2025-09-17T12:02:54","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Hi Barnabás,\n\nThanks for the patch.\n\nQuoting Barnabás Pőcze (2025-07-21 19:46:07)\n> Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n> reported by a camera for a given request. Previously, a `ControlList` was\n> used for this purpose. The reason for introducing a separate type is to\n> simplify the access to the returned metadata during the entire lifetime\n> of a request.\n> \n> Specifically, for early metadata completion to be easily usable it should be\n> guaranteed that any completed metadata item can be accessed and looked up\n> at least until the associated requested is reused with `Request::reuse()`.\n> \n> However, when a metadata item is completed early, the pipeline handler\n> might still work on the request in the `CameraManager`'s private thread,\n> therefore there is an inherent synchronization issue when an application\n> accesses early metadata.\n> \n> Restricting the user to only access the metadata items of a not yet completed\n> request in the early metadata availability signal handler by ways of\n> documenting or enforcing it at runtime could be an option, but it is not\n> too convenient for the user.\n> \n> The current `ControlList` implementation employs an `std::unordered_map`,\n> so pointers remain stable when the container is modified, so an application\n> could keep accessing particular metadata items outside the signal handler,\n> but this fact is far from obvious, and the user would still not be able\n> to make a copy of all metadata or do lookups based on the numeric ids or\n> the usual `libcamera::Control<>` objects, thus some type safety is lost.\n> \n> The above also requires that each metadata item is only completed once for\n> a given request, but this does not appear to be serious limitation,\n> and in fact, this restriction is enforced by `MetadataList`.\n> \n> The introduced `MetadataList` supports single writer - multiple reader\n> scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n> without introducing data races or other synchronization issues. This is\n> achieved by requiring the possible set of metadata items to be known\n> (such set is stored in a `MetadataListPlan` object). Based on the this\n> plan, a single contiguous allocation is made to accommodate all potential\n> metadata items. Due to this single contiguous allocation that is not modified\n> during the lifetime of a `MetadataList` and atomic modifications, it is\n> possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n> there is only ever a single writer.\n> \n> Signed-off-by: Barnabás Pőcze \n\nIn general, the ideas and design look good to me. Jacopo picked at a lot more\nthan I could.\n\nI only have a few comments/questions.\n\nNot sure if it's because it's still RFC phase or because everything is\ninternal, but most of this is \"I want documentation please\".\n\n> ---\n> changes in v2:\n> * remove multiple not strictly necessary functions\n> ---\n> include/libcamera/meson.build | 2 +\n> include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n> include/libcamera/metadata_list_plan.h | 130 ++++++\n> src/libcamera/meson.build | 1 +\n> src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n> test/controls/meson.build | 1 +\n> test/controls/metadata_list.cpp | 170 ++++++++\n> 7 files changed, 1195 insertions(+)\n> create mode 100644 include/libcamera/metadata_list.h\n> create mode 100644 include/libcamera/metadata_list_plan.h\n> create mode 100644 src/libcamera/metadata_list.cpp\n> create mode 100644 test/controls/metadata_list.cpp\n> \n> diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n> index 30ea76f94..410b548dd 100644\n> --- a/include/libcamera/meson.build\n> +++ b/include/libcamera/meson.build\n> @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n> 'framebuffer_allocator.h',\n> 'geometry.h',\n> 'logging.h',\n> + 'metadata_list.h',\n> + 'metadata_list_plan.h',\n> 'orientation.h',\n> 'pixel_format.h',\n> 'request.h',\n> diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n> new file mode 100644\n> index 000000000..7fe3dbbab\n> --- /dev/null\n> +++ b/include/libcamera/metadata_list.h\n> @@ -0,0 +1,547 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + *\n> + * Metadata list\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +\n> +// TODO: want this?\n\nGiven that we're manually doing a bunch of binary formatting I think we do want\nASAN.\n\n> +#if __has_include()\n> +#if __SANITIZE_ADDRESS__ /* gcc */\n> +#include \n> +#define HAS_ASAN 1\n> +#elif defined(__has_feature)\n> +#if __has_feature(address_sanitizer) /* clang */\n> +#include \n> +#define HAS_ASAN 1\n> +#endif\n> +#endif\n> +#endif\n> +\n> +namespace libcamera {\n> +\n> +class MetadataList\n> +{\n> +private:\n> + struct ValueParams {\n> + ControlType type;\n> + bool isArray;\n> + std::uint32_t numElements;\n> + };\n> +\n> + struct Entry {\n> + const std::uint32_t tag;\n> + const std::uint32_t capacity;\n> + const std::uint32_t alignment;\n> + const ControlType type;\n> + bool isArray;\n> +\n> + static constexpr std::uint32_t invalidOffset = -1;\n> + /*\n> + * Offset from the beginning of the allocation, and\n> + * and _not_ relative to `contentOffset_`.\n> + */\n> + std::atomic_uint32_t headerOffset = invalidOffset;\n> +\n> + [[nodiscard]] std::optional hasValue() const\n> + {\n> + auto offset = headerOffset.load(std::memory_order_relaxed);\n> + if (offset == invalidOffset)\n> + return {};\n> +\n> + return offset;\n> + }\n> +\n> + [[nodiscard]] std::optional acquireData() const\n> + {\n> + auto offset = hasValue();\n> + if (offset) {\n> + /* sync with release-store on `headerOffset` in `MetadataList::set()` */\n> + std::atomic_thread_fence(std::memory_order_acquire);\n> + }\n> +\n> + return offset;\n> + }\n> + };\n> +\n> + struct ValueHeader {\n> + std::uint32_t tag;\n> + std::uint32_t size;\n> + std::uint32_t alignment;\n> + ValueParams params;\n> + };\n\nI know these structs are internal but there are so many moving parts that I'm\njust barely able to hold on to the meanings of everything... can we have\ndocumentation please...? Especially for when our caches are all flushed when we\nneed to work on this in the future, I think documentation will speed up cache\nrefreshing.\n\n> +\n> + struct State {\n> + std::uint32_t count;\n> + std::uint32_t fill;\n> + };\n> +\n> +public:\n> + explicit MetadataList(const MetadataListPlan &plan)\n> + : capacity_(plan.size()),\n> + contentOffset_(MetadataList::contentOffset(capacity_)),\n> + alloc_(contentOffset_)\n> + {\n> + for (const auto &[tag, e] : plan) {\n> + alloc_ += sizeof(ValueHeader);\n> + alloc_ += e.alignment - 1; // XXX: this is the maximum\n> + alloc_ += e.size * e.numElements;\n> + alloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n\nWhy are we taking both the size and align of ValueHeader?\n\n> + }\n> +\n> + p_ = static_cast(::operator new(alloc_));\n> +\n> + auto *entries = reinterpret_cast(p_ + entriesOffset());\n> + auto it = plan.begin();\n> +\n> + for (std::size_t i = 0; i < capacity_; i++, ++it) {\n> + const auto &[tag, e] = *it;\n> +\n> + new (&entries[i]) Entry{\n> + .tag = tag,\n> + .capacity = e.size * e.numElements,\n> + .alignment = e.alignment,\n> + .type = e.type,\n> + .isArray = e.isArray,\n> + };\n> + }\n> +\n> +#if HAS_ASAN\n> + ::__sanitizer_annotate_contiguous_container(\n> + p_ + contentOffset_, p_ + alloc_,\n> + p_ + alloc_, p_ + contentOffset_\n> + );\n> +#endif\n> + }\n> +\n> + MetadataList(const MetadataList &) = delete;\n> + MetadataList(MetadataList &&) = delete;\n> +\n> + MetadataList &operator=(const MetadataList &) = delete;\n> + MetadataList &operator=(MetadataList &&) = delete;\n> +\n> + ~MetadataList()\n> + {\n> +#if HAS_ASAN\n> + /*\n> + * The documentation says the range apparently has to be\n> + * restored to its initial state before it is deallocated.\n> + */\n> + ::__sanitizer_annotate_contiguous_container(\n> + p_ + contentOffset_, p_ + alloc_,\n> + p_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n> + );\n> +#endif\n> +\n> + ::operator delete(p_, alloc_);\n> + }\n> +\n> + // TODO: want these?\n\nI think it's useful.\n\n> + [[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n> + [[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n> +\n> + enum class SetError {\n> + UnknownTag = 1,\n> + AlreadySet,\n> + SizeMismatch,\n> + TypeMismatch,\n> + };\n> +\n> + [[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n> + {\n> + auto *e = find(tag);\n> + if (!e)\n> + return SetError::UnknownTag;\n> +\n> + return set(*e, v);\n> + }\n> +\n> + template\n> + /* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> + {\n> + using TypeInfo = libcamera::details::control_type;\n> +\n> + if constexpr (TypeInfo::size > 0) {\n> + static_assert(std::is_trivially_copyable_v);\n> +\n> + return set(ctrl.id(), {\n> + TypeInfo::value,\n> + true,\n> + value.size(),\n> + reinterpret_cast(value.data()),\n> + });\n> + } else {\n> + static_assert(std::is_trivially_copyable_v);\n> +\n> + return set(ctrl.id(), {\n> + TypeInfo::value,\n> + false,\n> + 1,\n> + reinterpret_cast(&value),\n> + });\n> + }\n> + }\n> +\n> + template\n> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> + {\n> + ControlValueView v = get(ctrl.id());\n> +\n> + return v ? std::optional(v.get()) : std::nullopt;\n> + }\n> +\n> + // TODO: operator ControlListView() const ?\n> + // TODO: explicit operator ControlList() const ?\n> +\n> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> + {\n> + const auto *e = find(tag);\n> + if (!e)\n> + return {};\n> +\n> + return data_of(*e);\n\niirc Jacopo mentioned casing but this is a well-known common macro that imo\nit's ok to keep as-is. imo tho.\n\n> + }\n> +\n> + void clear()\n> + {\n> + for (auto &e : entries())\n> + e.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n> +\n> + [[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n> +\n> +#if HAS_ASAN\n> + ::__sanitizer_annotate_contiguous_container(\n> + p_ + contentOffset_, p_ + alloc_,\n> + p_ + contentOffset_ + s.fill, p_ + contentOffset_\n> + );\n> +#endif\n> + }\n> +\n> + class iterator\n> + {\n> + public:\n> + using difference_type = std::ptrdiff_t;\n> + using value_type = std::pair;\n> + using pointer = void;\n> + using reference = value_type;\n> + using iterator_category = std::forward_iterator_tag;\n> +\n> + iterator() = default;\n> +\n> + iterator& operator++()\n> + {\n> + const auto &h = header();\n> +\n> + p_ += sizeof(h);\n> + p_ = details::align::up(p_, h.alignment);\n> + p_ += h.size;\n> + p_ = details::align::up(p_, alignof(decltype(h)));\n\nOk I think this makes sense... I'd really like documentation on the binary\nformat somewhere though...\n\n> +\n> + return *this;\n> + }\n> +\n> + iterator operator++(int)\n> + {\n> + auto copy = *this;\n> + ++*this;\n> + return copy;\n> + }\n> +\n> + [[nodiscard]] reference operator*() const\n> + {\n> + const auto &h = header();\n> + const auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n> +\n> + return { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n> + }\n> +\n> + [[nodiscard]] bool operator==(const iterator &other) const\n> + {\n> + return p_ == other.p_;\n> + }\n> +\n> + [[nodiscard]] bool operator!=(const iterator &other) const\n> + {\n> + return !(*this == other);\n> + }\n> +\n> + private:\n> + iterator(const std::byte *p)\n> + : p_(p)\n> + {\n> + }\n> +\n> + [[nodiscard]] const ValueHeader &header() const\n> + {\n> + return *reinterpret_cast(p_);\n> + }\n> +\n> + friend MetadataList;\n> +\n> + const std::byte *p_ = nullptr;\n> + };\n> +\n> + [[nodiscard]] iterator begin() const\n> + {\n> + return { p_ + contentOffset_ };\n> + }\n> +\n> + [[nodiscard]] iterator end() const\n> + {\n> + return { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n> + }\n> +\n> + class Diff\n> + {\n> + public:\n> + // TODO: want these?\n> + [[nodiscard]] explicit operator bool() const { return !empty(); }\n> + [[nodiscard]] bool empty() const { return start_ == stop_; }\n> + [[nodiscard]] std::size_t size() const { return changed_; }\n> + [[nodiscard]] const MetadataList &list() const { return *l_; }\n\nI think they'd be helpful to have.\n\n> +\n> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> + {\n> + const auto *e = l_->find(tag);\n> + if (!e)\n> + return {};\n> +\n> + auto o = e->acquireData();\n> + if (!o)\n> + return {};\n> +\n> + if (!(start_ <= *o && *o < stop_))\n> + return {};\n> +\n> + return l_->data_of(*o);\n> + }\n> +\n> + template\n> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> + {\n> + ControlValueView v = get(ctrl.id());\n> +\n> + return v ? std::optional(v.get()) : std::nullopt;\n> + }\n> +\n> + [[nodiscard]] iterator begin() const\n> + {\n> + return { l_->p_ + start_ };\n> + }\n> +\n> + [[nodiscard]] iterator end() const\n> + {\n> + return { l_->p_ + stop_ };\n> + }\n> +\n> + private:\n> + Diff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n\nI can get what the meanings of these are... by reading Checkpoint below. imo\ndocumentation will help smooth out the process of understanding this.\n\n> + : l_(&l),\n> + changed_(changed),\n> + start_(l.contentOffset_ + oldFill),\n> + stop_(l.contentOffset_ + newFill)\n> + {\n> + }\n> +\n> + friend MetadataList;\n> + friend struct Checkpoint;\n> +\n> + const MetadataList *l_ = nullptr;\n> + std::size_t changed_;\n> + std::size_t start_;\n> + std::size_t stop_;\n> + };\n> +\n> + Diff merge(const ControlList &other)\n> + {\n> + // TODO: check id map of `other`?\n\nSince Diff can't be constructed by the user, I think we're guaranteed that the\nid maps match, so I don't think we need to spend resources on this. Or maybe we\ncould for a peace of mind; it's just a comparison of pointers, right?\n\nSpeaking of which, do we need to disable or privatize the default constructors?\n\n> +\n> + const auto c = checkpoint();\n> +\n> + for (const auto &[tag, value] : other) {\n> + auto *e = find(tag);\n> + if (e) {\n> + [[maybe_unused]] auto r = set(*e, value);\n> + assert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n\nTODO?\n\n> + }\n> + }\n> +\n> + return c.diffSince();\n> + }\n> +\n> + class Checkpoint\n> + {\n> + public:\n> + [[nodiscard]] Diff diffSince() const\n> + {\n> + /* sync with release-store on `state_` in `set()` */\n> + const auto curr = l_->state_.load(std::memory_order_acquire);\n> +\n> + assert(s_.count <= curr.count);\n> + assert(s_.fill <= curr.fill);\n> +\n> + return {\n> + *l_,\n> + curr.count - s_.count,\n> + s_.fill,\n> + curr.fill,\n> + };\n> + }\n> +\n> + private:\n> + Checkpoint(const MetadataList &l)\n> + : l_(&l),\n> + s_(l.state_.load(std::memory_order_relaxed))\n> + {\n> + }\n> +\n> + friend MetadataList;\n> +\n> + const MetadataList *l_ = nullptr;\n> + State s_ = {};\n> + };\n> +\n> + [[nodiscard]] Checkpoint checkpoint() const\n> + {\n> + return { *this };\n> + }\n> +\n> +private:\n> + [[nodiscard]] static constexpr std::size_t entriesOffset()\n> + {\n> + return 0;\n> + }\n> +\n> + [[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n> + {\n> + return details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n> + }\n> +\n> + [[nodiscard]] Span entries() const\n> + {\n> + return { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n> + }\n> +\n> + [[nodiscard]] Entry *find(std::uint32_t tag) const\n> + {\n> + const auto entries = this->entries();\n> + auto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n> + return e.tag < tag;\n> + });\n> +\n> + if (it == entries.end() || it->tag != tag)\n> + return nullptr;\n> +\n> + return &*it;\n> + }\n> +\n> + [[nodiscard]] ControlValueView data_of(const Entry &e) const\n> + {\n> + const auto o = e.acquireData();\n> + return o ? data_of(*o) : ControlValueView{ };\n> + }\n> +\n> + [[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n> + {\n> + assert(headerOffset <= alloc_ - sizeof(ValueHeader));\n> + assert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n> +\n> + const auto *vh = reinterpret_cast(p_ + headerOffset);\n> + const auto *p = reinterpret_cast(vh) + sizeof(*vh);\n> + std::size_t avail = p_ + alloc_ - p;\n> +\n> + const auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n> + assert(data);\n> +\n> + return { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n> + }\n\nThis... costs a lot to understand... I think documentation on the binary format\nwould help considerably.\n\n> +\n> + [[nodiscard]] SetError set(Entry &e, ControlValueView v)\n> + {\n> + if (e.hasValue())\n> + return SetError::AlreadySet;\n> + if (e.type != v.type() || e.isArray != v.isArray())\n> + return SetError::TypeMismatch;\n> +\n> + const auto src = v.data();\n> + if (e.isArray) {\n> + if (src.size_bytes() > e.capacity)\n> + return SetError::SizeMismatch;\n> + } else {\n> + if (src.size_bytes() != e.capacity)\n> + return SetError::SizeMismatch;\n> + }\n> +\n> + auto s = state_.load(std::memory_order_relaxed);\n> + std::byte *oldEnd = p_ + contentOffset_ + s.fill;\n> + std::byte *p = oldEnd;\n> +\n> + auto *headerPtr = details::align::up(p);\n> + auto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n> + details::align::up(0, alignof(ValueHeader), p);\n> +\n> +#if HAS_ASAN\n> + ::__sanitizer_annotate_contiguous_container(\n> + p_ + contentOffset_, p_ + alloc_,\n> + oldEnd, p\n> + );\n> +#endif\n> +\n> + new (headerPtr) ValueHeader{\n> + .tag = e.tag,\n> + .size = std::uint32_t(src.size_bytes()),\n> + .alignment = e.alignment,\n> + .params = {\n> + .type = v.type(),\n> + .isArray = v.isArray(),\n> + .numElements = std::uint32_t(v.numElements()),\n> + },\n> + };\n> + std::memcpy(dataPtr, src.data(), src.size_bytes());\n> + e.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n> +\n> + s.fill += p - oldEnd;\n> + s.count += 1;\n> +\n> + state_.store(s, std::memory_order_release);\n> +\n> + return {};\n> + }\n> +\n> + std::size_t capacity_ = 0;\n> + std::size_t contentOffset_ = -1;\n> + std::size_t alloc_ = 0;\n> + std::atomic state_ = State{};\n> + std::byte *p_ = nullptr;\n\nI... would like some documentation on these... I'm just barely able to hang on\nto the meanings... I think documentation would ease future maintenance when all\nour caches have been flushed. \n\n> + // TODO: ControlIdMap in any way shape or form?\n\nI don't think you need it? I think you've designed this in a way such that\nwhich ControlIdMap is being used doesn't really matter, since everything\noriginates from the MetadataListPlan.\n\n> +\n> + /*\n> + * If this is problematic on a 32-bit architecture, then\n> + * `count` can be stored in a separate atomic variable\n> + * but then `Diff::changed_` must be removed since the fill\n> + * level and item count cannot be retrieved atomically.\n> + */\n> + static_assert(decltype(state_)::is_always_lock_free);\n> +};\n> +\n> +} /* namespace libcamera */\n> +\n> +#undef HAS_ASAN\n> diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n> new file mode 100644\n> index 000000000..2ed35c54f\n> --- /dev/null\n> +++ b/include/libcamera/metadata_list_plan.h\n> @@ -0,0 +1,130 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + */\n> +\n> +#pragma once\n> +\n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +class MetadataListPlan\n> +{\n> +public:\n> + struct Entry {\n> + std::uint32_t size;\n> + std::uint32_t alignment; // TODO: is this necessary?\n> + std::uint32_t numElements;\n> + ControlType type;\n> + bool isArray;\n> + };\n> +\n> + [[nodiscard]] bool empty() const { return items_.empty(); }\n> + [[nodiscard]] std::size_t size() const { return items_.size(); }\n> + [[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n> + [[nodiscard]] decltype(auto) end() const { return items_.end(); }\n> + void clear() { items_.clear(); }\n> +\n> + template<\n> + typename T,\n> + std::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n> + >\n> + decltype(auto) set(const Control &ctrl)\n> + {\n> + if constexpr (libcamera::details::control_type::size > 0) {\n> + static_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n> +\n> + return set(\n> + ctrl.id(),\n> + libcamera::details::control_type::size,\n> + true\n> + );\n> + } else {\n> + return set(ctrl.id(), 1, false);\n> + }\n> + }\n> +\n> + template<\n> + typename T,\n> + std::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n> + >\n> + decltype(auto) set(const Control &ctrl, std::size_t numElements)\n> + {\n> + return set(ctrl.id(), numElements, true);\n> + }\n> +\n> + [[nodiscard]] bool set(std::uint32_t tag,\n> + std::size_t size, std::size_t alignment,\n> + std::size_t numElements, ControlType type, bool isArray)\n> + {\n> + if (size == 0 || size > std::numeric_limits::max())\n> + return false;\n> + if (alignment > std::numeric_limits::max())\n> + return false;\n> + if (!details::cxx20::has_single_bit(alignment))\n> + return false;\n> + if (numElements > std::numeric_limits::max() / size)\n> + return false;\n> + if (!isArray && numElements != 1)\n> + return false;\n> +\n> + items_.insert_or_assign(tag, Entry{\n> + .size = std::uint32_t(size),\n> + .alignment = std::uint32_t(alignment),\n> + .numElements = std::uint32_t(numElements),\n> + .type = type,\n> + .isArray = isArray,\n> + });\n> +\n> + return true;\n> + }\n> +\n> + [[nodiscard]] const Entry *get(std::uint32_t tag) const\n> + {\n> + auto it = items_.find(tag);\n> + if (it == items_.end())\n> + return nullptr;\n> +\n> + return &it->second;\n> + }\n> +\n> + [[nodiscard]] const Entry *get(const ControlId &cid) const\n> + {\n> + const auto *e = get(cid.id());\n> + if (!e)\n> + return nullptr;\n> +\n> + if (e->type != cid.type() || e->isArray != cid.isArray())\n> + return nullptr;\n> +\n> + return e;\n> + }\n> +\n> +private:\n> + std::map items_;\n> +\n> + template\n> + decltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> + {\n> + static_assert(std::is_trivially_copyable_v);\n> +\n> + [[maybe_unused]] bool ok = set(tag,\n> + sizeof(T), alignof(T),\n> + numElements, details::control_type::value, isArray);\n> + assert(ok);\n> +\n> + return *this;\n> + }\n> +};\n> +\n> +} /* namespace libcamera */\n> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> index de1eb99b2..8c5ce4503 100644\n> --- a/src/libcamera/meson.build\n> +++ b/src/libcamera/meson.build\n> @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n> 'framebuffer.cpp',\n> 'framebuffer_allocator.cpp',\n> 'geometry.cpp',\n> + 'metadata_list.cpp',\n> 'orientation.cpp',\n> 'pixel_format.cpp',\n> 'request.cpp',\n> diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n> new file mode 100644\n> index 000000000..ebefdfdad\n> --- /dev/null\n> +++ b/src/libcamera/metadata_list.cpp\n> @@ -0,0 +1,344 @@\n> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + */\n> +\n> +#include \n> +\n> +namespace libcamera {\n> +\n> +/**\n> + * \\class MetadataListPlan\n> + * \\brief Class to hold the possible set of metadata items for a MetadataList\n> + */\n> +\n> +/**\n> + * \\class MetadataListPlan::Entry\n> + * \\brief Details of a metadata item\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\var MetadataListPlan::Entry::size\n> + * \\brief Number of bytes in a single element\n> + *\n> + * \\var MetadataListPlan::Entry::alignment\n> + * \\brief Required alignment of the elements\n> + * \\endinternal\n> + *\n> + * \\var MetadataListPlan::Entry::numElements\n> + * \\brief Number of elements in the value\n> + * \\sa ControlValueView::numElements()\n> + *\n> + * \\var MetadataListPlan::Entry::type\n> + * \\brief The type of the value\n> + * \\sa ControlValueView::type()\n> + *\n> + * \\var MetadataListPlan::Entry::isArray\n> + * \\brief Whether or not the value is array-like\n> + * \\sa ControlValueView::isArray()\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::begin() const\n> + * \\brief Retrieve the begin iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::end() const\n> + * \\brief Retrieve the end iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::size() const\n> + * \\brief Retrieve the number of entries\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::empty() const\n> + * \\brief Check if empty\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::clear()\n> + * \\brief Remove all controls\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(const Control &ctrl)\n> + * \\brief Add an entry for the given control to the metadata list plan\n> + * \\param[in] ctrl The control\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n> + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n> + * \\param[in] ctrl The control\n> + * \\param[in] count The maximum number of elements\n> + *\n> + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n> + * capacity of \\a count elements.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataListPlan::set(std::uint32_t tag,\n> + * std::size_t size, std::size_t alignment,\n> + * std::size_t count, ControlType type, bool isArray)\n> + * \\brief Add an entry to the metadata list plan\n> + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n> + * would result in an invalid entry\n> + *\n> + * This functions adds an entry with essentially arbitrary parameters, without deriving\n> + * them from a given ControlId instance. This is mainly used when deserializing.\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n> + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n> + */\n> +\n> +/**\n> + * \\fn MetadataListPlan::get(const ControlId &cid) const\n> + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n> + *\n> + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n> + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n> + */\n> +\n> +/**\n> + * \\class MetadataList\n> + * \\brief Class to hold metadata items\n\nI'd like this to be expanded. What's special about it? Why can't we just use\nControlList? How is the data formatted in the custom data container (this could\nbe internal in the code)? What's the significance and difference between Entry\nand ValueHeader?\n\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n> + * \\brief Construct a metadata list according to \\a plan\n> + *\n> + * Construct a metadata list according to the provided \\a plan.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::size() const\n> + * \\brief Retrieve the number of controls\n> + * \\context This function is \\threadsafe.\n> + * \\note If the list is being modified, the return value may be out of\n> + * date by the time the function returns\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::empty() const\n> + * \\brief Check if empty\n> + * \\context This function is \\threadsafe.\n> + * \\note If the list is being modified, the return value may be out of\n> + * date by the time the function returns\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::clear()\n> + * \\brief Remove all items from the list\n> + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n> + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n> + * when this function is called.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::begin() const\n> + * \\brief Retrieve begin iterator\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::end() const\n> + * \\brief Retrieve end iterator\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::get(const Control &ctrl) const\n> + * \\brief Get the value of control \\a ctrl\n> + * \\return A std::optional containing the control value, or std::nullopt if\n> + * the control \\a ctrl is not present in the list\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::get(std::uint32_t tag) const\n> + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n> + * \\return A std::optional containing the control value, or std::nullopt if\n> + * the control is not present in the list\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> + * \\brief Set the value of control \\a ctrl to \\a value\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n> + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::merge(const ControlList &other)\n> + * \\brief Add all missing items from \\a other\n> + *\n> + * Add all items from \\a other that are not present in \\a this.\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\enum MetadataList::SetError\n> + * \\brief Error code returned by a set operation\n> + *\n> + * \\var MetadataList::SetError::UnknownTag\n> + * \\brief The tag is not supported by the metadata list\n> + * \\var MetadataList::SetError::AlreadySet\n> + * \\brief A value has already been added with the given tag\n> + * \\var MetadataList::SetError::SizeMismatch\n> + * \\brief The size of the data is not appropriate for the given tag\n> + * \\var MetadataList::SetError::TypeMismatch\n> + * \\brief The type of the value does not match the expected type\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::checkpoint() const\n> + * \\brief Create a checkpoint\n> + * \\context This function is \\threadsafe.\n> + */\n> +\n> +/**\n> + * \\class MetadataList::iterator\n> + * \\brief Iterator\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::difference_type\n> + * \\brief iterator's difference type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::value_type\n> + * \\brief iterator's value type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::pointer\n> + * \\brief iterator's pointer type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::reference\n> + * \\brief iterator's reference type\n> + */\n> +\n> +/**\n> + * \\typedef MetadataList::iterator::iterator_category\n> + * \\brief iterator's category\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator*()\n> + * \\brief Retrieve value at iterator\n> + * \\return A \\a ControlListView representing the value\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n> + * \\brief Check if two iterators are equal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n> + * \\brief Check if two iterators are not equal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator++(int)\n> + * \\brief Advance the iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::iterator::operator++()\n> + * \\brief Advance the iterator\n> + */\n> +\n> +/**\n> + * \\class MetadataList::Diff\n> + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n\nPersonally it was hard to figure out what this is meant to do and how it's\nmeant to use... until I read the test, then it all made sense. imo it would be\nuseful to expand the documentation to cover those so that people don't have to\ngo over to the test case to figure it out.\n\nI suppose the documentation for the functions below describe how to use it, so\nmaybe just adding the raison d'etre is good enough here. Expecially since this\nis meant to be used by applications, right?\n\n\nAs dense and intricate this is, I like the ideas here.\n\n\nThanks,\n\nPaul\n\n> + * \\sa Camera::metadataAvailable\n> + * \\internal\n> + * \\sa MetadataList::Checkpoint::diffSince()\n> + * \\endinternal\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::list() const\n> + * \\brief Retrieve the associated MetadataList\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::size() const\n> + * \\brief Retrieve the number of metadata items designated\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::empty() const\n> + * \\brief Check if any metadata items are designated\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::operator bool() const\n> + * \\copydoc MetadataList::Diff::empty() const\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n> + * \\copydoc MetadataList::get(const Control &ctrl) const\n> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> + * even if it is otherwise present in the backing MetadataList.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n> + * \\copydoc MetadataList::get(std::uint32_t tag) const\n> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> + * even if it is otherwise present in the backing MetadataList.\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::begin() const\n> + * \\brief Retrieve the begin iterator\n> + */\n> +\n> +/**\n> + * \\fn MetadataList::Diff::end() const\n> + * \\brief Retrieve the end iterator\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\class MetadataList::Checkpoint\n> + * \\brief Designates a point in the stream of metadata items\n> + *\n> + * A Checkpoint object designates a point in the stream of metadata items in the associated\n> + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n> + * added to the list after the designated point using diffSince().\n> + */\n> +\n> +/**\n> + * \\internal\n> + * \\fn MetadataList::Checkpoint::diffSince() const\n> + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n> + */\n> +\n> +} /* namespace libcamera */\n> diff --git a/test/controls/meson.build b/test/controls/meson.build\n> index 763f8905e..b68a4fc53 100644\n> --- a/test/controls/meson.build\n> +++ b/test/controls/meson.build\n> @@ -5,6 +5,7 @@ control_tests = [\n> {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n> {'name': 'control_list', 'sources': ['control_list.cpp']},\n> {'name': 'control_value', 'sources': ['control_value.cpp']},\n> + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n> ]\n> \n> foreach test : control_tests\n> diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n> new file mode 100644\n> index 000000000..f0872acd9\n> --- /dev/null\n> +++ b/test/controls/metadata_list.cpp\n> @@ -0,0 +1,170 @@\n> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> +/*\n> + * Copyright (C) 2025, Ideas On Board Oy\n> + *\n> + * MetadataList tests\n> + */\n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \n> +#include \n> +#include \n> +\n> +#include \"test.h\"\n> +\n> +using namespace std;\n> +using namespace libcamera;\n> +\n> +#define ASSERT(x) do { \\\n> + if (!static_cast(x)) { \\\n> + std::cerr << '`' << #x << \"` failed\" << std::endl; \\\n> + return TestFail; \\\n> + } \\\n> +} while (false)\n> +\n> +class MetadataListTest : public Test\n> +{\n> +public:\n> + MetadataListTest() = default;\n> +\n> +protected:\n> + int run() override\n> + {\n> + MetadataListPlan mlp;\n> + mlp.set(controls::ExposureTime);\n> + mlp.set(controls::ExposureValue);\n> + mlp.set(controls::ColourGains);\n> + mlp.set(controls::AfWindows, 10);\n> + mlp.set(controls::AeEnable);\n> + mlp.set(controls::SensorTimestamp);\n> +\n> + MetadataList ml(mlp);\n> +\n> + static_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n> + ASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n> +\n> + ASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n> + ASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n> +\n> + auto f1 = std::async(std::launch::async, [&] {\n> + using namespace std::chrono_literals;\n> +\n> + std::this_thread::sleep_for(500ms);\n> + ASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n> +\n> + std::this_thread::sleep_for(500ms);\n> + ASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n> +\n> + std::this_thread::sleep_for(500ms);\n> + ASSERT(ml.set(controls::ColourGains, std::array{\n> + 123.f,\n> + 456.f\n> + }) == MetadataList::SetError());\n> +\n> + std::this_thread::sleep_for(500ms);\n> + ASSERT(ml.set(controls::AfWindows, std::array{\n> + Rectangle(),\n> + Rectangle(1, 2, 3, 4),\n> + Rectangle(0x1111, 0x2222, 0x3333, 0x4444),\n> + }) == MetadataList::SetError());\n> +\n> + return TestPass;\n> + });\n> +\n> + auto f2 = std::async(std::launch::async, [&] {\n> + for (;;) {\n> + const auto x = ml.get(controls::ExposureTime);\n> + const auto y = ml.get(controls::ExposureValue);\n> + const auto z = ml.get(controls::ColourGains);\n> + const auto w = ml.get(controls::AfWindows);\n> +\n> + if (x)\n> + ASSERT(*x == 0x1111);\n> +\n> + if (y)\n> + ASSERT(*y == 1.0f);\n> +\n> + if (z) {\n> + ASSERT(z->size() == 2);\n> + ASSERT((*z)[0] == 123.f);\n> + ASSERT((*z)[1] == 456.f);\n> + }\n> +\n> + if (w) {\n> + ASSERT(w->size() == 3);\n> + ASSERT((*w)[0].isNull());\n> + ASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n> + ASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n> + }\n> +\n> + if (x && y && z && w)\n> + break;\n> + }\n> +\n> + return TestPass;\n> + });\n> +\n> + ASSERT(f1.get() == TestPass);\n> + ASSERT(f2.get() == TestPass);\n> +\n> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n> +\n> + ASSERT(ml.get(controls::ExposureTime) == 0x1111);\n> + ASSERT(ml.get(controls::ExposureValue) == 1);\n> +\n> + for (auto &&[tag, v] : ml)\n> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> +\n> + std::cout << std::endl;\n> +\n> + ml.clear();\n> + ASSERT(ml.empty());\n> + ASSERT(ml.size() == 0);\n> +\n> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n> + ASSERT(ml.get(controls::ExposureTime) == 0x2222);\n> +\n> + auto c = ml.checkpoint();\n> +\n> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n> + ASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n> +\n> + auto d = c.diffSince();\n> + ASSERT(&d.list() == &ml);\n> +\n> + ASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n> +\n> + ASSERT(d);\n> + ASSERT(!d.empty());\n> + ASSERT(d.size() == 2);\n> + ASSERT(!d.get(controls::ExposureTime));\n> + ASSERT(!d.get(controls::ColourGains));\n> + ASSERT(!d.get(controls::AfWindows));\n> + ASSERT(d.get(controls::ExposureValue) == 2);\n> + ASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n> +\n> + for (auto &&[tag, v] : d)\n> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> +\n> + /* Test if iterators work with algorithms. */\n> + std::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n> + return false;\n> + });\n> +\n> +#if 0\n> + {\n> + auto it = ml.begin();\n> + ml.clear();\n> + std::ignore = *it; /* Trigger ASAN. */\n> + }\n> +#endif\n> +\n> + return TestPass;\n> + }\n> +};\n> +\n> +TEST_REGISTER(MetadataListTest)\n> -- \n> 2.50.1\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 7B460BE173\n\tfor ;\n\tWed, 17 Sep 2025 12:03:22 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id F2CAB6936F;\n\tWed, 17 Sep 2025 14:03:20 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CCCAD69367\n\tfor ;\n\tWed, 17 Sep 2025 14:03:03 +0200 (CEST)","from neptunite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:b5ec:f217:6590:d4ee])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 9D0C06AE;\n\tWed, 17 Sep 2025 14:01:42 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"ZCNIp8nZ\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1758110503;\n\tbh=l/CdtY3nV11SJ4kDOO64W5pf40K8mT4kqTmOTe/Oub8=;\n\th=In-Reply-To:References:Subject:From:To:Date:From;\n\tb=ZCNIp8nZjtRoLODiY8bcevPUSD93+XJF8mYMee7sLb5Ja1dmsbwdeEtYhf1rpDMDM\n\tutsitPK5+zU1K9xIiqbeM8BrcrZklUXsVJQQevCPhBxqH4njkbMlnMrg8VdvQzmeHw\n\t1IpBtj5XPo4rbR/u1mDwOCNsZcg5l9eOOQrJOC2c=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","From":"Paul Elder ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Wed, 17 Sep 2025 21:02:54 +0900","Message-ID":"<175811057476.2127323.6616992472481774741@neptunite.rasen.tech>","User-Agent":"alot/0.0.0","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":35959,"web_url":"https://patchwork.libcamera.org/comment/35959/","msgid":"<2322dac9-04a3-49cf-8faa-bfe86ecadaee@ideasonboard.com>","date":"2025-09-24T08:59:04","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":216,"url":"https://patchwork.libcamera.org/api/people/216/","name":"Barnabás Pőcze","email":"barnabas.pocze@ideasonboard.com"},"content":"Hi\n\n2025. 09. 17. 14:02 keltezéssel, Paul Elder írta:\n> Hi Barnabás,\n> \n> Thanks for the patch.\n> \n> Quoting Barnabás Pőcze (2025-07-21 19:46:07)\n>> Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n>> reported by a camera for a given request. Previously, a `ControlList` was\n>> used for this purpose. The reason for introducing a separate type is to\n>> simplify the access to the returned metadata during the entire lifetime\n>> of a request.\n>>\n>> Specifically, for early metadata completion to be easily usable it should be\n>> guaranteed that any completed metadata item can be accessed and looked up\n>> at least until the associated requested is reused with `Request::reuse()`.\n>>\n>> However, when a metadata item is completed early, the pipeline handler\n>> might still work on the request in the `CameraManager`'s private thread,\n>> therefore there is an inherent synchronization issue when an application\n>> accesses early metadata.\n>>\n>> Restricting the user to only access the metadata items of a not yet completed\n>> request in the early metadata availability signal handler by ways of\n>> documenting or enforcing it at runtime could be an option, but it is not\n>> too convenient for the user.\n>>\n>> The current `ControlList` implementation employs an `std::unordered_map`,\n>> so pointers remain stable when the container is modified, so an application\n>> could keep accessing particular metadata items outside the signal handler,\n>> but this fact is far from obvious, and the user would still not be able\n>> to make a copy of all metadata or do lookups based on the numeric ids or\n>> the usual `libcamera::Control<>` objects, thus some type safety is lost.\n>>\n>> The above also requires that each metadata item is only completed once for\n>> a given request, but this does not appear to be serious limitation,\n>> and in fact, this restriction is enforced by `MetadataList`.\n>>\n>> The introduced `MetadataList` supports single writer - multiple reader\n>> scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n>> without introducing data races or other synchronization issues. This is\n>> achieved by requiring the possible set of metadata items to be known\n>> (such set is stored in a `MetadataListPlan` object). Based on the this\n>> plan, a single contiguous allocation is made to accommodate all potential\n>> metadata items. Due to this single contiguous allocation that is not modified\n>> during the lifetime of a `MetadataList` and atomic modifications, it is\n>> possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n>> there is only ever a single writer.\n>>\n>> Signed-off-by: Barnabás Pőcze \n> \n> In general, the ideas and design look good to me. Jacopo picked at a lot more\n> than I could.\n> \n> I only have a few comments/questions.\n> \n> Not sure if it's because it's still RFC phase or because everything is\n> internal, but most of this is \"I want documentation please\".\n> \n>> ---\n>> changes in v2:\n>> * remove multiple not strictly necessary functions\n>> ---\n>> include/libcamera/meson.build | 2 +\n>> include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n>> include/libcamera/metadata_list_plan.h | 130 ++++++\n>> src/libcamera/meson.build | 1 +\n>> src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n>> test/controls/meson.build | 1 +\n>> test/controls/metadata_list.cpp | 170 ++++++++\n>> 7 files changed, 1195 insertions(+)\n>> create mode 100644 include/libcamera/metadata_list.h\n>> create mode 100644 include/libcamera/metadata_list_plan.h\n>> create mode 100644 src/libcamera/metadata_list.cpp\n>> create mode 100644 test/controls/metadata_list.cpp\n>>\n>> diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n>> index 30ea76f94..410b548dd 100644\n>> --- a/include/libcamera/meson.build\n>> +++ b/include/libcamera/meson.build\n>> @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n>> 'framebuffer_allocator.h',\n>> 'geometry.h',\n>> 'logging.h',\n>> + 'metadata_list.h',\n>> + 'metadata_list_plan.h',\n>> 'orientation.h',\n>> 'pixel_format.h',\n>> 'request.h',\n>> diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n>> new file mode 100644\n>> index 000000000..7fe3dbbab\n>> --- /dev/null\n>> +++ b/include/libcamera/metadata_list.h\n>> @@ -0,0 +1,547 @@\n>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>> +/*\n>> + * Copyright (C) 2025, Ideas On Board Oy\n>> + *\n>> + * Metadata list\n>> + */\n>> +\n>> +#pragma once\n>> +\n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +\n>> +#include \n>> +#include \n>> +#include \n>> +\n>> +#include \n>> +#include \n>> +\n>> +// TODO: want this?\n> \n> Given that we're manually doing a bunch of binary formatting I think we do want\n> ASAN.\n> \n>> +#if __has_include()\n>> +#if __SANITIZE_ADDRESS__ /* gcc */\n>> +#include \n>> +#define HAS_ASAN 1\n>> +#elif defined(__has_feature)\n>> +#if __has_feature(address_sanitizer) /* clang */\n>> +#include \n>> +#define HAS_ASAN 1\n>> +#endif\n>> +#endif\n>> +#endif\n>> +\n>> +namespace libcamera {\n>> +\n>> +class MetadataList\n>> +{\n>> +private:\n>> + struct ValueParams {\n>> + ControlType type;\n>> + bool isArray;\n>> + std::uint32_t numElements;\n>> + };\n>> +\n>> + struct Entry {\n>> + const std::uint32_t tag;\n>> + const std::uint32_t capacity;\n>> + const std::uint32_t alignment;\n>> + const ControlType type;\n>> + bool isArray;\n>> +\n>> + static constexpr std::uint32_t invalidOffset = -1;\n>> + /*\n>> + * Offset from the beginning of the allocation, and\n>> + * and _not_ relative to `contentOffset_`.\n>> + */\n>> + std::atomic_uint32_t headerOffset = invalidOffset;\n>> +\n>> + [[nodiscard]] std::optional hasValue() const\n>> + {\n>> + auto offset = headerOffset.load(std::memory_order_relaxed);\n>> + if (offset == invalidOffset)\n>> + return {};\n>> +\n>> + return offset;\n>> + }\n>> +\n>> + [[nodiscard]] std::optional acquireData() const\n>> + {\n>> + auto offset = hasValue();\n>> + if (offset) {\n>> + /* sync with release-store on `headerOffset` in `MetadataList::set()` */\n>> + std::atomic_thread_fence(std::memory_order_acquire);\n>> + }\n>> +\n>> + return offset;\n>> + }\n>> + };\n>> +\n>> + struct ValueHeader {\n>> + std::uint32_t tag;\n>> + std::uint32_t size;\n>> + std::uint32_t alignment;\n>> + ValueParams params;\n>> + };\n> \n> I know these structs are internal but there are so many moving parts that I'm\n> just barely able to hold on to the meanings of everything... can we have\n> documentation please...? Especially for when our caches are all flushed when we\n> need to work on this in the future, I think documentation will speed up cache\n> refreshing.\n\nI can't seem to convince doxygen to generate documentation for private types...\n\n\n> \n>> +\n>> + struct State {\n>> + std::uint32_t count;\n>> + std::uint32_t fill;\n>> + };\n>> +\n>> +public:\n>> + explicit MetadataList(const MetadataListPlan &plan)\n>> + : capacity_(plan.size()),\n>> + contentOffset_(MetadataList::contentOffset(capacity_)),\n>> + alloc_(contentOffset_)\n>> + {\n>> + for (const auto &[tag, e] : plan) {\n>> + alloc_ += sizeof(ValueHeader);\n>> + alloc_ += e.alignment - 1; // XXX: this is the maximum\n>> + alloc_ += e.size * e.numElements;\n>> + alloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n> \n> Why are we taking both the size and align of ValueHeader?\n\nThe idea is that size+alignment-1 bytes are always enough to properly align\nthe thing regardless of the alignment of the underlying storage. Another\noption that is being considered is using a fixed alignment of 8, 16, etc.\nfor everything.\n\n\n> \n>> + }\n>> +\n>> + p_ = static_cast(::operator new(alloc_));\n>> +\n>> + auto *entries = reinterpret_cast(p_ + entriesOffset());\n>> + auto it = plan.begin();\n>> +\n>> + for (std::size_t i = 0; i < capacity_; i++, ++it) {\n>> + const auto &[tag, e] = *it;\n>> +\n>> + new (&entries[i]) Entry{\n>> + .tag = tag,\n>> + .capacity = e.size * e.numElements,\n>> + .alignment = e.alignment,\n>> + .type = e.type,\n>> + .isArray = e.isArray,\n>> + };\n>> + }\n>> +\n>> +#if HAS_ASAN\n>> + ::__sanitizer_annotate_contiguous_container(\n>> + p_ + contentOffset_, p_ + alloc_,\n>> + p_ + alloc_, p_ + contentOffset_\n>> + );\n>> +#endif\n>> + }\n>> +\n>> + MetadataList(const MetadataList &) = delete;\n>> + MetadataList(MetadataList &&) = delete;\n>> +\n>> + MetadataList &operator=(const MetadataList &) = delete;\n>> + MetadataList &operator=(MetadataList &&) = delete;\n>> +\n>> + ~MetadataList()\n>> + {\n>> +#if HAS_ASAN\n>> + /*\n>> + * The documentation says the range apparently has to be\n>> + * restored to its initial state before it is deallocated.\n>> + */\n>> + ::__sanitizer_annotate_contiguous_container(\n>> + p_ + contentOffset_, p_ + alloc_,\n>> + p_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n>> + );\n>> +#endif\n>> +\n>> + ::operator delete(p_, alloc_);\n>> + }\n>> +\n>> + // TODO: want these?\n> \n> I think it's useful.\n> \n>> + [[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n>> + [[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n>> +\n>> + enum class SetError {\n>> + UnknownTag = 1,\n>> + AlreadySet,\n>> + SizeMismatch,\n>> + TypeMismatch,\n>> + };\n>> +\n>> + [[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n>> + {\n>> + auto *e = find(tag);\n>> + if (!e)\n>> + return SetError::UnknownTag;\n>> +\n>> + return set(*e, v);\n>> + }\n>> +\n>> + template\n>> + /* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n>> + {\n>> + using TypeInfo = libcamera::details::control_type;\n>> +\n>> + if constexpr (TypeInfo::size > 0) {\n>> + static_assert(std::is_trivially_copyable_v);\n>> +\n>> + return set(ctrl.id(), {\n>> + TypeInfo::value,\n>> + true,\n>> + value.size(),\n>> + reinterpret_cast(value.data()),\n>> + });\n>> + } else {\n>> + static_assert(std::is_trivially_copyable_v);\n>> +\n>> + return set(ctrl.id(), {\n>> + TypeInfo::value,\n>> + false,\n>> + 1,\n>> + reinterpret_cast(&value),\n>> + });\n>> + }\n>> + }\n>> +\n>> + template\n>> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n>> + {\n>> + ControlValueView v = get(ctrl.id());\n>> +\n>> + return v ? std::optional(v.get()) : std::nullopt;\n>> + }\n>> +\n>> + // TODO: operator ControlListView() const ?\n>> + // TODO: explicit operator ControlList() const ?\n>> +\n>> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n>> + {\n>> + const auto *e = find(tag);\n>> + if (!e)\n>> + return {};\n>> +\n>> + return data_of(*e);\n> \n> iirc Jacopo mentioned casing but this is a well-known common macro that imo\n> it's ok to keep as-is. imo tho.\n> \n>> + }\n>> +\n>> + void clear()\n>> + {\n>> + for (auto &e : entries())\n>> + e.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n>> +\n>> + [[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n>> +\n>> +#if HAS_ASAN\n>> + ::__sanitizer_annotate_contiguous_container(\n>> + p_ + contentOffset_, p_ + alloc_,\n>> + p_ + contentOffset_ + s.fill, p_ + contentOffset_\n>> + );\n>> +#endif\n>> + }\n>> +\n>> + class iterator\n>> + {\n>> + public:\n>> + using difference_type = std::ptrdiff_t;\n>> + using value_type = std::pair;\n>> + using pointer = void;\n>> + using reference = value_type;\n>> + using iterator_category = std::forward_iterator_tag;\n>> +\n>> + iterator() = default;\n>> +\n>> + iterator& operator++()\n>> + {\n>> + const auto &h = header();\n>> +\n>> + p_ += sizeof(h);\n>> + p_ = details::align::up(p_, h.alignment);\n>> + p_ += h.size;\n>> + p_ = details::align::up(p_, alignof(decltype(h)));\n> \n> Ok I think this makes sense... I'd really like documentation on the binary\n> format somewhere though...\n> \n>> +\n>> + return *this;\n>> + }\n>> +\n>> + iterator operator++(int)\n>> + {\n>> + auto copy = *this;\n>> + ++*this;\n>> + return copy;\n>> + }\n>> +\n>> + [[nodiscard]] reference operator*() const\n>> + {\n>> + const auto &h = header();\n>> + const auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n>> +\n>> + return { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n>> + }\n>> +\n>> + [[nodiscard]] bool operator==(const iterator &other) const\n>> + {\n>> + return p_ == other.p_;\n>> + }\n>> +\n>> + [[nodiscard]] bool operator!=(const iterator &other) const\n>> + {\n>> + return !(*this == other);\n>> + }\n>> +\n>> + private:\n>> + iterator(const std::byte *p)\n>> + : p_(p)\n>> + {\n>> + }\n>> +\n>> + [[nodiscard]] const ValueHeader &header() const\n>> + {\n>> + return *reinterpret_cast(p_);\n>> + }\n>> +\n>> + friend MetadataList;\n>> +\n>> + const std::byte *p_ = nullptr;\n>> + };\n>> +\n>> + [[nodiscard]] iterator begin() const\n>> + {\n>> + return { p_ + contentOffset_ };\n>> + }\n>> +\n>> + [[nodiscard]] iterator end() const\n>> + {\n>> + return { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n>> + }\n>> +\n>> + class Diff\n>> + {\n>> + public:\n>> + // TODO: want these?\n>> + [[nodiscard]] explicit operator bool() const { return !empty(); }\n>> + [[nodiscard]] bool empty() const { return start_ == stop_; }\n>> + [[nodiscard]] std::size_t size() const { return changed_; }\n>> + [[nodiscard]] const MetadataList &list() const { return *l_; }\n> \n> I think they'd be helpful to have.\n> \n>> +\n>> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n>> + {\n>> + const auto *e = l_->find(tag);\n>> + if (!e)\n>> + return {};\n>> +\n>> + auto o = e->acquireData();\n>> + if (!o)\n>> + return {};\n>> +\n>> + if (!(start_ <= *o && *o < stop_))\n>> + return {};\n>> +\n>> + return l_->data_of(*o);\n>> + }\n>> +\n>> + template\n>> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n>> + {\n>> + ControlValueView v = get(ctrl.id());\n>> +\n>> + return v ? std::optional(v.get()) : std::nullopt;\n>> + }\n>> +\n>> + [[nodiscard]] iterator begin() const\n>> + {\n>> + return { l_->p_ + start_ };\n>> + }\n>> +\n>> + [[nodiscard]] iterator end() const\n>> + {\n>> + return { l_->p_ + stop_ };\n>> + }\n>> +\n>> + private:\n>> + Diff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n> \n> I can get what the meanings of these are... by reading Checkpoint below. imo\n> documentation will help smooth out the process of understanding this.\n> \n>> + : l_(&l),\n>> + changed_(changed),\n>> + start_(l.contentOffset_ + oldFill),\n>> + stop_(l.contentOffset_ + newFill)\n>> + {\n>> + }\n>> +\n>> + friend MetadataList;\n>> + friend struct Checkpoint;\n>> +\n>> + const MetadataList *l_ = nullptr;\n>> + std::size_t changed_;\n>> + std::size_t start_;\n>> + std::size_t stop_;\n>> + };\n>> +\n>> + Diff merge(const ControlList &other)\n>> + {\n>> + // TODO: check id map of `other`?\n> \n> Since Diff can't be constructed by the user, I think we're guaranteed that the\n> id maps match, so I don't think we need to spend resources on this. Or maybe we\n> could for a peace of mind; it's just a comparison of pointers, right?\n> \n> Speaking of which, do we need to disable or privatize the default constructors?\n\nNo default constructor is generated for either `Diff` or `Checkpoint`.\n\n\n> \n>> +\n>> + const auto c = checkpoint();\n>> +\n>> + for (const auto &[tag, value] : other) {\n>> + auto *e = find(tag);\n>> + if (e) {\n>> + [[maybe_unused]] auto r = set(*e, value);\n>> + assert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n> \n> TODO?\n\nThe question here is how to handle failure. They could be ignored entirely, or an\nassertion could be used to ensure no errors. Or a mix of the two like above. Or\n`merge()` could be probably made more \"transactional\" somehow so that it will either\nadd all items or none of them, and then the error could be directly reported to the\ncaller to handle it.\n\n\n> \n>> + }\n>> + }\n>> +\n>> + return c.diffSince();\n>> + }\n>> +\n>> + class Checkpoint\n>> + {\n>> + public:\n>> + [[nodiscard]] Diff diffSince() const\n>> + {\n>> + /* sync with release-store on `state_` in `set()` */\n>> + const auto curr = l_->state_.load(std::memory_order_acquire);\n>> +\n>> + assert(s_.count <= curr.count);\n>> + assert(s_.fill <= curr.fill);\n>> +\n>> + return {\n>> + *l_,\n>> + curr.count - s_.count,\n>> + s_.fill,\n>> + curr.fill,\n>> + };\n>> + }\n>> +\n>> + private:\n>> + Checkpoint(const MetadataList &l)\n>> + : l_(&l),\n>> + s_(l.state_.load(std::memory_order_relaxed))\n>> + {\n>> + }\n>> +\n>> + friend MetadataList;\n>> +\n>> + const MetadataList *l_ = nullptr;\n>> + State s_ = {};\n>> + };\n>> +\n>> + [[nodiscard]] Checkpoint checkpoint() const\n>> + {\n>> + return { *this };\n>> + }\n>> +\n>> +private:\n>> + [[nodiscard]] static constexpr std::size_t entriesOffset()\n>> + {\n>> + return 0;\n>> + }\n>> +\n>> + [[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n>> + {\n>> + return details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n>> + }\n>> +\n>> + [[nodiscard]] Span entries() const\n>> + {\n>> + return { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n>> + }\n>> +\n>> + [[nodiscard]] Entry *find(std::uint32_t tag) const\n>> + {\n>> + const auto entries = this->entries();\n>> + auto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n>> + return e.tag < tag;\n>> + });\n>> +\n>> + if (it == entries.end() || it->tag != tag)\n>> + return nullptr;\n>> +\n>> + return &*it;\n>> + }\n>> +\n>> + [[nodiscard]] ControlValueView data_of(const Entry &e) const\n>> + {\n>> + const auto o = e.acquireData();\n>> + return o ? data_of(*o) : ControlValueView{ };\n>> + }\n>> +\n>> + [[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n>> + {\n>> + assert(headerOffset <= alloc_ - sizeof(ValueHeader));\n>> + assert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n>> +\n>> + const auto *vh = reinterpret_cast(p_ + headerOffset);\n>> + const auto *p = reinterpret_cast(vh) + sizeof(*vh);\n>> + std::size_t avail = p_ + alloc_ - p;\n>> +\n>> + const auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n>> + assert(data);\n>> +\n>> + return { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n>> + }\n> \n> This... costs a lot to understand... I think documentation on the binary format\n> would help considerably.\n> \n>> +\n>> + [[nodiscard]] SetError set(Entry &e, ControlValueView v)\n>> + {\n>> + if (e.hasValue())\n>> + return SetError::AlreadySet;\n>> + if (e.type != v.type() || e.isArray != v.isArray())\n>> + return SetError::TypeMismatch;\n>> +\n>> + const auto src = v.data();\n>> + if (e.isArray) {\n>> + if (src.size_bytes() > e.capacity)\n>> + return SetError::SizeMismatch;\n>> + } else {\n>> + if (src.size_bytes() != e.capacity)\n>> + return SetError::SizeMismatch;\n>> + }\n>> +\n>> + auto s = state_.load(std::memory_order_relaxed);\n>> + std::byte *oldEnd = p_ + contentOffset_ + s.fill;\n>> + std::byte *p = oldEnd;\n>> +\n>> + auto *headerPtr = details::align::up(p);\n>> + auto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n>> + details::align::up(0, alignof(ValueHeader), p);\n>> +\n>> +#if HAS_ASAN\n>> + ::__sanitizer_annotate_contiguous_container(\n>> + p_ + contentOffset_, p_ + alloc_,\n>> + oldEnd, p\n>> + );\n>> +#endif\n>> +\n>> + new (headerPtr) ValueHeader{\n>> + .tag = e.tag,\n>> + .size = std::uint32_t(src.size_bytes()),\n>> + .alignment = e.alignment,\n>> + .params = {\n>> + .type = v.type(),\n>> + .isArray = v.isArray(),\n>> + .numElements = std::uint32_t(v.numElements()),\n>> + },\n>> + };\n>> + std::memcpy(dataPtr, src.data(), src.size_bytes());\n>> + e.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n>> +\n>> + s.fill += p - oldEnd;\n>> + s.count += 1;\n>> +\n>> + state_.store(s, std::memory_order_release);\n>> +\n>> + return {};\n>> + }\n>> +\n>> + std::size_t capacity_ = 0;\n>> + std::size_t contentOffset_ = -1;\n>> + std::size_t alloc_ = 0;\n>> + std::atomic state_ = State{};\n>> + std::byte *p_ = nullptr;\n> \n> I... would like some documentation on these... I'm just barely able to hang on\n> to the meanings... I think documentation would ease future maintenance when all\n> our caches have been flushed.\n> \n>> + // TODO: ControlIdMap in any way shape or form?\n> \n> I don't think you need it? I think you've designed this in a way such that\n> which ControlIdMap is being used doesn't really matter, since everything\n> originates from the MetadataListPlan.\n\nIt's probably not needed, but maybe useful to provide the same interface\nas `ControlList`, although I don't have a concrete use case in mind.\n\n\n> \n>> +\n>> + /*\n>> + * If this is problematic on a 32-bit architecture, then\n>> + * `count` can be stored in a separate atomic variable\n>> + * but then `Diff::changed_` must be removed since the fill\n>> + * level and item count cannot be retrieved atomically.\n>> + */\n>> + static_assert(decltype(state_)::is_always_lock_free);\n>> +};\n>> +\n>> +} /* namespace libcamera */\n>> +\n>> +#undef HAS_ASAN\n>> diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n>> new file mode 100644\n>> index 000000000..2ed35c54f\n>> --- /dev/null\n>> +++ b/include/libcamera/metadata_list_plan.h\n>> @@ -0,0 +1,130 @@\n>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>> +/*\n>> + * Copyright (C) 2025, Ideas On Board Oy\n>> + */\n>> +\n>> +#pragma once\n>> +\n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +#include \n>> +\n>> +#include \n>> +\n>> +#include \n>> +\n>> +namespace libcamera {\n>> +\n>> +class MetadataListPlan\n>> +{\n>> +public:\n>> + struct Entry {\n>> + std::uint32_t size;\n>> + std::uint32_t alignment; // TODO: is this necessary?\n>> + std::uint32_t numElements;\n>> + ControlType type;\n>> + bool isArray;\n>> + };\n>> +\n>> + [[nodiscard]] bool empty() const { return items_.empty(); }\n>> + [[nodiscard]] std::size_t size() const { return items_.size(); }\n>> + [[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n>> + [[nodiscard]] decltype(auto) end() const { return items_.end(); }\n>> + void clear() { items_.clear(); }\n>> +\n>> + template<\n>> + typename T,\n>> + std::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n>> + >\n>> + decltype(auto) set(const Control &ctrl)\n>> + {\n>> + if constexpr (libcamera::details::control_type::size > 0) {\n>> + static_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n>> +\n>> + return set(\n>> + ctrl.id(),\n>> + libcamera::details::control_type::size,\n>> + true\n>> + );\n>> + } else {\n>> + return set(ctrl.id(), 1, false);\n>> + }\n>> + }\n>> +\n>> + template<\n>> + typename T,\n>> + std::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n>> + >\n>> + decltype(auto) set(const Control &ctrl, std::size_t numElements)\n>> + {\n>> + return set(ctrl.id(), numElements, true);\n>> + }\n>> +\n>> + [[nodiscard]] bool set(std::uint32_t tag,\n>> + std::size_t size, std::size_t alignment,\n>> + std::size_t numElements, ControlType type, bool isArray)\n>> + {\n>> + if (size == 0 || size > std::numeric_limits::max())\n>> + return false;\n>> + if (alignment > std::numeric_limits::max())\n>> + return false;\n>> + if (!details::cxx20::has_single_bit(alignment))\n>> + return false;\n>> + if (numElements > std::numeric_limits::max() / size)\n>> + return false;\n>> + if (!isArray && numElements != 1)\n>> + return false;\n>> +\n>> + items_.insert_or_assign(tag, Entry{\n>> + .size = std::uint32_t(size),\n>> + .alignment = std::uint32_t(alignment),\n>> + .numElements = std::uint32_t(numElements),\n>> + .type = type,\n>> + .isArray = isArray,\n>> + });\n>> +\n>> + return true;\n>> + }\n>> +\n>> + [[nodiscard]] const Entry *get(std::uint32_t tag) const\n>> + {\n>> + auto it = items_.find(tag);\n>> + if (it == items_.end())\n>> + return nullptr;\n>> +\n>> + return &it->second;\n>> + }\n>> +\n>> + [[nodiscard]] const Entry *get(const ControlId &cid) const\n>> + {\n>> + const auto *e = get(cid.id());\n>> + if (!e)\n>> + return nullptr;\n>> +\n>> + if (e->type != cid.type() || e->isArray != cid.isArray())\n>> + return nullptr;\n>> +\n>> + return e;\n>> + }\n>> +\n>> +private:\n>> + std::map items_;\n>> +\n>> + template\n>> + decltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n>> + {\n>> + static_assert(std::is_trivially_copyable_v);\n>> +\n>> + [[maybe_unused]] bool ok = set(tag,\n>> + sizeof(T), alignof(T),\n>> + numElements, details::control_type::value, isArray);\n>> + assert(ok);\n>> +\n>> + return *this;\n>> + }\n>> +};\n>> +\n>> +} /* namespace libcamera */\n>> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n>> index de1eb99b2..8c5ce4503 100644\n>> --- a/src/libcamera/meson.build\n>> +++ b/src/libcamera/meson.build\n>> @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n>> 'framebuffer.cpp',\n>> 'framebuffer_allocator.cpp',\n>> 'geometry.cpp',\n>> + 'metadata_list.cpp',\n>> 'orientation.cpp',\n>> 'pixel_format.cpp',\n>> 'request.cpp',\n>> diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n>> new file mode 100644\n>> index 000000000..ebefdfdad\n>> --- /dev/null\n>> +++ b/src/libcamera/metadata_list.cpp\n>> @@ -0,0 +1,344 @@\n>> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n>> +/*\n>> + * Copyright (C) 2025, Ideas On Board Oy\n>> + */\n>> +\n>> +#include \n>> +\n>> +namespace libcamera {\n>> +\n>> +/**\n>> + * \\class MetadataListPlan\n>> + * \\brief Class to hold the possible set of metadata items for a MetadataList\n>> + */\n>> +\n>> +/**\n>> + * \\class MetadataListPlan::Entry\n>> + * \\brief Details of a metadata item\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\var MetadataListPlan::Entry::size\n>> + * \\brief Number of bytes in a single element\n>> + *\n>> + * \\var MetadataListPlan::Entry::alignment\n>> + * \\brief Required alignment of the elements\n>> + * \\endinternal\n>> + *\n>> + * \\var MetadataListPlan::Entry::numElements\n>> + * \\brief Number of elements in the value\n>> + * \\sa ControlValueView::numElements()\n>> + *\n>> + * \\var MetadataListPlan::Entry::type\n>> + * \\brief The type of the value\n>> + * \\sa ControlValueView::type()\n>> + *\n>> + * \\var MetadataListPlan::Entry::isArray\n>> + * \\brief Whether or not the value is array-like\n>> + * \\sa ControlValueView::isArray()\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::begin() const\n>> + * \\brief Retrieve the begin iterator\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::end() const\n>> + * \\brief Retrieve the end iterator\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::size() const\n>> + * \\brief Retrieve the number of entries\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::empty() const\n>> + * \\brief Check if empty\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataListPlan::clear()\n>> + * \\brief Remove all controls\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataListPlan::set(const Control &ctrl)\n>> + * \\brief Add an entry for the given control to the metadata list plan\n>> + * \\param[in] ctrl The control\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n>> + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n>> + * \\param[in] ctrl The control\n>> + * \\param[in] count The maximum number of elements\n>> + *\n>> + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n>> + * capacity of \\a count elements.\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataListPlan::set(std::uint32_t tag,\n>> + * std::size_t size, std::size_t alignment,\n>> + * std::size_t count, ControlType type, bool isArray)\n>> + * \\brief Add an entry to the metadata list plan\n>> + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n>> + * would result in an invalid entry\n>> + *\n>> + * This functions adds an entry with essentially arbitrary parameters, without deriving\n>> + * them from a given ControlId instance. This is mainly used when deserializing.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n>> + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataListPlan::get(const ControlId &cid) const\n>> + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n>> + *\n>> + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n>> + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n>> + */\n>> +\n>> +/**\n>> + * \\class MetadataList\n>> + * \\brief Class to hold metadata items\n> \n> I'd like this to be expanded. What's special about it? Why can't we just use\n> ControlList? How is the data formatted in the custom data container (this could\n> be internal in the code)? What's the significance and difference between Entry\n> and ValueHeader?\n\nI'm now wondering if the documentation in the next patch should be merged into this.\n\n\n> \n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n>> + * \\brief Construct a metadata list according to \\a plan\n>> + *\n>> + * Construct a metadata list according to the provided \\a plan.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::size() const\n>> + * \\brief Retrieve the number of controls\n>> + * \\context This function is \\threadsafe.\n>> + * \\note If the list is being modified, the return value may be out of\n>> + * date by the time the function returns\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::empty() const\n>> + * \\brief Check if empty\n>> + * \\context This function is \\threadsafe.\n>> + * \\note If the list is being modified, the return value may be out of\n>> + * date by the time the function returns\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::clear()\n>> + * \\brief Remove all items from the list\n>> + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n>> + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n>> + * when this function is called.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::begin() const\n>> + * \\brief Retrieve begin iterator\n>> + * \\context This function is \\threadsafe.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::end() const\n>> + * \\brief Retrieve end iterator\n>> + * \\context This function is \\threadsafe.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::get(const Control &ctrl) const\n>> + * \\brief Get the value of control \\a ctrl\n>> + * \\return A std::optional containing the control value, or std::nullopt if\n>> + * the control \\a ctrl is not present in the list\n>> + * \\context This function is \\threadsafe.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::get(std::uint32_t tag) const\n>> + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n>> + * \\return A std::optional containing the control value, or std::nullopt if\n>> + * the control is not present in the list\n>> + * \\context This function is \\threadsafe.\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n>> + * \\brief Set the value of control \\a ctrl to \\a value\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n>> + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::merge(const ControlList &other)\n>> + * \\brief Add all missing items from \\a other\n>> + *\n>> + * Add all items from \\a other that are not present in \\a this.\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\enum MetadataList::SetError\n>> + * \\brief Error code returned by a set operation\n>> + *\n>> + * \\var MetadataList::SetError::UnknownTag\n>> + * \\brief The tag is not supported by the metadata list\n>> + * \\var MetadataList::SetError::AlreadySet\n>> + * \\brief A value has already been added with the given tag\n>> + * \\var MetadataList::SetError::SizeMismatch\n>> + * \\brief The size of the data is not appropriate for the given tag\n>> + * \\var MetadataList::SetError::TypeMismatch\n>> + * \\brief The type of the value does not match the expected type\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::checkpoint() const\n>> + * \\brief Create a checkpoint\n>> + * \\context This function is \\threadsafe.\n>> + */\n>> +\n>> +/**\n>> + * \\class MetadataList::iterator\n>> + * \\brief Iterator\n>> + */\n>> +\n>> +/**\n>> + * \\typedef MetadataList::iterator::difference_type\n>> + * \\brief iterator's difference type\n>> + */\n>> +\n>> +/**\n>> + * \\typedef MetadataList::iterator::value_type\n>> + * \\brief iterator's value type\n>> + */\n>> +\n>> +/**\n>> + * \\typedef MetadataList::iterator::pointer\n>> + * \\brief iterator's pointer type\n>> + */\n>> +\n>> +/**\n>> + * \\typedef MetadataList::iterator::reference\n>> + * \\brief iterator's reference type\n>> + */\n>> +\n>> +/**\n>> + * \\typedef MetadataList::iterator::iterator_category\n>> + * \\brief iterator's category\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::iterator::operator*()\n>> + * \\brief Retrieve value at iterator\n>> + * \\return A \\a ControlListView representing the value\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n>> + * \\brief Check if two iterators are equal\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n>> + * \\brief Check if two iterators are not equal\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::iterator::operator++(int)\n>> + * \\brief Advance the iterator\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::iterator::operator++()\n>> + * \\brief Advance the iterator\n>> + */\n>> +\n>> +/**\n>> + * \\class MetadataList::Diff\n>> + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n> \n> Personally it was hard to figure out what this is meant to do and how it's\n> meant to use... until I read the test, then it all made sense. imo it would be\n> useful to expand the documentation to cover those so that people don't have to\n> go over to the test case to figure it out.\n\nI believe the documentation of Camera::metadataAvailable (added later) provides a\ngood example of how this type should be used. I didn't want to duplicate it and\nthat seemed like a better location, but not entirely sure which is the best location.\n\n\n> \n> I suppose the documentation for the functions below describe how to use it, so\n> maybe just adding the raison d'etre is good enough here. Expecially since this\n> is meant to be used by applications, right?\n\nThis type is meant to be used by applications, yes. Could you clarify what\nyou mean exactly?\n\n\nRegards,\nBarnabás Pőcze\n\n> \n> \n> As dense and intricate this is, I like the ideas here.\n> \n> \n> Thanks,\n> \n> Paul\n> \n>> + * \\sa Camera::metadataAvailable\n>> + * \\internal\n>> + * \\sa MetadataList::Checkpoint::diffSince()\n>> + * \\endinternal\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::list() const\n>> + * \\brief Retrieve the associated MetadataList\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::size() const\n>> + * \\brief Retrieve the number of metadata items designated\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::empty() const\n>> + * \\brief Check if any metadata items are designated\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::operator bool() const\n>> + * \\copydoc MetadataList::Diff::empty() const\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n>> + * \\copydoc MetadataList::get(const Control &ctrl) const\n>> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n>> + * even if it is otherwise present in the backing MetadataList.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n>> + * \\copydoc MetadataList::get(std::uint32_t tag) const\n>> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n>> + * even if it is otherwise present in the backing MetadataList.\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::begin() const\n>> + * \\brief Retrieve the begin iterator\n>> + */\n>> +\n>> +/**\n>> + * \\fn MetadataList::Diff::end() const\n>> + * \\brief Retrieve the end iterator\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\class MetadataList::Checkpoint\n>> + * \\brief Designates a point in the stream of metadata items\n>> + *\n>> + * A Checkpoint object designates a point in the stream of metadata items in the associated\n>> + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n>> + * added to the list after the designated point using diffSince().\n>> + */\n>> +\n>> +/**\n>> + * \\internal\n>> + * \\fn MetadataList::Checkpoint::diffSince() const\n>> + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n>> + */\n>> +\n>> +} /* namespace libcamera */\n>> diff --git a/test/controls/meson.build b/test/controls/meson.build\n>> index 763f8905e..b68a4fc53 100644\n>> --- a/test/controls/meson.build\n>> +++ b/test/controls/meson.build\n>> @@ -5,6 +5,7 @@ control_tests = [\n>> {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n>> {'name': 'control_list', 'sources': ['control_list.cpp']},\n>> {'name': 'control_value', 'sources': ['control_value.cpp']},\n>> + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n>> ]\n>> \n>> foreach test : control_tests\n>> diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n>> new file mode 100644\n>> index 000000000..f0872acd9\n>> --- /dev/null\n>> +++ b/test/controls/metadata_list.cpp\n>> @@ -0,0 +1,170 @@\n>> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n>> +/*\n>> + * Copyright (C) 2025, Ideas On Board Oy\n>> + *\n>> + * MetadataList tests\n>> + */\n>> +\n>> +#include \n>> +#include \n>> +#include \n>> +\n>> +#include \n>> +#include \n>> +#include \n>> +\n>> +#include \"test.h\"\n>> +\n>> +using namespace std;\n>> +using namespace libcamera;\n>> +\n>> +#define ASSERT(x) do { \\\n>> + if (!static_cast(x)) { \\\n>> + std::cerr << '`' << #x << \"` failed\" << std::endl; \\\n>> + return TestFail; \\\n>> + } \\\n>> +} while (false)\n>> +\n>> +class MetadataListTest : public Test\n>> +{\n>> +public:\n>> + MetadataListTest() = default;\n>> +\n>> +protected:\n>> + int run() override\n>> + {\n>> + MetadataListPlan mlp;\n>> + mlp.set(controls::ExposureTime);\n>> + mlp.set(controls::ExposureValue);\n>> + mlp.set(controls::ColourGains);\n>> + mlp.set(controls::AfWindows, 10);\n>> + mlp.set(controls::AeEnable);\n>> + mlp.set(controls::SensorTimestamp);\n>> +\n>> + MetadataList ml(mlp);\n>> +\n>> + static_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n>> + ASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n>> +\n>> + ASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n>> + ASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n>> +\n>> + auto f1 = std::async(std::launch::async, [&] {\n>> + using namespace std::chrono_literals;\n>> +\n>> + std::this_thread::sleep_for(500ms);\n>> + ASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n>> +\n>> + std::this_thread::sleep_for(500ms);\n>> + ASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n>> +\n>> + std::this_thread::sleep_for(500ms);\n>> + ASSERT(ml.set(controls::ColourGains, std::array{\n>> + 123.f,\n>> + 456.f\n>> + }) == MetadataList::SetError());\n>> +\n>> + std::this_thread::sleep_for(500ms);\n>> + ASSERT(ml.set(controls::AfWindows, std::array{\n>> + Rectangle(),\n>> + Rectangle(1, 2, 3, 4),\n>> + Rectangle(0x1111, 0x2222, 0x3333, 0x4444),\n>> + }) == MetadataList::SetError());\n>> +\n>> + return TestPass;\n>> + });\n>> +\n>> + auto f2 = std::async(std::launch::async, [&] {\n>> + for (;;) {\n>> + const auto x = ml.get(controls::ExposureTime);\n>> + const auto y = ml.get(controls::ExposureValue);\n>> + const auto z = ml.get(controls::ColourGains);\n>> + const auto w = ml.get(controls::AfWindows);\n>> +\n>> + if (x)\n>> + ASSERT(*x == 0x1111);\n>> +\n>> + if (y)\n>> + ASSERT(*y == 1.0f);\n>> +\n>> + if (z) {\n>> + ASSERT(z->size() == 2);\n>> + ASSERT((*z)[0] == 123.f);\n>> + ASSERT((*z)[1] == 456.f);\n>> + }\n>> +\n>> + if (w) {\n>> + ASSERT(w->size() == 3);\n>> + ASSERT((*w)[0].isNull());\n>> + ASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n>> + ASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n>> + }\n>> +\n>> + if (x && y && z && w)\n>> + break;\n>> + }\n>> +\n>> + return TestPass;\n>> + });\n>> +\n>> + ASSERT(f1.get() == TestPass);\n>> + ASSERT(f2.get() == TestPass);\n>> +\n>> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n>> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n>> +\n>> + ASSERT(ml.get(controls::ExposureTime) == 0x1111);\n>> + ASSERT(ml.get(controls::ExposureValue) == 1);\n>> +\n>> + for (auto &&[tag, v] : ml)\n>> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n>> +\n>> + std::cout << std::endl;\n>> +\n>> + ml.clear();\n>> + ASSERT(ml.empty());\n>> + ASSERT(ml.size() == 0);\n>> +\n>> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n>> + ASSERT(ml.get(controls::ExposureTime) == 0x2222);\n>> +\n>> + auto c = ml.checkpoint();\n>> +\n>> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n>> + ASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n>> +\n>> + auto d = c.diffSince();\n>> + ASSERT(&d.list() == &ml);\n>> +\n>> + ASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n>> +\n>> + ASSERT(d);\n>> + ASSERT(!d.empty());\n>> + ASSERT(d.size() == 2);\n>> + ASSERT(!d.get(controls::ExposureTime));\n>> + ASSERT(!d.get(controls::ColourGains));\n>> + ASSERT(!d.get(controls::AfWindows));\n>> + ASSERT(d.get(controls::ExposureValue) == 2);\n>> + ASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n>> +\n>> + for (auto &&[tag, v] : d)\n>> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n>> +\n>> + /* Test if iterators work with algorithms. */\n>> + std::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n>> + return false;\n>> + });\n>> +\n>> +#if 0\n>> + {\n>> + auto it = ml.begin();\n>> + ml.clear();\n>> + std::ignore = *it; /* Trigger ASAN. */\n>> + }\n>> +#endif\n>> +\n>> + return TestPass;\n>> + }\n>> +};\n>> +\n>> +TEST_REGISTER(MetadataListTest)\n>> -- \n>> 2.50.1\n>>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 660A0C328C\n\tfor ;\n\tWed, 24 Sep 2025 09:01:28 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id 7B2F76B5C4;\n\tWed, 24 Sep 2025 11:01:26 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[213.167.242.64])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id CFD2B613AB\n\tfor ;\n\tWed, 24 Sep 2025 11:00:08 +0200 (CEST)","from [192.168.33.12] (185.221.140.70.nat.pool.zt.hu\n\t[185.221.140.70])\n\tby perceval.ideasonboard.com (Postfix) with ESMTPSA id 90E14596;\n\tWed, 24 Sep 2025 10:57:44 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"eZcqVQK0\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1758704264;\n\tbh=4xgoK/E3npsQxak/La62NFzBFaAYLg/2ID70mFAhO3M=;\n\th=Date:Subject:To:References:From:In-Reply-To:From;\n\tb=eZcqVQK09m9PySbhS/PFGyChBPfkDL9iC0u2f8jPIcxxTBhdufAcaDP8QUQaMWdNb\n\tFvz02PPYRF/PsPppTSqSGUUVtN9DtUKYTot3/N+z21Um0UU515LKZydwiJecZxXZim\n\twRf7NJAsvzcGc2OVsIckG0Nsbw0fjoVSWqjgbF6E=","Message-ID":"<2322dac9-04a3-49cf-8faa-bfe86ecadaee@ideasonboard.com>","Date":"Wed, 24 Sep 2025 10:59:04 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","To":"Paul Elder ,\n\tlibcamera-devel@lists.libcamera.org","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>\n\t<175811057476.2127323.6616992472481774741@neptunite.rasen.tech>","From":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ","Content-Language":"en-US, hu-HU","In-Reply-To":"<175811057476.2127323.6616992472481774741@neptunite.rasen.tech>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}},{"id":36167,"web_url":"https://patchwork.libcamera.org/comment/36167/","msgid":"<175983658218.2583768.9921359216503718502@neptunite.rasen.tech>","date":"2025-10-07T11:29:42","subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","submitter":{"id":17,"url":"https://patchwork.libcamera.org/api/people/17/","name":"Paul Elder","email":"paul.elder@ideasonboard.com"},"content":"Quoting Barnabás Pőcze (2025-09-24 17:59:04)\n> Hi\n> \n> 2025. 09. 17. 14:02 keltezéssel, Paul Elder írta:\n> > Hi Barnabás,\n> > \n> > Thanks for the patch.\n> > \n> > Quoting Barnabás Pőcze (2025-07-21 19:46:07)\n> >> Add a dedicated `MetadataList` type, whose purpose is to store the metadata\n> >> reported by a camera for a given request. Previously, a `ControlList` was\n> >> used for this purpose. The reason for introducing a separate type is to\n> >> simplify the access to the returned metadata during the entire lifetime\n> >> of a request.\n> >>\n> >> Specifically, for early metadata completion to be easily usable it should be\n> >> guaranteed that any completed metadata item can be accessed and looked up\n> >> at least until the associated requested is reused with `Request::reuse()`.\n> >>\n> >> However, when a metadata item is completed early, the pipeline handler\n> >> might still work on the request in the `CameraManager`'s private thread,\n> >> therefore there is an inherent synchronization issue when an application\n> >> accesses early metadata.\n> >>\n> >> Restricting the user to only access the metadata items of a not yet completed\n> >> request in the early metadata availability signal handler by ways of\n> >> documenting or enforcing it at runtime could be an option, but it is not\n> >> too convenient for the user.\n> >>\n> >> The current `ControlList` implementation employs an `std::unordered_map`,\n> >> so pointers remain stable when the container is modified, so an application\n> >> could keep accessing particular metadata items outside the signal handler,\n> >> but this fact is far from obvious, and the user would still not be able\n> >> to make a copy of all metadata or do lookups based on the numeric ids or\n> >> the usual `libcamera::Control<>` objects, thus some type safety is lost.\n> >>\n> >> The above also requires that each metadata item is only completed once for\n> >> a given request, but this does not appear to be serious limitation,\n> >> and in fact, this restriction is enforced by `MetadataList`.\n> >>\n> >> The introduced `MetadataList` supports single writer - multiple reader\n> >> scenarios, and it can be set, looked-up, and copied in a wait-free fashion\n> >> without introducing data races or other synchronization issues. This is\n> >> achieved by requiring the possible set of metadata items to be known\n> >> (such set is stored in a `MetadataListPlan` object). Based on the this\n> >> plan, a single contiguous allocation is made to accommodate all potential\n> >> metadata items. Due to this single contiguous allocation that is not modified\n> >> during the lifetime of a `MetadataList` and atomic modifications, it is\n> >> possible to easily gaurantee thread-safe set, lookup, and copy; assuming\n> >> there is only ever a single writer.\n> >>\n> >> Signed-off-by: Barnabás Pőcze \n> > \n> > In general, the ideas and design look good to me. Jacopo picked at a lot more\n> > than I could.\n> > \n> > I only have a few comments/questions.\n> > \n> > Not sure if it's because it's still RFC phase or because everything is\n> > internal, but most of this is \"I want documentation please\".\n> > \n> >> ---\n> >> changes in v2:\n> >> * remove multiple not strictly necessary functions\n> >> ---\n> >> include/libcamera/meson.build | 2 +\n> >> include/libcamera/metadata_list.h | 547 +++++++++++++++++++++++++\n> >> include/libcamera/metadata_list_plan.h | 130 ++++++\n> >> src/libcamera/meson.build | 1 +\n> >> src/libcamera/metadata_list.cpp | 344 ++++++++++++++++\n> >> test/controls/meson.build | 1 +\n> >> test/controls/metadata_list.cpp | 170 ++++++++\n> >> 7 files changed, 1195 insertions(+)\n> >> create mode 100644 include/libcamera/metadata_list.h\n> >> create mode 100644 include/libcamera/metadata_list_plan.h\n> >> create mode 100644 src/libcamera/metadata_list.cpp\n> >> create mode 100644 test/controls/metadata_list.cpp\n> >>\n> >> diff --git a/include/libcamera/meson.build b/include/libcamera/meson.build\n> >> index 30ea76f94..410b548dd 100644\n> >> --- a/include/libcamera/meson.build\n> >> +++ b/include/libcamera/meson.build\n> >> @@ -12,6 +12,8 @@ libcamera_public_headers = files([\n> >> 'framebuffer_allocator.h',\n> >> 'geometry.h',\n> >> 'logging.h',\n> >> + 'metadata_list.h',\n> >> + 'metadata_list_plan.h',\n> >> 'orientation.h',\n> >> 'pixel_format.h',\n> >> 'request.h',\n> >> diff --git a/include/libcamera/metadata_list.h b/include/libcamera/metadata_list.h\n> >> new file mode 100644\n> >> index 000000000..7fe3dbbab\n> >> --- /dev/null\n> >> +++ b/include/libcamera/metadata_list.h\n> >> @@ -0,0 +1,547 @@\n> >> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >> +/*\n> >> + * Copyright (C) 2025, Ideas On Board Oy\n> >> + *\n> >> + * Metadata list\n> >> + */\n> >> +\n> >> +#pragma once\n> >> +\n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \n> >> +#include \n> >> +\n> >> +// TODO: want this?\n> > \n> > Given that we're manually doing a bunch of binary formatting I think we do want\n> > ASAN.\n> > \n> >> +#if __has_include()\n> >> +#if __SANITIZE_ADDRESS__ /* gcc */\n> >> +#include \n> >> +#define HAS_ASAN 1\n> >> +#elif defined(__has_feature)\n> >> +#if __has_feature(address_sanitizer) /* clang */\n> >> +#include \n> >> +#define HAS_ASAN 1\n> >> +#endif\n> >> +#endif\n> >> +#endif\n> >> +\n> >> +namespace libcamera {\n> >> +\n> >> +class MetadataList\n> >> +{\n> >> +private:\n> >> + struct ValueParams {\n> >> + ControlType type;\n> >> + bool isArray;\n> >> + std::uint32_t numElements;\n> >> + };\n> >> +\n> >> + struct Entry {\n> >> + const std::uint32_t tag;\n> >> + const std::uint32_t capacity;\n> >> + const std::uint32_t alignment;\n> >> + const ControlType type;\n> >> + bool isArray;\n> >> +\n> >> + static constexpr std::uint32_t invalidOffset = -1;\n> >> + /*\n> >> + * Offset from the beginning of the allocation, and\n> >> + * and _not_ relative to `contentOffset_`.\n> >> + */\n> >> + std::atomic_uint32_t headerOffset = invalidOffset;\n> >> +\n> >> + [[nodiscard]] std::optional hasValue() const\n> >> + {\n> >> + auto offset = headerOffset.load(std::memory_order_relaxed);\n> >> + if (offset == invalidOffset)\n> >> + return {};\n> >> +\n> >> + return offset;\n> >> + }\n> >> +\n> >> + [[nodiscard]] std::optional acquireData() const\n> >> + {\n> >> + auto offset = hasValue();\n> >> + if (offset) {\n> >> + /* sync with release-store on `headerOffset` in `MetadataList::set()` */\n> >> + std::atomic_thread_fence(std::memory_order_acquire);\n> >> + }\n> >> +\n> >> + return offset;\n> >> + }\n> >> + };\n> >> +\n> >> + struct ValueHeader {\n> >> + std::uint32_t tag;\n> >> + std::uint32_t size;\n> >> + std::uint32_t alignment;\n> >> + ValueParams params;\n> >> + };\n> > \n> > I know these structs are internal but there are so many moving parts that I'm\n> > just barely able to hold on to the meanings of everything... can we have\n> > documentation please...? Especially for when our caches are all flushed when we\n> > need to work on this in the future, I think documentation will speed up cache\n> > refreshing.\n> \n> I can't seem to convince doxygen to generate documentation for private types...\n\nimo it doesn't necessarily need to be in doxygen, just visible in the code is\nenough for private types.\n\n> \n> \n> > \n> >> +\n> >> + struct State {\n> >> + std::uint32_t count;\n> >> + std::uint32_t fill;\n> >> + };\n> >> +\n> >> +public:\n> >> + explicit MetadataList(const MetadataListPlan &plan)\n> >> + : capacity_(plan.size()),\n> >> + contentOffset_(MetadataList::contentOffset(capacity_)),\n> >> + alloc_(contentOffset_)\n> >> + {\n> >> + for (const auto &[tag, e] : plan) {\n> >> + alloc_ += sizeof(ValueHeader);\n> >> + alloc_ += e.alignment - 1; // XXX: this is the maximum\n> >> + alloc_ += e.size * e.numElements;\n> >> + alloc_ += alignof(ValueHeader) - 1; // XXX: this is the maximum\n> > \n> > Why are we taking both the size and align of ValueHeader?\n> \n> The idea is that size+alignment-1 bytes are always enough to properly align\n> the thing regardless of the alignment of the underlying storage. Another\n> option that is being considered is using a fixed alignment of 8, 16, etc.\n> for everything.\n\nOk, I see.\n\n> \n> \n> > \n> >> + }\n> >> +\n> >> + p_ = static_cast(::operator new(alloc_));\n> >> +\n> >> + auto *entries = reinterpret_cast(p_ + entriesOffset());\n> >> + auto it = plan.begin();\n> >> +\n> >> + for (std::size_t i = 0; i < capacity_; i++, ++it) {\n> >> + const auto &[tag, e] = *it;\n> >> +\n> >> + new (&entries[i]) Entry{\n> >> + .tag = tag,\n> >> + .capacity = e.size * e.numElements,\n> >> + .alignment = e.alignment,\n> >> + .type = e.type,\n> >> + .isArray = e.isArray,\n> >> + };\n> >> + }\n> >> +\n> >> +#if HAS_ASAN\n> >> + ::__sanitizer_annotate_contiguous_container(\n> >> + p_ + contentOffset_, p_ + alloc_,\n> >> + p_ + alloc_, p_ + contentOffset_\n> >> + );\n> >> +#endif\n> >> + }\n> >> +\n> >> + MetadataList(const MetadataList &) = delete;\n> >> + MetadataList(MetadataList &&) = delete;\n> >> +\n> >> + MetadataList &operator=(const MetadataList &) = delete;\n> >> + MetadataList &operator=(MetadataList &&) = delete;\n> >> +\n> >> + ~MetadataList()\n> >> + {\n> >> +#if HAS_ASAN\n> >> + /*\n> >> + * The documentation says the range apparently has to be\n> >> + * restored to its initial state before it is deallocated.\n> >> + */\n> >> + ::__sanitizer_annotate_contiguous_container(\n> >> + p_ + contentOffset_, p_ + alloc_,\n> >> + p_ + contentOffset_ + state_.load(std::memory_order_relaxed).fill, p_ + alloc_\n> >> + );\n> >> +#endif\n> >> +\n> >> + ::operator delete(p_, alloc_);\n> >> + }\n> >> +\n> >> + // TODO: want these?\n> > \n> > I think it's useful.\n> > \n> >> + [[nodiscard]] std::size_t size() const { return state_.load(std::memory_order_relaxed).count; }\n> >> + [[nodiscard]] bool empty() const { return state_.load(std::memory_order_relaxed).fill == 0; }\n> >> +\n> >> + enum class SetError {\n> >> + UnknownTag = 1,\n> >> + AlreadySet,\n> >> + SizeMismatch,\n> >> + TypeMismatch,\n> >> + };\n> >> +\n> >> + [[nodiscard]] SetError set(std::uint32_t tag, ControlValueView v)\n> >> + {\n> >> + auto *e = find(tag);\n> >> + if (!e)\n> >> + return SetError::UnknownTag;\n> >> +\n> >> + return set(*e, v);\n> >> + }\n> >> +\n> >> + template\n> >> + /* TODO: [[nodiscard]] */ SetError set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> >> + {\n> >> + using TypeInfo = libcamera::details::control_type;\n> >> +\n> >> + if constexpr (TypeInfo::size > 0) {\n> >> + static_assert(std::is_trivially_copyable_v);\n> >> +\n> >> + return set(ctrl.id(), {\n> >> + TypeInfo::value,\n> >> + true,\n> >> + value.size(),\n> >> + reinterpret_cast(value.data()),\n> >> + });\n> >> + } else {\n> >> + static_assert(std::is_trivially_copyable_v);\n> >> +\n> >> + return set(ctrl.id(), {\n> >> + TypeInfo::value,\n> >> + false,\n> >> + 1,\n> >> + reinterpret_cast(&value),\n> >> + });\n> >> + }\n> >> + }\n> >> +\n> >> + template\n> >> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> >> + {\n> >> + ControlValueView v = get(ctrl.id());\n> >> +\n> >> + return v ? std::optional(v.get()) : std::nullopt;\n> >> + }\n> >> +\n> >> + // TODO: operator ControlListView() const ?\n> >> + // TODO: explicit operator ControlList() const ?\n> >> +\n> >> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> >> + {\n> >> + const auto *e = find(tag);\n> >> + if (!e)\n> >> + return {};\n> >> +\n> >> + return data_of(*e);\n> > \n> > iirc Jacopo mentioned casing but this is a well-known common macro that imo\n> > it's ok to keep as-is. imo tho.\n> > \n> >> + }\n> >> +\n> >> + void clear()\n> >> + {\n> >> + for (auto &e : entries())\n> >> + e.headerOffset.store(Entry::invalidOffset, std::memory_order_relaxed);\n> >> +\n> >> + [[maybe_unused]] auto s = state_.exchange({}, std::memory_order_relaxed);\n> >> +\n> >> +#if HAS_ASAN\n> >> + ::__sanitizer_annotate_contiguous_container(\n> >> + p_ + contentOffset_, p_ + alloc_,\n> >> + p_ + contentOffset_ + s.fill, p_ + contentOffset_\n> >> + );\n> >> +#endif\n> >> + }\n> >> +\n> >> + class iterator\n> >> + {\n> >> + public:\n> >> + using difference_type = std::ptrdiff_t;\n> >> + using value_type = std::pair;\n> >> + using pointer = void;\n> >> + using reference = value_type;\n> >> + using iterator_category = std::forward_iterator_tag;\n> >> +\n> >> + iterator() = default;\n> >> +\n> >> + iterator& operator++()\n> >> + {\n> >> + const auto &h = header();\n> >> +\n> >> + p_ += sizeof(h);\n> >> + p_ = details::align::up(p_, h.alignment);\n> >> + p_ += h.size;\n> >> + p_ = details::align::up(p_, alignof(decltype(h)));\n> > \n> > Ok I think this makes sense... I'd really like documentation on the binary\n> > format somewhere though...\n> > \n> >> +\n> >> + return *this;\n> >> + }\n> >> +\n> >> + iterator operator++(int)\n> >> + {\n> >> + auto copy = *this;\n> >> + ++*this;\n> >> + return copy;\n> >> + }\n> >> +\n> >> + [[nodiscard]] reference operator*() const\n> >> + {\n> >> + const auto &h = header();\n> >> + const auto *data = details::align::up(p_ + sizeof(h), h.alignment);\n> >> +\n> >> + return { h.tag, { h.params.type, h.params.isArray, h.params.numElements, data } };\n> >> + }\n> >> +\n> >> + [[nodiscard]] bool operator==(const iterator &other) const\n> >> + {\n> >> + return p_ == other.p_;\n> >> + }\n> >> +\n> >> + [[nodiscard]] bool operator!=(const iterator &other) const\n> >> + {\n> >> + return !(*this == other);\n> >> + }\n> >> +\n> >> + private:\n> >> + iterator(const std::byte *p)\n> >> + : p_(p)\n> >> + {\n> >> + }\n> >> +\n> >> + [[nodiscard]] const ValueHeader &header() const\n> >> + {\n> >> + return *reinterpret_cast(p_);\n> >> + }\n> >> +\n> >> + friend MetadataList;\n> >> +\n> >> + const std::byte *p_ = nullptr;\n> >> + };\n> >> +\n> >> + [[nodiscard]] iterator begin() const\n> >> + {\n> >> + return { p_ + contentOffset_ };\n> >> + }\n> >> +\n> >> + [[nodiscard]] iterator end() const\n> >> + {\n> >> + return { p_ + contentOffset_ + state_.load(std::memory_order_acquire).fill };\n> >> + }\n> >> +\n> >> + class Diff\n> >> + {\n> >> + public:\n> >> + // TODO: want these?\n> >> + [[nodiscard]] explicit operator bool() const { return !empty(); }\n> >> + [[nodiscard]] bool empty() const { return start_ == stop_; }\n> >> + [[nodiscard]] std::size_t size() const { return changed_; }\n> >> + [[nodiscard]] const MetadataList &list() const { return *l_; }\n> > \n> > I think they'd be helpful to have.\n> > \n> >> +\n> >> + [[nodiscard]] ControlValueView get(std::uint32_t tag) const\n> >> + {\n> >> + const auto *e = l_->find(tag);\n> >> + if (!e)\n> >> + return {};\n> >> +\n> >> + auto o = e->acquireData();\n> >> + if (!o)\n> >> + return {};\n> >> +\n> >> + if (!(start_ <= *o && *o < stop_))\n> >> + return {};\n> >> +\n> >> + return l_->data_of(*o);\n> >> + }\n> >> +\n> >> + template\n> >> + [[nodiscard]] decltype(auto) get(const Control &ctrl) const\n> >> + {\n> >> + ControlValueView v = get(ctrl.id());\n> >> +\n> >> + return v ? std::optional(v.get()) : std::nullopt;\n> >> + }\n> >> +\n> >> + [[nodiscard]] iterator begin() const\n> >> + {\n> >> + return { l_->p_ + start_ };\n> >> + }\n> >> +\n> >> + [[nodiscard]] iterator end() const\n> >> + {\n> >> + return { l_->p_ + stop_ };\n> >> + }\n> >> +\n> >> + private:\n> >> + Diff(const MetadataList &l, std::size_t changed, std::size_t oldFill, std::size_t newFill)\n> > \n> > I can get what the meanings of these are... by reading Checkpoint below. imo\n> > documentation will help smooth out the process of understanding this.\n> > \n> >> + : l_(&l),\n> >> + changed_(changed),\n> >> + start_(l.contentOffset_ + oldFill),\n> >> + stop_(l.contentOffset_ + newFill)\n> >> + {\n> >> + }\n> >> +\n> >> + friend MetadataList;\n> >> + friend struct Checkpoint;\n> >> +\n> >> + const MetadataList *l_ = nullptr;\n> >> + std::size_t changed_;\n> >> + std::size_t start_;\n> >> + std::size_t stop_;\n> >> + };\n> >> +\n> >> + Diff merge(const ControlList &other)\n> >> + {\n> >> + // TODO: check id map of `other`?\n> > \n> > Since Diff can't be constructed by the user, I think we're guaranteed that the\n> > id maps match, so I don't think we need to spend resources on this. Or maybe we\n> > could for a peace of mind; it's just a comparison of pointers, right?\n> > \n> > Speaking of which, do we need to disable or privatize the default constructors?\n> \n> No default constructor is generated for either `Diff` or `Checkpoint`.\n\nOk.\n\n> \n> \n> > \n> >> +\n> >> + const auto c = checkpoint();\n> >> +\n> >> + for (const auto &[tag, value] : other) {\n> >> + auto *e = find(tag);\n> >> + if (e) {\n> >> + [[maybe_unused]] auto r = set(*e, value);\n> >> + assert(r == SetError() || r == SetError::AlreadySet); // TODO: ?\n> > \n> > TODO?\n> \n> The question here is how to handle failure. They could be ignored entirely, or an\n> assertion could be used to ensure no errors. Or a mix of the two like above. Or\n> `merge()` could be probably made more \"transactional\" somehow so that it will either\n> add all items or none of them, and then the error could be directly reported to the\n> caller to handle it.\n\nAh, good question. In theory I don't think we'll ever hit these errors since\nall the metadata is emitted by sane pipeline handlers (maybe unless we have a\nrogue IPA?). But if they do ever happen we do want to protect them from\nbreaking everything. Maybe if they do happen then it means something has gone\nhorribly wrong and it's ok to assert() and abort everything?\n\nOr maybe we can be nice and print errors and just ignore the controls that fail\nto merge. That might be a nicer middle ground. I personally don't think we need\nto go all-or-nothing.\n\nLet's see what other people think.\n\n> \n> \n> > \n> >> + }\n> >> + }\n> >> +\n> >> + return c.diffSince();\n> >> + }\n> >> +\n> >> + class Checkpoint\n> >> + {\n> >> + public:\n> >> + [[nodiscard]] Diff diffSince() const\n> >> + {\n> >> + /* sync with release-store on `state_` in `set()` */\n> >> + const auto curr = l_->state_.load(std::memory_order_acquire);\n> >> +\n> >> + assert(s_.count <= curr.count);\n> >> + assert(s_.fill <= curr.fill);\n> >> +\n> >> + return {\n> >> + *l_,\n> >> + curr.count - s_.count,\n> >> + s_.fill,\n> >> + curr.fill,\n> >> + };\n> >> + }\n> >> +\n> >> + private:\n> >> + Checkpoint(const MetadataList &l)\n> >> + : l_(&l),\n> >> + s_(l.state_.load(std::memory_order_relaxed))\n> >> + {\n> >> + }\n> >> +\n> >> + friend MetadataList;\n> >> +\n> >> + const MetadataList *l_ = nullptr;\n> >> + State s_ = {};\n> >> + };\n> >> +\n> >> + [[nodiscard]] Checkpoint checkpoint() const\n> >> + {\n> >> + return { *this };\n> >> + }\n> >> +\n> >> +private:\n> >> + [[nodiscard]] static constexpr std::size_t entriesOffset()\n> >> + {\n> >> + return 0;\n> >> + }\n> >> +\n> >> + [[nodiscard]] static constexpr std::size_t contentOffset(std::size_t entries)\n> >> + {\n> >> + return details::align::up(entriesOffset() + entries * sizeof(Entry), alignof(ValueHeader));\n> >> + }\n> >> +\n> >> + [[nodiscard]] Span entries() const\n> >> + {\n> >> + return { reinterpret_cast(p_ + entriesOffset()), capacity_ };\n> >> + }\n> >> +\n> >> + [[nodiscard]] Entry *find(std::uint32_t tag) const\n> >> + {\n> >> + const auto entries = this->entries();\n> >> + auto it = std::partition_point(entries.begin(), entries.end(), [&](const auto &e) {\n> >> + return e.tag < tag;\n> >> + });\n> >> +\n> >> + if (it == entries.end() || it->tag != tag)\n> >> + return nullptr;\n> >> +\n> >> + return &*it;\n> >> + }\n> >> +\n> >> + [[nodiscard]] ControlValueView data_of(const Entry &e) const\n> >> + {\n> >> + const auto o = e.acquireData();\n> >> + return o ? data_of(*o) : ControlValueView{ };\n> >> + }\n> >> +\n> >> + [[nodiscard]] ControlValueView data_of(std::size_t headerOffset) const\n> >> + {\n> >> + assert(headerOffset <= alloc_ - sizeof(ValueHeader));\n> >> + assert(details::align::is(p_ + headerOffset, alignof(ValueHeader)));\n> >> +\n> >> + const auto *vh = reinterpret_cast(p_ + headerOffset);\n> >> + const auto *p = reinterpret_cast(vh) + sizeof(*vh);\n> >> + std::size_t avail = p_ + alloc_ - p;\n> >> +\n> >> + const auto *data = details::align::up(vh->size, vh->alignment, p, &avail);\n> >> + assert(data);\n> >> +\n> >> + return { vh->params.type, vh->params.isArray, vh->params.numElements, data };\n> >> + }\n> > \n> > This... costs a lot to understand... I think documentation on the binary format\n> > would help considerably.\n> > \n> >> +\n> >> + [[nodiscard]] SetError set(Entry &e, ControlValueView v)\n> >> + {\n> >> + if (e.hasValue())\n> >> + return SetError::AlreadySet;\n> >> + if (e.type != v.type() || e.isArray != v.isArray())\n> >> + return SetError::TypeMismatch;\n> >> +\n> >> + const auto src = v.data();\n> >> + if (e.isArray) {\n> >> + if (src.size_bytes() > e.capacity)\n> >> + return SetError::SizeMismatch;\n> >> + } else {\n> >> + if (src.size_bytes() != e.capacity)\n> >> + return SetError::SizeMismatch;\n> >> + }\n> >> +\n> >> + auto s = state_.load(std::memory_order_relaxed);\n> >> + std::byte *oldEnd = p_ + contentOffset_ + s.fill;\n> >> + std::byte *p = oldEnd;\n> >> +\n> >> + auto *headerPtr = details::align::up(p);\n> >> + auto *dataPtr = details::align::up(src.size_bytes(), e.alignment, p);\n> >> + details::align::up(0, alignof(ValueHeader), p);\n> >> +\n> >> +#if HAS_ASAN\n> >> + ::__sanitizer_annotate_contiguous_container(\n> >> + p_ + contentOffset_, p_ + alloc_,\n> >> + oldEnd, p\n> >> + );\n> >> +#endif\n> >> +\n> >> + new (headerPtr) ValueHeader{\n> >> + .tag = e.tag,\n> >> + .size = std::uint32_t(src.size_bytes()),\n> >> + .alignment = e.alignment,\n> >> + .params = {\n> >> + .type = v.type(),\n> >> + .isArray = v.isArray(),\n> >> + .numElements = std::uint32_t(v.numElements()),\n> >> + },\n> >> + };\n> >> + std::memcpy(dataPtr, src.data(), src.size_bytes());\n> >> + e.headerOffset.store(reinterpret_cast(headerPtr) - p_, std::memory_order_release);\n> >> +\n> >> + s.fill += p - oldEnd;\n> >> + s.count += 1;\n> >> +\n> >> + state_.store(s, std::memory_order_release);\n> >> +\n> >> + return {};\n> >> + }\n> >> +\n> >> + std::size_t capacity_ = 0;\n> >> + std::size_t contentOffset_ = -1;\n> >> + std::size_t alloc_ = 0;\n> >> + std::atomic state_ = State{};\n> >> + std::byte *p_ = nullptr;\n> > \n> > I... would like some documentation on these... I'm just barely able to hang on\n> > to the meanings... I think documentation would ease future maintenance when all\n> > our caches have been flushed.\n> > \n> >> + // TODO: ControlIdMap in any way shape or form?\n> > \n> > I don't think you need it? I think you've designed this in a way such that\n> > which ControlIdMap is being used doesn't really matter, since everything\n> > originates from the MetadataListPlan.\n> \n> It's probably not needed, but maybe useful to provide the same interface\n> as `ControlList`, although I don't have a concrete use case in mind.\n\nafaik nobody actually uses the ControlIdMap in ControlList (except serializers\nI think). iirc it was to support V4L2 controls?\n\n> \n> \n> > \n> >> +\n> >> + /*\n> >> + * If this is problematic on a 32-bit architecture, then\n> >> + * `count` can be stored in a separate atomic variable\n> >> + * but then `Diff::changed_` must be removed since the fill\n> >> + * level and item count cannot be retrieved atomically.\n> >> + */\n> >> + static_assert(decltype(state_)::is_always_lock_free);\n> >> +};\n> >> +\n> >> +} /* namespace libcamera */\n> >> +\n> >> +#undef HAS_ASAN\n> >> diff --git a/include/libcamera/metadata_list_plan.h b/include/libcamera/metadata_list_plan.h\n> >> new file mode 100644\n> >> index 000000000..2ed35c54f\n> >> --- /dev/null\n> >> +++ b/include/libcamera/metadata_list_plan.h\n> >> @@ -0,0 +1,130 @@\n> >> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >> +/*\n> >> + * Copyright (C) 2025, Ideas On Board Oy\n> >> + */\n> >> +\n> >> +#pragma once\n> >> +\n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \n> >> +\n> >> +#include \n> >> +\n> >> +namespace libcamera {\n> >> +\n> >> +class MetadataListPlan\n> >> +{\n> >> +public:\n> >> + struct Entry {\n> >> + std::uint32_t size;\n> >> + std::uint32_t alignment; // TODO: is this necessary?\n> >> + std::uint32_t numElements;\n> >> + ControlType type;\n> >> + bool isArray;\n> >> + };\n> >> +\n> >> + [[nodiscard]] bool empty() const { return items_.empty(); }\n> >> + [[nodiscard]] std::size_t size() const { return items_.size(); }\n> >> + [[nodiscard]] decltype(auto) begin() const { return items_.begin(); }\n> >> + [[nodiscard]] decltype(auto) end() const { return items_.end(); }\n> >> + void clear() { items_.clear(); }\n> >> +\n> >> + template<\n> >> + typename T,\n> >> + std::enable_if_t::size != libcamera::dynamic_extent> * = nullptr\n> >> + >\n> >> + decltype(auto) set(const Control &ctrl)\n> >> + {\n> >> + if constexpr (libcamera::details::control_type::size > 0) {\n> >> + static_assert(libcamera::details::control_type::size != libcamera::dynamic_extent);\n> >> +\n> >> + return set(\n> >> + ctrl.id(),\n> >> + libcamera::details::control_type::size,\n> >> + true\n> >> + );\n> >> + } else {\n> >> + return set(ctrl.id(), 1, false);\n> >> + }\n> >> + }\n> >> +\n> >> + template<\n> >> + typename T,\n> >> + std::enable_if_t::size == libcamera::dynamic_extent> * = nullptr\n> >> + >\n> >> + decltype(auto) set(const Control &ctrl, std::size_t numElements)\n> >> + {\n> >> + return set(ctrl.id(), numElements, true);\n> >> + }\n> >> +\n> >> + [[nodiscard]] bool set(std::uint32_t tag,\n> >> + std::size_t size, std::size_t alignment,\n> >> + std::size_t numElements, ControlType type, bool isArray)\n> >> + {\n> >> + if (size == 0 || size > std::numeric_limits::max())\n> >> + return false;\n> >> + if (alignment > std::numeric_limits::max())\n> >> + return false;\n> >> + if (!details::cxx20::has_single_bit(alignment))\n> >> + return false;\n> >> + if (numElements > std::numeric_limits::max() / size)\n> >> + return false;\n> >> + if (!isArray && numElements != 1)\n> >> + return false;\n> >> +\n> >> + items_.insert_or_assign(tag, Entry{\n> >> + .size = std::uint32_t(size),\n> >> + .alignment = std::uint32_t(alignment),\n> >> + .numElements = std::uint32_t(numElements),\n> >> + .type = type,\n> >> + .isArray = isArray,\n> >> + });\n> >> +\n> >> + return true;\n> >> + }\n> >> +\n> >> + [[nodiscard]] const Entry *get(std::uint32_t tag) const\n> >> + {\n> >> + auto it = items_.find(tag);\n> >> + if (it == items_.end())\n> >> + return nullptr;\n> >> +\n> >> + return &it->second;\n> >> + }\n> >> +\n> >> + [[nodiscard]] const Entry *get(const ControlId &cid) const\n> >> + {\n> >> + const auto *e = get(cid.id());\n> >> + if (!e)\n> >> + return nullptr;\n> >> +\n> >> + if (e->type != cid.type() || e->isArray != cid.isArray())\n> >> + return nullptr;\n> >> +\n> >> + return e;\n> >> + }\n> >> +\n> >> +private:\n> >> + std::map items_;\n> >> +\n> >> + template\n> >> + decltype(auto) set(std::uint32_t tag, std::size_t numElements, bool isArray)\n> >> + {\n> >> + static_assert(std::is_trivially_copyable_v);\n> >> +\n> >> + [[maybe_unused]] bool ok = set(tag,\n> >> + sizeof(T), alignof(T),\n> >> + numElements, details::control_type::value, isArray);\n> >> + assert(ok);\n> >> +\n> >> + return *this;\n> >> + }\n> >> +};\n> >> +\n> >> +} /* namespace libcamera */\n> >> diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build\n> >> index de1eb99b2..8c5ce4503 100644\n> >> --- a/src/libcamera/meson.build\n> >> +++ b/src/libcamera/meson.build\n> >> @@ -9,6 +9,7 @@ libcamera_public_sources = files([\n> >> 'framebuffer.cpp',\n> >> 'framebuffer_allocator.cpp',\n> >> 'geometry.cpp',\n> >> + 'metadata_list.cpp',\n> >> 'orientation.cpp',\n> >> 'pixel_format.cpp',\n> >> 'request.cpp',\n> >> diff --git a/src/libcamera/metadata_list.cpp b/src/libcamera/metadata_list.cpp\n> >> new file mode 100644\n> >> index 000000000..ebefdfdad\n> >> --- /dev/null\n> >> +++ b/src/libcamera/metadata_list.cpp\n> >> @@ -0,0 +1,344 @@\n> >> +/* SPDX-License-Identifier: LGPL-2.1-or-later */\n> >> +/*\n> >> + * Copyright (C) 2025, Ideas On Board Oy\n> >> + */\n> >> +\n> >> +#include \n> >> +\n> >> +namespace libcamera {\n> >> +\n> >> +/**\n> >> + * \\class MetadataListPlan\n> >> + * \\brief Class to hold the possible set of metadata items for a MetadataList\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\class MetadataListPlan::Entry\n> >> + * \\brief Details of a metadata item\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\var MetadataListPlan::Entry::size\n> >> + * \\brief Number of bytes in a single element\n> >> + *\n> >> + * \\var MetadataListPlan::Entry::alignment\n> >> + * \\brief Required alignment of the elements\n> >> + * \\endinternal\n> >> + *\n> >> + * \\var MetadataListPlan::Entry::numElements\n> >> + * \\brief Number of elements in the value\n> >> + * \\sa ControlValueView::numElements()\n> >> + *\n> >> + * \\var MetadataListPlan::Entry::type\n> >> + * \\brief The type of the value\n> >> + * \\sa ControlValueView::type()\n> >> + *\n> >> + * \\var MetadataListPlan::Entry::isArray\n> >> + * \\brief Whether or not the value is array-like\n> >> + * \\sa ControlValueView::isArray()\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::begin() const\n> >> + * \\brief Retrieve the begin iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::end() const\n> >> + * \\brief Retrieve the end iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::size() const\n> >> + * \\brief Retrieve the number of entries\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::empty() const\n> >> + * \\brief Check if empty\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataListPlan::clear()\n> >> + * \\brief Remove all controls\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataListPlan::set(const Control &ctrl)\n> >> + * \\brief Add an entry for the given control to the metadata list plan\n> >> + * \\param[in] ctrl The control\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataListPlan::set(const Control &ctrl, std::size_t count)\n> >> + * \\brief Add an entry for the given dynamically-sized control to the metadata list plan\n> >> + * \\param[in] ctrl The control\n> >> + * \\param[in] count The maximum number of elements\n> >> + *\n> >> + * Add the dynamically-sized control \\a ctrl to the metadata list plan with a maximum\n> >> + * capacity of \\a count elements.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataListPlan::set(std::uint32_t tag,\n> >> + * std::size_t size, std::size_t alignment,\n> >> + * std::size_t count, ControlType type, bool isArray)\n> >> + * \\brief Add an entry to the metadata list plan\n> >> + * \\return \\a true if the entry has been added, or \\a false if the given parameters\n> >> + * would result in an invalid entry\n> >> + *\n> >> + * This functions adds an entry with essentially arbitrary parameters, without deriving\n> >> + * them from a given ControlId instance. This is mainly used when deserializing.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::get(std::uint32_t tag) const\n> >> + * \\brief Find the \\ref Entry \"entry\" with the given identifier\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataListPlan::get(const ControlId &cid) const\n> >> + * \\brief Find the \\ref Entry \"entry\" for the given ControlId\n> >> + *\n> >> + * The \\ref Entry \"entry\" is only returned if ControlId::type() and ControlId::isArray()\n> >> + * of \\a cid matches Entry::type and Entry::isArray, respectively.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\class MetadataList\n> >> + * \\brief Class to hold metadata items\n> > \n> > I'd like this to be expanded. What's special about it? Why can't we just use\n> > ControlList? How is the data formatted in the custom data container (this could\n> > be internal in the code)? What's the significance and difference between Entry\n> > and ValueHeader?\n> \n> I'm now wondering if the documentation in the next patch should be merged into this.\n\nI think it's better to keep the documentation in the next patch separate,\nmostly because its big but also because it's got more of the design stuff. It\nmight have just been me getting confused reading this patch and not yet having\nread the next one. That being said, I think it would be nice to add just a\nsliver of it here to document the class itself. At least the parts that are\nrelevating to an application developer.\n\n> \n> \n> > \n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::MetadataList(const MetadataListPlan &plan)\n> >> + * \\brief Construct a metadata list according to \\a plan\n> >> + *\n> >> + * Construct a metadata list according to the provided \\a plan.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::size() const\n> >> + * \\brief Retrieve the number of controls\n> >> + * \\context This function is \\threadsafe.\n> >> + * \\note If the list is being modified, the return value may be out of\n> >> + * date by the time the function returns\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::empty() const\n> >> + * \\brief Check if empty\n> >> + * \\context This function is \\threadsafe.\n> >> + * \\note If the list is being modified, the return value may be out of\n> >> + * date by the time the function returns\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::clear()\n> >> + * \\brief Remove all items from the list\n> >> + * \\note This function in effect resets the list to its original state. As a consequence it invalidates - among others -\n> >> + * all iterators, Checkpoint, and Diff objects that are associated with the list. No readers must exist\n> >> + * when this function is called.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::begin() const\n> >> + * \\brief Retrieve begin iterator\n> >> + * \\context This function is \\threadsafe.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::end() const\n> >> + * \\brief Retrieve end iterator\n> >> + * \\context This function is \\threadsafe.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::get(const Control &ctrl) const\n> >> + * \\brief Get the value of control \\a ctrl\n> >> + * \\return A std::optional containing the control value, or std::nullopt if\n> >> + * the control \\a ctrl is not present in the list\n> >> + * \\context This function is \\threadsafe.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::get(std::uint32_t tag) const\n> >> + * \\brief Get the value of pertaining to the numeric identifier \\a tag\n> >> + * \\return A std::optional containing the control value, or std::nullopt if\n> >> + * the control is not present in the list\n> >> + * \\context This function is \\threadsafe.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::set(const Control &ctrl, const details::cxx20::type_identity_t &value)\n> >> + * \\brief Set the value of control \\a ctrl to \\a value\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::set(std::uint32_t tag, ControlValueView v)\n> >> + * \\brief Set the value of pertaining to the numeric identifier \\a tag to \\a v\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::merge(const ControlList &other)\n> >> + * \\brief Add all missing items from \\a other\n> >> + *\n> >> + * Add all items from \\a other that are not present in \\a this.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\enum MetadataList::SetError\n> >> + * \\brief Error code returned by a set operation\n> >> + *\n> >> + * \\var MetadataList::SetError::UnknownTag\n> >> + * \\brief The tag is not supported by the metadata list\n> >> + * \\var MetadataList::SetError::AlreadySet\n> >> + * \\brief A value has already been added with the given tag\n> >> + * \\var MetadataList::SetError::SizeMismatch\n> >> + * \\brief The size of the data is not appropriate for the given tag\n> >> + * \\var MetadataList::SetError::TypeMismatch\n> >> + * \\brief The type of the value does not match the expected type\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::checkpoint() const\n> >> + * \\brief Create a checkpoint\n> >> + * \\context This function is \\threadsafe.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\class MetadataList::iterator\n> >> + * \\brief Iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\typedef MetadataList::iterator::difference_type\n> >> + * \\brief iterator's difference type\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\typedef MetadataList::iterator::value_type\n> >> + * \\brief iterator's value type\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\typedef MetadataList::iterator::pointer\n> >> + * \\brief iterator's pointer type\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\typedef MetadataList::iterator::reference\n> >> + * \\brief iterator's reference type\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\typedef MetadataList::iterator::iterator_category\n> >> + * \\brief iterator's category\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::iterator::operator*()\n> >> + * \\brief Retrieve value at iterator\n> >> + * \\return A \\a ControlListView representing the value\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::iterator::operator==(const iterator &other) const\n> >> + * \\brief Check if two iterators are equal\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::iterator::operator!=(const iterator &other) const\n> >> + * \\brief Check if two iterators are not equal\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::iterator::operator++(int)\n> >> + * \\brief Advance the iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::iterator::operator++()\n> >> + * \\brief Advance the iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\class MetadataList::Diff\n> >> + * \\brief Designates a set of consecutively added metadata items from a particular MetadataList\n> > \n> > Personally it was hard to figure out what this is meant to do and how it's\n> > meant to use... until I read the test, then it all made sense. imo it would be\n> > useful to expand the documentation to cover those so that people don't have to\n> > go over to the test case to figure it out.\n> \n> I believe the documentation of Camera::metadataAvailable (added later) provides a\n> good example of how this type should be used. I didn't want to duplicate it and\n> that seemed like a better location, but not entirely sure which is the best location.\n\nThen imo this could use a \\sa Camera::metadataAvailable... oh I see it now below :)\n\n> \n> \n> > \n> > I suppose the documentation for the functions below describe how to use it, so\n> > maybe just adding the raison d'etre is good enough here. Expecially since this\n> > is meant to be used by applications, right?\n> \n> This type is meant to be used by applications, yes. Could you clarify what\n> you mean exactly?\n\nThe design document covered it so no worries.\n\n\nThanks,\n\nPaul\n\n> \n> \n> Regards,\n> Barnabás Pőcze\n> \n> > \n> > \n> > As dense and intricate this is, I like the ideas here.\n> > \n> > \n> > Thanks,\n> > \n> > Paul\n> > \n> >> + * \\sa Camera::metadataAvailable\n> >> + * \\internal\n> >> + * \\sa MetadataList::Checkpoint::diffSince()\n> >> + * \\endinternal\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::list() const\n> >> + * \\brief Retrieve the associated MetadataList\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::size() const\n> >> + * \\brief Retrieve the number of metadata items designated\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::empty() const\n> >> + * \\brief Check if any metadata items are designated\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::operator bool() const\n> >> + * \\copydoc MetadataList::Diff::empty() const\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::get(const Control &ctrl) const\n> >> + * \\copydoc MetadataList::get(const Control &ctrl) const\n> >> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> >> + * even if it is otherwise present in the backing MetadataList.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::get(std::uint32_t tag) const\n> >> + * \\copydoc MetadataList::get(std::uint32_t tag) const\n> >> + * \\note The lookup will fail if the metadata item is not designated by this Diff object,\n> >> + * even if it is otherwise present in the backing MetadataList.\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::begin() const\n> >> + * \\brief Retrieve the begin iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\fn MetadataList::Diff::end() const\n> >> + * \\brief Retrieve the end iterator\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\class MetadataList::Checkpoint\n> >> + * \\brief Designates a point in the stream of metadata items\n> >> + *\n> >> + * A Checkpoint object designates a point in the stream of metadata items in the associated\n> >> + * MetadataList. Its main use to be able to retrieve the set of metadata items that were\n> >> + * added to the list after the designated point using diffSince().\n> >> + */\n> >> +\n> >> +/**\n> >> + * \\internal\n> >> + * \\fn MetadataList::Checkpoint::diffSince() const\n> >> + * \\brief Retrieve the set of metadata items added since the checkpoint was created\n> >> + */\n> >> +\n> >> +} /* namespace libcamera */\n> >> diff --git a/test/controls/meson.build b/test/controls/meson.build\n> >> index 763f8905e..b68a4fc53 100644\n> >> --- a/test/controls/meson.build\n> >> +++ b/test/controls/meson.build\n> >> @@ -5,6 +5,7 @@ control_tests = [\n> >> {'name': 'control_info_map', 'sources': ['control_info_map.cpp']},\n> >> {'name': 'control_list', 'sources': ['control_list.cpp']},\n> >> {'name': 'control_value', 'sources': ['control_value.cpp']},\n> >> + {'name': 'metadata_list', 'sources': ['metadata_list.cpp']},\n> >> ]\n> >> \n> >> foreach test : control_tests\n> >> diff --git a/test/controls/metadata_list.cpp b/test/controls/metadata_list.cpp\n> >> new file mode 100644\n> >> index 000000000..f0872acd9\n> >> --- /dev/null\n> >> +++ b/test/controls/metadata_list.cpp\n> >> @@ -0,0 +1,170 @@\n> >> +/* SPDX-License-Identifier: GPL-2.0-or-later */\n> >> +/*\n> >> + * Copyright (C) 2025, Ideas On Board Oy\n> >> + *\n> >> + * MetadataList tests\n> >> + */\n> >> +\n> >> +#include \n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \n> >> +#include \n> >> +#include \n> >> +\n> >> +#include \"test.h\"\n> >> +\n> >> +using namespace std;\n> >> +using namespace libcamera;\n> >> +\n> >> +#define ASSERT(x) do { \\\n> >> + if (!static_cast(x)) { \\\n> >> + std::cerr << '`' << #x << \"` failed\" << std::endl; \\\n> >> + return TestFail; \\\n> >> + } \\\n> >> +} while (false)\n> >> +\n> >> +class MetadataListTest : public Test\n> >> +{\n> >> +public:\n> >> + MetadataListTest() = default;\n> >> +\n> >> +protected:\n> >> + int run() override\n> >> + {\n> >> + MetadataListPlan mlp;\n> >> + mlp.set(controls::ExposureTime);\n> >> + mlp.set(controls::ExposureValue);\n> >> + mlp.set(controls::ColourGains);\n> >> + mlp.set(controls::AfWindows, 10);\n> >> + mlp.set(controls::AeEnable);\n> >> + mlp.set(controls::SensorTimestamp);\n> >> +\n> >> + MetadataList ml(mlp);\n> >> +\n> >> + static_assert(static_cast(properties::LOCATION) == controls::AE_ENABLE);\n> >> + ASSERT(ml.set(properties::Location, properties::CameraLocationFront) == MetadataList::SetError::TypeMismatch);\n> >> +\n> >> + ASSERT(ml.set(controls::AfWindows, std::array{}) == MetadataList::SetError::SizeMismatch);\n> >> + ASSERT(ml.set(controls::ColourTemperature, 123) == MetadataList::SetError::UnknownTag);\n> >> +\n> >> + auto f1 = std::async(std::launch::async, [&] {\n> >> + using namespace std::chrono_literals;\n> >> +\n> >> + std::this_thread::sleep_for(500ms);\n> >> + ASSERT(ml.set(controls::ExposureTime, 0x1111) == MetadataList::SetError());\n> >> +\n> >> + std::this_thread::sleep_for(500ms);\n> >> + ASSERT(ml.set(controls::ExposureValue, 1) == MetadataList::SetError());\n> >> +\n> >> + std::this_thread::sleep_for(500ms);\n> >> + ASSERT(ml.set(controls::ColourGains, std::array{\n> >> + 123.f,\n> >> + 456.f\n> >> + }) == MetadataList::SetError());\n> >> +\n> >> + std::this_thread::sleep_for(500ms);\n> >> + ASSERT(ml.set(controls::AfWindows, std::array{\n> >> + Rectangle(),\n> >> + Rectangle(1, 2, 3, 4),\n> >> + Rectangle(0x1111, 0x2222, 0x3333, 0x4444),\n> >> + }) == MetadataList::SetError());\n> >> +\n> >> + return TestPass;\n> >> + });\n> >> +\n> >> + auto f2 = std::async(std::launch::async, [&] {\n> >> + for (;;) {\n> >> + const auto x = ml.get(controls::ExposureTime);\n> >> + const auto y = ml.get(controls::ExposureValue);\n> >> + const auto z = ml.get(controls::ColourGains);\n> >> + const auto w = ml.get(controls::AfWindows);\n> >> +\n> >> + if (x)\n> >> + ASSERT(*x == 0x1111);\n> >> +\n> >> + if (y)\n> >> + ASSERT(*y == 1.0f);\n> >> +\n> >> + if (z) {\n> >> + ASSERT(z->size() == 2);\n> >> + ASSERT((*z)[0] == 123.f);\n> >> + ASSERT((*z)[1] == 456.f);\n> >> + }\n> >> +\n> >> + if (w) {\n> >> + ASSERT(w->size() == 3);\n> >> + ASSERT((*w)[0].isNull());\n> >> + ASSERT((*w)[1] == Rectangle(1, 2, 3, 4));\n> >> + ASSERT((*w)[2] == Rectangle(0x1111, 0x2222, 0x3333, 0x4444));\n> >> + }\n> >> +\n> >> + if (x && y && z && w)\n> >> + break;\n> >> + }\n> >> +\n> >> + return TestPass;\n> >> + });\n> >> +\n> >> + ASSERT(f1.get() == TestPass);\n> >> + ASSERT(f2.get() == TestPass);\n> >> +\n> >> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError::AlreadySet);\n> >> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError::AlreadySet);\n> >> +\n> >> + ASSERT(ml.get(controls::ExposureTime) == 0x1111);\n> >> + ASSERT(ml.get(controls::ExposureValue) == 1);\n> >> +\n> >> + for (auto &&[tag, v] : ml)\n> >> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> >> +\n> >> + std::cout << std::endl;\n> >> +\n> >> + ml.clear();\n> >> + ASSERT(ml.empty());\n> >> + ASSERT(ml.size() == 0);\n> >> +\n> >> + ASSERT(ml.set(controls::ExposureTime, 0x2222) == MetadataList::SetError());\n> >> + ASSERT(ml.get(controls::ExposureTime) == 0x2222);\n> >> +\n> >> + auto c = ml.checkpoint();\n> >> +\n> >> + ASSERT(ml.set(controls::ExposureValue, 2) == MetadataList::SetError());\n> >> + ASSERT(ml.set(controls::SensorTimestamp, 0x99999999) == MetadataList::SetError());\n> >> +\n> >> + auto d = c.diffSince();\n> >> + ASSERT(&d.list() == &ml);\n> >> +\n> >> + ASSERT(ml.set(controls::ColourGains, std::array{ 1.f, 2.f }) == MetadataList::SetError());\n> >> +\n> >> + ASSERT(d);\n> >> + ASSERT(!d.empty());\n> >> + ASSERT(d.size() == 2);\n> >> + ASSERT(!d.get(controls::ExposureTime));\n> >> + ASSERT(!d.get(controls::ColourGains));\n> >> + ASSERT(!d.get(controls::AfWindows));\n> >> + ASSERT(d.get(controls::ExposureValue) == 2);\n> >> + ASSERT(d.get(controls::SensorTimestamp) == 0x99999999);\n> >> +\n> >> + for (auto &&[tag, v] : d)\n> >> + std::cout << \"[\" << tag << \"] -> \" << v << '\\n';\n> >> +\n> >> + /* Test if iterators work with algorithms. */\n> >> + std::ignore = std::find_if(d.begin(), d.end(), [](const auto &) {\n> >> + return false;\n> >> + });\n> >> +\n> >> +#if 0\n> >> + {\n> >> + auto it = ml.begin();\n> >> + ml.clear();\n> >> + std::ignore = *it; /* Trigger ASAN. */\n> >> + }\n> >> +#endif\n> >> +\n> >> + return TestPass;\n> >> + }\n> >> +};\n> >> +\n> >> +TEST_REGISTER(MetadataListTest)\n> >> -- \n> >> 2.50.1\n> >>\n>","headers":{"Return-Path":"","X-Original-To":"parsemail@patchwork.libcamera.org","Delivered-To":"parsemail@patchwork.libcamera.org","Received":["from lancelot.ideasonboard.com (lancelot.ideasonboard.com\n\t[92.243.16.209])\n\tby patchwork.libcamera.org (Postfix) with ESMTPS id 78BFEBF415\n\tfor ;\n\tTue, 7 Oct 2025 11:29:53 +0000 (UTC)","from lancelot.ideasonboard.com (localhost [IPv6:::1])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTP id C27756B5F8;\n\tTue, 7 Oct 2025 13:29:51 +0200 (CEST)","from perceval.ideasonboard.com (perceval.ideasonboard.com\n\t[IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647])\n\tby lancelot.ideasonboard.com (Postfix) with ESMTPS id 81D1569367\n\tfor ;\n\tTue, 7 Oct 2025 13:29:49 +0200 (CEST)","from neptunite.rasen.tech (unknown\n\t[IPv6:2404:7a81:160:2100:859f:ca72:88bf:8f12])\n\tby perceval.ideasonboard.com (Postfix) with UTF8SMTPSA id 0A86F929;\n\tTue, 7 Oct 2025 13:28:14 +0200 (CEST)"],"Authentication-Results":"lancelot.ideasonboard.com; dkim=pass (1024-bit key;\n\tunprotected) header.d=ideasonboard.com header.i=@ideasonboard.com\n\theader.b=\"JyoIhARo\"; dkim-atps=neutral","DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com;\n\ts=mail; t=1759836496;\n\tbh=FHFGahDBlCSx10qMEk7P/aYK5RJW7FR+FmzvRNBNocY=;\n\th=In-Reply-To:References:Subject:From:To:Date:From;\n\tb=JyoIhARoiksHtT6w/vWeLgVAU2ESIwbjyZwOPuE2usc2f1Rua4+H6TKFGjACZ8+oJ\n\t9T8XYEP0QzOs5AhGMk6MvEgY+CWRl2VCQJGedUwGwRrE0Ei0LtkHJ6wHyznrcmoLVK\n\tkShKHXKvC/vDovcLAz8TcnZVKZRa4QmNrmQjLVAw=","Content-Type":"text/plain; charset=\"utf-8\"","MIME-Version":"1.0","Content-Transfer-Encoding":"quoted-printable","In-Reply-To":"<2322dac9-04a3-49cf-8faa-bfe86ecadaee@ideasonboard.com>","References":"<20250721104622.1550908-1-barnabas.pocze@ideasonboard.com>\n\t<20250721104622.1550908-8-barnabas.pocze@ideasonboard.com>\n\t<175811057476.2127323.6616992472481774741@neptunite.rasen.tech>\n\t<2322dac9-04a3-49cf-8faa-bfe86ecadaee@ideasonboard.com>","Subject":"Re: [RFC PATCH v2 07/22] libcamera: Add `MetadataList`","From":"Paul Elder ","To":"=?utf-8?q?Barnab=C3=A1s_P=C5=91cze?= ,\n\tlibcamera-devel@lists.libcamera.org","Date":"Tue, 07 Oct 2025 20:29:42 +0900","Message-ID":"<175983658218.2583768.9921359216503718502@neptunite.rasen.tech>","User-Agent":"alot/0.0.0","X-BeenThere":"libcamera-devel@lists.libcamera.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"libcamera-devel-bounces@lists.libcamera.org","Sender":"\"libcamera-devel\" "}}]