| Message ID | 20201105001546.1690179-8-niklas.soderlund@ragnatech.se |
|---|---|
| State | Superseded |
| Delegated to: | Niklas Söderlund |
| Headers | show |
| Series |
|
| Related | show |
Hi Niklas, On 05/11/2020 00:15, Niklas Söderlund wrote: > Add an empty IPA skeleton for the IPU3 pipeline. The skeleton IPA do > handle the flow of parameter and statistic buffers but does not read or s/do handle/handles/ > write anything in the buffers. It also allows the IPA to set sensor > controls but does not implement any logic to set optimal values and > instead set the V4L2 exposure and gain controls to max and keeps them at > that setting. > > This IPA is meant as a base to allow the pipeline handler to be wired up > to an IPA. The image algorithms can then later be added to the IPA > independent from also having to also add plumbing to the pipeline > handler. > > Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> As long as this doesn't break existing running of the pipeline, I'd pretty much go for an anything goes here, to build upon ;-) > --- > include/libcamera/ipa/ipu3.h | 22 ++++ > src/ipa/ipu3/ipu3.cpp | 237 +++++++++++++++++++++++++++++++++++ > src/ipa/ipu3/meson.build | 21 ++++ > src/ipa/meson.build | 2 +- > 4 files changed, 281 insertions(+), 1 deletion(-) > create mode 100644 include/libcamera/ipa/ipu3.h > create mode 100644 src/ipa/ipu3/ipu3.cpp > create mode 100644 src/ipa/ipu3/meson.build > > diff --git a/include/libcamera/ipa/ipu3.h b/include/libcamera/ipa/ipu3.h > new file mode 100644 > index 0000000000000000..7c240a46b97dee28 > --- /dev/null > +++ b/include/libcamera/ipa/ipu3.h > @@ -0,0 +1,22 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.h - Image Processing Algorithm interface for IPU3 > + */ > +#ifndef __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > +#define __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > + > +#ifndef __DOXYGEN__ > + > +enum IPU3Operations { > + IPU3_IPA_ACTION_SET_SENSOR_CONTROLS = 1, > + IPU3_IPA_ACTION_PARAM_FILLED = 2, > + IPU3_IPA_ACTION_METADATA_READY = 3, > + IPU3_IPA_EVENT_PARSE_STAT = 4, > + IPU3_IPA_EVENT_FILL_PARAMS = 5, > +}; > + > +#endif /* __DOXYGEN__ */ > + > +#endif /* __LIBCAMERA_IPA_INTERFACE_IPU3_H__ */ > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp > new file mode 100644 > index 0000000000000000..cdcd9b9ff013e0cc > --- /dev/null > +++ b/src/ipa/ipu3/ipu3.cpp > @@ -0,0 +1,237 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.cpp - IPU3 Image Processing Algorithms > + */ > + > +#include <algorithm> > +#include <math.h> > +#include <queue> > +#include <stdint.h> > +#include <string.h> > +#include <sys/mman.h> > + > +#include <linux/intel-ipu3.h> > +#include <linux/v4l2-controls.h> > + > +#include <libcamera/buffer.h> > +#include <libcamera/control_ids.h> > +#include <libcamera/ipa/ipa_interface.h> > +#include <libcamera/ipa/ipa_module_info.h> > +#include <libcamera/ipa/ipu3.h> > +#include <libcamera/request.h> > + > +#include <libipa/ipa_interface_wrapper.h> > + > +#include "libcamera/internal/log.h" > + > +namespace libcamera { > + > +LOG_DEFINE_CATEGORY(IPAIPU3) > + > +class IPAIPU3 : public IPAInterface > +{ > +public: > + int init([[maybe_unused]] const IPASettings &settings) override > + { > + return 0; > + } > + int start() override { return 0; } > + void stop() override {} > + > + void configure(const CameraSensorInfo &info, > + const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + const IPAOperationData &ipaConfig, > + IPAOperationData *response) override; > + void mapBuffers(const std::vector<IPABuffer> &buffers) override; > + void unmapBuffers(const std::vector<unsigned int> &ids) override; > + void processEvent(const IPAOperationData &event) override; > + > +private: > + void fillParams(unsigned int frame, ipu3_uapi_params *params, > + const ControlList &controls); > + > + void parseStatistics(unsigned int frame, > + const ipu3_uapi_stats_3a *stats); > + > + void setControls(unsigned int frame); > + > + std::map<unsigned int, FrameBuffer> buffers_; > + std::map<unsigned int, void *> buffersMemory_; > + > + ControlInfoMap ctrls_; > + > + /* Camera sensor controls. */ > + uint32_t exposure_; > + uint32_t minExposure_; > + uint32_t maxExposure_; > + uint32_t gain_; > + uint32_t minGain_; > + uint32_t maxGain_; > +}; > + > +void IPAIPU3::configure([[maybe_unused]] const CameraSensorInfo &info, > + [[maybe_unused]] const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + [[maybe_unused]] const IPAOperationData &ipaConfig, > + [[maybe_unused]] IPAOperationData *result) > +{ > + if (entityControls.empty()) > + return; > + > + ctrls_ = entityControls.at(0); > + > + const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); > + if (itExp == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find exposure control"; > + return; > + } > + > + const auto itGain = ctrls_.find(V4L2_CID_ANALOGUE_GAIN); > + if (itGain == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find gain control"; > + return; > + } I'm a little confused here, expecting the IPA to be dealing with libcamera controls and the translation to happen in the pipeline. Otherwise, are these v4l2 controls on the sensor? or the ISP? But maybe that doesn't matter. Anyway, anything goes still applies. This is initial skeleton, so I'll not argue strongly against this if it's just prototype place holder code to change later. > + > + minExposure_ = std::max<uint32_t>(itExp->second.min().get<int32_t>(), 1); > + maxExposure_ = itExp->second.max().get<int32_t>(); > + exposure_ = maxExposure_; > + > + minGain_ = std::max<uint32_t>(itGain->second.min().get<int32_t>(), 1); > + maxGain_ = itGain->second.max().get<int32_t>(); > + gain_ = maxGain_; > + > + setControls(0); > +} > + > +void IPAIPU3::mapBuffers(const std::vector<IPABuffer> &buffers) > +{ > + for (const IPABuffer &buffer : buffers) { > + auto elem = buffers_.emplace(std::piecewise_construct, > + std::forward_as_tuple(buffer.id), > + std::forward_as_tuple(buffer.planes)); > + const FrameBuffer &fb = elem.first->second; > + > + buffersMemory_[buffer.id] = mmap(NULL, > + fb.planes()[0].length, > + PROT_READ | PROT_WRITE, > + MAP_SHARED, > + fb.planes()[0].fd.fd(), > + 0); > + > + if (buffersMemory_[buffer.id] == MAP_FAILED) { > + int ret = -errno; > + LOG(IPAIPU3, Fatal) << "Failed to mmap buffer: " > + << strerror(-ret); > + } > + } > +} > + > +void IPAIPU3::unmapBuffers(const std::vector<unsigned int> &ids) > +{ > + for (unsigned int id : ids) { > + const auto fb = buffers_.find(id); > + if (fb == buffers_.end()) > + continue; > + > + munmap(buffersMemory_[id], fb->second.planes()[0].length); > + buffersMemory_.erase(id); > + buffers_.erase(id); > + } > +} > + > +void IPAIPU3::processEvent(const IPAOperationData &event) > +{ > + switch (event.operation) { > + case IPU3_IPA_EVENT_PARSE_STAT: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + const ipu3_uapi_stats_3a *stats = > + static_cast<ipu3_uapi_stats_3a *>(buffersMemory_[bufferId]); > + > + parseStatistics(frame, stats); > + break; > + } > + case IPU3_IPA_EVENT_FILL_PARAMS: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + ipu3_uapi_params *params = > + static_cast<ipu3_uapi_params *>(buffersMemory_[bufferId]); > + > + fillParams(frame, params, event.controls[0]); > + break; > + } > + default: > + LOG(IPAIPU3, Error) << "Unknown event " << event.operation; > + break; > + } > +} > + > +void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params, > + [[maybe_unused]] const ControlList &controls) > +{ > + /* Prepare parameters buffer. */ > + memset(params, 0, sizeof(*params)); > + > + /* \todo Fill in parameters buffer. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_PARAM_FILLED; > + > + queueFrameAction.emit(frame, op); > + > + /* \todo Calculate new values for exposure_ and gain_. */ > + setControls(frame); > +} > + > +void IPAIPU3::parseStatistics(unsigned int frame, > + [[maybe_unused]] const ipu3_uapi_stats_3a *stats) > +{ > + ControlList ctrls(controls::controls); > + > + /* \todo React to statistics and update internal state machine. */ > + /* \todo Add meta-data information to ctrls. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_METADATA_READY; > + op.controls.push_back(ctrls); > + > + queueFrameAction.emit(frame, op); > +} > + > +void IPAIPU3::setControls(unsigned int frame) > +{ > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_SET_SENSOR_CONTROLS; > + > + ControlList ctrls(ctrls_); > + ctrls.set(V4L2_CID_EXPOSURE, static_cast<int32_t>(exposure_)); > + ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast<int32_t>(gain_)); > + op.controls.push_back(ctrls); Does this happen for every frame? Ah - I see this is called from configure, with setControls(0). Anyway, I see this all as a starting point, not a finishing point. Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com> > + > + queueFrameAction.emit(frame, op); > +} > + > +/* > + * External IPA module interface > + */ > + > +extern "C" { > +const struct IPAModuleInfo ipaModuleInfo = { > + IPA_MODULE_API_VERSION, > + 1, > + "PipelineHandlerIPU3", > + "ipu3", > +}; > + > +struct ipa_context *ipaCreate() > +{ > + return new IPAInterfaceWrapper(std::make_unique<IPAIPU3>()); > +} > +} > + > +} /* namespace libcamera */ > diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build > new file mode 100644 > index 0000000000000000..a7e18f06b62bef3e > --- /dev/null > +++ b/src/ipa/ipu3/meson.build > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: CC0-1.0 > + > +ipa_name = 'ipa_ipu3' > + > +mod = shared_module(ipa_name, > + 'ipu3.cpp', > + name_prefix : '', > + include_directories : [ipa_includes, libipa_includes], > + dependencies : libcamera_dep, > + link_with : libipa, > + install : true, > + install_dir : ipa_install_dir) > + > +if ipa_sign_module > + custom_target(ipa_name + '.so.sign', > + input : mod, > + output : ipa_name + '.so.sign', > + command : [ ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@' ], > + install : false, > + build_by_default : true) > +endif > diff --git a/src/ipa/meson.build b/src/ipa/meson.build > index 5a5de267c1477d24..9d623f227a1f9feb 100644 > --- a/src/ipa/meson.build > +++ b/src/ipa/meson.build > @@ -19,7 +19,7 @@ subdir('libipa') > > ipa_sign = files('ipa-sign.sh') > > -ipas = ['raspberrypi', 'rkisp1', 'vimc'] > +ipas = ['ipu3', 'raspberrypi', 'rkisp1', 'vimc'] > ipa_names = [] > > foreach pipeline : get_option('pipelines') >
Hi Niklas, On Thu, Nov 05, 2020 at 01:15:42AM +0100, Niklas Söderlund wrote: > Add an empty IPA skeleton for the IPU3 pipeline. The skeleton IPA do > handle the flow of parameter and statistic buffers but does not read or > write anything in the buffers. It also allows the IPA to set sensor > controls but does not implement any logic to set optimal values and > instead set the V4L2 exposure and gain controls to max and keeps them at > that setting. > > This IPA is meant as a base to allow the pipeline handler to be wired up > to an IPA. The image algorithms can then later be added to the IPA > independent from also having to also add plumbing to the pipeline > handler. I would have expected this to be based on Paul's IPC work. Is it intentioanlly defined as a legacy IPA with the idea of moving it to the new framework later ? Paul, as you've done the conversion of all the legacy IPAs, what would it be the best course of actions in your opinion ? Thanks j > > Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> > --- > include/libcamera/ipa/ipu3.h | 22 ++++ > src/ipa/ipu3/ipu3.cpp | 237 +++++++++++++++++++++++++++++++++++ > src/ipa/ipu3/meson.build | 21 ++++ > src/ipa/meson.build | 2 +- > 4 files changed, 281 insertions(+), 1 deletion(-) > create mode 100644 include/libcamera/ipa/ipu3.h > create mode 100644 src/ipa/ipu3/ipu3.cpp > create mode 100644 src/ipa/ipu3/meson.build > > diff --git a/include/libcamera/ipa/ipu3.h b/include/libcamera/ipa/ipu3.h > new file mode 100644 > index 0000000000000000..7c240a46b97dee28 > --- /dev/null > +++ b/include/libcamera/ipa/ipu3.h > @@ -0,0 +1,22 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.h - Image Processing Algorithm interface for IPU3 > + */ > +#ifndef __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > +#define __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > + > +#ifndef __DOXYGEN__ > + > +enum IPU3Operations { > + IPU3_IPA_ACTION_SET_SENSOR_CONTROLS = 1, > + IPU3_IPA_ACTION_PARAM_FILLED = 2, > + IPU3_IPA_ACTION_METADATA_READY = 3, > + IPU3_IPA_EVENT_PARSE_STAT = 4, > + IPU3_IPA_EVENT_FILL_PARAMS = 5, > +}; > + > +#endif /* __DOXYGEN__ */ > + > +#endif /* __LIBCAMERA_IPA_INTERFACE_IPU3_H__ */ > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp > new file mode 100644 > index 0000000000000000..cdcd9b9ff013e0cc > --- /dev/null > +++ b/src/ipa/ipu3/ipu3.cpp > @@ -0,0 +1,237 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.cpp - IPU3 Image Processing Algorithms > + */ > + > +#include <algorithm> > +#include <math.h> > +#include <queue> > +#include <stdint.h> > +#include <string.h> > +#include <sys/mman.h> > + > +#include <linux/intel-ipu3.h> > +#include <linux/v4l2-controls.h> > + > +#include <libcamera/buffer.h> > +#include <libcamera/control_ids.h> > +#include <libcamera/ipa/ipa_interface.h> > +#include <libcamera/ipa/ipa_module_info.h> > +#include <libcamera/ipa/ipu3.h> > +#include <libcamera/request.h> > + > +#include <libipa/ipa_interface_wrapper.h> > + > +#include "libcamera/internal/log.h" > + > +namespace libcamera { > + > +LOG_DEFINE_CATEGORY(IPAIPU3) > + > +class IPAIPU3 : public IPAInterface > +{ > +public: > + int init([[maybe_unused]] const IPASettings &settings) override > + { > + return 0; > + } > + int start() override { return 0; } > + void stop() override {} > + > + void configure(const CameraSensorInfo &info, > + const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + const IPAOperationData &ipaConfig, > + IPAOperationData *response) override; > + void mapBuffers(const std::vector<IPABuffer> &buffers) override; > + void unmapBuffers(const std::vector<unsigned int> &ids) override; > + void processEvent(const IPAOperationData &event) override; > + > +private: > + void fillParams(unsigned int frame, ipu3_uapi_params *params, > + const ControlList &controls); > + > + void parseStatistics(unsigned int frame, > + const ipu3_uapi_stats_3a *stats); > + > + void setControls(unsigned int frame); > + > + std::map<unsigned int, FrameBuffer> buffers_; > + std::map<unsigned int, void *> buffersMemory_; > + > + ControlInfoMap ctrls_; > + > + /* Camera sensor controls. */ > + uint32_t exposure_; > + uint32_t minExposure_; > + uint32_t maxExposure_; > + uint32_t gain_; > + uint32_t minGain_; > + uint32_t maxGain_; > +}; > + > +void IPAIPU3::configure([[maybe_unused]] const CameraSensorInfo &info, > + [[maybe_unused]] const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + [[maybe_unused]] const IPAOperationData &ipaConfig, > + [[maybe_unused]] IPAOperationData *result) > +{ > + if (entityControls.empty()) > + return; > + > + ctrls_ = entityControls.at(0); > + > + const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); > + if (itExp == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find exposure control"; > + return; > + } > + > + const auto itGain = ctrls_.find(V4L2_CID_ANALOGUE_GAIN); > + if (itGain == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find gain control"; > + return; > + } > + > + minExposure_ = std::max<uint32_t>(itExp->second.min().get<int32_t>(), 1); > + maxExposure_ = itExp->second.max().get<int32_t>(); > + exposure_ = maxExposure_; > + > + minGain_ = std::max<uint32_t>(itGain->second.min().get<int32_t>(), 1); > + maxGain_ = itGain->second.max().get<int32_t>(); > + gain_ = maxGain_; > + > + setControls(0); > +} > + > +void IPAIPU3::mapBuffers(const std::vector<IPABuffer> &buffers) > +{ > + for (const IPABuffer &buffer : buffers) { > + auto elem = buffers_.emplace(std::piecewise_construct, > + std::forward_as_tuple(buffer.id), > + std::forward_as_tuple(buffer.planes)); > + const FrameBuffer &fb = elem.first->second; > + > + buffersMemory_[buffer.id] = mmap(NULL, > + fb.planes()[0].length, > + PROT_READ | PROT_WRITE, > + MAP_SHARED, > + fb.planes()[0].fd.fd(), > + 0); > + > + if (buffersMemory_[buffer.id] == MAP_FAILED) { > + int ret = -errno; > + LOG(IPAIPU3, Fatal) << "Failed to mmap buffer: " > + << strerror(-ret); > + } > + } > +} > + > +void IPAIPU3::unmapBuffers(const std::vector<unsigned int> &ids) > +{ > + for (unsigned int id : ids) { > + const auto fb = buffers_.find(id); > + if (fb == buffers_.end()) > + continue; > + > + munmap(buffersMemory_[id], fb->second.planes()[0].length); > + buffersMemory_.erase(id); > + buffers_.erase(id); > + } > +} > + > +void IPAIPU3::processEvent(const IPAOperationData &event) > +{ > + switch (event.operation) { > + case IPU3_IPA_EVENT_PARSE_STAT: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + const ipu3_uapi_stats_3a *stats = > + static_cast<ipu3_uapi_stats_3a *>(buffersMemory_[bufferId]); > + > + parseStatistics(frame, stats); > + break; > + } > + case IPU3_IPA_EVENT_FILL_PARAMS: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + ipu3_uapi_params *params = > + static_cast<ipu3_uapi_params *>(buffersMemory_[bufferId]); > + > + fillParams(frame, params, event.controls[0]); > + break; > + } > + default: > + LOG(IPAIPU3, Error) << "Unknown event " << event.operation; > + break; > + } > +} > + > +void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params, > + [[maybe_unused]] const ControlList &controls) > +{ > + /* Prepare parameters buffer. */ > + memset(params, 0, sizeof(*params)); > + > + /* \todo Fill in parameters buffer. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_PARAM_FILLED; > + > + queueFrameAction.emit(frame, op); > + > + /* \todo Calculate new values for exposure_ and gain_. */ > + setControls(frame); > +} > + > +void IPAIPU3::parseStatistics(unsigned int frame, > + [[maybe_unused]] const ipu3_uapi_stats_3a *stats) > +{ > + ControlList ctrls(controls::controls); > + > + /* \todo React to statistics and update internal state machine. */ > + /* \todo Add meta-data information to ctrls. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_METADATA_READY; > + op.controls.push_back(ctrls); > + > + queueFrameAction.emit(frame, op); > +} > + > +void IPAIPU3::setControls(unsigned int frame) > +{ > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_SET_SENSOR_CONTROLS; > + > + ControlList ctrls(ctrls_); > + ctrls.set(V4L2_CID_EXPOSURE, static_cast<int32_t>(exposure_)); > + ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast<int32_t>(gain_)); > + op.controls.push_back(ctrls); > + > + queueFrameAction.emit(frame, op); > +} > + > +/* > + * External IPA module interface > + */ > + > +extern "C" { > +const struct IPAModuleInfo ipaModuleInfo = { > + IPA_MODULE_API_VERSION, > + 1, > + "PipelineHandlerIPU3", > + "ipu3", > +}; > + > +struct ipa_context *ipaCreate() > +{ > + return new IPAInterfaceWrapper(std::make_unique<IPAIPU3>()); > +} > +} > + > +} /* namespace libcamera */ > diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build > new file mode 100644 > index 0000000000000000..a7e18f06b62bef3e > --- /dev/null > +++ b/src/ipa/ipu3/meson.build > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: CC0-1.0 > + > +ipa_name = 'ipa_ipu3' > + > +mod = shared_module(ipa_name, > + 'ipu3.cpp', > + name_prefix : '', > + include_directories : [ipa_includes, libipa_includes], > + dependencies : libcamera_dep, > + link_with : libipa, > + install : true, > + install_dir : ipa_install_dir) > + > +if ipa_sign_module > + custom_target(ipa_name + '.so.sign', > + input : mod, > + output : ipa_name + '.so.sign', > + command : [ ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@' ], > + install : false, > + build_by_default : true) > +endif > diff --git a/src/ipa/meson.build b/src/ipa/meson.build > index 5a5de267c1477d24..9d623f227a1f9feb 100644 > --- a/src/ipa/meson.build > +++ b/src/ipa/meson.build > @@ -19,7 +19,7 @@ subdir('libipa') > > ipa_sign = files('ipa-sign.sh') > > -ipas = ['raspberrypi', 'rkisp1', 'vimc'] > +ipas = ['ipu3', 'raspberrypi', 'rkisp1', 'vimc'] > ipa_names = [] > > foreach pipeline : get_option('pipelines') > -- > 2.29.2 > > _______________________________________________ > libcamera-devel mailing list > libcamera-devel@lists.libcamera.org > https://lists.libcamera.org/listinfo/libcamera-devel
Hi Niklas, On Thu, Nov 05, 2020 at 01:15:42AM +0100, Niklas Söderlund wrote: > Add an empty IPA skeleton for the IPU3 pipeline. The skeleton IPA do > handle the flow of parameter and statistic buffers but does not read or > write anything in the buffers. It also allows the IPA to set sensor > controls but does not implement any logic to set optimal values and > instead set the V4L2 exposure and gain controls to max and keeps them at > that setting. > > This IPA is meant as a base to allow the pipeline handler to be wired up > to an IPA. The image algorithms can then later be added to the IPA > independent from also having to also add plumbing to the pipeline > handler. > > Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> > --- > include/libcamera/ipa/ipu3.h | 22 ++++ > src/ipa/ipu3/ipu3.cpp | 237 +++++++++++++++++++++++++++++++++++ > src/ipa/ipu3/meson.build | 21 ++++ > src/ipa/meson.build | 2 +- > 4 files changed, 281 insertions(+), 1 deletion(-) > create mode 100644 include/libcamera/ipa/ipu3.h > create mode 100644 src/ipa/ipu3/ipu3.cpp > create mode 100644 src/ipa/ipu3/meson.build > > diff --git a/include/libcamera/ipa/ipu3.h b/include/libcamera/ipa/ipu3.h > new file mode 100644 > index 0000000000000000..7c240a46b97dee28 > --- /dev/null > +++ b/include/libcamera/ipa/ipu3.h > @@ -0,0 +1,22 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.h - Image Processing Algorithm interface for IPU3 > + */ > +#ifndef __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > +#define __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > + > +#ifndef __DOXYGEN__ > + > +enum IPU3Operations { > + IPU3_IPA_ACTION_SET_SENSOR_CONTROLS = 1, > + IPU3_IPA_ACTION_PARAM_FILLED = 2, > + IPU3_IPA_ACTION_METADATA_READY = 3, > + IPU3_IPA_EVENT_PARSE_STAT = 4, > + IPU3_IPA_EVENT_FILL_PARAMS = 5, > +}; > + > +#endif /* __DOXYGEN__ */ > + > +#endif /* __LIBCAMERA_IPA_INTERFACE_IPU3_H__ */ > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp > new file mode 100644 > index 0000000000000000..cdcd9b9ff013e0cc > --- /dev/null > +++ b/src/ipa/ipu3/ipu3.cpp > @@ -0,0 +1,237 @@ > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > +/* > + * Copyright (C) 2020, Google Inc. > + * > + * ipu3.cpp - IPU3 Image Processing Algorithms > + */ > + > +#include <algorithm> > +#include <math.h> > +#include <queue> > +#include <stdint.h> > +#include <string.h> > +#include <sys/mman.h> > + > +#include <linux/intel-ipu3.h> > +#include <linux/v4l2-controls.h> > + > +#include <libcamera/buffer.h> > +#include <libcamera/control_ids.h> > +#include <libcamera/ipa/ipa_interface.h> > +#include <libcamera/ipa/ipa_module_info.h> > +#include <libcamera/ipa/ipu3.h> > +#include <libcamera/request.h> > + > +#include <libipa/ipa_interface_wrapper.h> > + > +#include "libcamera/internal/log.h" > + > +namespace libcamera { > + > +LOG_DEFINE_CATEGORY(IPAIPU3) > + > +class IPAIPU3 : public IPAInterface > +{ > +public: > + int init([[maybe_unused]] const IPASettings &settings) override > + { > + return 0; > + } > + int start() override { return 0; } > + void stop() override {} > + > + void configure(const CameraSensorInfo &info, > + const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + const IPAOperationData &ipaConfig, > + IPAOperationData *response) override; > + void mapBuffers(const std::vector<IPABuffer> &buffers) override; > + void unmapBuffers(const std::vector<unsigned int> &ids) override; > + void processEvent(const IPAOperationData &event) override; > + > +private: > + void fillParams(unsigned int frame, ipu3_uapi_params *params, > + const ControlList &controls); > + > + void parseStatistics(unsigned int frame, > + const ipu3_uapi_stats_3a *stats); > + > + void setControls(unsigned int frame); > + > + std::map<unsigned int, FrameBuffer> buffers_; > + std::map<unsigned int, void *> buffersMemory_; > + > + ControlInfoMap ctrls_; > + > + /* Camera sensor controls. */ > + uint32_t exposure_; > + uint32_t minExposure_; > + uint32_t maxExposure_; > + uint32_t gain_; > + uint32_t minGain_; > + uint32_t maxGain_; > +}; > + > +void IPAIPU3::configure([[maybe_unused]] const CameraSensorInfo &info, > + [[maybe_unused]] const std::map<unsigned int, IPAStream> &streamConfig, > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > + [[maybe_unused]] const IPAOperationData &ipaConfig, > + [[maybe_unused]] IPAOperationData *result) > +{ > + if (entityControls.empty()) > + return; > + > + ctrls_ = entityControls.at(0); > + > + const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); Shouldn't we use libcamera controls ? Otherwise the pipeline handler has to implement the translation. The rest is good, with the above question clarified I'll send my tag. > + if (itExp == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find exposure control"; > + return; > + } > + > + const auto itGain = ctrls_.find(V4L2_CID_ANALOGUE_GAIN); > + if (itGain == ctrls_.end()) { > + LOG(IPAIPU3, Error) << "Can't find gain control"; > + return; > + } > + > + minExposure_ = std::max<uint32_t>(itExp->second.min().get<int32_t>(), 1); > + maxExposure_ = itExp->second.max().get<int32_t>(); > + exposure_ = maxExposure_; > + > + minGain_ = std::max<uint32_t>(itGain->second.min().get<int32_t>(), 1); > + maxGain_ = itGain->second.max().get<int32_t>(); > + gain_ = maxGain_; > + > + setControls(0); > +} > + > +void IPAIPU3::mapBuffers(const std::vector<IPABuffer> &buffers) > +{ > + for (const IPABuffer &buffer : buffers) { > + auto elem = buffers_.emplace(std::piecewise_construct, > + std::forward_as_tuple(buffer.id), > + std::forward_as_tuple(buffer.planes)); > + const FrameBuffer &fb = elem.first->second; > + > + buffersMemory_[buffer.id] = mmap(NULL, > + fb.planes()[0].length, > + PROT_READ | PROT_WRITE, > + MAP_SHARED, > + fb.planes()[0].fd.fd(), > + 0); > + > + if (buffersMemory_[buffer.id] == MAP_FAILED) { > + int ret = -errno; > + LOG(IPAIPU3, Fatal) << "Failed to mmap buffer: " > + << strerror(-ret); > + } > + } > +} > + > +void IPAIPU3::unmapBuffers(const std::vector<unsigned int> &ids) > +{ > + for (unsigned int id : ids) { > + const auto fb = buffers_.find(id); > + if (fb == buffers_.end()) > + continue; > + > + munmap(buffersMemory_[id], fb->second.planes()[0].length); > + buffersMemory_.erase(id); > + buffers_.erase(id); > + } > +} > + > +void IPAIPU3::processEvent(const IPAOperationData &event) > +{ > + switch (event.operation) { > + case IPU3_IPA_EVENT_PARSE_STAT: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + const ipu3_uapi_stats_3a *stats = > + static_cast<ipu3_uapi_stats_3a *>(buffersMemory_[bufferId]); > + > + parseStatistics(frame, stats); > + break; > + } > + case IPU3_IPA_EVENT_FILL_PARAMS: { > + unsigned int frame = event.data[0]; > + unsigned int bufferId = event.data[1]; > + > + ipu3_uapi_params *params = > + static_cast<ipu3_uapi_params *>(buffersMemory_[bufferId]); > + > + fillParams(frame, params, event.controls[0]); > + break; > + } > + default: > + LOG(IPAIPU3, Error) << "Unknown event " << event.operation; > + break; > + } > +} > + > +void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params, > + [[maybe_unused]] const ControlList &controls) > +{ > + /* Prepare parameters buffer. */ > + memset(params, 0, sizeof(*params)); > + > + /* \todo Fill in parameters buffer. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_PARAM_FILLED; > + > + queueFrameAction.emit(frame, op); > + > + /* \todo Calculate new values for exposure_ and gain_. */ > + setControls(frame); > +} > + > +void IPAIPU3::parseStatistics(unsigned int frame, > + [[maybe_unused]] const ipu3_uapi_stats_3a *stats) > +{ > + ControlList ctrls(controls::controls); > + > + /* \todo React to statistics and update internal state machine. */ > + /* \todo Add meta-data information to ctrls. */ > + > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_METADATA_READY; > + op.controls.push_back(ctrls); > + > + queueFrameAction.emit(frame, op); > +} > + > +void IPAIPU3::setControls(unsigned int frame) > +{ > + IPAOperationData op; > + op.operation = IPU3_IPA_ACTION_SET_SENSOR_CONTROLS; > + > + ControlList ctrls(ctrls_); > + ctrls.set(V4L2_CID_EXPOSURE, static_cast<int32_t>(exposure_)); > + ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast<int32_t>(gain_)); > + op.controls.push_back(ctrls); > + > + queueFrameAction.emit(frame, op); > +} > + > +/* > + * External IPA module interface > + */ > + > +extern "C" { > +const struct IPAModuleInfo ipaModuleInfo = { > + IPA_MODULE_API_VERSION, > + 1, > + "PipelineHandlerIPU3", > + "ipu3", > +}; > + > +struct ipa_context *ipaCreate() > +{ > + return new IPAInterfaceWrapper(std::make_unique<IPAIPU3>()); > +} > +} > + > +} /* namespace libcamera */ > diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build > new file mode 100644 > index 0000000000000000..a7e18f06b62bef3e > --- /dev/null > +++ b/src/ipa/ipu3/meson.build > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: CC0-1.0 > + > +ipa_name = 'ipa_ipu3' > + > +mod = shared_module(ipa_name, > + 'ipu3.cpp', > + name_prefix : '', > + include_directories : [ipa_includes, libipa_includes], > + dependencies : libcamera_dep, > + link_with : libipa, > + install : true, > + install_dir : ipa_install_dir) > + > +if ipa_sign_module > + custom_target(ipa_name + '.so.sign', > + input : mod, > + output : ipa_name + '.so.sign', > + command : [ ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@' ], > + install : false, > + build_by_default : true) > +endif > diff --git a/src/ipa/meson.build b/src/ipa/meson.build > index 5a5de267c1477d24..9d623f227a1f9feb 100644 > --- a/src/ipa/meson.build > +++ b/src/ipa/meson.build > @@ -19,7 +19,7 @@ subdir('libipa') > > ipa_sign = files('ipa-sign.sh') > > -ipas = ['raspberrypi', 'rkisp1', 'vimc'] > +ipas = ['ipu3', 'raspberrypi', 'rkisp1', 'vimc'] > ipa_names = [] > > foreach pipeline : get_option('pipelines') > -- > 2.29.2 > > _______________________________________________ > libcamera-devel mailing list > libcamera-devel@lists.libcamera.org > https://lists.libcamera.org/listinfo/libcamera-devel
Hi Niklas, Thank you for the patch. On Wed, Nov 18, 2020 at 04:41:09PM +0100, Jacopo Mondi wrote: > On Thu, Nov 05, 2020 at 01:15:42AM +0100, Niklas Söderlund wrote: > > Add an empty IPA skeleton for the IPU3 pipeline. The skeleton IPA do > > handle the flow of parameter and statistic buffers but does not read or s/do handle/handles/ > > write anything in the buffers. It also allows the IPA to set sensor > > controls but does not implement any logic to set optimal values and > > instead set the V4L2 exposure and gain controls to max and keeps them at s/set/sets/ > > that setting. > > > > This IPA is meant as a base to allow the pipeline handler to be wired up > > to an IPA. The image algorithms can then later be added to the IPA > > independent from also having to also add plumbing to the pipeline s/independent/independently/ s/to also/to/ > > handler. > > > > Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> > > --- > > include/libcamera/ipa/ipu3.h | 22 ++++ > > src/ipa/ipu3/ipu3.cpp | 237 +++++++++++++++++++++++++++++++++++ > > src/ipa/ipu3/meson.build | 21 ++++ > > src/ipa/meson.build | 2 +- > > 4 files changed, 281 insertions(+), 1 deletion(-) > > create mode 100644 include/libcamera/ipa/ipu3.h > > create mode 100644 src/ipa/ipu3/ipu3.cpp > > create mode 100644 src/ipa/ipu3/meson.build > > > > diff --git a/include/libcamera/ipa/ipu3.h b/include/libcamera/ipa/ipu3.h > > new file mode 100644 > > index 0000000000000000..7c240a46b97dee28 > > --- /dev/null > > +++ b/include/libcamera/ipa/ipu3.h > > @@ -0,0 +1,22 @@ > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > > +/* > > + * Copyright (C) 2020, Google Inc. > > + * > > + * ipu3.h - Image Processing Algorithm interface for IPU3 > > + */ > > +#ifndef __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > > +#define __LIBCAMERA_IPA_INTERFACE_IPU3_H__ > > + > > +#ifndef __DOXYGEN__ > > + > > +enum IPU3Operations { > > + IPU3_IPA_ACTION_SET_SENSOR_CONTROLS = 1, > > + IPU3_IPA_ACTION_PARAM_FILLED = 2, > > + IPU3_IPA_ACTION_METADATA_READY = 3, > > + IPU3_IPA_EVENT_PARSE_STAT = 4, As this is an event, should it be named stat ready instead of parse stat ? No big deal, this will be reworked with Paul's IPA IPC work anyway. > > + IPU3_IPA_EVENT_FILL_PARAMS = 5, > > +}; > > + > > +#endif /* __DOXYGEN__ */ > > + > > +#endif /* __LIBCAMERA_IPA_INTERFACE_IPU3_H__ */ > > diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp > > new file mode 100644 > > index 0000000000000000..cdcd9b9ff013e0cc > > --- /dev/null > > +++ b/src/ipa/ipu3/ipu3.cpp > > @@ -0,0 +1,237 @@ > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */ > > +/* > > + * Copyright (C) 2020, Google Inc. > > + * > > + * ipu3.cpp - IPU3 Image Processing Algorithms > > + */ > > + > > +#include <algorithm> > > +#include <math.h> > > +#include <queue> > > +#include <stdint.h> > > +#include <string.h> > > +#include <sys/mman.h> Some of these headers seem unused. > > + > > +#include <linux/intel-ipu3.h> > > +#include <linux/v4l2-controls.h> > > + > > +#include <libcamera/buffer.h> > > +#include <libcamera/control_ids.h> > > +#include <libcamera/ipa/ipa_interface.h> > > +#include <libcamera/ipa/ipa_module_info.h> > > +#include <libcamera/ipa/ipu3.h> You could include this file at the top, to ensure it gets compiled self-contained. > > +#include <libcamera/request.h> > > + > > +#include <libipa/ipa_interface_wrapper.h> > > + > > +#include "libcamera/internal/log.h" > > + > > +namespace libcamera { > > + > > +LOG_DEFINE_CATEGORY(IPAIPU3) > > + > > +class IPAIPU3 : public IPAInterface > > +{ > > +public: > > + int init([[maybe_unused]] const IPASettings &settings) override > > + { > > + return 0; > > + } > > + int start() override { return 0; } > > + void stop() override {} > > + > > + void configure(const CameraSensorInfo &info, > > + const std::map<unsigned int, IPAStream> &streamConfig, > > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > > + const IPAOperationData &ipaConfig, > > + IPAOperationData *response) override; > > + void mapBuffers(const std::vector<IPABuffer> &buffers) override; > > + void unmapBuffers(const std::vector<unsigned int> &ids) override; > > + void processEvent(const IPAOperationData &event) override; > > + > > +private: > > + void fillParams(unsigned int frame, ipu3_uapi_params *params, > > + const ControlList &controls); > > + > > + void parseStatistics(unsigned int frame, > > + const ipu3_uapi_stats_3a *stats); > > + > > + void setControls(unsigned int frame); > > + > > + std::map<unsigned int, FrameBuffer> buffers_; > > + std::map<unsigned int, void *> buffersMemory_; > > + > > + ControlInfoMap ctrls_; > > + > > + /* Camera sensor controls. */ > > + uint32_t exposure_; > > + uint32_t minExposure_; > > + uint32_t maxExposure_; > > + uint32_t gain_; > > + uint32_t minGain_; > > + uint32_t maxGain_; > > +}; > > + > > +void IPAIPU3::configure([[maybe_unused]] const CameraSensorInfo &info, > > + [[maybe_unused]] const std::map<unsigned int, IPAStream> &streamConfig, > > + const std::map<unsigned int, const ControlInfoMap &> &entityControls, > > + [[maybe_unused]] const IPAOperationData &ipaConfig, > > + [[maybe_unused]] IPAOperationData *result) > > +{ > > + if (entityControls.empty()) > > + return; > > + > > + ctrls_ = entityControls.at(0); > > + > > + const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); > > Shouldn't we use libcamera controls ? Otherwise the pipeline handler > has to implement the translation. > > The rest is good, with the above question clarified I'll send my tag. We do the same in the RPi IPA. This should be reworked to use controls specific to the CameraSensor class (we may reuse libcamera controls, but conceptually speaking, they're not the same, so we'll need a different ControlInfoMap to report the correct limits, libcamera::controls won't do). I'd rather fix this on top, for all IPA modules, as it requires a rework of the CameraSensor class. > > + if (itExp == ctrls_.end()) { > > + LOG(IPAIPU3, Error) << "Can't find exposure control"; > > + return; > > + } > > + > > + const auto itGain = ctrls_.find(V4L2_CID_ANALOGUE_GAIN); > > + if (itGain == ctrls_.end()) { > > + LOG(IPAIPU3, Error) << "Can't find gain control"; > > + return; > > + } > > + > > + minExposure_ = std::max<uint32_t>(itExp->second.min().get<int32_t>(), 1); > > + maxExposure_ = itExp->second.max().get<int32_t>(); > > + exposure_ = maxExposure_; > > + > > + minGain_ = std::max<uint32_t>(itGain->second.min().get<int32_t>(), 1); > > + maxGain_ = itGain->second.max().get<int32_t>(); > > + gain_ = maxGain_; > > + > > + setControls(0); I'm not sure we need to do so at configure() time, but this is a skeleton anyway, so it's no big deal. Let's however be careful on how this is handled on the pipeline handler side, an application could call configure() multiple times, and if this results in multiple calls to DelayedControls::push() before start(), I expect some trouble. This likely needs to be addressed on the pipeline handler side, and it's not a blocker as this is work in progress. > > +} > > + > > +void IPAIPU3::mapBuffers(const std::vector<IPABuffer> &buffers) > > +{ > > + for (const IPABuffer &buffer : buffers) { > > + auto elem = buffers_.emplace(std::piecewise_construct, > > + std::forward_as_tuple(buffer.id), > > + std::forward_as_tuple(buffer.planes)); > > + const FrameBuffer &fb = elem.first->second; > > + > > + buffersMemory_[buffer.id] = mmap(NULL, > > + fb.planes()[0].length, > > + PROT_READ | PROT_WRITE, > > + MAP_SHARED, > > + fb.planes()[0].fd.fd(), > > + 0); > > + > > + if (buffersMemory_[buffer.id] == MAP_FAILED) { > > + int ret = -errno; > > + LOG(IPAIPU3, Fatal) << "Failed to mmap buffer: " > > + << strerror(-ret); > > + } How about using MappedFrameBuffer ? The RPi IPA has moved to that since you've posted this series. > > + } > > +} > > + > > +void IPAIPU3::unmapBuffers(const std::vector<unsigned int> &ids) > > +{ > > + for (unsigned int id : ids) { > > + const auto fb = buffers_.find(id); > > + if (fb == buffers_.end()) > > + continue; > > + > > + munmap(buffersMemory_[id], fb->second.planes()[0].length); > > + buffersMemory_.erase(id); > > + buffers_.erase(id); > > + } > > +} > > + > > +void IPAIPU3::processEvent(const IPAOperationData &event) > > +{ > > + switch (event.operation) { > > + case IPU3_IPA_EVENT_PARSE_STAT: { > > + unsigned int frame = event.data[0]; > > + unsigned int bufferId = event.data[1]; > > + > > + const ipu3_uapi_stats_3a *stats = > > + static_cast<ipu3_uapi_stats_3a *>(buffersMemory_[bufferId]); > > + > > + parseStatistics(frame, stats); > > + break; > > + } > > + case IPU3_IPA_EVENT_FILL_PARAMS: { > > + unsigned int frame = event.data[0]; > > + unsigned int bufferId = event.data[1]; > > + > > + ipu3_uapi_params *params = > > + static_cast<ipu3_uapi_params *>(buffersMemory_[bufferId]); > > + > > + fillParams(frame, params, event.controls[0]); > > + break; > > + } > > + default: > > + LOG(IPAIPU3, Error) << "Unknown event " << event.operation; > > + break; > > + } > > +} > > + > > +void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params, > > + [[maybe_unused]] const ControlList &controls) > > +{ > > + /* Prepare parameters buffer. */ > > + memset(params, 0, sizeof(*params)); > > + > > + /* \todo Fill in parameters buffer. */ > > + > > + IPAOperationData op; > > + op.operation = IPU3_IPA_ACTION_PARAM_FILLED; > > + > > + queueFrameAction.emit(frame, op); > > + > > + /* \todo Calculate new values for exposure_ and gain_. */ > > + setControls(frame); > > +} > > + > > +void IPAIPU3::parseStatistics(unsigned int frame, > > + [[maybe_unused]] const ipu3_uapi_stats_3a *stats) > > +{ > > + ControlList ctrls(controls::controls); > > + > > + /* \todo React to statistics and update internal state machine. */ > > + /* \todo Add meta-data information to ctrls. */ > > + > > + IPAOperationData op; > > + op.operation = IPU3_IPA_ACTION_METADATA_READY; > > + op.controls.push_back(ctrls); > > + > > + queueFrameAction.emit(frame, op); > > +} > > + > > +void IPAIPU3::setControls(unsigned int frame) > > +{ > > + IPAOperationData op; > > + op.operation = IPU3_IPA_ACTION_SET_SENSOR_CONTROLS; > > + > > + ControlList ctrls(ctrls_); > > + ctrls.set(V4L2_CID_EXPOSURE, static_cast<int32_t>(exposure_)); > > + ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast<int32_t>(gain_)); > > + op.controls.push_back(ctrls); > > + > > + queueFrameAction.emit(frame, op); > > +} > > + > > +/* > > + * External IPA module interface > > + */ > > + > > +extern "C" { > > +const struct IPAModuleInfo ipaModuleInfo = { > > + IPA_MODULE_API_VERSION, > > + 1, > > + "PipelineHandlerIPU3", > > + "ipu3", > > +}; > > + > > +struct ipa_context *ipaCreate() > > +{ > > + return new IPAInterfaceWrapper(std::make_unique<IPAIPU3>()); > > +} > > +} > > + > > +} /* namespace libcamera */ > > diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build > > new file mode 100644 > > index 0000000000000000..a7e18f06b62bef3e > > --- /dev/null > > +++ b/src/ipa/ipu3/meson.build > > @@ -0,0 +1,21 @@ > > +# SPDX-License-Identifier: CC0-1.0 > > + > > +ipa_name = 'ipa_ipu3' > > + > > +mod = shared_module(ipa_name, > > + 'ipu3.cpp', > > + name_prefix : '', > > + include_directories : [ipa_includes, libipa_includes], Missing spaces after [ and before ]. > > + dependencies : libcamera_dep, > > + link_with : libipa, Based on 5d05418d9b53e1838692f687a6dc373dad45355c, I wonder if we should link with libatomic. > > + install : true, > > + install_dir : ipa_install_dir) > > + > > +if ipa_sign_module > > + custom_target(ipa_name + '.so.sign', > > + input : mod, > > + output : ipa_name + '.so.sign', > > + command : [ ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@' ], > > + install : false, > > + build_by_default : true) > > +endif > > diff --git a/src/ipa/meson.build b/src/ipa/meson.build > > index 5a5de267c1477d24..9d623f227a1f9feb 100644 > > --- a/src/ipa/meson.build > > +++ b/src/ipa/meson.build > > @@ -19,7 +19,7 @@ subdir('libipa') > > > > ipa_sign = files('ipa-sign.sh') > > > > -ipas = ['raspberrypi', 'rkisp1', 'vimc'] > > +ipas = ['ipu3', 'raspberrypi', 'rkisp1', 'vimc'] > > ipa_names = [] > > > > foreach pipeline : get_option('pipelines')
diff --git a/include/libcamera/ipa/ipu3.h b/include/libcamera/ipa/ipu3.h new file mode 100644 index 0000000000000000..7c240a46b97dee28 --- /dev/null +++ b/include/libcamera/ipa/ipu3.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipu3.h - Image Processing Algorithm interface for IPU3 + */ +#ifndef __LIBCAMERA_IPA_INTERFACE_IPU3_H__ +#define __LIBCAMERA_IPA_INTERFACE_IPU3_H__ + +#ifndef __DOXYGEN__ + +enum IPU3Operations { + IPU3_IPA_ACTION_SET_SENSOR_CONTROLS = 1, + IPU3_IPA_ACTION_PARAM_FILLED = 2, + IPU3_IPA_ACTION_METADATA_READY = 3, + IPU3_IPA_EVENT_PARSE_STAT = 4, + IPU3_IPA_EVENT_FILL_PARAMS = 5, +}; + +#endif /* __DOXYGEN__ */ + +#endif /* __LIBCAMERA_IPA_INTERFACE_IPU3_H__ */ diff --git a/src/ipa/ipu3/ipu3.cpp b/src/ipa/ipu3/ipu3.cpp new file mode 100644 index 0000000000000000..cdcd9b9ff013e0cc --- /dev/null +++ b/src/ipa/ipu3/ipu3.cpp @@ -0,0 +1,237 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +/* + * Copyright (C) 2020, Google Inc. + * + * ipu3.cpp - IPU3 Image Processing Algorithms + */ + +#include <algorithm> +#include <math.h> +#include <queue> +#include <stdint.h> +#include <string.h> +#include <sys/mman.h> + +#include <linux/intel-ipu3.h> +#include <linux/v4l2-controls.h> + +#include <libcamera/buffer.h> +#include <libcamera/control_ids.h> +#include <libcamera/ipa/ipa_interface.h> +#include <libcamera/ipa/ipa_module_info.h> +#include <libcamera/ipa/ipu3.h> +#include <libcamera/request.h> + +#include <libipa/ipa_interface_wrapper.h> + +#include "libcamera/internal/log.h" + +namespace libcamera { + +LOG_DEFINE_CATEGORY(IPAIPU3) + +class IPAIPU3 : public IPAInterface +{ +public: + int init([[maybe_unused]] const IPASettings &settings) override + { + return 0; + } + int start() override { return 0; } + void stop() override {} + + void configure(const CameraSensorInfo &info, + const std::map<unsigned int, IPAStream> &streamConfig, + const std::map<unsigned int, const ControlInfoMap &> &entityControls, + const IPAOperationData &ipaConfig, + IPAOperationData *response) override; + void mapBuffers(const std::vector<IPABuffer> &buffers) override; + void unmapBuffers(const std::vector<unsigned int> &ids) override; + void processEvent(const IPAOperationData &event) override; + +private: + void fillParams(unsigned int frame, ipu3_uapi_params *params, + const ControlList &controls); + + void parseStatistics(unsigned int frame, + const ipu3_uapi_stats_3a *stats); + + void setControls(unsigned int frame); + + std::map<unsigned int, FrameBuffer> buffers_; + std::map<unsigned int, void *> buffersMemory_; + + ControlInfoMap ctrls_; + + /* Camera sensor controls. */ + uint32_t exposure_; + uint32_t minExposure_; + uint32_t maxExposure_; + uint32_t gain_; + uint32_t minGain_; + uint32_t maxGain_; +}; + +void IPAIPU3::configure([[maybe_unused]] const CameraSensorInfo &info, + [[maybe_unused]] const std::map<unsigned int, IPAStream> &streamConfig, + const std::map<unsigned int, const ControlInfoMap &> &entityControls, + [[maybe_unused]] const IPAOperationData &ipaConfig, + [[maybe_unused]] IPAOperationData *result) +{ + if (entityControls.empty()) + return; + + ctrls_ = entityControls.at(0); + + const auto itExp = ctrls_.find(V4L2_CID_EXPOSURE); + if (itExp == ctrls_.end()) { + LOG(IPAIPU3, Error) << "Can't find exposure control"; + return; + } + + const auto itGain = ctrls_.find(V4L2_CID_ANALOGUE_GAIN); + if (itGain == ctrls_.end()) { + LOG(IPAIPU3, Error) << "Can't find gain control"; + return; + } + + minExposure_ = std::max<uint32_t>(itExp->second.min().get<int32_t>(), 1); + maxExposure_ = itExp->second.max().get<int32_t>(); + exposure_ = maxExposure_; + + minGain_ = std::max<uint32_t>(itGain->second.min().get<int32_t>(), 1); + maxGain_ = itGain->second.max().get<int32_t>(); + gain_ = maxGain_; + + setControls(0); +} + +void IPAIPU3::mapBuffers(const std::vector<IPABuffer> &buffers) +{ + for (const IPABuffer &buffer : buffers) { + auto elem = buffers_.emplace(std::piecewise_construct, + std::forward_as_tuple(buffer.id), + std::forward_as_tuple(buffer.planes)); + const FrameBuffer &fb = elem.first->second; + + buffersMemory_[buffer.id] = mmap(NULL, + fb.planes()[0].length, + PROT_READ | PROT_WRITE, + MAP_SHARED, + fb.planes()[0].fd.fd(), + 0); + + if (buffersMemory_[buffer.id] == MAP_FAILED) { + int ret = -errno; + LOG(IPAIPU3, Fatal) << "Failed to mmap buffer: " + << strerror(-ret); + } + } +} + +void IPAIPU3::unmapBuffers(const std::vector<unsigned int> &ids) +{ + for (unsigned int id : ids) { + const auto fb = buffers_.find(id); + if (fb == buffers_.end()) + continue; + + munmap(buffersMemory_[id], fb->second.planes()[0].length); + buffersMemory_.erase(id); + buffers_.erase(id); + } +} + +void IPAIPU3::processEvent(const IPAOperationData &event) +{ + switch (event.operation) { + case IPU3_IPA_EVENT_PARSE_STAT: { + unsigned int frame = event.data[0]; + unsigned int bufferId = event.data[1]; + + const ipu3_uapi_stats_3a *stats = + static_cast<ipu3_uapi_stats_3a *>(buffersMemory_[bufferId]); + + parseStatistics(frame, stats); + break; + } + case IPU3_IPA_EVENT_FILL_PARAMS: { + unsigned int frame = event.data[0]; + unsigned int bufferId = event.data[1]; + + ipu3_uapi_params *params = + static_cast<ipu3_uapi_params *>(buffersMemory_[bufferId]); + + fillParams(frame, params, event.controls[0]); + break; + } + default: + LOG(IPAIPU3, Error) << "Unknown event " << event.operation; + break; + } +} + +void IPAIPU3::fillParams(unsigned int frame, ipu3_uapi_params *params, + [[maybe_unused]] const ControlList &controls) +{ + /* Prepare parameters buffer. */ + memset(params, 0, sizeof(*params)); + + /* \todo Fill in parameters buffer. */ + + IPAOperationData op; + op.operation = IPU3_IPA_ACTION_PARAM_FILLED; + + queueFrameAction.emit(frame, op); + + /* \todo Calculate new values for exposure_ and gain_. */ + setControls(frame); +} + +void IPAIPU3::parseStatistics(unsigned int frame, + [[maybe_unused]] const ipu3_uapi_stats_3a *stats) +{ + ControlList ctrls(controls::controls); + + /* \todo React to statistics and update internal state machine. */ + /* \todo Add meta-data information to ctrls. */ + + IPAOperationData op; + op.operation = IPU3_IPA_ACTION_METADATA_READY; + op.controls.push_back(ctrls); + + queueFrameAction.emit(frame, op); +} + +void IPAIPU3::setControls(unsigned int frame) +{ + IPAOperationData op; + op.operation = IPU3_IPA_ACTION_SET_SENSOR_CONTROLS; + + ControlList ctrls(ctrls_); + ctrls.set(V4L2_CID_EXPOSURE, static_cast<int32_t>(exposure_)); + ctrls.set(V4L2_CID_ANALOGUE_GAIN, static_cast<int32_t>(gain_)); + op.controls.push_back(ctrls); + + queueFrameAction.emit(frame, op); +} + +/* + * External IPA module interface + */ + +extern "C" { +const struct IPAModuleInfo ipaModuleInfo = { + IPA_MODULE_API_VERSION, + 1, + "PipelineHandlerIPU3", + "ipu3", +}; + +struct ipa_context *ipaCreate() +{ + return new IPAInterfaceWrapper(std::make_unique<IPAIPU3>()); +} +} + +} /* namespace libcamera */ diff --git a/src/ipa/ipu3/meson.build b/src/ipa/ipu3/meson.build new file mode 100644 index 0000000000000000..a7e18f06b62bef3e --- /dev/null +++ b/src/ipa/ipu3/meson.build @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: CC0-1.0 + +ipa_name = 'ipa_ipu3' + +mod = shared_module(ipa_name, + 'ipu3.cpp', + name_prefix : '', + include_directories : [ipa_includes, libipa_includes], + dependencies : libcamera_dep, + link_with : libipa, + install : true, + install_dir : ipa_install_dir) + +if ipa_sign_module + custom_target(ipa_name + '.so.sign', + input : mod, + output : ipa_name + '.so.sign', + command : [ ipa_sign, ipa_priv_key, '@INPUT@', '@OUTPUT@' ], + install : false, + build_by_default : true) +endif diff --git a/src/ipa/meson.build b/src/ipa/meson.build index 5a5de267c1477d24..9d623f227a1f9feb 100644 --- a/src/ipa/meson.build +++ b/src/ipa/meson.build @@ -19,7 +19,7 @@ subdir('libipa') ipa_sign = files('ipa-sign.sh') -ipas = ['raspberrypi', 'rkisp1', 'vimc'] +ipas = ['ipu3', 'raspberrypi', 'rkisp1', 'vimc'] ipa_names = [] foreach pipeline : get_option('pipelines')
Add an empty IPA skeleton for the IPU3 pipeline. The skeleton IPA do handle the flow of parameter and statistic buffers but does not read or write anything in the buffers. It also allows the IPA to set sensor controls but does not implement any logic to set optimal values and instead set the V4L2 exposure and gain controls to max and keeps them at that setting. This IPA is meant as a base to allow the pipeline handler to be wired up to an IPA. The image algorithms can then later be added to the IPA independent from also having to also add plumbing to the pipeline handler. Signed-off-by: Niklas Söderlund <niklas.soderlund@ragnatech.se> --- include/libcamera/ipa/ipu3.h | 22 ++++ src/ipa/ipu3/ipu3.cpp | 237 +++++++++++++++++++++++++++++++++++ src/ipa/ipu3/meson.build | 21 ++++ src/ipa/meson.build | 2 +- 4 files changed, 281 insertions(+), 1 deletion(-) create mode 100644 include/libcamera/ipa/ipu3.h create mode 100644 src/ipa/ipu3/ipu3.cpp create mode 100644 src/ipa/ipu3/meson.build