| Message ID | 20261021080000.16624-1-mzamazal@redhat.com |
|---|---|
| Headers | show |
| Series |
|
| Related | show |
Hi 2026. 10. 21. 9:59 keltezéssel, Milan Zamazal írta: > This is another attempt to introduce autofocus to software ISP. It is > not a complete implementation; autofocus stats are missing, to be > addressed in another patch series, to keep the patch series relatively > short. > > It is posted as RFC to gather initial comments on the approach. > I'm sharing this in case it turns out to be in any way useful. I have also started experimenting with contrast based autofocus, but with two significant differences: * on the rkisp1 + imx708 * without reusing the ipu3 algorithm. The rkisp1 has a hardware block to collect statistics for auto focus using the "tenengrad focus measure", which uses the Sobel operator. * https://opencv.org/autofocus-using-opencv-a-comparative-study-of-focus-measures-for-sharpness-assessment/ * https://en.wikipedia.org/wiki/Sobel_operator The decision not to copy&paste the ipu3 was twofold: * I wanted to do a from scratch implementation, * and I wanted to create the implementation with most of the many auto focus related controls in mind Here it is: https://gitlab.freedesktop.org/pobrn/libcamera/-/tree/af-prototype The basic idea is that a single unit-less "sharpness" value is provided, and the lens is moved in such a way that the maximum sharpness value is achieved. At the moment it implements manual and continuous mode. No auto focus window support for the time being. And it is of course a prototype, a lot of fine tuning is probably necessary to get the thresholds and all that right, and of course it uses a very trivial two phase scan. > What this series implements: > > - Software ISP autofocus is based on the IPU3 implementation. Generic > parts of the IPU3 autofocus algorithm are moved to libipa for the > purpose. > > - Software ISP autofocus algorithm based on that. > > - Autofocus (lens) controls added to software ISP. > > - Rudimentary support for providing the autofocus stats value from stats > (without actually implementing it). > > What should be added in future patch series: > > - Software ISP autofocus stats. While it may look easy, it’s not, there > are several things to address: accuracy, efficiency, frequency, > collisions with exposure adjustments, focus area, etc. There is some > former work available in Hans’s tree: > https://github.com/jwrdegoede/libcamera/tree/software-af-done > > - Manual focus support. This should be easy, maybe old Pavel’s patches > could be reused in some way? > > Milan Zamazal (7): > libcamera: software_isp: Let a single point decide about computing > stats > ipa: ipu3: Move non-specific parts of Af to libipa > libcamera: software_isp: Add variance to stats > ipa: softisp: Remove unused imports from ipa_context.h > ipa: softisp: Add Af algorithm > libcamera: software_isp: Add autofocus controls > ipa: libipa: Use autofocus limits if available > > .../internal/software_isp/software_isp.h | 4 +- > .../internal/software_isp/swisp_stats.h | 4 + > .../internal/software_isp/swstats_cpu.h | 17 +- > include/libcamera/ipa/softisp.mojom | 3 +- > src/ipa/ipu3/algorithms/af.cpp | 219 +----------- > src/ipa/ipu3/algorithms/af.h | 27 +- > src/ipa/ipu3/ipa_context.cpp | 17 +- > src/ipa/ipu3/ipa_context.h | 8 +- > src/ipa/libipa/af.cpp | 315 ++++++++++++++++++ > src/ipa/libipa/af.h | 72 ++++ > src/ipa/libipa/meson.build | 2 + > src/ipa/softisp/algorithms/af.cpp | 52 +++ > src/ipa/softisp/algorithms/af.h | 39 +++ > src/ipa/softisp/algorithms/meson.build | 1 + > src/ipa/softisp/data/uncalibrated.yaml | 1 + > src/ipa/softisp/ipa_context.h | 6 +- > src/ipa/softisp/softisp.cpp | 7 +- > src/libcamera/pipeline/simple/simple.cpp | 13 +- > src/libcamera/software_isp/debayer_egl.cpp | 24 +- > src/libcamera/software_isp/software_isp.cpp | 4 +- > src/libcamera/software_isp/swstats_cpu.cpp | 19 +- > 21 files changed, 560 insertions(+), 294 deletions(-) > create mode 100644 src/ipa/libipa/af.cpp > create mode 100644 src/ipa/libipa/af.h > create mode 100644 src/ipa/softisp/algorithms/af.cpp > create mode 100644 src/ipa/softisp/algorithms/af.h >
Hi Barnabás, Barnabás Pőcze <barnabas.pocze@ideasonboard.com> writes: > I'm sharing this in case it turns out to be in any way useful. It is, thank you. > I have also started experimenting with contrast based autofocus, but > with two significant differences: > > * on the rkisp1 + imx708 > * without reusing the ipu3 algorithm. > > The rkisp1 has a hardware block to collect statistics for auto focus using > the "tenengrad focus measure", which uses the Sobel operator. > > * https://opencv.org/autofocus-using-opencv-a-comparative-study-of-focus-measures-for-sharpness-assessment/ Interesting as for which algorithms fail badly. But otherwise the scene doesn't look very representative to me: no colours, no real background, no other objects than text, ... But ... > * https://en.wikipedia.org/wiki/Sobel_operator ... if everybody mentions this, it must be the right choice. :-) Anyway, implementing the algorithm itself looks to me like the easier part about the stats. > > The decision not to copy&paste the ipu3 was twofold: > > * I wanted to do a from scratch implementation, > * and I wanted to create the implementation with most of the many > auto focus related controls in mind Makes sense. My aim is in a different direction: There have been several autofocus efforts but no real progress towards actually getting one in software ISP. We should really do something about it, now. So I try to go simple, with reuse, splitting the whole stuff to several shorter patch series. Even if it may mean sort of wasted work, which will be replaced by something else before it gets merged or later in the future. > > Here it is: https://gitlab.freedesktop.org/pobrn/libcamera/-/tree/af-prototype Looks quite differently indeed. I plan to go on with my effort, let's see how things evolve. As said above, I don't mind if it is temporary, more important to me is to get some usable autofocus soon. > > The basic idea is that a single unit-less "sharpness" value is provided, and > the lens is moved in such a way that the maximum sharpness value is achieved. Yes. > At the moment it implements manual and continuous mode. No auto focus window > support for the time being. It's desirable to be able to handle the typical situation of a relatively small object against large background (e.g. a face in a meeting) automatically. > And it is of course a prototype, a lot of fine tuning is probably > necessary to get the thresholds and all that right, Yes, I'm afraid. > and of course it uses a very trivial two phase scan.
This is another attempt to introduce autofocus to software ISP. It is not a complete implementation; autofocus stats are missing, to be addressed in another patch series, to keep the patch series relatively short. It is posted as RFC to gather initial comments on the approach. What this series implements: - Software ISP autofocus is based on the IPU3 implementation. Generic parts of the IPU3 autofocus algorithm are moved to libipa for the purpose. - Software ISP autofocus algorithm based on that. - Autofocus (lens) controls added to software ISP. - Rudimentary support for providing the autofocus stats value from stats (without actually implementing it). What should be added in future patch series: - Software ISP autofocus stats. While it may look easy, it’s not, there are several things to address: accuracy, efficiency, frequency, collisions with exposure adjustments, focus area, etc. There is some former work available in Hans’s tree: https://github.com/jwrdegoede/libcamera/tree/software-af-done - Manual focus support. This should be easy, maybe old Pavel’s patches could be reused in some way? Milan Zamazal (7): libcamera: software_isp: Let a single point decide about computing stats ipa: ipu3: Move non-specific parts of Af to libipa libcamera: software_isp: Add variance to stats ipa: softisp: Remove unused imports from ipa_context.h ipa: softisp: Add Af algorithm libcamera: software_isp: Add autofocus controls ipa: libipa: Use autofocus limits if available .../internal/software_isp/software_isp.h | 4 +- .../internal/software_isp/swisp_stats.h | 4 + .../internal/software_isp/swstats_cpu.h | 17 +- include/libcamera/ipa/softisp.mojom | 3 +- src/ipa/ipu3/algorithms/af.cpp | 219 +----------- src/ipa/ipu3/algorithms/af.h | 27 +- src/ipa/ipu3/ipa_context.cpp | 17 +- src/ipa/ipu3/ipa_context.h | 8 +- src/ipa/libipa/af.cpp | 315 ++++++++++++++++++ src/ipa/libipa/af.h | 72 ++++ src/ipa/libipa/meson.build | 2 + src/ipa/softisp/algorithms/af.cpp | 52 +++ src/ipa/softisp/algorithms/af.h | 39 +++ src/ipa/softisp/algorithms/meson.build | 1 + src/ipa/softisp/data/uncalibrated.yaml | 1 + src/ipa/softisp/ipa_context.h | 6 +- src/ipa/softisp/softisp.cpp | 7 +- src/libcamera/pipeline/simple/simple.cpp | 13 +- src/libcamera/software_isp/debayer_egl.cpp | 24 +- src/libcamera/software_isp/software_isp.cpp | 4 +- src/libcamera/software_isp/swstats_cpu.cpp | 19 +- 21 files changed, 560 insertions(+), 294 deletions(-) create mode 100644 src/ipa/libipa/af.cpp create mode 100644 src/ipa/libipa/af.h create mode 100644 src/ipa/softisp/algorithms/af.cpp create mode 100644 src/ipa/softisp/algorithms/af.h