| Message ID | 20220421072848.6752-1-hpa@redhat.com |
|---|---|
| Headers | show |
| Series |
|
| Related | show |
Hi Kate Thanks for doing all this work on autofocus! There are still some questions I'm struggling with so I was wondering if I might ask you whether you had any opinions. On Thu, 21 Apr 2022 at 08:29, Kate Hsuan via libcamera-devel <libcamera-devel@lists.libcamera.org> wrote: > > Some of the algorithm, such as autofocus needs the information from the > device to configure the range of the search or the maximum values of the > configurations. Currently, this information, such as VCM attributes was > hardcoded in the code so it can only be used for specific hardware. If the > device attributes can be got through the codes, the algorithm can be > dynamically configured based on the device attributes got from the kernel. Yes, retrieving values from the lens driver is clearly much better than hard-coding them! One problem I have is that the appropriate range for the lens is determined not only by the lens driver but by the physical characteristics of the module. For example, I had a module where the driver advertises a range of 0 to 1023, but the module's physical characteristics meant that anything outside about 500 to 800 was not useful (either you couldn't move the lens that far, or it was beyond infinity etc.). Also things like the hyperfocal position would depend on the module too. I'm not clear how we find the correct lens information. I suppose one might consider manipulating the device tree to change those numbers, but I can't escape the feeling that we need to supply some kind of configuration information (even just the module name) when libcamera starts. I imagine that this information would allow us to load the correct "lens helper" that would give us all this information. It would also convert between "application lens positions" and VCM driver units (applications are not supposed to know about the driver units, AIUI). Would this sort of scheme work for you? Does anyone else have any thoughts on this? Thanks again! David > > This work proposed an implementation that can be used to get the device > attribute from the kernel and allows the algorithm has a method to get the > device attribute without a specific call. Therefore, for example, the AF > algorithm can get the necessary maximum VCM steps value to configure itself. > > During the AF algorithm configuration, the prefetched VCM steps will be > set to the configInfo and based on this the corresponding variable will > also be initiated. > > Kate Hsuan (4): > ipa: ipu3: Variables to hold the maximum VCM steps > libcamera: camera_lens: Get maximum VCM steps > libcamera: pipeline: ipu3: Get and set VCM information > ipa: ipu3: af: Remove hardcoded maximum VCM steps > > include/libcamera/internal/camera_lens.h | 1 + > include/libcamera/ipa/core.mojom | 7 +++++++ > src/ipa/ipu3/algorithms/af.cpp | 13 ++++--------- > src/ipa/ipu3/algorithms/af.h | 2 ++ > src/libcamera/camera_lens.cpp | 13 +++++++++++++ > src/libcamera/pipeline/ipu3/ipu3.cpp | 15 +++++++++++++++ > 6 files changed, 42 insertions(+), 9 deletions(-) > > -- > 2.35.1 >
Hi David, On Thu, Apr 21, 2022 at 4:50 PM David Plowman <david.plowman@raspberrypi.com> wrote: > > Hi Kate > > Thanks for doing all this work on autofocus! There are still some > questions I'm struggling with so I was wondering if I might ask you > whether you had any opinions. > > On Thu, 21 Apr 2022 at 08:29, Kate Hsuan via libcamera-devel > <libcamera-devel@lists.libcamera.org> wrote: > > > > Some of the algorithm, such as autofocus needs the information from the > > device to configure the range of the search or the maximum values of the > > configurations. Currently, this information, such as VCM attributes was > > hardcoded in the code so it can only be used for specific hardware. If the > > device attributes can be got through the codes, the algorithm can be > > dynamically configured based on the device attributes got from the kernel. > > Yes, retrieving values from the lens driver is clearly much better > than hard-coding them! > > One problem I have is that the appropriate range for the lens is > determined not only by the lens driver but by the physical > characteristics of the module. For example, I had a module where the > driver advertises a range of 0 to 1023, but the module's physical > characteristics meant that anything outside about 500 to 800 was not > useful (either you couldn't move the lens that far, or it was beyond > infinity etc.). Also things like the hyperfocal position would depend > on the module too. You are right, the current AF range is determined by the VCM spec itself but not by a datasheet of the whole image sensor module (sensor + VCM). > > I'm not clear how we find the correct lens information. I suppose one > might consider manipulating the device tree to change those numbers, > but I can't escape the feeling that we need to supply some kind of > configuration information (even just the module name) when libcamera > starts. I imagine that this information would allow us to load the > correct "lens helper" that would give us all this information. It > would also convert between "application lens positions" and VCM driver > units (applications are not supposed to know about the driver units, > AIUI). I think the "helper" could be a spec provider which provides a detailed spec of the sensor+VCM module. Based on the information, the lens information could be set to the AF algorithm. Therefore, AF algorithm could move the lens from the minimum to the maximum step. For example, the lens was moved from step 500 to step 800. It is easier than translating logical lens position to physical position. Thank you :) > > Would this sort of scheme work for you? Does anyone else have any > thoughts on this? > > Thanks again! > > David > > > > > This work proposed an implementation that can be used to get the device > > attribute from the kernel and allows the algorithm has a method to get the > > device attribute without a specific call. Therefore, for example, the AF > > algorithm can get the necessary maximum VCM steps value to configure itself. > > > > During the AF algorithm configuration, the prefetched VCM steps will be > > set to the configInfo and based on this the corresponding variable will > > also be initiated. > > > > Kate Hsuan (4): > > ipa: ipu3: Variables to hold the maximum VCM steps > > libcamera: camera_lens: Get maximum VCM steps > > libcamera: pipeline: ipu3: Get and set VCM information > > ipa: ipu3: af: Remove hardcoded maximum VCM steps > > > > include/libcamera/internal/camera_lens.h | 1 + > > include/libcamera/ipa/core.mojom | 7 +++++++ > > src/ipa/ipu3/algorithms/af.cpp | 13 ++++--------- > > src/ipa/ipu3/algorithms/af.h | 2 ++ > > src/libcamera/camera_lens.cpp | 13 +++++++++++++ > > src/libcamera/pipeline/ipu3/ipu3.cpp | 15 +++++++++++++++ > > 6 files changed, 42 insertions(+), 9 deletions(-) > > > > -- > > 2.35.1 > > > -- BR, Kate
Hi Kate, David, On 21/04/2022 13:02, Kate Hsuan via libcamera-devel wrote: > Hi David, > > On Thu, Apr 21, 2022 at 4:50 PM David Plowman > <david.plowman@raspberrypi.com> wrote: >> >> Hi Kate >> >> Thanks for doing all this work on autofocus! There are still some >> questions I'm struggling with so I was wondering if I might ask you >> whether you had any opinions. >> >> On Thu, 21 Apr 2022 at 08:29, Kate Hsuan via libcamera-devel >> <libcamera-devel@lists.libcamera.org> wrote: >>> >>> Some of the algorithm, such as autofocus needs the information from the >>> device to configure the range of the search or the maximum values of the >>> configurations. Currently, this information, such as VCM attributes was >>> hardcoded in the code so it can only be used for specific hardware. If the >>> device attributes can be got through the codes, the algorithm can be >>> dynamically configured based on the device attributes got from the kernel. >> >> Yes, retrieving values from the lens driver is clearly much better >> than hard-coding them! >> >> One problem I have is that the appropriate range for the lens is >> determined not only by the lens driver but by the physical >> characteristics of the module. For example, I had a module where the >> driver advertises a range of 0 to 1023, but the module's physical >> characteristics meant that anything outside about 500 to 800 was not >> useful (either you couldn't move the lens that far, or it was beyond >> infinity etc.). Also things like the hyperfocal position would depend >> on the module too. > > You are right, the current AF range is determined by the VCM spec > itself but not by a datasheet of the whole image sensor module (sensor > + VCM). > I think, David, I have the same module ;-). I have conducted a few tests here, and used a tape meter to roughly determine the focal distance corresponding to a given lens position. The capture for lens position of 450 [1] and the corresponding chart [2] are interesting. We should not forget, though, that depending on the steps between the positions, the lens may not be exactly at the same position. The tests here show that it is not varying a lot for the lens I have, but I suppose it is not a general rule ;-). [1]: https://pasteboard.co/6dQ94QYw3HtG.png [2]: https://pasteboard.co/0MGqr0wOJ7lr.png This is a *very* rough estimation in not that good conditions, but works pretty well. We could determine the mathematical curve from this, and use it in a lens helper. I have a few patches with a basic lens helper for RPi, but it is, again, highly experimental and needs to be improved. I will try to send it as RFC in a short notice and continue the discussion ? David, do you know a tuning environment (tool, box, charts, whatever) which would be able to determine the lens characteristics ? >> >> I'm not clear how we find the correct lens information. I suppose one >> might consider manipulating the device tree to change those numbers, >> but I can't escape the feeling that we need to supply some kind of >> configuration information (even just the module name) when libcamera >> starts. I imagine that this information would allow us to load the >> correct "lens helper" that would give us all this information. It >> would also convert between "application lens positions" and VCM driver >> units (applications are not supposed to know about the driver units, >> AIUI). > > I think the "helper" could be a spec provider which provides a > detailed spec of the sensor+VCM module. Based on the information, the > lens information could be set to the AF algorithm. Therefore, AF > algorithm could move the lens from the minimum to the maximum step. > For example, the lens was moved from step 500 to step 800. It is > easier than translating logical lens position to physical position. > > Thank you :) > >> >> Would this sort of scheme work for you? Does anyone else have any >> thoughts on this? >> >> Thanks again! >> >> David >> >>> >>> This work proposed an implementation that can be used to get the device >>> attribute from the kernel and allows the algorithm has a method to get the >>> device attribute without a specific call. Therefore, for example, the AF >>> algorithm can get the necessary maximum VCM steps value to configure itself. >>> >>> During the AF algorithm configuration, the prefetched VCM steps will be >>> set to the configInfo and based on this the corresponding variable will >>> also be initiated. >>> >>> Kate Hsuan (4): >>> ipa: ipu3: Variables to hold the maximum VCM steps >>> libcamera: camera_lens: Get maximum VCM steps >>> libcamera: pipeline: ipu3: Get and set VCM information >>> ipa: ipu3: af: Remove hardcoded maximum VCM steps >>> >>> include/libcamera/internal/camera_lens.h | 1 + >>> include/libcamera/ipa/core.mojom | 7 +++++++ >>> src/ipa/ipu3/algorithms/af.cpp | 13 ++++--------- >>> src/ipa/ipu3/algorithms/af.h | 2 ++ >>> src/libcamera/camera_lens.cpp | 13 +++++++++++++ >>> src/libcamera/pipeline/ipu3/ipu3.cpp | 15 +++++++++++++++ >>> 6 files changed, 42 insertions(+), 9 deletions(-) >>> >>> -- >>> 2.35.1 >>> >> > > > -- > BR, > Kate >