[libcamera-devel,RFC,0/2] andriod: Basic NV12 thumbnailer

10157 mbox series

Still RFC as couple of \todos still pending to get address; partly which
I need help with in order to know what should be the forward path from
here on.

One of the hurdle after writing the thumbnailer component was to use the
existing encoder to compress it. The encoder uses FrameBuffer parameter
to encode as of now. The result of thumbnailer is raw bytes, so either
we need to get a FrameBuffer/MappedFrameBuffer instance created out of
those raw bytes(which seems difficult) OR make the encoder flexible
enough so that we can encode directly from bytes. Patch 1/2 implements
the latter (discussed with Kieran last week).

Coming to patch 2/2, is the thumbnailer which is generated and
compressed in the PostProcessorJpeg layer. How would we embed this
in the Exif data, is still the missing piece of the puzzle. I am still
unsure where I should (thumnail + compress) in PostProcessor layer
or the Encoder layer. Both these places has it's own set of challenges
according to me. Please refer to the \todo and see if you can help me
with any ideas.

(The actual thumbnailer(.cpp / .h) has been reviewed before and I might
have not addressed all the comments from it. Although there a few
changes in there, I will make sure that the reviews are addressed when
I'll submit a non-RFC version).

Thanks!

Umang Jain (2):
  android: jpeg: encoder_libjpeg Allow encoding raw frame bytes
  android: jpeg: Add a basic NV12 image thumbnailer

 src/android/jpeg/encoder_libjpeg.cpp     |  37 +++++----
 src/android/jpeg/encoder_libjpeg.h       |   7 +-
 src/android/jpeg/post_processor_jpeg.cpp |  39 +++++++++
 src/android/jpeg/post_processor_jpeg.h   |   3 +
 src/android/jpeg/thumbnailer.cpp         | 100 +++++++++++++++++++++++
 src/android/jpeg/thumbnailer.h           |  40 +++++++++
 src/android/meson.build                  |   1 +
 7 files changed, 211 insertions(+), 16 deletions(-)
 create mode 100644 src/android/jpeg/thumbnailer.cpp
 create mode 100644 src/android/jpeg/thumbnailer.h

Hi Umang,

On 21/10/2020 09:08, Umang Jain wrote:
> Still RFC as couple of \todos still pending to get address; partly which
> I need help with in order to know what should be the forward path from
> here on.
> 
> One of the hurdle after writing the thumbnailer component was to use the
> existing encoder to compress it. The encoder uses FrameBuffer parameter
> to encode as of now. The result of thumbnailer is raw bytes, so either
> we need to get a FrameBuffer/MappedFrameBuffer instance created out of
> those raw bytes(which seems difficult) OR make the encoder flexible
> enough so that we can encode directly from bytes. Patch 1/2 implements
> the latter (discussed with Kieran last week).

I like this ;-)

> Coming to patch 2/2, is the thumbnailer which is generated and
> compressed in the PostProcessorJpeg layer. How would we embed this
> in the Exif data, is still the missing piece of the puzzle. I am still
> unsure where I should (thumnail + compress) in PostProcessor layer
> or the Encoder layer. Both these places has it's own set of challenges

Certainly in the PostProcessorJpeg layer in my opinion.

Encoder is just about creating compressing a single image into a
compressed/encoded form. In this instance a JPEG.

The PostProcessorJpeg is responsible for doing all the work 'on top' of
that - i.e. handling the exif, and compression, and therefore it's also
respoonsible for creating any thumbnail required.

It sounds like the only missing piece is actually getting this now
encoded thumbnail *into* the exif.

Perhaps splitting the thumbnailer out to it's own patch would mean the
updates to the encoder interface, and the thumbnailer can retain their
tags now. Then we just need to solve inserting the newly encoded
thumbnail correctly into the exif structure with the current interfaces
we have.

That might possibly require extending the exif class of course, so
splitting the thumbnailer out might make that path clearer: i.e.:

- android: jpeg: encoder_libjpeg Allow encoding raw frame bytes
- android: jpeg: Implement a simple thumbnailer
- android: exif: Extend exif API to support addition of thumbnails
- android: jpeg: PostProcessorJpeg: Generate thumbnail and add to exif

> according to me. Please refer to the \todo and see if you can help me
> with any ideas.
> 
> (The actual thumbnailer(.cpp / .h) has been reviewed before and I might
> have not addressed all the comments from it. Although there a few
> changes in there, I will make sure that the reviews are addressed when
> I'll submit a non-RFC version).
> 
> Thanks!
> 
> Umang Jain (2):
>   android: jpeg: encoder_libjpeg Allow encoding raw frame bytes
>   android: jpeg: Add a basic NV12 image thumbnailer
> 
>  src/android/jpeg/encoder_libjpeg.cpp     |  37 +++++----
>  src/android/jpeg/encoder_libjpeg.h       |   7 +-
>  src/android/jpeg/post_processor_jpeg.cpp |  39 +++++++++
>  src/android/jpeg/post_processor_jpeg.h   |   3 +
>  src/android/jpeg/thumbnailer.cpp         | 100 +++++++++++++++++++++++
>  src/android/jpeg/thumbnailer.h           |  40 +++++++++
>  src/android/meson.build                  |   1 +
>  7 files changed, 211 insertions(+), 16 deletions(-)
>  create mode 100644 src/android/jpeg/thumbnailer.cpp
>  create mode 100644 src/android/jpeg/thumbnailer.h
>