[libcamera-devel,RFC,1/6] utils: ipc: Add script to extract doxygen docs from mojom files

12370 diff mbox series

Add a script to extract doxygen documentation comments from mojom files.
It matches based on ^\/\*\*$ for start of block and ^ \*\/$ for end of
block, and simply copies the comments to the output file along with a
header and the libcamera namespace.

Also add it to the meson file so it is usable by other meson files.

Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
---
 utils/ipc/extract-docs.py | 77 +++++++++++++++++++++++++++++++++++++++
 utils/ipc/meson.build     |  2 +
 2 files changed, 79 insertions(+)
 create mode 100755 utils/ipc/extract-docs.py

Hi Paul,

Thanks for this work. The script, essentially the meat of the series, 
looks great!

On 5/24/21 2:10 PM, Paul Elder wrote:
> Add a script to extract doxygen documentation comments from mojom files.
> It matches based on ^\/\*\*$ for start of block and ^ \*\/$ for end of
> block, and simply copies the comments to the output file along with a
> header and the libcamera namespace.
>
> Also add it to the meson file so it is usable by other meson files.
>
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Acked-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
>   utils/ipc/extract-docs.py | 77 +++++++++++++++++++++++++++++++++++++++
>   utils/ipc/meson.build     |  2 +
>   2 files changed, 79 insertions(+)
>   create mode 100755 utils/ipc/extract-docs.py
>
> diff --git a/utils/ipc/extract-docs.py b/utils/ipc/extract-docs.py
> new file mode 100755
> index 00000000..b6667057
> --- /dev/null
> +++ b/utils/ipc/extract-docs.py
> @@ -0,0 +1,77 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (C) 2021, Google Inc.
> +#
> +# Author: Paul Elder <paul.elder@ideasonboard.com>
> +#
> +# extract-docs.py - Extract doxygen documentation from mojom files
> +
> +import argparse
> +import re
> +import sys
> +
> +regex_block_start = re.compile('^\/\*\*$')
> +regex_block_end = re.compile('^ \*\/$')
> +
> +
> +def main(argv):
> +
> +    # Parse command line arguments
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument('-o', dest='output', metavar='file', type=str,
> +                        help='Output file name. Defaults to standard output if not specified.')
> +    parser.add_argument('input', type=str,
> +                        help='Input file name.')
> +    args = parser.parse_args(argv[1:])
> +
> +    lines = open(args.input, 'r').readlines()
> +    pipeline = args.input.split('/')[-1].replace('.mojom', '')
> +    data = f'''\
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Google Inc.
> + *
> + * {pipeline}_ipa_interface.cpp - Docs file for generated {pipeline}.mojom
> + *
> + * This file is auto-generated. Do not edit.
> + */
> +
> +namespace libcamera {{
> +
> +'''
> +
> +    in_block = False
> +    comment = ''
> +    for lineno, line in enumerate(lines, start=1):
> +        if regex_block_start.match(line):
> +            if in_block:
> +                raise SyntaxError('Expected end of comment',
> +                                  (args.input, lineno, 1, line))
> +            in_block = True
> +            comment = line
> +            continue
> +
> +        if regex_block_end.match(line):
> +            if in_block:
> +                comment += line
> +                data += comment + '\n'
> +            in_block = False
> +            continue
> +
> +        if in_block:
> +            comment += line
> +

The only bit I was wondering, if we needed a '\n' after every block is 
extracted. But I guess, it makes /no/ difference to the doxygen if a \n 
is present between blocks.
> +    data += '} /* namespace libcamera */\n'
> +
> +    if args.output:
> +        output = open(args.output, 'wb')
> +        output.write(data.encode('utf-8'))
> +        output.close()
> +    else:
> +        sys.stdout.write(data)
> +
> +    return 0
> +
> +
> +if __name__ == '__main__':
> +    sys.exit(main(sys.argv))
> diff --git a/utils/ipc/meson.build b/utils/ipc/meson.build
> index 4a41b9c1..973a5417 100644
> --- a/utils/ipc/meson.build
> +++ b/utils/ipc/meson.build
> @@ -8,6 +8,8 @@ mojom_parser = find_program('./parser.py')
>   
>   mojom_generator = find_program('./generate.py')
>   
> +mojom_docs_extractor = find_program('./extract-docs.py')
> +
>   mojom_templates = custom_target('mojom_templates',
>                                   input : mojom_template_files,
>                                   output : 'libcamera_templates.zip',

Hi Umang,

On Mon, May 24, 2021 at 03:12:06PM +0530, Umang Jain wrote:
> Hi Paul,
> 
> Thanks for this work. The script, essentially the meat of the series, looks
> great!

\o/ thanks!

> 
> On 5/24/21 2:10 PM, Paul Elder wrote:
> > Add a script to extract doxygen documentation comments from mojom files.
> > It matches based on ^\/\*\*$ for start of block and ^ \*\/$ for end of
> > block, and simply copies the comments to the output file along with a
> > header and the libcamera namespace.
> > 
> > Also add it to the meson file so it is usable by other meson files.
> > 
> > Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> Acked-by: Umang Jain <umang.jain@ideasonboard.com>
> > ---
> >   utils/ipc/extract-docs.py | 77 +++++++++++++++++++++++++++++++++++++++
> >   utils/ipc/meson.build     |  2 +
> >   2 files changed, 79 insertions(+)
> >   create mode 100755 utils/ipc/extract-docs.py
> > 
> > diff --git a/utils/ipc/extract-docs.py b/utils/ipc/extract-docs.py
> > new file mode 100755
> > index 00000000..b6667057
> > --- /dev/null
> > +++ b/utils/ipc/extract-docs.py
> > @@ -0,0 +1,77 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +# Copyright (C) 2021, Google Inc.
> > +#
> > +# Author: Paul Elder <paul.elder@ideasonboard.com>
> > +#
> > +# extract-docs.py - Extract doxygen documentation from mojom files
> > +
> > +import argparse
> > +import re
> > +import sys
> > +
> > +regex_block_start = re.compile('^\/\*\*$')
> > +regex_block_end = re.compile('^ \*\/$')
> > +
> > +
> > +def main(argv):
> > +
> > +    # Parse command line arguments
> > +    parser = argparse.ArgumentParser()
> > +    parser.add_argument('-o', dest='output', metavar='file', type=str,
> > +                        help='Output file name. Defaults to standard output if not specified.')
> > +    parser.add_argument('input', type=str,
> > +                        help='Input file name.')
> > +    args = parser.parse_args(argv[1:])
> > +
> > +    lines = open(args.input, 'r').readlines()
> > +    pipeline = args.input.split('/')[-1].replace('.mojom', '')
> > +    data = f'''\
> > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > +/*
> > + * Copyright (C) 2021, Google Inc.
> > + *
> > + * {pipeline}_ipa_interface.cpp - Docs file for generated {pipeline}.mojom
> > + *
> > + * This file is auto-generated. Do not edit.
> > + */
> > +
> > +namespace libcamera {{
> > +
> > +'''
> > +
> > +    in_block = False
> > +    comment = ''
> > +    for lineno, line in enumerate(lines, start=1):
> > +        if regex_block_start.match(line):
> > +            if in_block:
> > +                raise SyntaxError('Expected end of comment',
> > +                                  (args.input, lineno, 1, line))
> > +            in_block = True
> > +            comment = line
> > +            continue
> > +
> > +        if regex_block_end.match(line):
> > +            if in_block:
> > +                comment += line
> > +                data += comment + '\n'
> > +            in_block = False
> > +            continue
> > +
> > +        if in_block:
> > +            comment += line
> > +
> 
> The only bit I was wondering, if we needed a '\n' after every block is
> extracted. But I guess, it makes /no/ difference to the doxygen if a \n is
> present between blocks.

It's not really necessary, but I like generated code to be at least
readable, and I think a blank line in between blocks helps with that.

I have a bunch of those in the generated code for IPC too.


Paul

> > +    data += '} /* namespace libcamera */\n'
> > +
> > +    if args.output:
> > +        output = open(args.output, 'wb')
> > +        output.write(data.encode('utf-8'))
> > +        output.close()
> > +    else:
> > +        sys.stdout.write(data)
> > +
> > +    return 0
> > +
> > +
> > +if __name__ == '__main__':
> > +    sys.exit(main(sys.argv))
> > diff --git a/utils/ipc/meson.build b/utils/ipc/meson.build
> > index 4a41b9c1..973a5417 100644
> > --- a/utils/ipc/meson.build
> > +++ b/utils/ipc/meson.build
> > @@ -8,6 +8,8 @@ mojom_parser = find_program('./parser.py')
> >   mojom_generator = find_program('./generate.py')
> > +mojom_docs_extractor = find_program('./extract-docs.py')
> > +
> >   mojom_templates = custom_target('mojom_templates',
> >                                   input : mojom_template_files,
> >                                   output : 'libcamera_templates.zip',
>

Hi Paul,

Thank you for the patch.

On Mon, May 24, 2021 at 05:40:24PM +0900, Paul Elder wrote:
> Add a script to extract doxygen documentation comments from mojom files.
> It matches based on ^\/\*\*$ for start of block and ^ \*\/$ for end of
> block, and simply copies the comments to the output file along with a
> header and the libcamera namespace.
> 
> Also add it to the meson file so it is usable by other meson files.
> 
> Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
> ---
>  utils/ipc/extract-docs.py | 77 +++++++++++++++++++++++++++++++++++++++
>  utils/ipc/meson.build     |  2 +
>  2 files changed, 79 insertions(+)
>  create mode 100755 utils/ipc/extract-docs.py
> 
> diff --git a/utils/ipc/extract-docs.py b/utils/ipc/extract-docs.py
> new file mode 100755
> index 00000000..b6667057
> --- /dev/null
> +++ b/utils/ipc/extract-docs.py
> @@ -0,0 +1,77 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (C) 2021, Google Inc.
> +#
> +# Author: Paul Elder <paul.elder@ideasonboard.com>
> +#
> +# extract-docs.py - Extract doxygen documentation from mojom files
> +
> +import argparse
> +import re
> +import sys
> +
> +regex_block_start = re.compile('^\/\*\*$')
> +regex_block_end = re.compile('^ \*\/$')

This may be a bit fragile, but it should do for now.

> +
> +
> +def main(argv):
> +
> +    # Parse command line arguments
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument('-o', dest='output', metavar='file', type=str,
> +                        help='Output file name. Defaults to standard output if not specified.')

You could use argparse's native support for files:

    parser.add_argument('-o', '--output', metavar='file',
                        type=argparse.FileType('w', encoding='utf-8'),
                        default=sys.stdout,
                        help='Output file name (default: standard output)')

> +    parser.add_argument('input', type=str,
> +                        help='Input file name.')
> +    args = parser.parse_args(argv[1:])
> +
> +    lines = open(args.input, 'r').readlines()
> +    pipeline = args.input.split('/')[-1].replace('.mojom', '')
> +    data = f'''\
> +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> +/*
> + * Copyright (C) 2021, Google Inc.
> + *
> + * {pipeline}_ipa_interface.cpp - Docs file for generated {pipeline}.mojom
> + *
> + * This file is auto-generated. Do not edit.
> + */
> +
> +namespace libcamera {{
> +
> +'''
> +
> +    in_block = False
> +    comment = ''
> +    for lineno, line in enumerate(lines, start=1):
> +        if regex_block_start.match(line):
> +            if in_block:
> +                raise SyntaxError('Expected end of comment',
> +                                  (args.input, lineno, 1, line))
> +            in_block = True
> +            comment = line
> +            continue
> +
> +        if regex_block_end.match(line):
> +            if in_block:
> +                comment += line
> +                data += comment + '\n'
> +            in_block = False
> +            continue
> +
> +        if in_block:
> +            comment += line
> +
> +    data += '} /* namespace libcamera */\n'
> +
> +    if args.output:
> +        output = open(args.output, 'wb')
> +        output.write(data.encode('utf-8'))
> +        output.close()
> +    else:
> +        sys.stdout.write(data)

And all this would become

    args.output.write(data)

You could also call args.output.write() to write the header, the
comments and the footer seperately, instead of accumulating comments in
the data variable. Up to you.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +
> +    return 0
> +
> +
> +if __name__ == '__main__':
> +    sys.exit(main(sys.argv))
> diff --git a/utils/ipc/meson.build b/utils/ipc/meson.build
> index 4a41b9c1..973a5417 100644
> --- a/utils/ipc/meson.build
> +++ b/utils/ipc/meson.build
> @@ -8,6 +8,8 @@ mojom_parser = find_program('./parser.py')
>  
>  mojom_generator = find_program('./generate.py')
>  
> +mojom_docs_extractor = find_program('./extract-docs.py')
> +
>  mojom_templates = custom_target('mojom_templates',
>                                  input : mojom_template_files,
>                                  output : 'libcamera_templates.zip',