[PATCH] kernel-doc: Handle returning pointers to pointers

[PATCH] kernel-doc: Handle returning pointers to pointers

[Matthew Wilcox]

Clearly nobody ever tried to build the documentation for the radix tree
before:

include/linux/radix-tree.h:400: warning: cannot understand function
prototype: 'void ** radix_tree_iter_init(struct radix_tree_iter *iter,
unsigned long start) '

Indeed, the regexes only handled a single '*', not one-or-more.  I have
tried to fix that, but now I have perl regexes all over my hands, and
I fear I shall never be clean again.

Signed-off-by: Matthew Wilcox <mawilcox@microsoft.com>
---
 scripts/kernel-doc | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 030fc633acd4..b193b0337d1b 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2533,21 +2533,21 @@ sub dump_function($$) {
         $noret = 1;
     } elsif ($prototype =~ m/^()([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
 	$prototype =~ m/^(\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
-	$prototype =~ m/^(\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
+	$prototype =~ m/^(\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+\s+\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
-	$prototype =~ m/^(\w+\s+\w+\s+\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
+	$prototype =~ m/^(\w+\s+\w+\s+\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\(]*)\)/ ||
 	$prototype =~ m/^()([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
 	$prototype =~ m/^(\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
-	$prototype =~ m/^(\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
+	$prototype =~ m/^(\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
-	$prototype =~ m/^(\w+\s+\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
+	$prototype =~ m/^(\w+\s+\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+\s+\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
-	$prototype =~ m/^(\w+\s+\w+\s+\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
+	$prototype =~ m/^(\w+\s+\w+\s+\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
 	$prototype =~ m/^(\w+\s+\w+\s+\w+\s+\w+)\s+([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
-	$prototype =~ m/^(\w+\s+\w+\s+\w+\s+\w+\s*\*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
-	$prototype =~ m/^(\w+\s+\w+\s*\*\s*\w+\s*\*\s*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/)  {
+	$prototype =~ m/^(\w+\s+\w+\s+\w+\s+\w+\s*\*+)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/ ||
+	$prototype =~ m/^(\w+\s+\w+\s*\*+\s*\w+\s*\*+\s*)\s*([a-zA-Z0-9_~:]+)\s*\(([^\{]*)\)/)  {
 	$return_type = $1;
 	$declaration_name = $2;
 	my $args = $3;
-- 
2.11.0.301.g722e3be85.dirty

Re: [PATCH] kernel-doc: Handle returning pointers to pointers

[Markus Heiser]

Am 23.01.2017 um 09:18 schrieb Matthew Wilcox <mawilcox@microsoft.com>:

> Clearly nobody ever tried to build the documentation for the radix tree
> before:
> 
> include/linux/radix-tree.h:400: warning: cannot understand function
> prototype: 'void ** radix_tree_iter_init(struct radix_tree_iter *iter,
> unsigned long start) '
> 
> Indeed, the regexes only handled a single '*', not one-or-more.

Hi Matthew !

short answer: Thanks a lot
    Acked-by: Markus Heiser <markus.heiser@darmarIT.de>

to be more verbose:
   what I have tested and what I recommend  ...

I maintain my own stack of "linuxdoc" with a python version
of the kernel-doc script (hosted on github). It uses the same
regexes as the perl version (using a python rewrite here has some
other benefits, one you will see below). I merged your patch:

  https://github.com/return42/linuxdoc/commit/6c1b85b

> I have tried to fix that, but now I have perl regexes all over my hands, and
> I fear I shall never be clean again.


Another part of my stack is the 'sphkerneldoc' repository
with reST, generated from whole kernel source tree. Here is
the diff of what your patch produce on the whole source tree:  

  https://github.com/return42/sphkerneldoc/commit/e64c19822

So I we see, everything is fine with your patch (no reason to
fear any regression).

And to complete the roundtrip, here is the HTML rendered radix-tree.h

  https://h2626237.stratoserver.net/kernel/linux_src_doc/include/linux/radix-tree_h.html
  
with the new 'radix_tree_iter_init', 'radix_tree_next_chunk' etc. in.

BTW: the kernel-doc python variant does some *linting*, here is what
it says:

$ kernel-lintdoc include/linux/radix-tree.h
...
/share/linux-docs-next/include/linux/radix-tree.h:218: :ERROR: odd construct, gathering documentation of a function outside of the main block?!?
/share/linux-docs-next/include/linux/radix-tree.h:219: :ERROR: Comment without header was found split-state --> 4
/share/linux-docs-next/include/linux/radix-tree.h:219: :WARN: Incorrect use of kernel-doc format:  * radix_tree_deref_slot        - dereference a slot
/share/linux-docs-next/include/linux/radix-tree.h:167: :WARN: no description found for parameter 'pslot'
/share/linux-docs-next/include/linux/radix-tree.h:167: :WARN: function name from comment differs:  Radix <--> radix_tree_deref_slot
/share/linux-docs-next/include/linux/radix-tree.h:167: :WARN: no description found for return-value of function 'radix_tree_deref_slot()'
/share/linux-docs-next/include/linux/radix-tree.h:237: :WARN: no description found for parameter 'treelock'
/share/linux-docs-next/include/linux/radix-tree.h:398: :WARN: no description found for return-value of function 'radix_tree_iter_retry()'
/share/linux-docs-next/include/linux/radix-tree.h:571: :WARN: total errors: 2 / total warnings: 6

Don't look to close at all these warnings and errors since they are
consecutive faults of the first, a missing 'DOC:' in the *overall*
header file's comment block.

Here is my patch for this. May you like to add this to your patch:  

diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
index 5dea8f6..33cbc1b 100644
--- a/include/linux/radix-tree.h
+++ b/include/linux/radix-tree.h
@@ -164,7 +164,7 @@ static inline unsigned int iter_shift(const struct radix_tree_iter *iter)
 }

 /**
- * Radix-tree synchronization
+ * DOC: Radix-tree synchronization
  *
  * The radix-tree API requires that users provide all synchronisation (with
  * specific exceptions, noted below).

with this small patch, we also get header's 'DOC: comment' as HTML.

Unfortunately the comment block of func 'radix_tree_iter' is on top of
the header file. To be perfect, move the 'DOC: comment' block on top of
the header file, so that it is the first comment block (vice versa, move
radix_tree_iter below DOC).
  
Again, thanks a lot.

  -- Markus --

PS: I hope there comes a day we merge the kernel-doc.py into kernel's
source tree, make linting and other stuff available to all developers
and get rid of the complicated steps I needed above. :-o

RE: [PATCH] kernel-doc: Handle returning pointers to pointers

[Matthew Wilcox]

From: Markus Heiser [mailto:markus.heiser@darmarit.de]
> Am 23.01.2017 um 09:18 schrieb Matthew Wilcox
> <mawilcox@microsoft.com>:
> Hi Matthew !
> 
> short answer: Thanks a lot
>     Acked-by: Markus Heiser <markus.heiser@darmarIT.de>

Excellent!

> to be more verbose:
>    what I have tested and what I recommend  ...
> 
> I maintain my own stack of "linuxdoc" with a python version
> of the kernel-doc script (hosted on github). It uses the same
> regexes as the perl version (using a python rewrite here has some
> other benefits, one you will see below). I merged your patch:

Are there plans to merge that?  It feels so odd to have a python script calling a perl script ...

> Another part of my stack is the 'sphkerneldoc' repository
> with reST, generated from whole kernel source tree. Here is
> the diff of what your patch produce on the whole source tree:

Oh, that's funny.  Other places had the same problem ... I guess nobody had tried to fix that warning yet :-)

> Here is my patch for this. May you like to add this to your patch:
> 
> diff --git a/include/linux/radix-tree.h b/include/linux/radix-tree.h
> index 5dea8f6..33cbc1b 100644
> --- a/include/linux/radix-tree.h
> +++ b/include/linux/radix-tree.h
> @@ -164,7 +164,7 @@ static inline unsigned int iter_shift(const struct
> radix_tree_iter *iter)
>  }
> 
>  /**
> - * Radix-tree synchronization
> + * DOC: Radix-tree synchronization
>   *
>   * The radix-tree API requires that users provide all synchronisation (with
>   * specific exceptions, noted below).
> 
> with this small patch, we also get header's 'DOC: comment' as HTML.

I have that and much, much more in my tree right now ... I wrote a radix-tree.rst on the plane, and I've been tidying up all kinds of problems that sphinx noticed, or just reading the documentation made glaringly obvious.

Re: [PATCH] kernel-doc: Handle returning pointers to pointers

[Jonathan Corbet]

On Mon, 23 Jan 2017 15:14:51 +0000
Matthew Wilcox <mawilcox@microsoft.com> wrote:

> > I maintain my own stack of "linuxdoc" with a python version
> > of the kernel-doc script (hosted on github). It uses the same
> > regexes as the perl version (using a python rewrite here has some
> > other benefits, one you will see below). I merged your patch:  
> 
> Are there plans to merge that?  It feels so odd to have a python script calling a perl script ...

I pushed back pretty hard on it last year; my feeling at the time was that
the Sphinx transition had enough moving parts as it was.  I do think we
can reconsider it now, though.  It's not as if the perl kerneldoc script
is a thing of great beauty in need of preservation.

Markus, would you consider sending out a new patch set for review?  What I
would like to do see is something adding the new script for the Sphinx
toolchain, while leaving the DocBook build unchanged, using the old
script.  We could then delete it once the last template file has moved
over. 

4.12 would be the probable merge target; it's a big enough change that I'd
like to have it in -next for a full development cycle.

Thanks,

jon

Re: [PATCH] kernel-doc: Handle returning pointers to pointers

[Markus Heiser]

Am 23.01.2017 um 16:24 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Mon, 23 Jan 2017 15:14:51 +0000
> Matthew Wilcox <mawilcox@microsoft.com> wrote:
> 
>>> I maintain my own stack of "linuxdoc" with a python version
>>> of the kernel-doc script (hosted on github). It uses the same
>>> regexes as the perl version (using a python rewrite here has some
>>> other benefits, one you will see below). I merged your patch:  
>> 
>> Are there plans to merge that?  It feels so odd to have a python script calling a perl script ...
> 
> I pushed back pretty hard on it last year; my feeling at the time was that
> the Sphinx transition had enough moving parts as it was.

I know what you mean ;)

>  I do think we
> can reconsider it now, though.  It's not as if the perl kerneldoc script
> is a thing of great beauty in need of preservation.
> 
> Markus, would you consider sending out a new patch set for review?  

Yes, I send RFC soon ...

> What I
> would like to do see is something adding the new script for the Sphinx
> toolchain, while leaving the DocBook build unchanged, using the old
> script.  We could then delete it once the last template file has moved
> over.

agree ...

> 4.12 would be the probable merge target; it's a big enough change that I'd
> like to have it in -next for a full development cycle.

agree once more.

Thanks for choosing this!

  -- Markus --

RE: [PATCH] kernel-doc: Handle returning pointers to pointers

[Matthew Wilcox]

From: Markus Heiser [mailto:markus.heiser@darmarit.de]
> Am 23.01.2017 um 16:24 schrieb Jonathan Corbet <corbet@lwn.net>:
> > Markus, would you consider sending out a new patch set for review?
> 
> Yes, I send RFC soon ...

Could I ask for some features?  I'd've been trying to add them to the perl script, but since I am terrible at writing both Perl and Python, and I have your attention right now ...

1. An option to select which functions are output by regular expression.  I would like to be able to say:

.. kernel-doc:: lib/radix-tree.c
   :functions: radix.*

to avoid the IDA/IDR functions which now live in radix-tree.c from having their kernel-doc output.

2. An option to output everything *except* the /** DOC: */ comments.  I want to be able to do something like this in my .rst file:

--- 8< ---

Locking
-------

.. kernel-doc:: include/linux/radix-tree.h
   :doc: Radix-tree synchronization

The Public API
==============

The public API can be found in ``<linux/radix-tree.h>``.  To use a
radix tree in your data structure, embed a :c:type:`struct radix_tree_root`
in it, and initialise it using ``INIT_RADIX_TREE``.  You can also use
a file-local or global radix tree by defining a :c:type:`RADIX_TREE` as you
would a :c:type:`LIST_HEAD`.

.. kernel-doc:: include/linux/radix-tree.h
   :nodoc:

--- >8 ---

3. I think it would also make sense, as well as being able to ask for 'all exported symbols', to be able to ask for 'all non-static symbols'; we want to document __radix_tree_lookup(), but we only want to export it as a symbol if a modular user shows up (this isn't critical for me since I want to filter out the idr/ida functions as well, but it might be useful for other files).

Re: [PATCH] kernel-doc: Handle returning pointers to pointers

[Markus Heiser]

Am 24.01.2017 um 16:35 schrieb Matthew Wilcox <mawilcox@microsoft.com>:

> From: Markus Heiser [mailto:markus.heiser@darmarit.de]
>> Am 23.01.2017 um 16:24 schrieb Jonathan Corbet <corbet@lwn.net>:
>>> Markus, would you consider sending out a new patch set for review?
>> 
>> Yes, I send RFC soon ...
> 
> Could I ask for some features?

Yes, if you help me ;) ... I send a RFC [1] and placed you in the 'to:'
list. Could you view/test the RFC series? After, lets discuss additions
in the RFC ML thread / thanks!

-- Markus --

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09016.html

Re: [PATCH] kernel-doc: Handle returning pointers to pointers

[Jonathan Corbet]

On Mon, 23 Jan 2017 00:18:10 -0800
Matthew Wilcox <mawilcox@microsoft.com> wrote:

> Clearly nobody ever tried to build the documentation for the radix tree
> before:
> 
> include/linux/radix-tree.h:400: warning: cannot understand function
> prototype: 'void ** radix_tree_iter_init(struct radix_tree_iter *iter,
> unsigned long start) '
> 
> Indeed, the regexes only handled a single '*', not one-or-more.  I have
> tried to fix that, but now I have perl regexes all over my hands, and
> I fear I shall never be clean again.

Applied, thanks.

About your hands, try Dr Bronner's soap; at least, that's what I use for
bicycle grease.  What might be harder to shake is your new reputation as
somebody who knows how to fix the kernel-doc script...:)

jon