[PATCH 0/2] docs: A couple of automarkup improvements

[PATCH 0/2] docs: A couple of automarkup improvements

[Jonathan Corbet]

Building the docs takes far too long.  Much of the blame for that lies with
Sphinx, but some of it is under our own control.  These two patches
eliminate a bunch of useless work in the automarkup extension.

Benchmarking Sphinx runs is hard, as the run time can be rather variable.
I have consistently found about a 15% speedup from these patches, though,
with no output changes.

Testing was done with Sphinx 4.4.0 and 5.0.2.  My systems all have Python
3.10 installed, and there does not appear, alas, to be a straightforward
way to get Sphinx 3.x to run on that version.  Versions of Sphinx prior to
3.0 will not be affected by these changes.

Jonathan Corbet (2):
  docs: automarkup: track failed cross-reference attempts
  docs: automarkup: do not look up symbols twice

 Documentation/sphinx/automarkup.py | 56 +++++++++++++++++-------------
 1 file changed, 32 insertions(+), 24 deletions(-)

-- 
2.36.1

[PATCH 1/2] docs: automarkup: track failed cross-reference attempts

[Jonathan Corbet]

The automarkup code tries to create a lot of cross-references that don't
exist.  Cross-reference lookups are expensive, especially in later versions
of Sphinx, so there is value in avoiding unnecessary ones.  Remember
attempts that failed and do not retry them.

This improves the htmldocs build time by 5-10% depending on the phase of
the moon and other factors.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/sphinx/automarkup.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index cc348b219fca..5b1f83e6192f 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -120,6 +120,16 @@ def markup_refs(docname, app, node):
         repl.append(nodes.Text(t[done:]))
     return repl
 
+#
+# Keep track of cross-reference lookups that failed so we don't have to
+# do them again.
+#
+failed_lookups = { }
+def failure_seen(target, reftype):
+    return (target + '::' + reftype) in failed_lookups
+def note_failure(target, reftype):
+    failed_lookups[target + '::' + reftype] = True
+
 #
 # In sphinx3 we can cross-reference to C macro and function, each one with its
 # own C role, but both match the same regex, so we try both.
@@ -145,6 +155,8 @@ def markup_func_ref_sphinx3(docname, app, match):
         for target in possible_targets:
             if target not in Skipfuncs:
                 for class_s, reftype_s in zip(class_str, reftype_str):
+                    if failure_seen(target, reftype_s):
+                        continue
                     lit_text = nodes.literal(classes=['xref', 'c', class_s])
                     lit_text += target_text
                     pxref = addnodes.pending_xref('', refdomain = 'c',
@@ -164,6 +176,7 @@ def markup_func_ref_sphinx3(docname, app, match):
 
                     if xref:
                         return xref
+                    note_failure(target, reftype_s)
 
     return target_text
 
-- 
2.36.1

[PATCH 2/2] docs: automarkup: do not look up symbols twice

[Jonathan Corbet]

The automarkup code tries to look up symbols once as a function, and once
as a macro.  The Sphinx C domain code, though, totally ignores that
distinction and will return the same results either way.  So just look
things up once and be done with it; the resulting output does not change,
but htmldocs build time drops by about 5%.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/sphinx/automarkup.py | 57 ++++++++++++++----------------
 1 file changed, 26 insertions(+), 31 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 5b1f83e6192f..06b34740bf90 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -125,19 +125,16 @@ def markup_refs(docname, app, node):
 # do them again.
 #
 failed_lookups = { }
-def failure_seen(target, reftype):
-    return (target + '::' + reftype) in failed_lookups
-def note_failure(target, reftype):
-    failed_lookups[target + '::' + reftype] = True
+def failure_seen(target):
+    return (target) in failed_lookups
+def note_failure(target):
+    failed_lookups[target] = True
 
 #
 # In sphinx3 we can cross-reference to C macro and function, each one with its
 # own C role, but both match the same regex, so we try both.
 #
 def markup_func_ref_sphinx3(docname, app, match):
-    class_str = ['c-func', 'c-macro']
-    reftype_str = ['function', 'macro']
-
     cdom = app.env.domains['c']
     #
     # Go through the dance of getting an xref out of the C domain
@@ -153,30 +150,28 @@ def markup_func_ref_sphinx3(docname, app, match):
 
     if base_target not in Skipnames:
         for target in possible_targets:
-            if target not in Skipfuncs:
-                for class_s, reftype_s in zip(class_str, reftype_str):
-                    if failure_seen(target, reftype_s):
-                        continue
-                    lit_text = nodes.literal(classes=['xref', 'c', class_s])
-                    lit_text += target_text
-                    pxref = addnodes.pending_xref('', refdomain = 'c',
-                                                  reftype = reftype_s,
-                                                  reftarget = target, modname = None,
-                                                  classname = None)
-                    #
-                    # XXX The Latex builder will throw NoUri exceptions here,
-                    # work around that by ignoring them.
-                    #
-                    try:
-                        xref = cdom.resolve_xref(app.env, docname, app.builder,
-                                                 reftype_s, target, pxref,
-                                                 lit_text)
-                    except NoUri:
-                        xref = None
-
-                    if xref:
-                        return xref
-                    note_failure(target, reftype_s)
+            if (target not in Skipfuncs) and not failure_seen(target):
+                lit_text = nodes.literal(classes=['xref', 'c', 'c-func'])
+                lit_text += target_text
+                pxref = addnodes.pending_xref('', refdomain = 'c',
+                                              reftype = 'function',
+                                              reftarget = target,
+                                              modname = None,
+                                              classname = None)
+                #
+                # XXX The Latex builder will throw NoUri exceptions here,
+                # work around that by ignoring them.
+                #
+                try:
+                    xref = cdom.resolve_xref(app.env, docname, app.builder,
+                                             'function', target, pxref,
+                                             lit_text)
+                except NoUri:
+                    xref = None
+
+                if xref:
+                    return xref
+                note_failure(target)
 
     return target_text
 
-- 
2.36.1

Re: [PATCH 0/2] docs: A couple of automarkup improvements

[Akira Yokosawa]

On Thu, 30 Jun 2022 10:36:28 -0600, Jonathan Corbet wrote:
> Building the docs takes far too long.  Much of the blame for that lies with
> Sphinx, but some of it is under our own control.  These two patches
> eliminate a bunch of useless work in the automarkup extension.
> 
> Benchmarking Sphinx runs is hard, as the run time can be rather variable.
> I have consistently found about a 15% speedup from these patches, though,
> with no output changes.
> 
> Testing was done with Sphinx 4.4.0 and 5.0.2.  My systems all have Python
> 3.10 installed, and there does not appear, alas, to be a straightforward
> way to get Sphinx 3.x to run on that version.  Versions of Sphinx prior to
> 3.0 will not be affected by these changes.

They look nice improvements, indeed!

On my rather non-performant machine with Python 3.9.2 + Sphinx 5.0.2,
elapsed time of "make htmldocs" decreased about 12% without any difference
in generated HTML files.

For the series:

Tested-by: Akira Yokosawa <akiyks@gmail.com>

        Thanks, Akira
> 
> Jonathan Corbet (2):
>   docs: automarkup: track failed cross-reference attempts
>   docs: automarkup: do not look up symbols twice
> 
>  Documentation/sphinx/automarkup.py | 56 +++++++++++++++++-------------
>  1 file changed, 32 insertions(+), 24 deletions(-)
> 
> -- 
> 2.36.1