From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2626C2DF151; Mon, 1 Sep 2025 15:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756740838; cv=none; b=a+NCkZdeLrgTdP+Cviy6aPzTubfrlVVWNTmWnThxBFqaBeZ90D5Pu7O7nW/Dn13CSJ12fexVdzSgD8ElDl38/NoGOMBNnp2kU0lLG1i4i8ojg51AUb7Vqmeorzx1py8bk8kxDSob03OCxVAiDihhohZb1TwDNlUX9j7faxZKnVo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756740838; c=relaxed/simple; bh=xoCXkgDpfSGLUQkbKhJPhF5dmK3td3hHAgXtwYxjVi8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=im+Hhtn+J9ZM+MhjnXS92d5g3+ed0eNf4bcNVyfCuRl+k/dP6S/7N958DwgLlATocOKxlvpAlzh9aFN7X67d2RKAk9U3nz84s+YjharZOtrTEDFv7+CINq8LOjSn36I0Fu3uL+sfTLsnD8e5OtWjhujjhh5pjg3YewqvamowAgU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NOPDHF1A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NOPDHF1A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 970D4C16AAE; Mon, 1 Sep 2025 15:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756740837; bh=xoCXkgDpfSGLUQkbKhJPhF5dmK3td3hHAgXtwYxjVi8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NOPDHF1AhMAJJMdiWKoym+L46gtQzH9rnxTDai88KfBiOwD7AU2epaxxjpoOX1B9Q ipPcAe3UEeZahZ1U9ZfYSTIP3Horh7Ym/N5QiC3ePLsg/LldG0rXS0Mn74SgilzZZf uHRonZk8Fwy2jqxzYIGlEEEs6G21npURqULe9qcZ2OcSyRfdfhsPbRGqDKWWLHTTRH 7RH8K0S/1pSbgEj2G5Dlhs/YBkRQwakEf9kvr3iRAQTYlcPS3KWt5WtoFChHDkt9xV dn6QKNmp+CrzIiMMSRduGWu0QWOIBpBMK815F1fSoExNZuFs51Y3O9CE/qUN60sm1U 8S5mKNS3IrfPw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1ut6Xb-00000003HMQ-3g3O; Mon, 01 Sep 2025 17:33:55 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , Mauro Carvalho Chehab , linux-kernel@vger.kernel.org Subject: [PATCH RESEND v3 13/15] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel Date: Mon, 1 Sep 2025 17:33:48 +0200 Message-ID: <20eff278c995cd39535d723f538cff9a368ff8e1.1756740314.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Content-Type: text/plain; charset="utf-8" Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Use POSIX jobserver when available or -j to run PDF=0D builds in parallel, restoring pdf build performance. Yet,=0D running it when debugging troubles is a bad idea, so, when=0D calling directly via command line, except if "-j" is splicitly=0D requested, it will serialize the build.=0D =0D With such change, a PDF doc builds now takes around 5 minutes=0D on a Ryzen 9 machine with 32 cpu threads:=0D =0D # Explicitly paralelize both Sphinx and LaTeX pdf builds=0D $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs -j 33=0D =0D real 5m17.901s=0D user 15m1.499s=0D sys 2m31.482s=0D =0D # Use POSIX jobserver to paralelize both sphinx-build and LaTeX=0D $ make cleandocs; time make pdfdocs=0D =0D real 5m22.369s=0D user 15m9.076s=0D sys 2m31.419s=0D =0D # Serializes PDF build, while keeping Sphinx parallelized.=0D # it is equivalent of passing -jauto via command line=0D $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs=0D =0D real 11m20.901s=0D user 13m2.910s=0D sys 1m44.553s=0D =0D Signed-off-by: Mauro Carvalho Chehab =0D ---=0D tools/docs/sphinx-build-wrapper | 158 +++++++++++++++++++++++---------=0D 1 file changed, 117 insertions(+), 41 deletions(-)=0D =0D diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per=0D index 580582a76c93..c884022ad733 100755=0D --- a/tools/docs/sphinx-build-wrapper=0D +++ b/tools/docs/sphinx-build-wrapper=0D @@ -52,6 +52,8 @@ import shutil=0D import subprocess=0D import sys=0D =0D +from concurrent import futures=0D +=0D from lib.python_version import PythonVersion=0D =0D LIB_DIR =3D "../../scripts/lib"=0D @@ -278,6 +280,82 @@ class SphinxBuilder:=0D except (OSError, IOError) as e:=0D print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr)= =0D =0D + def build_pdf_file(self, latex_cmd, from_dir, path):=0D + """Builds a single pdf file using latex_cmd"""=0D + try:=0D + subprocess.run(latex_cmd + [path],=0D + cwd=3Dfrom_dir, check=3DTrue)=0D +=0D + return True=0D + except subprocess.CalledProcessError:=0D + return False=0D +=0D + def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs)= :=0D + """Build PDF files in parallel if possible"""=0D + builds =3D {}=0D + build_failed =3D False=0D + max_len =3D 0=0D + has_tex =3D False=0D +=0D + #=0D + # LaTeX PDF error code is almost useless for us:=0D + # any warning makes it non-zero. For kernel doc builds it always r= eturn=0D + # non-zero even when build succeeds. So, let's do the best next th= ing:=0D + # Ignore build errors. At the end, check if all PDF files were bui= lt,=0D + # printing a summary with the built ones and returning 0 if all of= =0D + # them were actually built.=0D + #=0D + with futures.ThreadPoolExecutor(max_workers=3Dn_jobs) as executor:= =0D + jobs =3D {}=0D +=0D + for from_dir, pdf_dir, entry in tex_files:=0D + name =3D entry.name=0D +=0D + if not name.endswith(tex_suffix):=0D + continue=0D +=0D + name =3D name[:-len(tex_suffix)]=0D +=0D + max_len =3D max(max_len, len(name))=0D +=0D + has_tex =3D True=0D +=0D + future =3D executor.submit(self.build_pdf_file, latex_cmd,= =0D + from_dir, entry.path)=0D + jobs[future] =3D (from_dir, pdf_dir, name)=0D +=0D + for future in futures.as_completed(jobs):=0D + from_dir, pdf_dir, name =3D jobs[future]=0D +=0D + pdf_name =3D name + ".pdf"=0D + pdf_from =3D os.path.join(from_dir, pdf_name)=0D +=0D + try:=0D + success =3D future.result()=0D +=0D + if success and os.path.exists(pdf_from):=0D + pdf_to =3D os.path.join(pdf_dir, pdf_name)=0D +=0D + os.rename(pdf_from, pdf_to)=0D + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir)=0D + else:=0D + builds[name] =3D "FAILED"=0D + build_failed =3D True=0D + except futures.Error as e:=0D + builds[name] =3D f"FAILED ({repr(e)})"=0D + build_failed =3D True=0D +=0D + #=0D + # Handle case where no .tex files were found=0D + #=0D + if not has_tex:=0D + name =3D "Sphinx LaTeX builder"=0D + max_len =3D max(max_len, len(name))=0D + builds[name] =3D "FAILED (no .tex file was generated)"=0D + build_failed =3D True=0D +=0D + return builds, build_failed, max_len=0D +=0D def handle_pdf(self, output_dirs):=0D """=0D Extra steps for PDF output.=0D @@ -288,7 +366,9 @@ class SphinxBuilder:=0D """=0D builds =3D {}=0D max_len =3D 0=0D + tex_suffix =3D ".tex"=0D =0D + tex_files =3D []=0D for from_dir in output_dirs:=0D pdf_dir =3D os.path.join(from_dir, "../pdf")=0D os.makedirs(pdf_dir, exist_ok=3DTrue)=0D @@ -300,55 +380,51 @@ class SphinxBuilder:=0D =0D latex_cmd.extend(shlex.split(self.latexopts))=0D =0D - tex_suffix =3D ".tex"=0D -=0D - #=0D - # Process each .tex file=0D - #=0D -=0D - has_tex =3D False=0D - build_failed =3D False=0D + # Get a list of tex files to process=0D with os.scandir(from_dir) as it:=0D for entry in it:=0D - if not entry.name.endswith(tex_suffix):=0D - continue=0D + if entry.name.endswith(tex_suffix):=0D + tex_files.append((from_dir, pdf_dir, entry))=0D =0D - name =3D entry.name[:-len(tex_suffix)]=0D - has_tex =3D True=0D + #=0D + # When using make, this won't be used, as the number of jobs comes= =0D + # from POSIX jobserver. So, this covers the case where build comes= =0D + # from command line. On such case, serialize by default, except if= =0D + # the user explicitly sets the number of jobs.=0D + #=0D + n_jobs =3D 1=0D =0D - #=0D - # LaTeX PDF error code is almost useless for us:=0D - # any warning makes it non-zero. For kernel doc builds= it=0D - # always return non-zero even when build succeeds.=0D - # So, let's do the best next thing: check if all PDF=0D - # files were built. If they're, print a summary and=0D - # return 0 at the end of this function=0D - #=0D - try:=0D - subprocess.run(latex_cmd + [entry.path],=0D - cwd=3Dfrom_dir, check=3DTrue)=0D - except subprocess.CalledProcessError:=0D - pass=0D + # n_jobs is either an integer or "auto". Only use it if it is a nu= mber=0D + if self.n_jobs:=0D + try:=0D + n_jobs =3D int(self.n_jobs)=0D + except ValueError:=0D + pass=0D =0D - pdf_name =3D name + ".pdf"=0D - pdf_from =3D os.path.join(from_dir, pdf_name)=0D - pdf_to =3D os.path.join(pdf_dir, pdf_name)=0D + #=0D + # When using make, jobserver.claim is the number of jobs that were= =0D + # used with "-j" and that aren't used by other make targets=0D + #=0D + with JobserverExec() as jobserver:=0D + n_jobs =3D 1=0D =0D - if os.path.exists(pdf_from):=0D - os.rename(pdf_from, pdf_to)=0D - builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir)=0D - else:=0D - builds[name] =3D "FAILED"=0D - build_failed =3D True=0D + #=0D + # Handle the case when a parameter is passed via command line,= =0D + # using it as default, if jobserver doesn't claim anything=0D + #=0D + if self.n_jobs:=0D + try:=0D + n_jobs =3D int(self.n_jobs)=0D + except ValueError:=0D + pass=0D =0D - name =3D entry.name.removesuffix(".tex")=0D - max_len =3D max(max_len, len(name))=0D + if jobserver.claim:=0D + n_jobs =3D jobserver.claim=0D =0D - if not has_tex:=0D - name =3D os.path.basename(from_dir)=0D - max_len =3D max(max_len, len(name))=0D - builds[name] =3D "FAILED (no .tex)"=0D - build_failed =3D True=0D + builds, build_failed, max_len =3D self.pdf_parallel_build(tex_= suffix,=0D + latex_= cmd,=0D + tex_fi= les,=0D + n_jobs= )=0D =0D msg =3D "Summary"=0D msg +=3D "\n" + "=3D" * len(msg)=0D -- =0D 2.51.0=0D =0D