[BUG] Build failure in next-20261007 and next-20261008

[BUG] Build failure in next-20261007 and next-20261008

[Paul E. McKenney]

Hello!

Building a kernel on CentOS Stream 9 from next-20261007 and next-20261008
(and maybe others?)  on an x86 or arm64 system results in a strange
build failure featuring:

WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')

However, a kernel really is built, and restarting the build immediately
after a failure succeeds and generates a good kernel.  Doing a "make
clean" causes the next build to fail, as does changing a sufficient
number of source files.

My reproducer is:

tools/testing/selftests/rcutorture/bin/kvm.sh --build-only --allcpus --configs TREE02 --trust-make

But a simple "make -jN" suffices.

Bisection converged on this commit:

48ebe1cbbeb5 ("kbuild: compress the kernel with pigz if available")

Reverting this commit gets rid of the bug.

The bug does not happen on my ancient Ubuntu laptop.

The failing systems have a pigz command (which complains bitterly when
run without arguments), and the immune system does not.  On the failing
systems, "pigz --version" gives me "pigz 2.5".

Do I need a specific version of pigz?  Does this commit need a
version-number check?  Are my systems pigz-intolerant?  ;-)

Less flippantly, is this a known problem?  If not, what debugging
information can I provide?

							Thanx, Paul

Re: [BUG] Build failure in next-20261007 and next-20261008

[Kees Cook]

On Thu, Oct 08, 2026 at 05:56:21PM -0700, Paul E. McKenney wrote:
> Building a kernel on CentOS Stream 9 from next-20261007 and next-20261008
> (and maybe others?)  on an x86 or arm64 system results in a strange
> build failure featuring:
> 
> WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')

What are your make and python versions? (Oh, the answer is going to be
"CentOS-patched make 4.3")

> The failing systems have a pigz command (which complains bitterly when
> run without arguments), and the immune system does not.  On the failing
> systems, "pigz --version" gives me "pigz 2.5".
> 
> Do I need a specific version of pigz?  Does this commit need a
> version-number check?  Are my systems pigz-intolerant?  ;-)
> 
> Less flippantly, is this a known problem?  If not, what debugging
> information can I provide?

Having pigz just means the parallelism logic kicks in, so I'm suspecting
something in scripts/jobserver-exec. That would point to something
specific to python or make.

I haven't seen this reported before, no.

Ah, and before I could even finish writing this email, my ever helpful
LLM has reported: "pre-4.4 make (like CentOS's 4.3) closes jobserver
pipe descriptors in non-recursive recipes but still exports
--jobserver-auth=R,W via MAKEFLAGS, whereas 4.4+ uses a named FIFO
instead, explaining why it doesn't appear here." And has further found
that this is CentOS-specific, since they appear to have incorrectly
backported a patch ("make-4.3-cloexec.patch"), as stock make 4.3 doesn't
have this problem.

I'll go figure out if there is some proper way to work around CentOS's
broken make...

(And given this, I would suspect you can't do a '-j...' build of the
"htmldocs" target either?)

-Kees

-- 
Kees Cook

Re: [BUG] Build failure in next-20261007 and next-20261008

[Paul E. McKenney]

On Thu, Oct 08, 2026 at 10:11:14PM -0700, Kees Cook wrote:
> On Thu, Oct 08, 2026 at 05:56:21PM -0700, Paul E. McKenney wrote:
> > Building a kernel on CentOS Stream 9 from next-20261007 and next-20261008
> > (and maybe others?)  on an x86 or arm64 system results in a strange
> > build failure featuring:
> > 
> > WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')
> 
> What are your make and python versions? (Oh, the answer is going to be
> "CentOS-patched make 4.3")

I get "GNU Make 4.3", so maybe I need to patch or upgrade or something?

And "Python 3.12.15+meta", so maybe I need to talk to people here.  ;-)

> > The failing systems have a pigz command (which complains bitterly when
> > run without arguments), and the immune system does not.  On the failing
> > systems, "pigz --version" gives me "pigz 2.5".
> > 
> > Do I need a specific version of pigz?  Does this commit need a
> > version-number check?  Are my systems pigz-intolerant?  ;-)
> > 
> > Less flippantly, is this a known problem?  If not, what debugging
> > information can I provide?
> 
> Having pigz just means the parallelism logic kicks in, so I'm suspecting
> something in scripts/jobserver-exec. That would point to something
> specific to python or make.
> 
> I haven't seen this reported before, no.
> 
> Ah, and before I could even finish writing this email, my ever helpful
> LLM has reported: "pre-4.4 make (like CentOS's 4.3) closes jobserver
> pipe descriptors in non-recursive recipes but still exports
> --jobserver-auth=R,W via MAKEFLAGS, whereas 4.4+ uses a named FIFO
> instead, explaining why it doesn't appear here." And has further found
> that this is CentOS-specific, since they appear to have incorrectly
> backported a patch ("make-4.3-cloexec.patch"), as stock make 4.3 doesn't
> have this problem.
> 
> I'll go figure out if there is some proper way to work around CentOS's
> broken make...
> 
> (And given this, I would suspect you can't do a '-j...' build of the
> "htmldocs" target either?)

I will install sphinx &c and give it a try.  ;-)

							Thanx, Paul

Re: [BUG] Build failure in next-20261007 and next-20261008

[Kees Cook]

On Fri, Oct 09, 2026 at 08:57:02AM -0700, Paul E. McKenney wrote:
> On Thu, Oct 08, 2026 at 10:11:14PM -0700, Kees Cook wrote:
> > On Thu, Oct 08, 2026 at 05:56:21PM -0700, Paul E. McKenney wrote:
> > > Building a kernel on CentOS Stream 9 from next-20261007 and next-20261008
> > > (and maybe others?)  on an x86 or arm64 system results in a strange
> > > build failure featuring:
> > > 
> > > WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')
> > 
> > What are your make and python versions? (Oh, the answer is going to be
> > "CentOS-patched make 4.3")
> 
> I get "GNU Make 4.3", so maybe I need to patch or upgrade or something?
> 
> And "Python 3.12.15+meta", so maybe I need to talk to people here.  ;-)

I meant, it looks like CentOS has a patched version of "make". But I
think that just changes the nature of the bug; it's still a bug that
needs to be fixed even with unpatched "make".

-- 
Kees Cook

Re: [BUG] Build failure in next-20261007 and next-20261008

[Paul E. McKenney]

On Fri, Oct 09, 2026 at 07:23:29PM -0700, Kees Cook wrote:
> On Fri, Oct 09, 2026 at 08:57:02AM -0700, Paul E. McKenney wrote:
> > On Thu, Oct 08, 2026 at 10:11:14PM -0700, Kees Cook wrote:
> > > On Thu, Oct 08, 2026 at 05:56:21PM -0700, Paul E. McKenney wrote:
> > > > Building a kernel on CentOS Stream 9 from next-20261007 and next-20261008
> > > > (and maybe others?)  on an x86 or arm64 system results in a strange
> > > > build failure featuring:
> > > > 
> > > > WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')
> > > 
> > > What are your make and python versions? (Oh, the answer is going to be
> > > "CentOS-patched make 4.3")
> > 
> > I get "GNU Make 4.3", so maybe I need to patch or upgrade or something?
> > 
> > And "Python 3.12.15+meta", so maybe I need to talk to people here.  ;-)
> 
> I meant, it looks like CentOS has a patched version of "make". But I
> think that just changes the nature of the bug; it's still a bug that
> needs to be fixed even with unpatched "make".

Whew!!!

And thank you very much for looking into this!

							Thanx, Paul