[BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[Paul E. McKenney]

Hello!

Running rcutorture scenario TREE01 on next-20240820 on a dual-socket x86
system (and for that matter on my laptop) got me this warning during boot:

sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

The number at the end varies from run to run.

Bisection landed here:

b188c57af2b5 ("workqueue: Split alloc_workqueue into internal function and lockdep init")

My reproducer is to run this at the top of the Linux-kernel file tree:

tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make

Reverting this commit on top of next-20240820 gets me clean runs.

I cannot see how the patch would make this happen, but then again,
I don't know this code at all well.

Thoughts?

						Thanx, Paul

Re: [BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[Tejun Heo]

On Tue, Aug 20, 2024 at 07:16:00PM -0700, Paul E. McKenney wrote:
> Running rcutorture scenario TREE01 on next-20240820 on a dual-socket x86
> system (and for that matter on my laptop) got me this warning during boot:
> 
> sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'
> 
> The number at the end varies from run to run.
> 
> Bisection landed here:
> 
> b188c57af2b5 ("workqueue: Split alloc_workqueue into internal function and lockdep init")
> 
> My reproducer is to run this at the top of the Linux-kernel file tree:
> 
> tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
> 
> Reverting this commit on top of next-20240820 gets me clean runs.
> 
> I cannot see how the patch would make this happen, but then again,
> I don't know this code at all well.
> 
> Thoughts?

Hmmm... I have a hard time imagining that change causing that error. My bet
is unreliable reproducer. I'll try the repro.

Thanks.

-- 
tejun

Re: [BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[Tejun Heo]

On Wed, Aug 21, 2024 at 06:18:45AM -1000, Tejun Heo wrote:
...
> > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
...
> Hmmm... I have a hard time imagining that change causing that error. My bet
> is unreliable reproducer. I'll try the repro.

I'm probably doing something wrong but all the tests say that init
segfaulted in the log file:

  [    5.505986] init[1]: segfault at 28 ip 000000000040101f sp 00007fff1bbf6fc0 error 4 in init[101f,401000+1000] likely on CPU 12 (core 12, socket 0)
  [    5.509949] Code: Unable to access opcode bytes at 0x400ff5.
  [    5.512314] coredump: 1(init): over coredump resource limit, skipping core dump
  [    5.514976] coredump: 1(init): coredump has not been created, error -7
  [    5.516911] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

Pasting the command output:

  $ tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
  tools/testing/selftests/rcutorture/initrd/init already exists, no need to create it
  Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
  tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
  ----Start batch 1: Wed Aug 21 06:31:03 AM HST 2024
  TREE01 8: Starting build. Wed Aug 21 06:31:03 AM HST 2024
  TREE01 8: Waiting for build to complete. Wed Aug 21 06:31:03 AM HST 2024
  TREE01 8: Build complete. Wed Aug 21 06:31:30 AM HST 2024
  TREE01.2 8: Starting build. Wed Aug 21 06:31:30 AM HST 2024
  TREE01.2 8: Waiting for build to complete. Wed Aug 21 06:31:30 AM HST 2024
  TREE01.2 8: Build complete. Wed Aug 21 06:31:31 AM HST 2024
  ---- TREE01 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
  ---- TREE01.2 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
  ---- Starting kernels. Wed Aug 21 06:31:31 AM HST 2024
  ---- All kernel runs complete. Wed Aug 21 06:31:42 AM HST 2024
  ---- TREE01 8: Build/run results:
   --- Wed Aug 21 06:31:03 AM HST 2024: Starting build, PID 456070
   --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
  CPU-hotplug kernel, adding rcutorture onoff.
   --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483593
  Monitoring qemu job at pid 483623 Wed Aug 21 06:31:41 AM HST 2024
  TREE01 no success message, 0 successful version messages
  WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
  WARNING: Summary: Call Traces: 1
  ---- TREE01.2 8: Build/run results:
   --- Wed Aug 21 06:31:30 AM HST 2024: Starting build, PID 482122
   --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
  CPU-hotplug kernel, adding rcutorture onoff.
   --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483619
  Monitoring qemu job at pid 483648 Wed Aug 21 06:31:41 AM HST 2024
  TREE01 no success message, 0 successful version messages
  WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01
  WARNING: Summary: Call Traces: 1


   --- Wed Aug 21 06:31:42 AM HST 2024 Test summary:
  Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
  tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
  TREE01 ------- 
  TREE01 no success message, 0 successful version messages
  WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
  WARNING: Summary: Call Traces: 1
  Completed in 10 vs. 60
  TREE01.2 ------- 
  TREE01.2 no success message, 0 successful version messages
  WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01.2
  WARNING: Summary: Call Traces: 1
  Completed in 10 vs. 60
  2 runs with build errors.
  2 runs with runtime errors.
   --- Done at Wed Aug 21 06:31:42 AM HST 2024 (39 seconds) exitcode 3

--
tejun

Re: [BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[Paul E. McKenney]

On Wed, Aug 21, 2024 at 06:35:41AM -1000, Tejun Heo wrote:
> On Wed, Aug 21, 2024 at 06:18:45AM -1000, Tejun Heo wrote:
> ...
> > > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
> ...
> > Hmmm... I have a hard time imagining that change causing that error. My bet
> > is unreliable reproducer. I'll try the repro.

I am glad that it was not just me being blind, then.  ;-)

And thank you for trying this out!

> I'm probably doing something wrong but all the tests say that init
> segfaulted in the log file:
> 
>   [    5.505986] init[1]: segfault at 28 ip 000000000040101f sp 00007fff1bbf6fc0 error 4 in init[101f,401000+1000] likely on CPU 12 (core 12, socket 0)
>   [    5.509949] Code: Unable to access opcode bytes at 0x400ff5.
>   [    5.512314] coredump: 1(init): over coredump resource limit, skipping core dump
>   [    5.514976] coredump: 1(init): coredump has not been created, error -7
>   [    5.516911] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

You are right, that is not the expected result.

> Pasting the command output:
> 
>   $ tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
>   tools/testing/selftests/rcutorture/initrd/init already exists, no need to create it

There was already an "init" program, so one thing to try is to remove it
and re-run.  The kvm.sh script will then rebuild it.

I just tried that in case some recent change made this fail to work, and
it "worked for me":

Creating a statically linked C-language initrd
Done creating a statically linked C-language initrd

I will also be testing next-20240821 in case it magically got better.
Stranger things have happened...

							Thanx, Paul

>   Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
>   tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
>   ----Start batch 1: Wed Aug 21 06:31:03 AM HST 2024
>   TREE01 8: Starting build. Wed Aug 21 06:31:03 AM HST 2024
>   TREE01 8: Waiting for build to complete. Wed Aug 21 06:31:03 AM HST 2024
>   TREE01 8: Build complete. Wed Aug 21 06:31:30 AM HST 2024
>   TREE01.2 8: Starting build. Wed Aug 21 06:31:30 AM HST 2024
>   TREE01.2 8: Waiting for build to complete. Wed Aug 21 06:31:30 AM HST 2024
>   TREE01.2 8: Build complete. Wed Aug 21 06:31:31 AM HST 2024
>   ---- TREE01 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
>   ---- TREE01.2 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
>   ---- Starting kernels. Wed Aug 21 06:31:31 AM HST 2024
>   ---- All kernel runs complete. Wed Aug 21 06:31:42 AM HST 2024
>   ---- TREE01 8: Build/run results:
>    --- Wed Aug 21 06:31:03 AM HST 2024: Starting build, PID 456070
>    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
>   CPU-hotplug kernel, adding rcutorture onoff.
>    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483593
>   Monitoring qemu job at pid 483623 Wed Aug 21 06:31:41 AM HST 2024
>   TREE01 no success message, 0 successful version messages
>   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
>   WARNING: Summary: Call Traces: 1
>   ---- TREE01.2 8: Build/run results:
>    --- Wed Aug 21 06:31:30 AM HST 2024: Starting build, PID 482122
>    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
>   CPU-hotplug kernel, adding rcutorture onoff.
>    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483619
>   Monitoring qemu job at pid 483648 Wed Aug 21 06:31:41 AM HST 2024
>   TREE01 no success message, 0 successful version messages
>   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01
>   WARNING: Summary: Call Traces: 1
> 
> 
>    --- Wed Aug 21 06:31:42 AM HST 2024 Test summary:
>   Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
>   tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
>   TREE01 ------- 
>   TREE01 no success message, 0 successful version messages
>   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
>   WARNING: Summary: Call Traces: 1
>   Completed in 10 vs. 60
>   TREE01.2 ------- 
>   TREE01.2 no success message, 0 successful version messages
>   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01.2
>   WARNING: Summary: Call Traces: 1
>   Completed in 10 vs. 60
>   2 runs with build errors.
>   2 runs with runtime errors.
>    --- Done at Wed Aug 21 06:31:42 AM HST 2024 (39 seconds) exitcode 3
> 
> --
> tejun

Re: [BUG bisected] sysfs: cannot create duplicate filename '/devices/virtual/workqueue/scsi_tmf_1073822488'

[Paul E. McKenney]

On Wed, Aug 21, 2024 at 10:37:21AM -0700, Paul E. McKenney wrote:
> On Wed, Aug 21, 2024 at 06:35:41AM -1000, Tejun Heo wrote:
> > On Wed, Aug 21, 2024 at 06:18:45AM -1000, Tejun Heo wrote:
> > ...
> > > > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
> > ...
> > > Hmmm... I have a hard time imagining that change causing that error. My bet
> > > is unreliable reproducer. I'll try the repro.
> 
> I am glad that it was not just me being blind, then.  ;-)
> 
> And thank you for trying this out!
> 
> > I'm probably doing something wrong but all the tests say that init
> > segfaulted in the log file:
> > 
> >   [    5.505986] init[1]: segfault at 28 ip 000000000040101f sp 00007fff1bbf6fc0 error 4 in init[101f,401000+1000] likely on CPU 12 (core 12, socket 0)
> >   [    5.509949] Code: Unable to access opcode bytes at 0x400ff5.
> >   [    5.512314] coredump: 1(init): over coredump resource limit, skipping core dump
> >   [    5.514976] coredump: 1(init): coredump has not been created, error -7
> >   [    5.516911] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> 
> You are right, that is not the expected result.
> 
> > Pasting the command output:
> > 
> >   $ tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs "2*TREE01" --trust-make
> >   tools/testing/selftests/rcutorture/initrd/init already exists, no need to create it
> 
> There was already an "init" program, so one thing to try is to remove it
> and re-run.  The kvm.sh script will then rebuild it.
> 
> I just tried that in case some recent change made this fail to work, and
> it "worked for me":
> 
> Creating a statically linked C-language initrd
> Done creating a statically linked C-language initrd
> 
> I will also be testing next-20240821 in case it magically got better.
> Stranger things have happened...

And it magically got better.  Yay?

Unfortunately not the case for the other bug I was chasing, but you
cannot have everything...

Apologies for the noise!

							Thanx, Paul

> >   Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
> >   tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
> >   ----Start batch 1: Wed Aug 21 06:31:03 AM HST 2024
> >   TREE01 8: Starting build. Wed Aug 21 06:31:03 AM HST 2024
> >   TREE01 8: Waiting for build to complete. Wed Aug 21 06:31:03 AM HST 2024
> >   TREE01 8: Build complete. Wed Aug 21 06:31:30 AM HST 2024
> >   TREE01.2 8: Starting build. Wed Aug 21 06:31:30 AM HST 2024
> >   TREE01.2 8: Waiting for build to complete. Wed Aug 21 06:31:30 AM HST 2024
> >   TREE01.2 8: Build complete. Wed Aug 21 06:31:31 AM HST 2024
> >   ---- TREE01 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
> >   ---- TREE01.2 8: Kernel present. Wed Aug 21 06:31:31 AM HST 2024
> >   ---- Starting kernels. Wed Aug 21 06:31:31 AM HST 2024
> >   ---- All kernel runs complete. Wed Aug 21 06:31:42 AM HST 2024
> >   ---- TREE01 8: Build/run results:
> >    --- Wed Aug 21 06:31:03 AM HST 2024: Starting build, PID 456070
> >    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
> >   CPU-hotplug kernel, adding rcutorture onoff.
> >    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483593
> >   Monitoring qemu job at pid 483623 Wed Aug 21 06:31:41 AM HST 2024
> >   TREE01 no success message, 0 successful version messages
> >   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
> >   WARNING: Summary: Call Traces: 1
> >   ---- TREE01.2 8: Build/run results:
> >    --- Wed Aug 21 06:31:30 AM HST 2024: Starting build, PID 482122
> >    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel
> >   CPU-hotplug kernel, adding rcutorture onoff.
> >    --- Wed Aug 21 06:31:31 AM HST 2024: Starting kernel, PID 483619
> >   Monitoring qemu job at pid 483648 Wed Aug 21 06:31:41 AM HST 2024
> >   TREE01 no success message, 0 successful version messages
> >   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01
> >   WARNING: Summary: Call Traces: 1
> > 
> > 
> >    --- Wed Aug 21 06:31:42 AM HST 2024 Test summary:
> >   Results directory: /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03
> >   tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 1m --configs 2*TREE01 --trust-make
> >   TREE01 ------- 
> >   TREE01 no success message, 0 successful version messages
> >   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01/console.log TREE01
> >   WARNING: Summary: Call Traces: 1
> >   Completed in 10 vs. 60
> >   TREE01.2 ------- 
> >   TREE01.2 no success message, 0 successful version messages
> >   WARNING: Assertion failure in /home/htejun/os/linux-next/tools/testing/selftests/rcutorture/res/2024.08.21-06.31.03/TREE01.2/console.log TREE01.2
> >   WARNING: Summary: Call Traces: 1
> >   Completed in 10 vs. 60
> >   2 runs with build errors.
> >   2 runs with runtime errors.
> >    --- Done at Wed Aug 21 06:31:42 AM HST 2024 (39 seconds) exitcode 3
> > 
> > --
> > tejun