Re: [RFC][PATCH 0/9] smp: irq_work / smp_call_function rework

["Paul E. McKenney" <paulmck@kernel.org>]

On Wed, Jul 22, 2020 at 05:01:49PM +0200, Peter Zijlstra wrote:
> Hi,
> 
> Here are a number of patches that continue the irq_work / smp_call_function
> integration / cleanup.
> 
> One of the biggest warts in this area is smp_call_function_single_async(); it
> looks like a useful function but is incredibly hard to use correctly due to the
> test-and-set LOCK on the csd not being atomic. This means you need to be
> incredibly careful to not corrupt the csd.
> 
> Simple patterns like allowing any CPU to IPI any other CPU end up requiring
> nr_cpu^2 storage because of this.
> 
> On top of that the csd has external data, vs the more common internal/embedded
> data pattern.
> 
> Now, irq_work has the embedded data pattern, but requires arch support for
> self-IPI. But because irq_work for remote CPUs relies on the smp_call_function
> infrastructure we can implement a generic irq_work_queue_remote().
> 
> Then it goes a bit ugly, and I introduce irq_work_queue_remote_static() that is
> non-atomic in exactly the same way smp_call_function_single_async() is now, but
> at least it has embedded data. A few performance sensitive users of
> smp_call_function_single_async() are converted.
> 
> Finally, smp_call_function_single_async() is made safer by using an atomic
> test-and-set.
> 
> TL;DR, I think at least the first few patches should go in the next round, but
> the rest can use some feedback.

And scftorture doesn't much like the full set of patches.  Though this
is early enough during boot that I am not sure that scftorture had much
chance to do any real damage.

I started with next-20200722, and reverted these three commits per our
IRC discussion:

c1c8004b7415 ("smp: Make symbol 'csd_bug_count' static")
1af4b06012bd ("kernel/smp: Provide CSD lock timeout diagnostics")
e327200d9d75 ("smp: Add source and destination CPUs to __call_single_data")

I ran scftorture using the following command on my 12-hardware-thread
x86 laptop:

tools/testing/selftests/rcutorture/bin/kvm.sh --torture scf --allcpus --duration 2 --trust-make

Rerunning this on top of the last of the above reverts completed with
no drama, as in no complaints and "echo $?" said "0".

Does this reproduce at your end?  If not, any debug code I should apply
or debug options I should enable?

						Thanx, Paul

------------------------------------------------------------------------

[    1.230832] BUG: unable to handle page fault for address: 00000002000009f1
[    1.231342] #PF: supervisor read access in kernel mode
[    1.231705] #PF: error_code(0x0000) - not-present page
[    1.231818] PGD 0 P4D 0 
[    1.231818] Oops: 0000 [#1] SMP PTI
[    1.231818] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.8.0-rc6-next-20200722+ #4
[    1.231818] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1ubuntu1 04/01/2014
[    1.231818] RIP: 0010:nohz_work_func+0x0/0xa0
[    1.231818] Code: e8 35 52 ff ff 48 89 c3 eb 93 8b 43 70 89 44 24 04 eb c2 48 c7 c0 ea ff ff ff eb 94 e8 c9 02 a7 00 66 0f 1f 84 00 00 00 00 00 <4c> 63 8f f0 09 00 00 48 c7 c1 00 91 02 00 48 89 ca 4a 03 14 cd e0
[    1.231818] RSP: 0000:ffff8fa4c0130fb8 EFLAGS: 00010086
[    1.231818] RAX: ffff8d505f2a9118 RBX: 0000000000000000 RCX: ffff8d505f2a9118
[    1.231818] RDX: ffffffff8f692330 RSI: 0000000000000000 RDI: 0000000200000001
[    1.231818] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000028980
[    1.231818] R10: ffff8fa4c0077dc0 R11: 0000000000000000 R12: 0000000000000000
[    1.231818] R13: ffff8d505f2a9118 R14: 0000000000000000 R15: 0000000000000000
[    1.238055] FS:  0000000000000000(0000) GS:ffff8d505f280000(0000) knlGS:0000000000000000
[    1.238055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.238055] CR2: 00000002000009f1 CR3: 000000001440a000 CR4: 00000000000006e0
[    1.238055] Call Trace:
[    1.238055]  <IRQ>
[    1.238055]  flush_smp_call_function_queue+0xc4/0x1e0
[    1.238055]  __sysvec_call_function_single+0x28/0xb0
[    1.238055]  asm_call_on_stack+0x12/0x20
[    1.238055]  </IRQ>
[    1.238055]  sysvec_call_function_single+0x52/0x80
[    1.238055]  asm_sysvec_call_function_single+0x12/0x20
[    1.238055] RIP: 0010:default_idle+0x20/0x140
[    1.238055] Code: 9f ff ff cc cc cc cc cc cc cc 41 55 41 54 55 53 65 8b 2d e3 8c f0 6f 0f 1f 44 00 00 e9 07 00 00 00 0f 00 2d 54 d4 4f 00 fb f4 <65> 8b 2d c9 8c f0 6f 0f 1f 44 00 00 5b 5d 41 5c 41 5d c3 65 8b 05
[    1.238055] RSP: 0000:ffff8fa4c0077ed0 EFLAGS: 00000246
[    1.238055] RAX: ffffffff90108670 RBX: 0000000000000002 RCX: 7fffffffb80ea43f
[    1.238055] RDX: 00000000000cc596 RSI: 0000000000000002 RDI: ffff8d505f29e460
[    1.238055] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
[    1.238055] R10: ffff8fa4c0077eb0 R11: 0000000000000000 R12: 0000000000000000
[    1.238055] R13: 0000000000000000 R14: ffffffffffffffff R15: ffff8d505ed3a700
[    1.238055]  ? __sched_text_end+0x7/0x7
[    1.238055]  do_idle+0x1ca/0x2b0
[    1.238055]  cpu_startup_entry+0x14/0x20
[    1.238055]  secondary_startup_64+0xb6/0xc0
[    1.238055] Modules linked in:
[    1.238055] CR2: 00000002000009f1
[    1.238055] ---[ end trace d53a363244ab3d2f ]---
[    1.238055] RIP: 0010:nohz_work_func+0x0/0xa0
[    1.238055] Code: e8 35 52 ff ff 48 89 c3 eb 93 8b 43 70 89 44 24 04 eb c2 48 c7 c0 ea ff ff ff eb 94 e8 c9 02 a7 00 66 0f 1f 84 00 00 00 00 00 <4c> 63 8f f0 09 00 00 48 c7 c1 00 91 02 00 48 89 ca 4a 03 14 cd e0
[    1.238055] RSP: 0000:ffff8fa4c0130fb8 EFLAGS: 00010086
[    1.238055] RAX: ffff8d505f2a9118 RBX: 0000000000000000 RCX: ffff8d505f2a9118
[    1.238055] RDX: ffffffff8f692330 RSI: 0000000000000000 RDI: 0000000200000001
[    1.238055] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000028980
[    1.238055] R10: ffff8fa4c0077dc0 R11: 0000000000000000 R12: 0000000000000000
[    1.238055] R13: ffff8d505f2a9118 R14: 0000000000000000 R15: 0000000000000000
[    1.238055] FS:  0000000000000000(0000) GS:ffff8d505f280000(0000) knlGS:0000000000000000
[    1.238055] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.238055] CR2: 00000002000009f1 CR3: 000000001440a000 CR4: 00000000000006e0
[    1.238055] Kernel panic - not syncing: Fatal exception in interrupt
[    1.238055] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---