* BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
@ 2021-01-19 11:13 Daniel Bristot de Oliveira
2021-01-19 16:45 ` Waiman Long
0 siblings, 1 reply; 4+ messages in thread
From: Daniel Bristot de Oliveira @ 2021-01-19 11:13 UTC (permalink / raw)
To: Waiman Long
Cc: Peter Zijlstra, Valentin Schneider, Juri Lelli, Dietmar Eggemann,
Vincent Guittot, Ingo Molnar, LKML
Hi Waiman,
Are you aware of this issue:
----- %< -----
[ 88.307857] BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
[ 88.308796] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 801, name: sh
[ 88.309785] 6 locks held by sh/801:
[ 88.310265] #0: ffff9f008c575460 (sb_writers#7){.+.+}-{0:0}, at: ksys_write+0x58/0xd0
[ 88.310906] #1: ffff9f008e9dd088 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write+0xa5/0x1c0
[ 88.311672] #2: ffff9f0092164a88 (kn->active#195){.+.+}-{0:0}, at: kernfs_fop_write+0xad/0x1c0
[ 88.312456] #3: ffffffffbac68310 (cpu_hotplug_lock){++++}-{0:0}, at: sched_partition_write+0x72/0x2f0
[ 88.313280] #4: ffffffffbae37090 (&cpuset_rwsem){++++}-{0:0}, at: sched_partition_write+0x7e/0x2f0
[ 88.314095] #5: ffffffffbad89140 (rcu_read_lock){....}-{1:3}, at: update_sibling_cpumasks+0x5/0x140
[ 88.314806] Preemption disabled at:
[ 88.314810] [<ffffffffb900454d>] preempt_schedule_thunk+0x16/0x18
[ 88.315815] CPU: 1 PID: 801 Comm: sh Not tainted 5.10.0-rc5+ #10
[ 88.316203] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-2.fc32 04/01/2014
[ 88.316714] Call Trace:
[ 88.316875] dump_stack+0x8b/0xb0
[ 88.317087] ___might_sleep.cold+0x102/0x116
[ 88.317354] stop_one_cpu+0x82/0xa0
[ 88.317578] ? set_cpus_allowed_ptr+0x10/0x10
[ 88.317858] __set_cpus_allowed_ptr+0x1e6/0x1f0
[ 88.318144] update_tasks_cpumask+0x25/0x50
[ 88.318415] update_cpumasks_hier+0x257/0x840
[ 88.318687] update_sibling_cpumasks+0x96/0x140
[ 88.318968] update_prstate+0x1a0/0x1f0
[ 88.319210] sched_partition_write+0x9f/0x2f0
[ 88.319482] kernfs_fop_write+0xdc/0x1c0
[ 88.319730] vfs_write+0xea/0x3b0
[ 88.319943] ksys_write+0x58/0xd0
[ 88.320156] do_syscall_64+0x33/0x40
[ 88.320382] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 88.320692] RIP: 0033:0x7fbbd79be537
[ 88.320915] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04
25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3
48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 88.322028] RSP: 002b:00007ffd44cc8398 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 88.322479] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fbbd79be537
[ 88.322910] RDX: 0000000000000005 RSI: 0000558ae69200a0 RDI: 0000000000000001
[ 88.323342] RBP: 0000558ae69200a0 R08: 000000000000000a R09: 0000000000000004
[ 88.323775] R10: 0000558ae6921ba0 R11: 0000000000000246 R12: 0000000000000005
[ 88.325046] R13: 00007fbbd7a90500 R14: 0000000000000005 R15: 00007fbbd7a90700
----- >% -----
it happens when I run this (from Dietmar):
----- %< -----
cd /sys/fs/cgroup/
echo $$ > cgroup.procs
echo +cpuset > cgroup.subtree_control
chrt -d --sched-period 1000000000 --sched-runtime 100000000 0 sleep 500 &
PID1=$!
chrt -d --sched-period 1000000000 --sched-runtime 100000000 0 sleep 500 &
PID2=$!
sleep 3
mkdir A
echo 0 > ./A/cpuset.mems
echo 0 > ./A/cpuset.cpus
echo $PID2 > ./A/cgroup.procs
echo root > ./A/cpuset.cpus.partition
echo $PID2 > ./A/cgroup.procs
cat /proc/$PID1/status | grep Cpus_allowed_list | awk '{print $2}'
cat /proc/$PID2/status | grep Cpus_allowed_list | awk '{print $2}'
----- >% -----
with this kernel config:
https://bristot.me/lkml/20200119/config.txt
on Fedora 32 (x86 64 vm).
full dmesg here:
https://bristot.me/lkml/20200119/dmesg.txt
-- Daniel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
2021-01-19 11:13 BUG: sleeping function called from invalid context at kernel/stop_machine.c:135 Daniel Bristot de Oliveira
@ 2021-01-19 16:45 ` Waiman Long
2021-01-19 16:49 ` Peter Zijlstra
0 siblings, 1 reply; 4+ messages in thread
From: Waiman Long @ 2021-01-19 16:45 UTC (permalink / raw)
To: Daniel Bristot de Oliveira
Cc: Peter Zijlstra, Valentin Schneider, Juri Lelli, Dietmar Eggemann,
Vincent Guittot, Ingo Molnar, LKML
On 1/19/21 6:13 AM, Daniel Bristot de Oliveira wrote:
> Hi Waiman,
>
> Are you aware of this issue:
> ----- %< -----
> [ 88.307857] BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
> [ 88.308796] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 801, name: sh
> [ 88.309785] 6 locks held by sh/801:
> [ 88.310265] #0: ffff9f008c575460 (sb_writers#7){.+.+}-{0:0}, at: ksys_write+0x58/0xd0
> [ 88.310906] #1: ffff9f008e9dd088 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write+0xa5/0x1c0
> [ 88.311672] #2: ffff9f0092164a88 (kn->active#195){.+.+}-{0:0}, at: kernfs_fop_write+0xad/0x1c0
> [ 88.312456] #3: ffffffffbac68310 (cpu_hotplug_lock){++++}-{0:0}, at: sched_partition_write+0x72/0x2f0
> [ 88.313280] #4: ffffffffbae37090 (&cpuset_rwsem){++++}-{0:0}, at: sched_partition_write+0x7e/0x2f0
> [ 88.314095] #5: ffffffffbad89140 (rcu_read_lock){....}-{1:3}, at: update_sibling_cpumasks+0x5/0x140
> [ 88.314806] Preemption disabled at:
> [ 88.314810] [<ffffffffb900454d>] preempt_schedule_thunk+0x16/0x18
> [ 88.315815] CPU: 1 PID: 801 Comm: sh Not tainted 5.10.0-rc5+ #10
> [ 88.316203] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-2.fc32 04/01/2014
> [ 88.316714] Call Trace:
> [ 88.316875] dump_stack+0x8b/0xb0
> [ 88.317087] ___might_sleep.cold+0x102/0x116
> [ 88.317354] stop_one_cpu+0x82/0xa0
> [ 88.317578] ? set_cpus_allowed_ptr+0x10/0x10
> [ 88.317858] __set_cpus_allowed_ptr+0x1e6/0x1f0
> [ 88.318144] update_tasks_cpumask+0x25/0x50
> [ 88.318415] update_cpumasks_hier+0x257/0x840
> [ 88.318687] update_sibling_cpumasks+0x96/0x140
> [ 88.318968] update_prstate+0x1a0/0x1f0
> [ 88.319210] sched_partition_write+0x9f/0x2f0
> [ 88.319482] kernfs_fop_write+0xdc/0x1c0
> [ 88.319730] vfs_write+0xea/0x3b0
> [ 88.319943] ksys_write+0x58/0xd0
> [ 88.320156] do_syscall_64+0x33/0x40
> [ 88.320382] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 88.320692] RIP: 0033:0x7fbbd79be537
> [ 88.320915] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04
> 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3
> 48 83 ec 28 48 89 54 24 18 48 89 74 24
> [ 88.322028] RSP: 002b:00007ffd44cc8398 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> [ 88.322479] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fbbd79be537
> [ 88.322910] RDX: 0000000000000005 RSI: 0000558ae69200a0 RDI: 0000000000000001
> [ 88.323342] RBP: 0000558ae69200a0 R08: 000000000000000a R09: 0000000000000004
> [ 88.323775] R10: 0000558ae6921ba0 R11: 0000000000000246 R12: 0000000000000005
> [ 88.325046] R13: 00007fbbd7a90500 R14: 0000000000000005 R15: 00007fbbd7a90700
> ----- >% -----
I am not aware of that.
void ___might_sleep(const char *file, int line, int preempt_offset)
:
if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
!is_idle_task(current) && !current->non_block_count) ||
system_state == SYSTEM_BOOTING || system_state >
SYSTEM_RUNNING ||
oops_in_progress)
return;
I think the failing test was preempt_count_equals(preempt_offset).
static inline int preempt_count_equals(int preempt_offset)
{
int nested = preempt_count() + rcu_preempt_depth();
return (nested == preempt_offset);
}
preempt_count() is 0 (in_atomic() == 0) and preempt_offset is 0, but
rcu_preempt_depth() should be at least 1 as a rcu_read_lock was held. I
don't think we should prevent sleeping if a rcu_read_lock is held. We
need to look at the reason why rcu_preempt_depth() is included in this test.
Cheers,
Longman
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
2021-01-19 16:45 ` Waiman Long
@ 2021-01-19 16:49 ` Peter Zijlstra
2021-01-19 16:55 ` Waiman Long
0 siblings, 1 reply; 4+ messages in thread
From: Peter Zijlstra @ 2021-01-19 16:49 UTC (permalink / raw)
To: Waiman Long
Cc: Daniel Bristot de Oliveira, Valentin Schneider, Juri Lelli,
Dietmar Eggemann, Vincent Guittot, Ingo Molnar, LKML
On Tue, Jan 19, 2021 at 11:45:02AM -0500, Waiman Long wrote:
> On 1/19/21 6:13 AM, Daniel Bristot de Oliveira wrote:
> > Hi Waiman,
> >
> > Are you aware of this issue:
> > ----- %< -----
> > [ 88.307857] BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
> > [ 88.308796] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 801, name: sh
> > [ 88.309785] 6 locks held by sh/801:
> > [ 88.310265] #0: ffff9f008c575460 (sb_writers#7){.+.+}-{0:0}, at: ksys_write+0x58/0xd0
> > [ 88.310906] #1: ffff9f008e9dd088 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write+0xa5/0x1c0
> > [ 88.311672] #2: ffff9f0092164a88 (kn->active#195){.+.+}-{0:0}, at: kernfs_fop_write+0xad/0x1c0
> > [ 88.312456] #3: ffffffffbac68310 (cpu_hotplug_lock){++++}-{0:0}, at: sched_partition_write+0x72/0x2f0
> > [ 88.313280] #4: ffffffffbae37090 (&cpuset_rwsem){++++}-{0:0}, at: sched_partition_write+0x7e/0x2f0
> > [ 88.314095] #5: ffffffffbad89140 (rcu_read_lock){....}-{1:3}, at: update_sibling_cpumasks+0x5/0x140
> > [ 88.314806] Preemption disabled at:
> > [ 88.314810] [<ffffffffb900454d>] preempt_schedule_thunk+0x16/0x18
> > [ 88.315815] CPU: 1 PID: 801 Comm: sh Not tainted 5.10.0-rc5+ #10
> > [ 88.316203] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-2.fc32 04/01/2014
> > [ 88.316714] Call Trace:
> > [ 88.316875] dump_stack+0x8b/0xb0
> > [ 88.317087] ___might_sleep.cold+0x102/0x116
> > [ 88.317354] stop_one_cpu+0x82/0xa0
> > [ 88.317578] ? set_cpus_allowed_ptr+0x10/0x10
> > [ 88.317858] __set_cpus_allowed_ptr+0x1e6/0x1f0
> > [ 88.318144] update_tasks_cpumask+0x25/0x50
> > [ 88.318415] update_cpumasks_hier+0x257/0x840
> > [ 88.318687] update_sibling_cpumasks+0x96/0x140
> > [ 88.318968] update_prstate+0x1a0/0x1f0
> > [ 88.319210] sched_partition_write+0x9f/0x2f0
> > [ 88.319482] kernfs_fop_write+0xdc/0x1c0
> > [ 88.319730] vfs_write+0xea/0x3b0
> > [ 88.319943] ksys_write+0x58/0xd0
> > [ 88.320156] do_syscall_64+0x33/0x40
> > [ 88.320382] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 88.320692] RIP: 0033:0x7fbbd79be537
> > [ 88.320915] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04
> > 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3
> > 48 83 ec 28 48 89 54 24 18 48 89 74 24
> > [ 88.322028] RSP: 002b:00007ffd44cc8398 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> > [ 88.322479] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fbbd79be537
> > [ 88.322910] RDX: 0000000000000005 RSI: 0000558ae69200a0 RDI: 0000000000000001
> > [ 88.323342] RBP: 0000558ae69200a0 R08: 000000000000000a R09: 0000000000000004
> > [ 88.323775] R10: 0000558ae6921ba0 R11: 0000000000000246 R12: 0000000000000005
> > [ 88.325046] R13: 00007fbbd7a90500 R14: 0000000000000005 R15: 00007fbbd7a90700
> > ----- >% -----
>
> I am not aware of that.
>
> void ___might_sleep(const char *file, int line, int preempt_offset)
> :
> if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
> !is_idle_task(current) && !current->non_block_count) ||
> system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING
> ||
> oops_in_progress)
> return;
>
> I think the failing test was preempt_count_equals(preempt_offset).
>
> static inline int preempt_count_equals(int preempt_offset)
> {
> int nested = preempt_count() + rcu_preempt_depth();
>
> return (nested == preempt_offset);
> }
>
> preempt_count() is 0 (in_atomic() == 0) and preempt_offset is 0, but
> rcu_preempt_depth() should be at least 1 as a rcu_read_lock was held. I
> don't think we should prevent sleeping if a rcu_read_lock is held. We need
> to look at the reason why rcu_preempt_depth() is included in this test.
You're not allowed to sleep with rcu_read_lock() held. With config
PREEMPT=y you're allowed to get preempted with rcu_read_lock() held, but
never to explicitly block.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
2021-01-19 16:49 ` Peter Zijlstra
@ 2021-01-19 16:55 ` Waiman Long
0 siblings, 0 replies; 4+ messages in thread
From: Waiman Long @ 2021-01-19 16:55 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Daniel Bristot de Oliveira, Valentin Schneider, Juri Lelli,
Dietmar Eggemann, Vincent Guittot, Ingo Molnar, LKML
On 1/19/21 11:49 AM, Peter Zijlstra wrote:
> On Tue, Jan 19, 2021 at 11:45:02AM -0500, Waiman Long wrote:
>> On 1/19/21 6:13 AM, Daniel Bristot de Oliveira wrote:
>>> Hi Waiman,
>>>
>>> Are you aware of this issue:
>>> ----- %< -----
>>> [ 88.307857] BUG: sleeping function called from invalid context at kernel/stop_machine.c:135
>>> [ 88.308796] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 801, name: sh
>>> [ 88.309785] 6 locks held by sh/801:
>>> [ 88.310265] #0: ffff9f008c575460 (sb_writers#7){.+.+}-{0:0}, at: ksys_write+0x58/0xd0
>>> [ 88.310906] #1: ffff9f008e9dd088 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write+0xa5/0x1c0
>>> [ 88.311672] #2: ffff9f0092164a88 (kn->active#195){.+.+}-{0:0}, at: kernfs_fop_write+0xad/0x1c0
>>> [ 88.312456] #3: ffffffffbac68310 (cpu_hotplug_lock){++++}-{0:0}, at: sched_partition_write+0x72/0x2f0
>>> [ 88.313280] #4: ffffffffbae37090 (&cpuset_rwsem){++++}-{0:0}, at: sched_partition_write+0x7e/0x2f0
>>> [ 88.314095] #5: ffffffffbad89140 (rcu_read_lock){....}-{1:3}, at: update_sibling_cpumasks+0x5/0x140
>>> [ 88.314806] Preemption disabled at:
>>> [ 88.314810] [<ffffffffb900454d>] preempt_schedule_thunk+0x16/0x18
>>> [ 88.315815] CPU: 1 PID: 801 Comm: sh Not tainted 5.10.0-rc5+ #10
>>> [ 88.316203] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-2.fc32 04/01/2014
>>> [ 88.316714] Call Trace:
>>> [ 88.316875] dump_stack+0x8b/0xb0
>>> [ 88.317087] ___might_sleep.cold+0x102/0x116
>>> [ 88.317354] stop_one_cpu+0x82/0xa0
>>> [ 88.317578] ? set_cpus_allowed_ptr+0x10/0x10
>>> [ 88.317858] __set_cpus_allowed_ptr+0x1e6/0x1f0
>>> [ 88.318144] update_tasks_cpumask+0x25/0x50
>>> [ 88.318415] update_cpumasks_hier+0x257/0x840
>>> [ 88.318687] update_sibling_cpumasks+0x96/0x140
>>> [ 88.318968] update_prstate+0x1a0/0x1f0
>>> [ 88.319210] sched_partition_write+0x9f/0x2f0
>>> [ 88.319482] kernfs_fop_write+0xdc/0x1c0
>>> [ 88.319730] vfs_write+0xea/0x3b0
>>> [ 88.319943] ksys_write+0x58/0xd0
>>> [ 88.320156] do_syscall_64+0x33/0x40
>>> [ 88.320382] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 88.320692] RIP: 0033:0x7fbbd79be537
>>> [ 88.320915] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04
>>> 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3
>>> 48 83 ec 28 48 89 54 24 18 48 89 74 24
>>> [ 88.322028] RSP: 002b:00007ffd44cc8398 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>>> [ 88.322479] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fbbd79be537
>>> [ 88.322910] RDX: 0000000000000005 RSI: 0000558ae69200a0 RDI: 0000000000000001
>>> [ 88.323342] RBP: 0000558ae69200a0 R08: 000000000000000a R09: 0000000000000004
>>> [ 88.323775] R10: 0000558ae6921ba0 R11: 0000000000000246 R12: 0000000000000005
>>> [ 88.325046] R13: 00007fbbd7a90500 R14: 0000000000000005 R15: 00007fbbd7a90700
>>> ----- >% -----
>> I am not aware of that.
>>
>> void ___might_sleep(const char *file, int line, int preempt_offset)
>> Â :
>> Â Â Â Â Â Â Â if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
>> Â Â Â Â Â Â Â Â Â Â Â Â !is_idle_task(current) && !current->non_block_count) ||
>> Â Â Â Â Â Â Â Â Â Â Â system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING
>> ||
>> Â Â Â Â Â Â Â Â Â Â Â oops_in_progress)
>> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â return;
>>
>> I think the failing test was preempt_count_equals(preempt_offset).
>>
>> static inline int preempt_count_equals(int preempt_offset)
>> {
>> Â Â Â Â Â Â Â int nested = preempt_count() + rcu_preempt_depth();
>>
>> Â Â Â Â Â Â Â return (nested == preempt_offset);
>> }
>>
>> preempt_count() is 0 (in_atomic() == 0) and preempt_offset is 0, but
>> rcu_preempt_depth() should be at least 1 as a rcu_read_lock was held. I
>> don't think we should prevent sleeping if a rcu_read_lock is held. We need
>> to look at the reason why rcu_preempt_depth() is included in this test.
> You're not allowed to sleep with rcu_read_lock() held. With config
> PREEMPT=y you're allowed to get preempted with rcu_read_lock() held, but
> never to explicitly block.
You are right. Sleep shouldn't be allowed with rcu_read_lock() held in
non-preempt kernel.
However, the kernel that Daniel tested did have CONFIG_PREEMPT=y set. So
perhaps we shouldn't do the rcu_preempt_depth() check in this particular
case.
Cheers,
Longman
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-01-19 18:28 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-19 11:13 BUG: sleeping function called from invalid context at kernel/stop_machine.c:135 Daniel Bristot de Oliveira
2021-01-19 16:45 ` Waiman Long
2021-01-19 16:49 ` Peter Zijlstra
2021-01-19 16:55 ` Waiman Long
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®