* [PATCH 1/3] x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled
2017-10-20 9:16 [PATCH 0/3] x86/intel_rdt: Locking and initialization fixes Reinette Chatre
@ 2017-10-20 9:16 ` Reinette Chatre
2017-10-21 15:18 ` [tip:x86/cache] " tip-bot for Reinette Chatre
2017-10-20 9:16 ` [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount Reinette Chatre
2017-10-20 9:16 ` [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount Reinette Chatre
2 siblings, 1 reply; 9+ messages in thread
From: Reinette Chatre @ 2017-10-20 9:16 UTC (permalink / raw)
To: tglx
Cc: fenghua.yu, tony.luck, vikas.shivappa, mingo, hpa, x86,
linux-kernel, Reinette Chatre
The platform informs us via CPUID.(EAX=0x10, ECX=res#):EBX[31:0] (valid
res# are only 1 for L3 and 2 for L2) which unit of the allocation may be
used by other entities in the platform. This information is valid whether
CDP (Code and Data Prioritization) is enabled or not.
Ensure that the bitmask of shareable resource is initialized when CDP is
enabled.
Fixes: 0dd2d7494cd8 ("x86/intel_rdt: Show bitmask of shareable resource
with other executing units"
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
Acked-by: Tony Luck <tony.luck@intel.com>
---
arch/x86/kernel/cpu/intel_rdt.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/cpu/intel_rdt.c b/arch/x86/kernel/cpu/intel_rdt.c
index cd5fc61ba450..88dcf8479013 100644
--- a/arch/x86/kernel/cpu/intel_rdt.c
+++ b/arch/x86/kernel/cpu/intel_rdt.c
@@ -267,6 +267,7 @@ static void rdt_get_cdp_l3_config(int type)
r->num_closid = r_l3->num_closid / 2;
r->cache.cbm_len = r_l3->cache.cbm_len;
r->default_ctrl = r_l3->default_ctrl;
+ r->cache.shareable_bits = r_l3->cache.shareable_bits;
r->data_width = (r->cache.cbm_len + 3) / 4;
r->alloc_capable = true;
/*
--
2.13.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [tip:x86/cache] x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled
2017-10-20 9:16 ` [PATCH 1/3] x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled Reinette Chatre
@ 2017-10-21 15:18 ` tip-bot for Reinette Chatre
0 siblings, 0 replies; 9+ messages in thread
From: tip-bot for Reinette Chatre @ 2017-10-21 15:18 UTC (permalink / raw)
To: linux-tip-commits
Cc: vikas.shivappa, tglx, tony.luck, fenghua.yu, reinette.chatre,
hpa, mingo, linux-kernel
Commit-ID: 95953034fb24c16ad0047a98b16427e5935830c4
Gitweb: https://git.kernel.org/tip/95953034fb24c16ad0047a98b16427e5935830c4
Author: Reinette Chatre <reinette.chatre@intel.com>
AuthorDate: Fri, 20 Oct 2017 02:16:57 -0700
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 21 Oct 2017 17:12:21 +0200
x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled
The platform informs via CPUID.(EAX=0x10, ECX=res#):EBX[31:0] (valid res#
are only 1 for L3 and 2 for L2) which unit of the allocation may be used by
other entities in the platform. This information is valid whether CDP (Code
and Data Prioritization) is enabled or not.
Ensure that the bitmask of shareable resource is initialized when CDP is
enabled.
Fixes: 0dd2d7494cd8 ("x86/intel_rdt: Show bitmask of shareable resource with other executing units"
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
Acked-by: Tony Luck <tony.luck@intel.com>
Link: https://lkml.kernel.org/r/815747bddc820ca221a8924edaf4d1a7324547e4.1508490116.git.reinette.chatre@intel.com
---
arch/x86/kernel/cpu/intel_rdt.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/cpu/intel_rdt.c b/arch/x86/kernel/cpu/intel_rdt.c
index cd5fc61..88dcf84 100644
--- a/arch/x86/kernel/cpu/intel_rdt.c
+++ b/arch/x86/kernel/cpu/intel_rdt.c
@@ -267,6 +267,7 @@ static void rdt_get_cdp_l3_config(int type)
r->num_closid = r_l3->num_closid / 2;
r->cache.cbm_len = r_l3->cache.cbm_len;
r->default_ctrl = r_l3->default_ctrl;
+ r->cache.shareable_bits = r_l3->cache.shareable_bits;
r->data_width = (r->cache.cbm_len + 3) / 4;
r->alloc_capable = true;
/*
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount
2017-10-20 9:16 [PATCH 0/3] x86/intel_rdt: Locking and initialization fixes Reinette Chatre
2017-10-20 9:16 ` [PATCH 1/3] x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled Reinette Chatre
@ 2017-10-20 9:16 ` Reinette Chatre
2017-10-20 18:35 ` Yu, Fenghua
2017-10-21 15:19 ` [tip:x86/cache] " tip-bot for Reinette Chatre
2017-10-20 9:16 ` [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount Reinette Chatre
2 siblings, 2 replies; 9+ messages in thread
From: Reinette Chatre @ 2017-10-20 9:16 UTC (permalink / raw)
To: tglx
Cc: fenghua.yu, tony.luck, vikas.shivappa, mingo, hpa, x86,
linux-kernel, Reinette Chatre
Lockdep is warning us about a potential deadlock:
[ 66.782842] ======================================================
[ 66.782888] WARNING: possible circular locking dependency detected
[ 66.782937] 4.14.0-rc2-test-test+ #48 Not tainted
[ 66.782983] ------------------------------------------------------
[ 66.783052] umount/336 is trying to acquire lock:
[ 66.783117] (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff81032395>] rdt_kill_sb+0x215/0x390
[ 66.783193]
but task is already holding lock:
[ 66.783244] (rdtgroup_mutex){+.+.}, at: [<ffffffff810321b6>] rdt_kill_sb+0x36/0x390
[ 66.783305]
which lock already depends on the new lock.
[ 66.783364]
the existing dependency chain (in reverse order) is:
[ 66.783419]
-> #3 (rdtgroup_mutex){+.+.}:
[ 66.783467] __lock_acquire+0x1293/0x13f0
[ 66.783509] lock_acquire+0xaf/0x220
[ 66.783543] __mutex_lock+0x71/0x9b0
[ 66.783575] mutex_lock_nested+0x1b/0x20
[ 66.783610] intel_rdt_online_cpu+0x3b/0x430
[ 66.783649] cpuhp_invoke_callback+0xab/0x8e0
[ 66.783687] cpuhp_thread_fun+0x7a/0x150
[ 66.783722] smpboot_thread_fn+0x1cc/0x270
[ 66.783764] kthread+0x16e/0x190
[ 66.783794] ret_from_fork+0x27/0x40
[ 66.783825]
-> #2 (cpuhp_state){+.+.}:
[ 66.783870] __lock_acquire+0x1293/0x13f0
[ 66.783906] lock_acquire+0xaf/0x220
[ 66.783938] cpuhp_issue_call+0x102/0x170
[ 66.783974] __cpuhp_setup_state_cpuslocked+0x154/0x2a0
[ 66.784023] __cpuhp_setup_state+0xc7/0x170
[ 66.784061] page_writeback_init+0x43/0x67
[ 66.784097] pagecache_init+0x43/0x4a
[ 66.784131] start_kernel+0x3ad/0x3f7
[ 66.784165] x86_64_start_reservations+0x2a/0x2c
[ 66.784204] x86_64_start_kernel+0x72/0x75
[ 66.784241] verify_cpu+0x0/0xfb
[ 66.784270]
-> #1 (cpuhp_state_mutex){+.+.}:
[ 66.784319] __lock_acquire+0x1293/0x13f0
[ 66.784355] lock_acquire+0xaf/0x220
[ 66.784387] __mutex_lock+0x71/0x9b0
[ 66.784419] mutex_lock_nested+0x1b/0x20
[ 66.784454] __cpuhp_setup_state_cpuslocked+0x52/0x2a0
[ 66.784497] __cpuhp_setup_state+0xc7/0x170
[ 66.784535] page_alloc_init+0x28/0x30
[ 66.784569] start_kernel+0x148/0x3f7
[ 66.784602] x86_64_start_reservations+0x2a/0x2c
[ 66.784642] x86_64_start_kernel+0x72/0x75
[ 66.784678] verify_cpu+0x0/0xfb
[ 66.784707]
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
[ 66.784759] check_prev_add+0x32f/0x6e0
[ 66.784794] __lock_acquire+0x1293/0x13f0
[ 66.784830] lock_acquire+0xaf/0x220
[ 66.784863] cpus_read_lock+0x3d/0xb0
[ 66.784896] rdt_kill_sb+0x215/0x390
[ 66.784930] deactivate_locked_super+0x3e/0x70
[ 66.784968] deactivate_super+0x40/0x60
[ 66.785003] cleanup_mnt+0x3f/0x80
[ 66.785034] __cleanup_mnt+0x12/0x20
[ 66.785070] task_work_run+0x8b/0xc0
[ 66.785103] exit_to_usermode_loop+0x94/0xa0
[ 66.786804] syscall_return_slowpath+0xe8/0x150
[ 66.788502] entry_SYSCALL_64_fastpath+0xab/0xad
[ 66.790194]
other info that might help us debug this:
[ 66.795139] Chain exists of:
cpu_hotplug_lock.rw_sem --> cpuhp_state --> rdtgroup_mutex
[ 66.800035] Possible unsafe locking scenario:
[ 66.803267] CPU0 CPU1
[ 66.804867] ---- ----
[ 66.806443] lock(rdtgroup_mutex);
[ 66.808002] lock(cpuhp_state);
[ 66.809565] lock(rdtgroup_mutex);
[ 66.811110] lock(cpu_hotplug_lock.rw_sem);
[ 66.812608]
*** DEADLOCK ***
[ 66.816983] 2 locks held by umount/336:
[ 66.818418] #0: (&type->s_umount_key#35){+.+.}, at: [<ffffffff81229738>] deactivate_super+0x38/0x60
[ 66.819922] #1: (rdtgroup_mutex){+.+.}, at: [<ffffffff810321b6>] rdt_kill_sb+0x36/0x390
When the resctrl filesystem is unmounted we should obtain the locks in
the same order as was done when the cpus came online, cpu_hotplug_lock
before rdtgroup_mutex. With cpu_hotplug_lock now obtained earlier we also
modify the static_branch_disable() calls to the cpuslocked variant.
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
Acked-by: Tony Luck <tony.luck@intel.com>
---
arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index 8a61b20c7e51..fc27f55e90dc 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -1364,9 +1364,7 @@ static void rmdir_all_sub(void)
kfree(rdtgrp);
}
/* Notify online CPUs to update per cpu storage and PQR_ASSOC MSR */
- get_online_cpus();
update_closid_rmid(cpu_online_mask, &rdtgroup_default);
- put_online_cpus();
kernfs_remove(kn_info);
kernfs_remove(kn_mongrp);
@@ -1377,6 +1375,7 @@ static void rdt_kill_sb(struct super_block *sb)
{
struct rdt_resource *r;
+ get_online_cpus();
mutex_lock(&rdtgroup_mutex);
/*Put everything back to default values. */
@@ -1384,11 +1383,12 @@ static void rdt_kill_sb(struct super_block *sb)
reset_all_ctrls(r);
cdp_disable();
rmdir_all_sub();
- static_branch_disable(&rdt_alloc_enable_key);
- static_branch_disable(&rdt_mon_enable_key);
- static_branch_disable(&rdt_enable_key);
+ static_branch_disable_cpuslocked(&rdt_alloc_enable_key);
+ static_branch_disable_cpuslocked(&rdt_mon_enable_key);
+ static_branch_disable_cpuslocked(&rdt_enable_key);
kernfs_kill_sb(sb);
mutex_unlock(&rdtgroup_mutex);
+ put_online_cpus();
}
static struct file_system_type rdt_fs_type = {
--
2.13.5
^ permalink raw reply [flat|nested] 9+ messages in thread* RE: [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount
2017-10-20 9:16 ` [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount Reinette Chatre
@ 2017-10-20 18:35 ` Yu, Fenghua
2017-10-21 15:19 ` [tip:x86/cache] " tip-bot for Reinette Chatre
1 sibling, 0 replies; 9+ messages in thread
From: Yu, Fenghua @ 2017-10-20 18:35 UTC (permalink / raw)
To: Chatre, Reinette, tglx
Cc: Luck, Tony, vikas.shivappa, mingo, hpa, x86, linux-kernel
> From: Chatre, Reinette on Friday, October 20, 2017 2:17 AM
> Subject: [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl
> unmount
>
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [tip:x86/cache] x86/intel_rdt: Fix potential deadlock during resctrl unmount
2017-10-20 9:16 ` [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount Reinette Chatre
2017-10-20 18:35 ` Yu, Fenghua
@ 2017-10-21 15:19 ` tip-bot for Reinette Chatre
1 sibling, 0 replies; 9+ messages in thread
From: tip-bot for Reinette Chatre @ 2017-10-21 15:19 UTC (permalink / raw)
To: linux-tip-commits
Cc: sai.praneeth.prakhya, tony.luck, fenghua.yu, mingo,
vikas.shivappa, reinette.chatre, linux-kernel, hpa, tglx
Commit-ID: 36b6f9fcb8928c06b6638a4cf91bc9d69bb49aa2
Gitweb: https://git.kernel.org/tip/36b6f9fcb8928c06b6638a4cf91bc9d69bb49aa2
Author: Reinette Chatre <reinette.chatre@intel.com>
AuthorDate: Fri, 20 Oct 2017 02:16:58 -0700
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 21 Oct 2017 17:12:21 +0200
x86/intel_rdt: Fix potential deadlock during resctrl unmount
Lockdep warns about a potential deadlock:
[ 66.782842] ======================================================
[ 66.782888] WARNING: possible circular locking dependency detected
[ 66.782937] 4.14.0-rc2-test-test+ #48 Not tainted
[ 66.782983] ------------------------------------------------------
[ 66.783052] umount/336 is trying to acquire lock:
[ 66.783117] (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff81032395>] rdt_kill_sb+0x215/0x390
[ 66.783193]
but task is already holding lock:
[ 66.783244] (rdtgroup_mutex){+.+.}, at: [<ffffffff810321b6>] rdt_kill_sb+0x36/0x390
[ 66.783305]
which lock already depends on the new lock.
[ 66.783364]
the existing dependency chain (in reverse order) is:
[ 66.783419]
-> #3 (rdtgroup_mutex){+.+.}:
[ 66.783467] __lock_acquire+0x1293/0x13f0
[ 66.783509] lock_acquire+0xaf/0x220
[ 66.783543] __mutex_lock+0x71/0x9b0
[ 66.783575] mutex_lock_nested+0x1b/0x20
[ 66.783610] intel_rdt_online_cpu+0x3b/0x430
[ 66.783649] cpuhp_invoke_callback+0xab/0x8e0
[ 66.783687] cpuhp_thread_fun+0x7a/0x150
[ 66.783722] smpboot_thread_fn+0x1cc/0x270
[ 66.783764] kthread+0x16e/0x190
[ 66.783794] ret_from_fork+0x27/0x40
[ 66.783825]
-> #2 (cpuhp_state){+.+.}:
[ 66.783870] __lock_acquire+0x1293/0x13f0
[ 66.783906] lock_acquire+0xaf/0x220
[ 66.783938] cpuhp_issue_call+0x102/0x170
[ 66.783974] __cpuhp_setup_state_cpuslocked+0x154/0x2a0
[ 66.784023] __cpuhp_setup_state+0xc7/0x170
[ 66.784061] page_writeback_init+0x43/0x67
[ 66.784097] pagecache_init+0x43/0x4a
[ 66.784131] start_kernel+0x3ad/0x3f7
[ 66.784165] x86_64_start_reservations+0x2a/0x2c
[ 66.784204] x86_64_start_kernel+0x72/0x75
[ 66.784241] verify_cpu+0x0/0xfb
[ 66.784270]
-> #1 (cpuhp_state_mutex){+.+.}:
[ 66.784319] __lock_acquire+0x1293/0x13f0
[ 66.784355] lock_acquire+0xaf/0x220
[ 66.784387] __mutex_lock+0x71/0x9b0
[ 66.784419] mutex_lock_nested+0x1b/0x20
[ 66.784454] __cpuhp_setup_state_cpuslocked+0x52/0x2a0
[ 66.784497] __cpuhp_setup_state+0xc7/0x170
[ 66.784535] page_alloc_init+0x28/0x30
[ 66.784569] start_kernel+0x148/0x3f7
[ 66.784602] x86_64_start_reservations+0x2a/0x2c
[ 66.784642] x86_64_start_kernel+0x72/0x75
[ 66.784678] verify_cpu+0x0/0xfb
[ 66.784707]
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
[ 66.784759] check_prev_add+0x32f/0x6e0
[ 66.784794] __lock_acquire+0x1293/0x13f0
[ 66.784830] lock_acquire+0xaf/0x220
[ 66.784863] cpus_read_lock+0x3d/0xb0
[ 66.784896] rdt_kill_sb+0x215/0x390
[ 66.784930] deactivate_locked_super+0x3e/0x70
[ 66.784968] deactivate_super+0x40/0x60
[ 66.785003] cleanup_mnt+0x3f/0x80
[ 66.785034] __cleanup_mnt+0x12/0x20
[ 66.785070] task_work_run+0x8b/0xc0
[ 66.785103] exit_to_usermode_loop+0x94/0xa0
[ 66.786804] syscall_return_slowpath+0xe8/0x150
[ 66.788502] entry_SYSCALL_64_fastpath+0xab/0xad
[ 66.790194]
other info that might help us debug this:
[ 66.795139] Chain exists of:
cpu_hotplug_lock.rw_sem --> cpuhp_state --> rdtgroup_mutex
[ 66.800035] Possible unsafe locking scenario:
[ 66.803267] CPU0 CPU1
[ 66.804867] ---- ----
[ 66.806443] lock(rdtgroup_mutex);
[ 66.808002] lock(cpuhp_state);
[ 66.809565] lock(rdtgroup_mutex);
[ 66.811110] lock(cpu_hotplug_lock.rw_sem);
[ 66.812608]
*** DEADLOCK ***
[ 66.816983] 2 locks held by umount/336:
[ 66.818418] #0: (&type->s_umount_key#35){+.+.}, at: [<ffffffff81229738>] deactivate_super+0x38/0x60
[ 66.819922] #1: (rdtgroup_mutex){+.+.}, at: [<ffffffff810321b6>] rdt_kill_sb+0x36/0x390
When the resctrl filesystem is unmounted the locks should be obtain in the
locks in the same order as was done when the cpus came online:
cpu_hotplug_lock before rdtgroup_mutex.
This also requires to switch the static_branch_disable() calls to the
_cpulocked variant because now cpu hotplug lock is held already.
[ tglx: Switched to cpus_read_[un]lock ]
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
Acked-by: Tony Luck <tony.luck@intel.com>
Link: https://lkml.kernel.org/r/cc292e76be073f7260604651711c47b09fd0dc81.1508490116.git.reinette.chatre@intel.com
---
arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index 8a61b20..8ce5d03 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -1364,9 +1364,7 @@ static void rmdir_all_sub(void)
kfree(rdtgrp);
}
/* Notify online CPUs to update per cpu storage and PQR_ASSOC MSR */
- get_online_cpus();
update_closid_rmid(cpu_online_mask, &rdtgroup_default);
- put_online_cpus();
kernfs_remove(kn_info);
kernfs_remove(kn_mongrp);
@@ -1377,6 +1375,7 @@ static void rdt_kill_sb(struct super_block *sb)
{
struct rdt_resource *r;
+ cpus_read_lock();
mutex_lock(&rdtgroup_mutex);
/*Put everything back to default values. */
@@ -1384,11 +1383,12 @@ static void rdt_kill_sb(struct super_block *sb)
reset_all_ctrls(r);
cdp_disable();
rmdir_all_sub();
- static_branch_disable(&rdt_alloc_enable_key);
- static_branch_disable(&rdt_mon_enable_key);
- static_branch_disable(&rdt_enable_key);
+ static_branch_disable_cpuslocked(&rdt_alloc_enable_key);
+ static_branch_disable_cpuslocked(&rdt_mon_enable_key);
+ static_branch_disable_cpuslocked(&rdt_enable_key);
kernfs_kill_sb(sb);
mutex_unlock(&rdtgroup_mutex);
+ cpus_read_unlock();
}
static struct file_system_type rdt_fs_type = {
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount
2017-10-20 9:16 [PATCH 0/3] x86/intel_rdt: Locking and initialization fixes Reinette Chatre
2017-10-20 9:16 ` [PATCH 1/3] x86/intel_rdt: Initialize bitmask of shareable resource if CDP enabled Reinette Chatre
2017-10-20 9:16 ` [PATCH 2/3] x86/intel_rdt: Fix potential deadlock during resctrl unmount Reinette Chatre
@ 2017-10-20 9:16 ` Reinette Chatre
2017-10-20 18:41 ` Yu, Fenghua
2017-10-21 15:19 ` [tip:x86/cache] " tip-bot for Reinette Chatre
2 siblings, 2 replies; 9+ messages in thread
From: Reinette Chatre @ 2017-10-20 9:16 UTC (permalink / raw)
To: tglx
Cc: fenghua.yu, tony.luck, vikas.shivappa, mingo, hpa, x86,
linux-kernel, Reinette Chatre
Sai reported a warning during some MBA tests:
[ 236.755559] ======================================================
[ 236.762443] WARNING: possible circular locking dependency detected
[ 236.769328] 4.14.0-rc4-yocto-standard #8 Not tainted
[ 236.774857] ------------------------------------------------------
[ 236.781738] mount/10091 is trying to acquire lock:
[ 236.787071] (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8117f892>] static_key_enable+0x12/0x30
[ 236.797058]
but task is already holding lock:
[ 236.803552] (&type->s_umount_key#37/1){+.+.}, at: [<ffffffff81208b2f>] sget_userns+0x32f/0x520
[ 236.813247]
which lock already depends on the new lock.
[ 236.822353]
the existing dependency chain (in reverse order) is:
[ 236.830686]
-> #4 (&type->s_umount_key#37/1){+.+.}:
[ 236.837756] __lock_acquire+0x1100/0x11a0
[ 236.842799] lock_acquire+0xdf/0x1d0
[ 236.847363] down_write_nested+0x46/0x80
[ 236.852310] sget_userns+0x32f/0x520
[ 236.856873] kernfs_mount_ns+0x7e/0x1f0
[ 236.861728] rdt_mount+0x30c/0x440
[ 236.866096] mount_fs+0x38/0x150
[ 236.870262] vfs_kern_mount+0x67/0x150
[ 236.875015] do_mount+0x1df/0xd50
[ 236.879286] SyS_mount+0x95/0xe0
[ 236.883464] entry_SYSCALL_64_fastpath+0x18/0xad
[ 236.889183]
-> #3 (rdtgroup_mutex){+.+.}:
[ 236.895292] __lock_acquire+0x1100/0x11a0
[ 236.900337] lock_acquire+0xdf/0x1d0
[ 236.904899] __mutex_lock+0x80/0x8f0
[ 236.909459] mutex_lock_nested+0x1b/0x20
[ 236.914407] intel_rdt_online_cpu+0x3b/0x4a0
[ 236.919745] cpuhp_invoke_callback+0xce/0xb80
[ 236.925177] cpuhp_thread_fun+0x1c5/0x230
[ 236.930222] smpboot_thread_fn+0x11a/0x1e0
[ 236.935362] kthread+0x152/0x190
[ 236.939536] ret_from_fork+0x27/0x40
[ 236.944097]
-> #2 (cpuhp_state-up){+.+.}:
[ 236.950199] __lock_acquire+0x1100/0x11a0
[ 236.955241] lock_acquire+0xdf/0x1d0
[ 236.959800] cpuhp_issue_call+0x12e/0x1c0
[ 236.964845] __cpuhp_setup_state_cpuslocked+0x13b/0x2f0
[ 236.971242] __cpuhp_setup_state+0xa7/0x120
[ 236.976483] page_writeback_init+0x43/0x67
[ 236.981623] pagecache_init+0x38/0x3b
[ 236.986281] start_kernel+0x3c6/0x41a
[ 236.990931] x86_64_start_reservations+0x2a/0x2c
[ 236.996650] x86_64_start_kernel+0x72/0x75
[ 237.001793] verify_cpu+0x0/0xfb
[ 237.005966]
-> #1 (cpuhp_state_mutex){+.+.}:
[ 237.012364] __lock_acquire+0x1100/0x11a0
[ 237.017408] lock_acquire+0xdf/0x1d0
[ 237.021969] __mutex_lock+0x80/0x8f0
[ 237.026527] mutex_lock_nested+0x1b/0x20
[ 237.031475] __cpuhp_setup_state_cpuslocked+0x54/0x2f0
[ 237.037777] __cpuhp_setup_state+0xa7/0x120
[ 237.043013] page_alloc_init+0x28/0x30
[ 237.047769] start_kernel+0x148/0x41a
[ 237.052425] x86_64_start_reservations+0x2a/0x2c
[ 237.058145] x86_64_start_kernel+0x72/0x75
[ 237.063284] verify_cpu+0x0/0xfb
[ 237.067456]
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
[ 237.074436] check_prev_add+0x401/0x800
[ 237.079286] __lock_acquire+0x1100/0x11a0
[ 237.084330] lock_acquire+0xdf/0x1d0
[ 237.088890] cpus_read_lock+0x42/0x90
[ 237.093548] static_key_enable+0x12/0x30
[ 237.098496] rdt_mount+0x406/0x440
[ 237.102862] mount_fs+0x38/0x150
[ 237.107035] vfs_kern_mount+0x67/0x150
[ 237.111787] do_mount+0x1df/0xd50
[ 237.116058] SyS_mount+0x95/0xe0
[ 237.120233] entry_SYSCALL_64_fastpath+0x18/0xad
[ 237.125952]
other info that might help us debug this:
[ 237.134867] Chain exists of:
cpu_hotplug_lock.rw_sem --> rdtgroup_mutex --> &type->s_umount_key#37/1
[ 237.148425] Possible unsafe locking scenario:
[ 237.155015] CPU0 CPU1
[ 237.160057] ---- ----
[ 237.165100] lock(&type->s_umount_key#37/1);
[ 237.169952] lock(rdtgroup_mutex);
[ 237.176641]
lock(&type->s_umount_key#37/1);
[ 237.184287] lock(cpu_hotplug_lock.rw_sem);
[ 237.189041]
*** DEADLOCK ***
When the resctrl filesystem is mounted we should obtain the locks in the
same order as was done when the cpus came online, cpu_hotplug_lock
before rdtgroup_mutex. Since we now obtain cpu_hotplug_lock earlier we
modify the static_branch_enable calls to the cpuslocked variants.
Reported-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
---
arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index fc27f55e90dc..26c7602512a0 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -1149,6 +1149,7 @@ static struct dentry *rdt_mount(struct file_system_type *fs_type,
struct dentry *dentry;
int ret;
+ get_online_cpus();
mutex_lock(&rdtgroup_mutex);
/*
* resctrl file system can only be mounted once.
@@ -1198,12 +1199,12 @@ static struct dentry *rdt_mount(struct file_system_type *fs_type,
goto out_mondata;
if (rdt_alloc_capable)
- static_branch_enable(&rdt_alloc_enable_key);
+ static_branch_enable_cpuslocked(&rdt_alloc_enable_key);
if (rdt_mon_capable)
- static_branch_enable(&rdt_mon_enable_key);
+ static_branch_enable_cpuslocked(&rdt_mon_enable_key);
if (rdt_alloc_capable || rdt_mon_capable)
- static_branch_enable(&rdt_enable_key);
+ static_branch_enable_cpuslocked(&rdt_enable_key);
if (is_mbm_enabled()) {
r = &rdt_resources_all[RDT_RESOURCE_L3];
@@ -1226,6 +1227,7 @@ static struct dentry *rdt_mount(struct file_system_type *fs_type,
out:
rdt_last_cmd_clear();
mutex_unlock(&rdtgroup_mutex);
+ put_online_cpus();
return dentry;
}
--
2.13.5
^ permalink raw reply [flat|nested] 9+ messages in thread* RE: [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount
2017-10-20 9:16 ` [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount Reinette Chatre
@ 2017-10-20 18:41 ` Yu, Fenghua
2017-10-21 15:19 ` [tip:x86/cache] " tip-bot for Reinette Chatre
1 sibling, 0 replies; 9+ messages in thread
From: Yu, Fenghua @ 2017-10-20 18:41 UTC (permalink / raw)
To: Chatre, Reinette, tglx
Cc: Luck, Tony, vikas.shivappa, mingo, hpa, x86, linux-kernel
> From: Chatre, Reinette on Friday, October 20, 2017 2:17 AM
> Subject: [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl
> mount
Acked-by: Fenghua Yu <fenghua.yu@intel.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [tip:x86/cache] x86/intel_rdt: Fix potential deadlock during resctrl mount
2017-10-20 9:16 ` [PATCH 3/3] x86/intel_rdt: Fix potential deadlock during resctrl mount Reinette Chatre
2017-10-20 18:41 ` Yu, Fenghua
@ 2017-10-21 15:19 ` tip-bot for Reinette Chatre
1 sibling, 0 replies; 9+ messages in thread
From: tip-bot for Reinette Chatre @ 2017-10-21 15:19 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, mingo, vikas.shivappa, reinette.chatre, tglx,
sai.praneeth.prakhya, linux-kernel
Commit-ID: 87943db7dfb0c5ee5aa74a9ac06346fadd9695c8
Gitweb: https://git.kernel.org/tip/87943db7dfb0c5ee5aa74a9ac06346fadd9695c8
Author: Reinette Chatre <reinette.chatre@intel.com>
AuthorDate: Fri, 20 Oct 2017 02:16:59 -0700
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 21 Oct 2017 17:12:22 +0200
x86/intel_rdt: Fix potential deadlock during resctrl mount
Sai reported a warning during some MBA tests:
[ 236.755559] ======================================================
[ 236.762443] WARNING: possible circular locking dependency detected
[ 236.769328] 4.14.0-rc4-yocto-standard #8 Not tainted
[ 236.774857] ------------------------------------------------------
[ 236.781738] mount/10091 is trying to acquire lock:
[ 236.787071] (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8117f892>] static_key_enable+0x12/0x30
[ 236.797058]
but task is already holding lock:
[ 236.803552] (&type->s_umount_key#37/1){+.+.}, at: [<ffffffff81208b2f>] sget_userns+0x32f/0x520
[ 236.813247]
which lock already depends on the new lock.
[ 236.822353]
the existing dependency chain (in reverse order) is:
[ 236.830686]
-> #4 (&type->s_umount_key#37/1){+.+.}:
[ 236.837756] __lock_acquire+0x1100/0x11a0
[ 236.842799] lock_acquire+0xdf/0x1d0
[ 236.847363] down_write_nested+0x46/0x80
[ 236.852310] sget_userns+0x32f/0x520
[ 236.856873] kernfs_mount_ns+0x7e/0x1f0
[ 236.861728] rdt_mount+0x30c/0x440
[ 236.866096] mount_fs+0x38/0x150
[ 236.870262] vfs_kern_mount+0x67/0x150
[ 236.875015] do_mount+0x1df/0xd50
[ 236.879286] SyS_mount+0x95/0xe0
[ 236.883464] entry_SYSCALL_64_fastpath+0x18/0xad
[ 236.889183]
-> #3 (rdtgroup_mutex){+.+.}:
[ 236.895292] __lock_acquire+0x1100/0x11a0
[ 236.900337] lock_acquire+0xdf/0x1d0
[ 236.904899] __mutex_lock+0x80/0x8f0
[ 236.909459] mutex_lock_nested+0x1b/0x20
[ 236.914407] intel_rdt_online_cpu+0x3b/0x4a0
[ 236.919745] cpuhp_invoke_callback+0xce/0xb80
[ 236.925177] cpuhp_thread_fun+0x1c5/0x230
[ 236.930222] smpboot_thread_fn+0x11a/0x1e0
[ 236.935362] kthread+0x152/0x190
[ 236.939536] ret_from_fork+0x27/0x40
[ 236.944097]
-> #2 (cpuhp_state-up){+.+.}:
[ 236.950199] __lock_acquire+0x1100/0x11a0
[ 236.955241] lock_acquire+0xdf/0x1d0
[ 236.959800] cpuhp_issue_call+0x12e/0x1c0
[ 236.964845] __cpuhp_setup_state_cpuslocked+0x13b/0x2f0
[ 236.971242] __cpuhp_setup_state+0xa7/0x120
[ 236.976483] page_writeback_init+0x43/0x67
[ 236.981623] pagecache_init+0x38/0x3b
[ 236.986281] start_kernel+0x3c6/0x41a
[ 236.990931] x86_64_start_reservations+0x2a/0x2c
[ 236.996650] x86_64_start_kernel+0x72/0x75
[ 237.001793] verify_cpu+0x0/0xfb
[ 237.005966]
-> #1 (cpuhp_state_mutex){+.+.}:
[ 237.012364] __lock_acquire+0x1100/0x11a0
[ 237.017408] lock_acquire+0xdf/0x1d0
[ 237.021969] __mutex_lock+0x80/0x8f0
[ 237.026527] mutex_lock_nested+0x1b/0x20
[ 237.031475] __cpuhp_setup_state_cpuslocked+0x54/0x2f0
[ 237.037777] __cpuhp_setup_state+0xa7/0x120
[ 237.043013] page_alloc_init+0x28/0x30
[ 237.047769] start_kernel+0x148/0x41a
[ 237.052425] x86_64_start_reservations+0x2a/0x2c
[ 237.058145] x86_64_start_kernel+0x72/0x75
[ 237.063284] verify_cpu+0x0/0xfb
[ 237.067456]
-> #0 (cpu_hotplug_lock.rw_sem){++++}:
[ 237.074436] check_prev_add+0x401/0x800
[ 237.079286] __lock_acquire+0x1100/0x11a0
[ 237.084330] lock_acquire+0xdf/0x1d0
[ 237.088890] cpus_read_lock+0x42/0x90
[ 237.093548] static_key_enable+0x12/0x30
[ 237.098496] rdt_mount+0x406/0x440
[ 237.102862] mount_fs+0x38/0x150
[ 237.107035] vfs_kern_mount+0x67/0x150
[ 237.111787] do_mount+0x1df/0xd50
[ 237.116058] SyS_mount+0x95/0xe0
[ 237.120233] entry_SYSCALL_64_fastpath+0x18/0xad
[ 237.125952]
other info that might help us debug this:
[ 237.134867] Chain exists of:
cpu_hotplug_lock.rw_sem --> rdtgroup_mutex --> &type->s_umount_key#37/1
[ 237.148425] Possible unsafe locking scenario:
[ 237.155015] CPU0 CPU1
[ 237.160057] ---- ----
[ 237.165100] lock(&type->s_umount_key#37/1);
[ 237.169952] lock(rdtgroup_mutex);
[ 237.176641]
lock(&type->s_umount_key#37/1);
[ 237.184287] lock(cpu_hotplug_lock.rw_sem);
[ 237.189041]
*** DEADLOCK ***
When the resctrl filesystem is mounted the locks must be acquired in the
same order as was done when the cpus came online:
cpu_hotplug_lock before rdtgroup_mutex.
This also requires to switch the static_branch_enable() calls to the
_cpulocked variant because now cpu hotplug lock is held already.
[ tglx: Switched to cpus_read_[un]lock ]
Reported-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Acked-by: Vikas Shivappa <vikas.shivappa@linux.intel.com>
Cc: fenghua.yu@intel.com
Cc: tony.luck@intel.com
Link: https://lkml.kernel.org/r/9c41b91bc2f47d9e95b62b213ecdb45623c47a9f.1508490116.git.reinette.chatre@intel.com
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/kernel/cpu/intel_rdt_rdtgroup.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index 8ce5d03..64c5ff9 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -1149,6 +1149,7 @@ static struct dentry *rdt_mount(struct file_system_type *fs_type,
struct dentry *dentry;
int ret;
+ cpus_read_lock();
mutex_lock(&rdtgroup_mutex);
/*
* resctrl file system can only be mounted once.
@@ -1198,12 +1199,12 @@ static struct dentry *rdt_mount(struct file_system_type *fs_type,
goto out_mondata;
if (rdt_alloc_capable)
- static_branch_enable(&rdt_alloc_enable_key);
+ static_branch_enable_cpuslocked(&rdt_alloc_enable_key);
if (rdt_mon_capable)
- static_branch_enable(&rdt_mon_enable_key);
+ static_branch_enable_cpuslocked(&rdt_mon_enable_key);
if (rdt_alloc_capable || rdt_mon_capable)
- static_branch_enable(&rdt_enable_key);
+ static_branch_enable_cpuslocked(&rdt_enable_key);
if (is_mbm_enabled()) {
r = &rdt_resources_all[RDT_RESOURCE_L3];
@@ -1226,6 +1227,7 @@ out_cdp:
out:
rdt_last_cmd_clear();
mutex_unlock(&rdtgroup_mutex);
+ cpus_read_unlock();
return dentry;
}
^ permalink raw reply [flat|nested] 9+ messages in thread