From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760756AbdKQS21 (ORCPT ); Fri, 17 Nov 2017 13:28:27 -0500 Received: from mail-db5eur01on0123.outbound.protection.outlook.com ([104.47.2.123]:12873 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1760684AbdKQS1s (ORCPT ); Fri, 17 Nov 2017 13:27:48 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=ktkhai@virtuozzo.com; Subject: [PATCH RFC 03/25] net: Introduce net_sem for protection of pernet_list From: Kirill Tkhai To: davem@davemloft.net, vyasevic@redhat.com, kstewart@linuxfoundation.org, pombredanne@nexb.com, vyasevich@gmail.com, mark.rutland@arm.com, gregkh@linuxfoundation.org, adobriyan@gmail.com, fw@strlen.de, nicolas.dichtel@6wind.com, xiyou.wangcong@gmail.com, roman.kapl@sysgo.com, paul@paul-moore.com, dsahern@gmail.com, daniel@iogearbox.net, lucien.xin@gmail.com, mschiffer@universe-factory.net, rshearma@brocade.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, ktkhai@virtuozzo.com, ebiederm@xmission.com, avagin@virtuozzo.com, gorcunov@virtuozzo.com, eric.dumazet@gmail.com, stephen@networkplumber.org, ktkhai@virtuozzo.com Date: Fri, 17 Nov 2017 21:27:41 +0300 Message-ID: <151094326157.20009.5480759391073404834.stgit@localhost.localdomain> In-Reply-To: <151094119999.20009.6955267140148739392.stgit@localhost.localdomain> References: <151094119999.20009.6955267140148739392.stgit@localhost.localdomain> User-Agent: StGit/0.18 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [195.214.232.6] X-ClientProxiedBy: HE1PR0701CA0071.eurprd07.prod.outlook.com (2603:10a6:3:64::15) To HE1PR0801MB1338.eurprd08.prod.outlook.com (2603:10a6:3:39::28) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4f7c8b41-c28c-4bf8-d470-08d52de8e558 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(4534020)(4602075)(7168020)(4627115)(201703031133081)(201702281549075)(2017052603199);SRVR:HE1PR0801MB1338; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;3:so9V4NUsbuQ4xIpXffV5jBPB8GkaVCv7UIssWXsaahQ4GxcFrp+AbK7/odCBh1SDiSWBYBcIOjr2xg86Wx0TlOtJjbaH9jR88m0pq8d2nqQGFkqHLLXzZkk6Y1GLnjXWjB5JpTAcnpW8Mx93yI1bhm+SUJiHXgbkfomhHVzd0eekIEWvY4ZHBx5IbpuqHgwT9XNCmdUOy+KdLFHUjapbtdFbLCD7TBQeTQKyWxeiVgEJEbyYrhUp8rMwoMOguu2q;25:Y6uuddXAM03hZWne4aM4+mC44oQhtPMN+Wrx4atmRDivLpRrZ0ADjF8uQKTsZVhz/FdYA8VlJjpptTQ37dDr99aaaK+WXNr683+R+FRf6sc167yRUu5syNTVMQgLK/UU4imdbollXRZ2auhmBq+BCl4u25tQcW/31qdU9IRbjSvpexrpLCOyviX7loFUI2lAH/Uihwi0MLdlc/ytRr8ztynvzWnabb5ul3uE1sYNahR/5EHr9rs99eZWKSie3d5b6SQoSeYjcexkvpuzf3ss8jjK5NT4UzQW/MV9GMXdi8GppmSc+gFU1WBn4lnC3aTblpe96tJCwNpVrV97rrtX9A==;31:ftt98xUCwOJWhPPD0yHiZLoMO9jCvFiClLlwOtLtVR0Qbq5YC3umv6ZuTZd1++dA0C7VQKMi1/hgsjgQy+T0nIk7IEt43ACHEjhHNX+gzw49VWNtRSkfrc66C5b/UZuIDQuD9xQC+0CGfAtR8Z5hZ5LKWduxqqk0C3VTa9BBqnuANLEq8P/TZ6xWPwwoSQ2otRCDsRotQ/ZGgZQZYVNrhtowOuhvABWLXT8JbmuDm/o= X-MS-TrafficTypeDiagnostic: HE1PR0801MB1338: X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;20:YrwYGhNQUxS1Ck29yJP7WWcdNnrGHXUq85IsOFCi9PgdBvyl0CCCNKwRgBKQj2oxwoOA+IDN0I6eDiLj1i24ryo0GmJdU9DJ4pNwht5aA0j/tUHB4G0jR1iobFtuUX0I4WimPNAtDVlmJ8waNB+Mx3Wa/DB0G5sdoGbksfX7+yHQWDwYtrW1i76K4hP1YEkrj+4CSMp+F+xh7x6Fi3Ib88ni+LoIwIvSRxa17y9xOLVVdjWYDUi2DX5OptXImL9GHxDp4a2AobHbNt3tFHGSLscfin63k12GJYDzfJt9bVtFqCMnvAsN2tnl98Hl+cUTDXlhExwKxj849Z93QbIVtPoIuz1BkOExUwD0TavxWchc9I3WxRvo6NS8LtfULomFmKLgfMVcbb6EPeZoRzOT6VJA0BnOr3vQv00qlVr1BbA=;4:oe+P2UNQh3xE60ViLVu6cxN9bBlRxWvgCZjAUHVz/m82NhWruqyww4E1/5fHMKiTi6/QGKo1xz2RtWZHzOs7W9kDXtPicOR3pavdreW+VK3HgSfOM/cShw6s1rTSp2YSpLfm0iL9WOZgOrkVzKwg2rW1o74YcApRdUXSwUWe7Hq1nia4CaulKjwg9ZnUwHTyd35/OZwQpl+aSO7TD1cuWD8TIZLUixNIQwdAzaCgvrdePadSkGhuqgtmOZVQxYz/RbTB0nSBF/VEfctP9CVBgA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231022)(10201501046)(3002001)(100000703101)(100105400095)(6041248)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123560025)(20161123555025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:HE1PR0801MB1338;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:HE1PR0801MB1338; X-Forefront-PRVS: 049486C505 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(6069001)(6009001)(346002)(376002)(199003)(189002)(86362001)(53936002)(103116003)(39060400002)(6306002)(316002)(8936002)(2906002)(9686003)(55016002)(16526018)(55236003)(189998001)(50466002)(6506006)(47776003)(76176999)(54356999)(7416002)(50986999)(66066001)(83506002)(5660300001)(2950100002)(305945005)(3846002)(6116002)(33646002)(97736004)(230700001)(58126008)(8676002)(7736002)(68736007)(61506002)(478600001)(106356001)(81166006)(105586002)(81156014)(23676003)(101416001)(25786009)(966005)(921003)(1121003)(217873001);DIR:OUT;SFP:1102;SCL:1;SRVR:HE1PR0801MB1338;H:localhost.localdomain;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtIRTFQUjA4MDFNQjEzMzg7MjM6ZHRmZ0ZkV3hpMWljcWR6SmRIVmt5TDcw?= =?utf-8?B?RzYvQ0pwUmVzdlB3VUlIODF2cU81ZUpSSXhyRjkwOGtMdmEwYjRHb2R6ZUZL?= =?utf-8?B?V05nMU1UWFlaaUw1R0NIY0JwZktaamdKaW4zd2dWQnNqd0FVS1ZFcWY1eFZ6?= =?utf-8?B?S0JCT2l2RkhTUkNLcGY4U3JZSVFSOVpSTFRIbU5kdmhCSDhkdXlieDlSK1lH?= =?utf-8?B?aHlzdGQxVlAvYklKZHdSNDJ4aHk2NTBHVlVnMTJ4bTlvajVMMCtBZ0psb2FE?= =?utf-8?B?RnFmbWR2MWF2SVhYOUplZHlwNWhyakVEWDRScnJRNGdHN2l4WVZYVDVYMDhN?= =?utf-8?B?bjAxeEtKSnlzRGU2K3JOY0VCeklTd09ZYTc0Rk05NjVWUXpKQkhwUkk1eFdt?= =?utf-8?B?dmlBdE9pTWQ0M2dKMS8xNE45djRMZ0lFS25oTC93ZHg5L3NVUXhiK29JQUJQ?= =?utf-8?B?bk1Gc1pmeldUdHZWL1I3SHBvSnV5QVVoQTFGNFBWVmJiYTZ5NmZuSlB2dGpk?= =?utf-8?B?YmxmNGNRempneHp3NVRhSEh6Zk00OFNuMVdONWZlUHZ4TzR2dkxIcjRMUVRs?= =?utf-8?B?TDZUQXo1NkxTYWFONWpka1oxTFZRUnpqNjA0b1g2aFhTZTNpK3d5YzlnYmhh?= =?utf-8?B?TWw1bG94NnhmRUcxOG96alp2MUwzaUZLTmZyQkhDdTFUZm5MYmszdGtuWGVU?= =?utf-8?B?Qi9ObThKbXZER2dRemZZcE1yYXR0cE5PZCtZVm5VZEQzRjBTRElwZTZHZkIx?= =?utf-8?B?OTk0ZE5BSkVPL094emZIN0NkandwMExZelQ1SzM0eWc4bG83WU5nSWRPMnE2?= =?utf-8?B?Z2ZYUGZmREZFRmFDN05WNHNVN3d3Y1JnQ1Ricm50eERpT3ZFSExpb3c3SXFl?= =?utf-8?B?UmNjQllmTnFVUWE4aFpUM0V0Z0NKTkJadGxMdHBGbEM2SXFVYTZkNjBuL2ZE?= =?utf-8?B?Wm1PNVkya0haUFdZWXVnZzZPSkZXdnZVTlU5T0IxRUtlaXpucUQ4alUySVRk?= =?utf-8?B?MzBqOWVOcnlEb1FPeGx2Z2VqZU4xZlpGdlBQOFlVa0VzVHBHL1FyVHN4TnBj?= =?utf-8?B?TlZsTnNtekZrVFB5YktDYzJmRlp3TERNekFPOHF2ck1LQUowWXVEOW5TRnE0?= =?utf-8?B?TW55VjlQQ2dMUnhlZnZFNUtTMTlQMnpaMnV6d2JvUUFpektnS0VkSDA2WWU4?= =?utf-8?B?dldRaHRBSXFEQTdlbUIvTkV1SmlHS3FZa2RwRDJERzBEU0UyUU05bjlFb2l5?= =?utf-8?B?cE4zYXlZcnEybll2VW5jK0QzV1pDUTdYcGw3T0pHZUx1U3RnUlRsYWhTUUM1?= =?utf-8?B?MS9MSmtNenM4c0ZMUUxKMDhaUVhDU0ZMbGxlR1VIRjNidkU3VFV1Q3hYOFVx?= =?utf-8?B?OEVPTVd4TWNveFFaLzIrV1c1b3ZGekdxclUybGZKK0JJSTJmb25JRmxTSHZ1?= =?utf-8?B?QllDdmg0ODFNL0VpRzlpMnptdTZlOEo1Vit1Rm5LN3F5bkhEcURDYjB0SE1T?= =?utf-8?B?VjdYUlQ1a04yeGRkWEh5bDcwWlQ4LzJyb0dWSzJ1SXRENEloOGJpb2s5bWdL?= =?utf-8?B?dXZ0WXZldWgzaS9TRnIyRjdjMlV3M2d3SmNIcWtnVGYvZm42MVdvcFhLQkFh?= =?utf-8?B?dlZFeURMUGhLOVlOdUh5dXFObXlqL1FnVnBCdEdQejFiSjJqK1NNZDZZcUE9?= =?utf-8?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;6:MqJ0O/QnIdmarYgkR3qkylKsvhbTDwITX39KwzfMj4gklIYzLOTrTVTCN/cfvw+yQ7solBfhXiRKDUafm6qxa83njUp+AE5lsVOrXQQfJdIsnGt7OFZepB8Ms1Mn3jYID0/TojSOe9SEknFdE+Hhmt1XsrP9E6TLOsPL76cZQ3YcVSsfWDGmswLnKxueFaGGeK1pm37lh/t/UcawxfjqgcGwTtVqafalANcS6HX7hMX/lfZqyrnqxlVjaaM70vpIhkd3FztylD3G8Cpv/kDSrxkC95g4VYAk4I3X+nZysnOoogjkfts6BUwI/x8OOJ62vXtCypssDdac1uYn0esROt+wIXHhPf46mJOTQsCObbE=;5:Nfq0f7F4yZJM4/FQ1FbB1VSvHKwiS4xY+RCeIAksq9ALJ3L0m8G1DBse7qIZhunKcPJPFcc8mWRKHTEEGAu28p5CY6OkB1aeeHAkWyfVMv/Ek2zIDVjhewGK4OQd9HFDIqOG9Zqq/KavEtRt5iAG82SRTJUrTM/x3UE1tNDa2jk=;24:R9NvnKw5qFZZPZARvRI4+Qfft0FyXpAtzHdWRNdqTZKhdh9i5eM4SO+fi7A0yusjCQSX1q6OX2V+Dn3un8vRACaYNtBydcfBzLIXGO51OAg=;7:P+Gq4qTh0hdazVxH2AaWyib3azUcAKcvfnvV1ckjZt04Hsx0lWmp6IbOeXL7iYg3hctejffDkrIEMJyqe5alnPYCWVgS0jf/iTjL7sDvJQct0AD3XTrO8mTKF4pMukkoLp2tVfRxQIzqB/Q+ce6YfPzRaD0bWAEYnJaF3tnH898RTpqGhI4K276lv/17TIJzhbHX9pS3DWFnrKsR/79n3LlniAQOXgQm2KUAO1H5p2jmqIN7kFdJIOEljHSXebFO SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;20:3vyjVY8OuxO66kdfAd8Cp9eAfg5BdwwBzGw8kE+N56gwh8fq3722eQifpmyejLH8MVe+9xO2WuxWpxMi/PLg0s376ruavmBNo0cM2RSw8vvsyhipMbHqXtTLrnPLka1wOoHxWdOAcJeEk1arKG9mase/JiJWJCuFME5++kxUSoQ= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2017 18:27:43.8447 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4f7c8b41-c28c-4bf8-d470-08d52de8e558 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1338 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Curently mutex is used to protect pernet operations list. It makes cleanup_net() to execute ->exit methods of the same operations set, which was used on the time of ->init, even after net namespace is unlinked from net_namespace_list. But the problem is it's need to synchronize_rcu() after net is removed from net_namespace_list(): Destroy net_ns: cleanup_net() mutex_lock(&net_mutex) list_del_rcu(&net->list) synchronize_rcu() <--- Sleep there for ages list_for_each_entry_reverse(ops, &pernet_list, list) ops_exit_list(ops, &net_exit_list) list_for_each_entry_reverse(ops, &pernet_list, list) ops_free_list(ops, &net_exit_list) mutex_unlock(&net_mutex) This primitive is not fast, especially on the systems with many processors and/or when preemptible RCU is enabled in config. So, all the time, while cleanup_net() is waiting for RCU grace period, creation of new net namespaces is not possible, the tasks, who makes it, are sleeping on the same mutex: Create net_ns: copy_net_ns() mutex_lock_killable(&net_mutex) <--- Sleep there for ages I observed 20-30 seconds hangs of "unshare -n" on ordinary 8-cpu laptop with preemptible RCU enabled. The solution is to convert net_mutex to the rw_semaphore and add small locks to really small number of pernet_operations, what really need them. Then, pernet_operations::init/::exit methods, modifying the net-related data, will require down_read() locking only, while down_write() will be used for changing pernet_list. This gives signify performance increase, like you may see here: https://www.spinics.net/lists/netdev/msg467095.html It's 4.6 times performance increase on one-thread test. Multi-thread tests increase may be close to 4.6 multiplied to number of threads. This patch starts replacing net_mutex to net_sem. It adds rw_semaphore, describes the variables it protects, and makes to use where appropriate. net_mutex is still present, and next patches will kick it out step-by-step. Signed-off-by: Kirill Tkhai --- include/linux/rtnetlink.h | 1 + net/core/net_namespace.c | 37 +++++++++++++++++++++++++------------ net/core/rtnetlink.c | 4 ++-- 3 files changed, 28 insertions(+), 14 deletions(-) diff --git a/include/linux/rtnetlink.h b/include/linux/rtnetlink.h index 2032ce2eb20b..f640fc87fe1d 100644 --- a/include/linux/rtnetlink.h +++ b/include/linux/rtnetlink.h @@ -35,6 +35,7 @@ extern int rtnl_is_locked(void); extern wait_queue_head_t netdev_unregistering_wq; extern struct mutex net_mutex; +extern struct rw_semaphore net_sem; #ifdef CONFIG_PROVE_LOCKING extern bool lockdep_rtnl_is_held(void); diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index 2e512965bf42..2254b1639209 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -41,6 +41,11 @@ struct net init_net = { EXPORT_SYMBOL(init_net); static bool init_net_initialized; +/* + * net_sem: protects: pernet_list, net_generic_ids, + * init_net_initialized and first_* pointers. + */ +DECLARE_RWSEM(net_sem); #define MIN_PERNET_OPS_ID \ ((sizeof(struct net_generic) + sizeof(void *) - 1) / sizeof(void *)) @@ -411,12 +416,16 @@ struct net *copy_net_ns(unsigned long flags, net->ucounts = ucounts; get_user_ns(user_ns); - rv = mutex_lock_killable(&net_mutex); + rv = down_read_killable(&net_sem); if (rv < 0) goto put_userns; - + rv = mutex_lock_killable(&net_mutex); + if (rv < 0) + goto up_read; rv = setup_net(net, user_ns); mutex_unlock(&net_mutex); +up_read: + up_read(&net_sem); if (rv < 0) { put_userns: put_user_ns(user_ns); @@ -443,6 +452,7 @@ static void cleanup_net(struct work_struct *work) list_replace_init(&cleanup_list, &net_kill_list); spin_unlock_irq(&cleanup_list_lock); + down_read(&net_sem); mutex_lock(&net_mutex); /* Don't let anyone else find us. */ @@ -484,6 +494,7 @@ static void cleanup_net(struct work_struct *work) ops_free_list(ops, &net_exit_list); mutex_unlock(&net_mutex); + up_read(&net_sem); /* Ensure there are no outstanding rcu callbacks using this * network namespace. @@ -510,8 +521,10 @@ static void cleanup_net(struct work_struct *work) */ void net_ns_barrier(void) { + down_write(&net_sem); mutex_lock(&net_mutex); mutex_unlock(&net_mutex); + up_write(&net_sem); } EXPORT_SYMBOL(net_ns_barrier); @@ -838,12 +851,12 @@ static int __init net_ns_init(void) rcu_assign_pointer(init_net.gen, ng); - mutex_lock(&net_mutex); + down_write(&net_sem); if (setup_net(&init_net, &init_user_ns)) panic("Could not setup the initial network namespace"); init_net_initialized = true; - mutex_unlock(&net_mutex); + up_write(&net_sem); register_pernet_subsys(&net_ns_ops); @@ -983,9 +996,9 @@ static void unregister_pernet_operations(struct pernet_operations *ops) int register_pernet_subsys(struct pernet_operations *ops) { int error; - mutex_lock(&net_mutex); + down_write(&net_sem); error = register_pernet_operations(first_device, ops); - mutex_unlock(&net_mutex); + up_write(&net_sem); return error; } EXPORT_SYMBOL_GPL(register_pernet_subsys); @@ -1001,9 +1014,9 @@ EXPORT_SYMBOL_GPL(register_pernet_subsys); */ void unregister_pernet_subsys(struct pernet_operations *ops) { - mutex_lock(&net_mutex); + down_write(&net_sem); unregister_pernet_operations(ops); - mutex_unlock(&net_mutex); + up_write(&net_sem); } EXPORT_SYMBOL_GPL(unregister_pernet_subsys); @@ -1029,11 +1042,11 @@ EXPORT_SYMBOL_GPL(unregister_pernet_subsys); int register_pernet_device(struct pernet_operations *ops) { int error; - mutex_lock(&net_mutex); + down_write(&net_sem); error = register_pernet_operations(&pernet_list, ops); if (!error && (first_device == &pernet_list)) first_device = &ops->list; - mutex_unlock(&net_mutex); + up_write(&net_sem); return error; } EXPORT_SYMBOL_GPL(register_pernet_device); @@ -1049,11 +1062,11 @@ EXPORT_SYMBOL_GPL(register_pernet_device); */ void unregister_pernet_device(struct pernet_operations *ops) { - mutex_lock(&net_mutex); + down_write(&net_sem); if (&ops->list == first_device) first_device = first_device->next; unregister_pernet_operations(ops); - mutex_unlock(&net_mutex); + up_write(&net_sem); } EXPORT_SYMBOL_GPL(unregister_pernet_device); diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index dabba2a91fc8..cb06d43c4230 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -390,11 +390,11 @@ static void rtnl_lock_unregistering_all(void) void rtnl_link_unregister(struct rtnl_link_ops *ops) { /* Close the race with cleanup_net() */ - mutex_lock(&net_mutex); + down_write(&net_sem); rtnl_lock_unregistering_all(); __rtnl_link_unregister(ops); rtnl_unlock(); - mutex_unlock(&net_mutex); + up_write(&net_sem); } EXPORT_SYMBOL_GPL(rtnl_link_unregister);