From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758723AbcHDOPi (ORCPT ); Thu, 4 Aug 2016 10:15:38 -0400 Received: from mail-db5eur01on0099.outbound.protection.outlook.com ([104.47.2.99]:53472 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1757009AbcHDOPK (ORCPT ); Thu, 4 Aug 2016 10:15:10 -0400 X-Greylist: delayed 6302 seconds by postgrey-1.27 at vger.kernel.org; Thu, 04 Aug 2016 10:15:09 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=skinsbursky@virtuozzo.com; Reply-To: Subject: Re: [RFC PATCH] sunrpc: do not allow process to freeze within RPC state machine References: <20160803165412.22407.47399.stgit@localhost.localdomain> <1470245777.13804.34.camel@redhat.com> <1470316606.22052.17.camel@redhat.com> To: Jeff Layton , , , CC: , , , , , From: Stanislav Kinsburskiy Message-ID: <1638fe19-af90-6683-f04e-6fb94b8b8f67@virtuozzo.com> Date: Thu, 4 Aug 2016 15:57:24 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.1.0 MIME-Version: 1.0 In-Reply-To: <1470316606.22052.17.camel@redhat.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [178.19.210.162] X-ClientProxiedBy: AMSPR04CA0060.eurprd04.prod.outlook.com (10.242.87.178) To VI1PR0802MB2462.eurprd08.prod.outlook.com (10.175.26.17) X-MS-Office365-Filtering-Correlation-Id: 3583feb5-eb02-4222-f45f-08d3bc6f4514 X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2462;2:NNWw1L4+TRmvR1gjIytiTtokx8pgj3KbUafP6ElAojhuVp4PM5Q2STwelt8471QgFUy1Q+QZ+inYZKogdkuYfAX6sq71i+dBXNgZD8ThuHTIxs/mslpxTAuGKqs6/x8HLXhLoQkl56YnyjBqIw2ss52EuLtR6kZkqmAf5tPFp6+hZNNZBuB5tF4oCZWR//B5;3:poUE92xpFS10hRMlBNBKIZZxMTrqohml40CKqRrxFzgjCs200F/l/ontJ8Kn3Mn0hN/IAhJT6i2B1mJSqbZltn0i6Z+YiiXm8ypDF9Uh40IMDuvCMlnLKmdr0E1hDxOc;25:zTj4DqEJlvLrOoljzbKN7JMn0Lh5HQbt0A2OyP0LKdiB74ohNY2I1T2GC0wB4xtKSVeRDFBbdgc69fs0zGO0VnhmckHH/f3wci7v1B+A7VncPxvlhM3hhZgjwUBaGsMkHK/8KL0CpPbtiTW9fCDdY0ac0/JtOGhWSnesBzYX2JbXR6rjc2S6kmBaC6GHOkBiq2urhWKxnEJICLRpOOOP8YwbtxAE1t7z/zGVKe0iyo9OZOWpoXARNZZi2fOOF2kyZOj492wcUOKDp02GIrg0aj80wscbVDPlDf6bHxrmzHvJtEuU/AcOfoPjP75W25QbxHuHZzOQNNZEv54nE//G8VTwL2EdOIpDMGSpxxsUhUY5N6TXSnj7MCYrlp+HVtjBpYY4qjqNhNONZgx5pHDc9/ubHo7o05N4aFqzbToQ5gg= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0802MB2462; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2462;31:iLCDLXUtgg+gZDunIhRb+SObWU4rLBczRDKcRpCAsHxYa7/nS0NMQgmo5bYqiifqhJ1JuJz36MpqnQjCaGYmjGOn9GgqVb14mN3ToPq24B0dXTIim824LzY7ZgwXThqU2XX25+6TaeQEOkj/O5H42lRTYNIWcjOSPiJmvQNVn8ijeAXF2cHkJWxz4iNCWE50HPd7jawEA9WGkbktjjuVtS4E/dCTI515e7iQmg3yCFA=;4:QFS7fsWQIxUeUbYXKfUUysDHd+d7qLidffWb/bL5WNrnaFypZhjbYRK7kzIUsKMUOIQd8BG2XCjvrXMoFVLM8fHasbUyKmxbL3qBI92H+ME+YdJGfORMZktnmmog3JG+ks5T4DlCnNEdN/VM/yG+fJ9SN7SSaGUw2p1WngR4Y0J+FAYhM2rMwvBK5ucz/KoT4XeU5jryoFQbPTbO3qs28Yr5zRTnZPnvc9FqBmWUmPcZWrQM6F3a23JXWy3E5B9qJYRmN3I1s8t/S09GfEqsND2JeebbSaeNYewNqnnlM4oysC0l0IBLTx9jKEzlUi2euIDTjxKIrkxQd6Lrr1YKiY1nl6/q8S1sPaPLAEWKn9gG8pAi4/t79fIYR5jqYGtPVzIabbi48axw2X0vee0mrCVPavZyMFaHbXaLqzHvmxChy5gW8NqQ5jA1utkiC3/qDi/ppsvSPLi4GKK0cL2JDoVsiEyASTXXckVpEAG95T5ztP7vbs8EAd+X8KxMfbUc X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(20558992708506); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040130)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041072)(6043046);SRVR:VI1PR0802MB2462;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0802MB2462; X-Forefront-PRVS: 00246AB517 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(7916002)(377424004)(189002)(199003)(24454002)(3450700001)(101416001)(105586002)(54356999)(23676002)(53806999)(4001350100001)(76176999)(31696002)(81166006)(5001770100001)(586003)(43066003)(107886002)(97736004)(4001430100002)(66066001)(50986999)(77096005)(2870700001)(2950100001)(189998001)(2201001)(2906002)(93886004)(64126003)(7846002)(68736007)(6116002)(36756003)(7736002)(47776003)(50466002)(19580405001)(19580395003)(31686004)(83506001)(106356001)(305945005)(4326007)(3846002)(42186005)(33646002)(8676002)(81156014)(86362001);DIR:OUT;SFP:1102;SCL:1;SRVR:VI1PR0802MB2462;H:[10.134.193.232];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA4MDJNQjI0NjI7MjM6ZUc1bVpaR2w2UTl1anVYZkRhem9HOTla?= =?utf-8?B?a2ZLRE5oa0R0THgza2xyWCtveUtHK1ljL3VJbXcvTXFsaXVzWWJ2b3R1K2Nk?= =?utf-8?B?ajFFQk0vWHl4RWlPdWoxUTZQd0dlZzBhTngxZ1VYWGFPbFp3UzdoWkE5VVVF?= =?utf-8?B?NGZQdEcyaWlqMVE0SFpXWkxySVRrYWtsRFdKdDFPaXN0c1pBcVFWWFRnaTND?= =?utf-8?B?NXZsbmx3TE04SzRhU2N1STYwcWdQUFNhaXBPRUxkclpCaEI5all5Mzd2ak1H?= =?utf-8?B?MmVQV0JrWnZkV0pZSFB5UDFCVGZOaTJTUDFxTzlSV0dUdnlwem16SzBsU3Rz?= =?utf-8?B?L0VWVExkRUE1T1V6NnZJY0wzeEp4TW5tWVdwQW0zL3h0Q0ZmZlBMdm1KdFRy?= =?utf-8?B?dFZYK1p2azBOMk5lZEJ2UGRUV0ZIbXBpSS9qQW5Hbms4bk8zWjVkQWhkOFdl?= =?utf-8?B?RmVJcVdEdU5YOUgxVVhEajh5NmUyaFM0TzBQRkdoYk9BSDRXWWN4SGtCR1RW?= =?utf-8?B?MjJmbWNra0hHeHhvdnRuTHViNld6K2hhcE95Z3crMzdJOTFZMzIwSDhYVnNO?= =?utf-8?B?V3lxU2p4Y2JSLzBnS2dFTWV4K1RNVmlxbElrZkxYT2ZoaGhXWlJOaHFDMEdI?= =?utf-8?B?OVlCODA1bzlXdHZuL1RNeHp4RlNPZG5iczIyY21vVnMvKzVJM3Iwb2JxM1VZ?= =?utf-8?B?WUN5eVpmM2gyUE1mVE0xZGhXYVRBRUwyVVFiWnV1MjBkMnMyY1FmRFZ3S0d3?= =?utf-8?B?aXNXZmR4ZEtPdjhUNFRjL09WdGxZNXR5V1VnQ2xHZTZJWHU1OFRjbDAvd3BO?= =?utf-8?B?dUdncVYwbE5jV2NYRDFtNzAzcEpHMkxKTDQ2QkZIazlyZTJ4UUxIU0tpVFRk?= =?utf-8?B?S3UrT1czZmNueVM4emZBYjVYVmNRa1R0bmlnSDRvTE83MFBIWnlwRVpjbWJY?= =?utf-8?B?Y3hKTWJJRmtsVk5XVXJ6TjdmSExjcTkrR01POXo3aFZnYUNrUUJKclJOYXVZ?= =?utf-8?B?cVkyOGZybXFrMDRWMXg2dFJjRm9Pa1pUZEdHUVNlQTNUYjRUMnBuWlJPK2pQ?= =?utf-8?B?RW9malpObWFZQ2ZuTlZDQ1RXY2h1dDVxejAzcWlZSksvUHc3eXhYSGtnNHFJ?= =?utf-8?B?VHFjckc1UGM1NVRnVHBqd0Rrckp5SEF1ckUyV1JhNHRQRSswVDFiRWppeTFG?= =?utf-8?B?RWlPTWorVzc4cmV6OUg1b0pqaGpGVWozcTFxZlErUjZLNGtneXloajZ4SmlS?= =?utf-8?B?WTJQUU5zUldIcExpY0lWRVFJTitjK3AzRkltZS9Hd2dteEtrRmVlV2ZKSTY3?= =?utf-8?B?eVJxcmhsVTZiZGlZaitrTHAyb20vdlNBREthdDNqcmRYN1BiTE5teGNMaHpi?= =?utf-8?B?ZXlVVTZObEUxZ25xOWVBOTF1SW1PeC8reENOaDEwVDRKSWthcUcxaFl2QVp5?= =?utf-8?B?Z0s3RVQzVlRmNWJJbC9FN0FHRmt4bWM5WllqNU9xMmcxcGRjZFdxUVFMRm5K?= =?utf-8?B?VkR0OU9tNkJEK0FvUy9STnpnekxCU1pndzhlc0pITDBYbEw5RmhnVE1UUDhu?= =?utf-8?B?NjYrMlJia3hXK0ZGQTlyQjZLZm9QVFgvRis3c2tFMDRnYkdOODF3d0dWVDlL?= =?utf-8?B?elE0NjBHVFJUTTBwYm5PeU8xSVRhclIrRkNhc3VFMVowMElKU2lVY1c2QWpP?= =?utf-8?B?QXBwR3dBMEhickFhSFZKQXRXUHBpMFdvbVczeHVoZEpOUWRoR2MwSTBFdEhG?= =?utf-8?B?QUtYL0VCeHcyejlXS3hoZVlld21BYnRwWVYwOHlPZ01BNU1lOS85SVN3cmlz?= =?utf-8?B?YkhIU3JpYnJCK0hxZldaUVMzUkVXUytxdHhvd0RielVoRC9IZz09?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2462;6:wWVC3X53Vjl5lkki1R4NskStzGXCSMHwprS9V8fju9nwBe9JCfnRajZKwjEAap1qI4qblVGjsPnEdTm+opLSKA90C0N6EVlQNyRFMHrVL8TsVOYwFWpWxwuFopAk1qNJQ9aKFR11TANESxL/7iRV2Cy9fEAdlT0YBLqLRSFMUlPCtG+l2ybfDKw/5XGXoi9oiaYiDKnYKNnompRS2r7fiT9yaC76hdm/oLpD09yaBMA+A0NWcvAU8P0ljfX5YOUhZKZEkGjMjdBx9bsEgtXj+nbj+MhsqvPpiYAFDA3Pz2MnlCSPcTEsvlROUtvxWHrP;5:ibi4V+La0b5Tax4UbQQ3B49KVy+3p1Qusm2bY/MdXrjGU5dy/VqmLn3SwxtgbImyJGoHPMe6oM4J1/LMBCNFTlUmIL2ll+s5fH/Atdw59mdGu847TJFeFvl3CFsfKYJqiAtfZEdauk0oQIMqN1d27g==;24:yg1VUe9Krw5VF9oEd3FWv3j1LDbYkKJVMApgz47Srspzezq1mr5fAoyVHAaETy+K+riFkPdkLx4Wrgl5GWPx6VzEFjfMsB0QZY7OhhMcl14=;7:zvEHbcKwC7PcPmLVI5rG+ED7YNxuMTkNY4Pc+UFLIfuYszs9MhmFakOqbV0XO3FIFSyugpMS+iY6m2vfW4bbRUDx2gjGN+gmOZA0qtfdbad1vlK5Eh9Ml2wf1lTBfDCPHFBkzZd2vETq8qxqWfyvDm/XThFreueDNFUxdDP8+0mdCY4WYcDyg2FAQb9mwVkAingALmSGoFZJjUMKoAM198XltAW72AGKZomKzFZ0FsDuQ9V57OnXSKpq4cHOIDLp SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;VI1PR0802MB2462;20:twWdJx6+qwDEwgFmHb5zzsVVEFRWNs2eByXJKllgKKnczknUyi+B1M0VPChWadPtS0tVUVMz2hYUd+prnevuPumQXVWCcZuP5uo5kP07hQU1nouJewcYgqWWKXgNfgLdt8+2A8ZOiCexr0fgBv703hysBB0lZQTLT+vYGihciTw= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Aug 2016 13:57:27.0100 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2462 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 04.08.2016 15:16, Jeff Layton пишет: > On Thu, 2016-08-04 at 12:55 +0200, Stanislav Kinsburskiy wrote: >> 03.08.2016 19:36, Jeff Layton пишет: >>> On Wed, 2016-08-03 at 20:54 +0400, Stanislav Kinsburskiy wrote: >>>> Otherwise freezer cgroup state might never become "FROZEN". >>>> >>>> Here is a deadlock scheme for 2 processes in one freezer cgroup, >>>> which is >>>> freezing: >>>> >>>> CPU 0 CPU 1 >>>> -------- -------- >>>> do_last >>>> inode_lock(dir->d_inode) >>>> vfs_create >>>> nfs_create >>>> ... >>>> __rpc_execute >>>> rpc_wait_bit_killable >>>> __refrigerator >>>> do_last >>>> inode_lock(dir->d_inode) >>>> >>>> So, the problem is that one process takes directory inode mutex, >>>> executes >>>> creation request and goes to refrigerator. >>>> Another one waits till directory lock is released, remains "thawed" >>>> and thus >>>> freezer cgroup state never becomes "FROZEN". >>>> >>>> Notes: >>>> 1) Interesting, that this is not a pure deadlock: one can thaw cgroup >>>> and then >>>> freeze it again. >>>> 2) The issue was introduced by commit >>>> d310310cbff18ec385c6ab4d58f33b100192a96a. >>>> 3) This patch is not aimed to fix the issue, but to show the problem >>>> root. >>>> Look like this problem moght be applicable to other hunks from the >>>> commit, >>>> mentioned above. >>>> >>>> >>>>>>> Signed-off-by: Stanislav Kinsburskiy >>>> --- >>>> net/sunrpc/sched.c | 1 - >>>> 1 file changed, 1 deletion(-) >>>> >>>> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c >>>> index 9ae5885..ec7ccc1 100644 >>>> --- a/net/sunrpc/sched.c >>>> +++ b/net/sunrpc/sched.c >>>> @@ -253,7 +253,6 @@ EXPORT_SYMBOL_GPL(rpc_destroy_wait_queue); >>>> >>>> static int rpc_wait_bit_killable(struct wait_bit_key *key, int mode) >>>> { >>>>>>> - freezable_schedule_unsafe(); >>>>>>> if (signal_pending_state(mode, current)) >>>>>>> return -ERESTARTSYS; >>>>>>> return 0; >>> Ummm...so what actually does the schedule() with this patch? >> Schedule() replaces freezable_schedule_unsafe() of course, sorry for this. >> >>> There was a bit of discussion on this recently -- see the thread with >>> this subject line in linux-nfs: >>> >>> Re: Hang due to nfs letting tasks freeze with locked inodes >> Thanks, had a look. >> >>> Basically it comes down to this: >>> >>> All of the proposals so far to fix this problem just switch out the >>> freezable_schedule_unsafe (and similar) calls for those that don't >>> allow the process to freeze. >>> >>> The problem there is that we originally added that stuff in response to >>> bug reports about machines failing to suspend. What often happens is >>> that the network interfaces come down, and then the freezer runs over >>> all of the processes, which never return because they're blocked >>> waiting on the server to reply. >> I probably don't understand something, but this sounds somewhat wrong to >> me: freezing processes _after_ network is down. >> >> >>> >>> ...shrug... >>> >>> Maybe we should just go ahead and do it (and to CIFS as well). Just be >>> prepared for the inevitable complaints about laptops failing to suspend >>> once you do. >> The worst part in all of this, from my POW, is that current behavior >> makes NFS non-freezable in a generic case, even in case of freezing a >> container, which has it's own net ns and NFS mount. >> So, I would say, that returning of previous logic would make the >> world better. >> >>> Part of the fix, I think is to add a return code (similar to >>> ERESTARTSYS) that gets interpreted near the kernel-userland boundary >>> as: "allow the process to be frozen, and then retry the call once it's >>> resumed". >>> >>> With that, filesystems could return the error code when they want to >>> redrive the entire syscall from that level. That won't work for non- >>> idempotent requests though. We'd need to do something more elaborate >>> there. >>> >> Might be, that breaking rpc request is something that should be avoided >> at all. >> With all these locks being held, almost all (any?) of the requests to >> remote server >> should be considered as an atomic operation from freezer point of view. >> The process always can be frozen on signal handling. >> >> IOW, I might worth considering a scenario, when NFS is not freezable at all, >> and any problems with suspend on laptops/whatever have to solved in >> suspend code. >> >> > Fair enough. At this point, I don't care much one way or another. Maybe > if we make this change and laptops start failing to suspend, we'll be > able to use that as leverage pursue other avenues to make the > suspend/resume subsystem work with NFS. > > That said, the patch you have really isn't sufficient. There are places > where the NFS client can sleep while waiting for things other than RPC > calls. Sure. As I said, this patch wasn't aimed to fix the issue but rather start the discussion. Thanks for your patch.