From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755211AbcHSP4z (ORCPT ); Fri, 19 Aug 2016 11:56:55 -0400 Received: from mail-cys01nam02on0066.outbound.protection.outlook.com ([104.47.37.66]:57520 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753120AbcHSP4w (ORCPT ); Fri, 19 Aug 2016 11:56:52 -0400 Authentication-Results: spf=pass (sender IP is 63.163.107.225) smtp.mailfrom=sandisk.com; broadcom.com; dkim=none (message not signed) header.d=none;broadcom.com; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: 0ac94371-952c29800000f08d-39-57b72a89f9d4 Subject: Re: Observing Softlockup's while running heavy IOs To: Sreekanth Reddy , "Elliott, Robert (Persistent Memory)" References: CC: "linux-scsi@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "irqbalance@lists.infradead.org" , "Kashyap Desai" , Sathya Prakash Veerichetty , Chaitra Basappa , Suganath Prabu Subramani From: Bart Van Assche Message-ID: <6b7930ca-092c-a03c-d745-b49153aa174c@sandisk.com> Date: Fri, 19 Aug 2016 08:56:34 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpjkeLIzCtJLcpLzFFi42LhOum8SLdLa3u4wbIL3BZv325msphxfA6b xeUjPxktDsw/x2pxeReQ2319B5vFhnm3WCxmfn3KbvHsTIwDp8es+2fZPHbtamT32Lyk3uPz JrkAligum5TUnMyy1CJ9uwSujCt7rzEXrOWpOPH7MmsD43KuLkZODgkBE4kjjddYQWwhgaVM End7PboYuYDsHYwS/XdOscIU7euZxASRWMoocXzbQ3aQhLCAtcSKhh9ACQ4OEYFciYNztSBq XjFK/O0/xgLiMAssZpa4tfEkI0gDm4CRxLf3M1lAbF4BO4m5U7azgdgsAqoSvX+mgsVFBcIk pt19yw5RIyhxcuYTsDinQKDEqccQFzELWEjMnH+eEcKWl9j+dg4zyDIJgbOsEmsnXGaH+Edd 4uSS+UwTGIVnIZk1C0n/LCT9CxiZVzGK5WbmFOempxYYmugVJ+alZBZn6yXn525ihMRO4Q7G 17e9DzEKcDAq8fA6SG0PF2JNLCuuzD3EKMHBrCTCqweMPCHelMTKqtSi/Pii0pzU4kOM0hws SuK8F6M/hgkJpCeWpGanphakFsFkmTg4pRoY434ffmw/zd+MbXvM/VXa2bF1ORd6j012WXVH UGFVXcrNw5wnvx3v7Daea2g+NTBkh6sCp9+PZ32rZSefffP81d3Zx29dTH5wh3+exTShXceF Ff4q11sd1D3wehXPA0b/mfrvev8xZ1+JuVl7LOzvkfhP5nqNHSfruZbI3Zz+36Qrao3+m6mr apVYijMSDbWYi4oTAWAIZz6ZAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLJMWRmVeSWpSXmKPExsXCtZEjRddYZ3u4wc/dBhZv325msphxfA6b xeUjPxktDsw/x2pxeReQ2319B5vFhnm3WCxmfn3KbvHsTIwDp8es+2fZPHbtamT32Lyk3uPz JrkAligum5TUnMyy1CJ9uwSujCt7rzEXrOWpOPH7MmsD43KuLkZODgkBE4l9PZOYuhi5OIQE FjNKHHz4jhEkISxgLbGi4QcTiC0ikCvx8d13qKJXjBJ/+4+xgDjMAouZJebuO8kCUsUmYCTx 7f1MMJtXwE5i7pTtbCA2i4CqRO+fqWBxUYEwiWl337JD1AhKnJz5BCzOKRAocerxKVYQm1nA TGLe5ofMELa8xPa3c5gnMPLNQtIyC0nZLCRlCxiZVzGK5WbmFOemZxYYGuoVJ+alZBZn6yXn 525iBAcwZ+QOxqcTzQ8xMnFwSjUwRh1/9Ke0eWe2ts+MzAzllM3yT7OSbv5P8Wxu4by8fT3H wXeenXN191tJVGTf//5jeuq+n5qbM9KjtL17Tpb73vlv8vj/gpvHpySaK/Jar3qwPP6sonT/ 82jvvKhHT5bPFJyqHM1q/ahj66GV8wp+CRuZ6b2b9mfKNDufUx5ufVtzz7/mSGw+rMRSnJFo qMVcVJwIAJwI8NAQAgAA X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:63.163.107.225;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(377454003)(199003)(24454002)(189002)(68736007)(97736004)(11100500001)(4326007)(92566002)(2950100001)(4001350100001)(5001770100001)(81166006)(77096005)(31686004)(87936001)(189998001)(53416004)(2270400002)(106466001)(7846002)(31696002)(65956001)(76176999)(65806001)(64126003)(69596002)(83506001)(50986999)(47776003)(36756003)(23676002)(33646002)(2906002)(8936002)(86362001)(356003)(7696003)(305945005)(8676002)(230700001)(626004)(575784001)(54356999)(81156014)(586003)(50466002)(65826006);DIR:OUT;SFP:1101;SCL:1;SRVR:BN3PR0201MB0900;H:milsmgep14.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BN1BFFO11FD007;1:zSuK4j9vcqVoT8UHLx16QQL/D/baRpWD+yoVXI6AlhaODHpDR81B9peLF3nV6qpwz/LykWsmbKerOpWbvL6gM5N64X6MMJAEnHVqSvfAXN/QPdbQJoMzvGkQRth72CBtAehNuOgza3+gHIKUs1nuGmSMfl8AukHh9rX0321PKbQCriP0WEInqU2/OIHMHhJAx4ykdd2JHbdJAnk3p8uOG0nfZE5qcW7dX9zg+AA6GorBopNgs3DH6xxtFQTpS2e8MDPWMu4cIfajxrCuM1WC+/2R3I2csHEfhTY0qJsFe5pG9dbQgLiGba8bc1l0NdmwBjf2cgEU8q+3RMVpLLaqJNNpw+8JCsflqm3jfd4pwFH8xIVHUz3E0FNwnn/LLu+Qam6yrvWP8UDiaC3q7H3Kjmf5o6bU7Jpk2d/aQ5mH5U8ByVLrANeQH0v6Kr5d89/emYfzeMdXMoDpE1GMPOrGuOZl0obu7oRQpBH4djYq0hI= X-MS-Office365-Filtering-Correlation-Id: 5046deee-99ed-4025-d040-08d3c84969ea X-Microsoft-Exchange-Diagnostics: 1;BN3PR0201MB0900;2:lZz6JywEYEclPeRZAHE04s01TnSNgPxR7msFWmL3DIej12yo+My6FZiYgOSjW+trQXd1jdBpm3/gua/PfkUcREkVNkhZpc4EJYzNeq35B0gqJjrgagmLoun/M5YAC21htE+qkRbrdikx/AGgRgnhZX9tD/JvRiBNHa/EWtj3GKjFAYoQQlRFbu92xQb0/moA;3:xFilMrdjbVbDr3vpG5sEIbBA18YWe9Ch+g3UThHf59On9CkZqHoIb2nG/hZpXbyuQyIpjNgzEaa3PHOq4Mjy+9sWP6NUL++HDDuOoYeGxS4OM/kFelvncN2994dNNIbgDUKMbNRkU0HZtSSWMRlwRG6VhkvY1SnrznP3WPIL3cfgT8Vih2c3i4yhxoNXkZxTQ2NRHMSmQrx2JNdrtQWuFXt/6mQRJRohB1XIVVonA9sToxdqHoAUMs74h9/U8EED7RHs6kzl2+KRBxd9Rcti4Q==;25:64BLMyaV1V2n5+09MQUpoOrFmpA4Sr332Ppl1KZhGDhaQqYJRLNS+xTEhjhYwU2VG1szq0L1D4+OJM8Wmk6oabFhlwnmUd6gkeGwV2sbcsB4MSPX8CC6am8IezZsFkSox2Ere3FrGRPuXThAZlwak9C1AIj33wZMU7+z+mGjSNW0YYbqW2sGXHZAu4tmx2b8RwYyahfJ5bpHN2VQbCj3YvM8ke2AXICCk91t12aAdh/prs8AbcAUkCoWvkiW+gKfWmDFLuXw6aIoS6Wrxi6JNMJ3vK8/F4UNhfGr33vmr6G5OFqAOqsFvFdga5HRKAqFk78vBFGKV/kcevUDkLk2m/Wr5dBUkf3Y7Ql8VsiENi64iEBhl71hsVxeO7mCbd3vzARMFJnhZjcUwMv745BAxDUbCBFzpJl239RyPkqQwHc= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:BN3PR0201MB0900; X-Microsoft-Exchange-Diagnostics: 1;BN3PR0201MB0900;31:9flRH79kkoa1rCpS5GE8//5IBLFMd+vYwo/SZSiZvqCAisvtADlJkNAfz1uQTDJlv2ImYfNy/39/nAVd8Q5L8WKodRqY0Iij/ka1rCtRwMOES6bgWT4pBsVMG8AKOVdM4JkmXSisqGVAxPfRZ5zYZIHW0k6AJ9hGcMRgEvbxk1+W5v/Oi6/vpbZh5g/3Z+a1aVyv0T0St3/Ka0Bq+N5nnxsTy2jI9Uacr+TqsoB1UyA=;20:ubrZJUBYGuM9J0cSqAMIxRicjls0Cv6cVQE5V0CaQN3z2xLFy9/IpBR75bvoqqAFHV4IP0OPVxw/TTHG0ZxdWhIr9nnw43jZjgo51+FQZrB0oNVc0u7hFPthutYnvGjI3VvIXNsQhHsl8nFBkCd4HJZDSVT+m3dyCpjjZ/YRQYOhPrgi921JSB0OfmZGnrznOgdAa4fngtnPMxvaPJmLwzLiwbjB7A/3hV9r6WBvh0yx8Vl9w5igZwQzWeHOVBUYrbmqdPdGReoXzzasyr0ymPpWAMppgkjWbJkiYUOUJH00Z/oS7ZDw4Gd5ynqkBA1egsevl60G8k60mZbH5EqkynxKf2qTsZ0XmOwsz6Ie5YUCAd1awqDg6zDnmMl5m4G2VzxE3G70QMQCXdruk/glU1hSmY4DA7bW6UnMedFcjAwEFhVHsaM/kKAQOT3jSsmEeNiUsovuTl3f+ZCcZh6tEqCSuQkKkabnkBrb0OOeerYRVN2f+SBGamjCGWTjRZsK X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(13018025)(8121501046)(13016025)(5005006)(10201501046)(3002001)(6055026);SRVR:BN3PR0201MB0900;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0201MB0900; X-Microsoft-Exchange-Diagnostics: 1;BN3PR0201MB0900;4:TVsxhqRrVYIJN2Q8X6UQ2Jps5ooRntVzSmhA86uCUCphhq17RcevS3WKtxk29YUkI+zLPXQrmmeCrwIEyemIFiHOgnJYGlVtbCnf6G8oRTvjLzoL7GaKlwpG5+6fZJovmtbaWV9R1UlMXsP5z0Hwf75xetIBVwithdmSXHenL6Z89Nj4C7fWW9FpZnBb0JloaHhvySl5byPF6FElJUULAYtH2jNselDMTpT/N1W/KKEH1RH0D+g9XEahOLdRi4n8NuUObMJpj49+F5thY1ij498fRJxOfatiMGmHtmOi4w4Qi5TJ0fbbYIegV19B30w5rK1tKUa2w922VbzB4SjUh49tZ3xkf/CFjXpFRWzaVDxL1bMQ6mg3yhyIM5wNGNWFbJ1KbpFneMbOhmEhGiaeyl/hg2HcRZMkjXG+7SAuABq1vZvu4zCVNzyLjcQXaZFN X-Forefront-PRVS: 0039C6E5C5 X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjNQUjAyMDFNQjA5MDA7MjM6aVV2QnZSSG44blR5b2Jqc0xIdjlzRThB?= =?utf-8?B?TDNMYUs5aWJ0L0NGZXJWdlhiWTNTOWY1UTFsWmdyUXRENkFxcXoybSt4WHNS?= =?utf-8?B?VVpwVXdhbmhFLzlrcW1YZEJqc01pTGZNZUZ6TVVEbkRxSWNvOWlYN1BocllY?= =?utf-8?B?QjB4bEc5OTdtT3dQR0ZRcCtuVGV4WnlBKzFxQjhzbnB6WGZwWlVYUi9sdDJL?= =?utf-8?B?clRlRHdFQUhOb294QURzUGlDUHlBL1g5OHg0NlR5bnlvNk1iZEVrbnR0dVNu?= =?utf-8?B?a1U2aUNXTDBhWGtwa0JZUC9kOGU5UGVKODY4R1cxVHU0S28xbFZMaEx1enR0?= =?utf-8?B?djloWFprdmtQRmpaMDY0VW9LTmFvRzJEMUJ5c1hmcS9KWGJndXRiWlcyR1NN?= =?utf-8?B?VWpkZDB3YjBkWTFaYVpIVjdvbXVCek5kV2JPQi9yMGtFYnpCVE94UDhRUkpZ?= =?utf-8?B?bERmdnRIb0FvZGk5SEtTL3VaUzlMZithZzV5K25vNWRxS2ZxOC8zWG5vVWpz?= =?utf-8?B?TEdEVXFhbExUci9wSXNZaEs5cy93UklSdmUwaGRLZEZTZmpsRjFXQlN3OEVC?= =?utf-8?B?YXdYUjdsWkl6WTc0TGZURWJaVkdBRGJmNTJ0M1VtUkdKT2wyUEE0d3lzcSt2?= =?utf-8?B?YllIVzhuR2xEVGkzN3RyR0FKNEY2dFFTODQwUDAvNnhVZTVSdmVHczg2NTFn?= =?utf-8?B?RWRUb3FSQ0dqV1FlQ0htY1FEZUZuQzRXcE1Nd2NKTFMzS1RyZS9vMUZkcUhJ?= =?utf-8?B?S05pYkVwTzJsbndlYlUvMi9Nd2tBS3YwTmloRk1zR1FtcFdyZldlVFY0TXhH?= =?utf-8?B?SE82azd3ZjBpamlzUnA0U2lvdktkUmNBTFNYc1V4ZElJUktqbXo5TzRORll1?= =?utf-8?B?b3NROEVGWnllMmFNTlBKRnNMS3Y0Uk9YYVNjV25ZNHN0UTNEOU14QUFZWWdi?= =?utf-8?B?T3gzQldmRGJpMVNPdEJnTU51L0RneEdEVUhVbGdiTVBWb0w4aTNYV1o4bW4r?= =?utf-8?B?bm5uSklGS2lzUitkZ3JzNXRsV2VvMEx0MWdkcWE4enA3bzkyajJnV0l4THV5?= =?utf-8?B?SGxZZnU2RDlrY1NkNzVRLytpUjdxVVJuVE5GNzc5d2NtUVRvWExCOVRxNkZh?= =?utf-8?B?VXZPUTA3c2NYZ05JclNHOElydk9ZYjNHc2NQNjB6QmExdHh5dWgwdnlvT1hE?= =?utf-8?B?NXFadk1hQkZlZjVBTlBxby9BdmhXaG9Mb1FzU1FVQXhnaUVlVTNXaDUvYlZy?= =?utf-8?B?Q3l5Ym9QTEhHcnhvNndIL2VWUzdpN1NDRWdTSjR1aWh2L1RoOEpzeVRBMVNP?= =?utf-8?B?UjFZdDdMMzJmREFkTlZzYXNyOHkyamRPZEYvSnJHOUtTMTQyM2RVZGNibS91?= =?utf-8?B?R0l4bFk3UCsxTVEzWXFBdWxMQiszQ0hGZkx3VE1KZWJ3SUNlV3l0MlJIcG9T?= =?utf-8?B?cHpxWlJRU2M5NnlRdUppaTBYZ2tIeTJtR1Z6dUlnVnhrZmUreUcxQm9GdGpt?= =?utf-8?B?b1ZGTk94ZVRCV2xpenBCYUFKeHBEQWdyK0QrYko4YW1uOVdQcGZObEd2dW52?= =?utf-8?B?YTd4SjFnZTFJSjIxenhiVkRaL1RlVGdOdXdxeXZoTkJQelRVVXBGMWNCS0J4?= =?utf-8?B?NzZwbGVFYTRLMzlKVGhBK0lsaXpoZjZkKzR5dVFRbU5PSDFCS3NTVnVmR2NE?= =?utf-8?Q?aMxBYnAwgnboAz3w5oM8=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN3PR0201MB0900;6:BWbCRDYju3VbBqavQdkp6IjPxIYaKYf88BnWqQwJgvncyW/VEmbqqAseMSQpp/FJiP0MWe4zzqpBSOICg//hOAVfWuIsqjSkRS6byWJpekNu488SAozpx705kEXWYjjOvctzP6rRk1wnjIIMxEhrWiBSxt4HMrayxGtKOLYPcikeKx0j107PQGnZAsPXO6NfbKFNf3g3dl9X+vG4t7ciAxfNdESREuY6k7E/DljCKp3TQgtUsc5k2mMtvKnBaWAzUQOIL5VpQcESlxut3lEdrEViMl0csgKu1CKCsGyZacl5BeC+53GO5Nv6CpTOIbnzu+6ZBNaek0cdg4DpHKGLIQ==;5:5VyqBUG4BaVcJW8qwbluhHQ2r6XMZgjtAr8C/ETj4JpC7YPbmVFz8prwEa/riTEeYMLiIEvY51V1t6vjnsvnBVPB6ee9D6GodhO/OgW6/QcrjURLATqg+CjFLthiNCnof+xCSK1hl8koo8tjnBSEYQ==;24:f2GpJ54WRToDrWczJ9DWM/hnHHGVtZwb7QB6MEzJiApsHZ3yoIh0sEp8DwDLKzkDjy92GAIQd80HzqOxGuXiFuzZihAkdSHIKz0wb0W1aP0=;7:aJvemn1Macvu+ZWC5zOD3NdK94Q8tvWprp6dpYuHHsYXf5WUeuX5fRx73OkV4rzOYFpshqqg4ZaYlgUrFpvD2U2ZcqMoL4Pl+n8bt4H7elwaJYWXcPQg6U/V4flldMKUxip1DGav/DoPqNQzReaN8aCDbawYdsVqPFdOG4vhbPM4yzHrBuJ5/yBtMRIzF81PQ5Vaj/drxB00ab7uiQL1zxKzIrw6FiMMEmbX3vy52GphzXUOAOS3ExqEKJm8S6M4 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN3PR0201MB0900;20:8Y2m5Fb8Xolx3iQj9Z1kSaM9GElTv0b+eglggrpwRTxZt4SKfqK3ingpuAGAEa01ar/+sGU0cMWplhUtotJbwGim8OjfT1GxAwT7j0NCWUpcPCMD3/DTx8/GAvRTz2jDgC5sSpdjTsVMxZD8DxmGpkAbfjzsBCCSaw2xiCoWnFmyzFox+xAAJMXmS2tT4dlNaYfYZye1Vc0GFn4WqJ41OjfSgtLHxpiePeOpoM97zxjfua0G+Uf+8pOE8STI691I X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2016 15:56:41.5800 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.225];Helo=[milsmgep14.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0201MB0900 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 08/19/2016 04:44 AM, Sreekanth Reddy wrote: > [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms > [ ... ] > [ +0.000005] [] ? finish_task_switch+0x6b/0x200 > [ +0.000006] [] __schedule+0x36c/0x950 > [ +0.000002] [] schedule+0x37/0x80 > [ +0.000006] [] futex_wait_queue_me+0xbc/0x120 > [ +0.000004] [] futex_wait+0x119/0x270 > [ +0.000004] [] ? futex_wake+0x90/0x180 > [ +0.000003] [] do_futex+0x12b/0xb00 > [ +0.000005] [] ? set_next_entity+0x23e/0x440 > [ +0.000007] [] ? __switch_to+0x261/0x4b0 > [ +0.000004] [] SyS_futex+0x81/0x180 > [ +0.000002] [] ? schedule+0x37/0x80 > [ +0.000004] [] entry_SYSCALL_64_fastpath+0x12/0x71 Hello Sreekanth, If a "soft lockup" is reported that often means that kernel code is iterating too long in a loop without giving up the CPU. Inserting a cond_resched() call in such loops usually resolves these soft lockup complaints. However, your latest e-mail shows that the soft lockup complaint was reported on other code than __blk_mq_run_hw_queue(). I'm afraid this means that the CPU on which the soft lockup was reported is hammered so hard with interrupts that hardly any time remains for the scheduler to run code on that CPU. You will have to follow Robert Elliott's advice and reduce the time that is spent per CPU in interrupt context. Bart.