From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753573AbdEIPPp (ORCPT ); Tue, 9 May 2017 11:15:45 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:51196 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751653AbdEIPPn (ORCPT ); Tue, 9 May 2017 11:15:43 -0400 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=fb.com; Subject: Re: get/put_cpu() usage in block/blk-mq.c To: Mike Galbraith References: <1494310029.20252.44.camel@gmx.de> <1494342256.4440.3.camel@gmx.de> <842b9fc5-5a8e-d147-0b14-3c9f9540336d@fb.com> CC: LKML From: Jens Axboe Message-ID: Date: Tue, 9 May 2017 09:15:34 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <842b9fc5-5a8e-d147-0b14-3c9f9540336d@fb.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [2620:10d:c090:180::1:59fe] X-ClientProxiedBy: MWHPR20CA0005.namprd20.prod.outlook.com (10.171.151.143) To MWHPR15MB1200.namprd15.prod.outlook.com (10.175.2.142) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 36921b0e-bf66-48e0-7a06-08d496ee3f5b X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:MWHPR15MB1200; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1200;3:4EhGby+WuhtCzFTbUm0lrbdCxOwK4g3vhSOR4uSJb3JeL3GIrNJk81NTXt/ILYN1jPwK/bfu68giDJTcU0PcxXOMuluBvl0FTYSZuBbysKODSIj2jX4B8+HM3QJXYvsU5MaDnHkbRbEq3+2HMIDiQT8/a7EETCubHFK7Fk6JQpQyd38FWjuhjyNylWS8HxMsjfucfpXJViyGKCrFmrXewhJsbt4T177z9nk2ViGrezHQzhfQ5mxCYLqCUs5cNJeXZ5ihfCyWESPNwX/eQHqBQc1k1x03FofSKfuJa8KIkvTNWev49yUKw0spKUKjvrSp3Z2jSrjjVO009tyb7ZzHYg==;25:q0Vu92K7si2/5Op50CESobaKJA55ik0Ks47uR1bVEpvymkmtmQOHqPOAdwEIXi/uHWauYKKdIsDgUm1/+PsXb6OKCwVNyzYTsV+ezQi+YMxFn0sSf5YpkSJwa9CB7dsAzSUvQS1vIYWWaPqw5HdJJvwuBVNSnYn6+NsYLzFttIbA4kXDoAv8VSO6S5TpY4g0oM9GXz4KZSWM8A8jEp3WNkMbruBCUc2LhP/GZCTkB4SipT3f9krv5nyxJDJ8natrDy2VLUeInObj1zDHBebkI/onjZw6hXx/NoN61iD4Gy8wxa7yLgw8BaH8rDHMD7ewmsa4Sv5q7dWIiXj1mFoEw7xUBkp4HTZxn2Hs208MCfd4YUiKiF0+N4JbnmnEVOIrvhvgC+776sZKNIuIFDFG4AWn/eN+VdcoQsCDLH9EgSZcft/18094l3YL5G58KtU+CikawCyHymjkciVTYcsqHgID1IpI2RNho9RHw1GVBeA= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1200;31:qGjZ238sNtYLd9OJRRIjwsoK/Cf23iYqXkXwyx43Sfty/dQiybZ07ZbLtMoKCL77+GzmgiGnVJc9nnvA1NJh0C4zKeKkbtXCovj/TRU8rS3BG6Id9BB1P0gM6Ee2vh10sL3TVSPt/Q0ojxsnT68C9lsPZtnE1rJ1HL4lKqGjm3XclC737e1mFJQd5SvijxtbLkGO5TLecJYpnTRlEGB4FtXFafzpGVIgIDrgkG3Xpp/y6uqp8CAjyHaCyTQ9YUXdRL73ALkYyVTwwkIBEyQXbozDckloefuZ0OB8H7Zasnw=;20:iS9rBPna0aVKvtU1qLJsf54xmzpD/7jMe0JzKdMhu3oA8ayBSeCcHP0jxKfwQ4hiBStmNWNMu+uWlEe4qJRVYYK/MAZF3yITpjXsGGMCvYXF1g4qkD50Vae9pjCqIDhdgY5rbXqyZVYpDjoPgELAui8jcaiMxz1EjudWL+JspRm7QAjAu5GUO8QLMxkw6QG+G7AQ0fHzrHeaOugIr0ZYjCV9Df38UILZCXBezZVddW81p65ZEapMOxdgyCpHyWIvkPZ+Q+03Mc3FoiBG5nMODGQMyyvS3LIetU6CXLl39WSJoNfInW0MkkQTa4KncruKKXSauW2iG6HG+YcPZN+uWz09Nrb6toIx09V1m/eyuAQhvoLzLpCQBtN5CRbtRcoY6zyTzEo7B+cNNDAuW84Va1rpCyTkACE/h7jmcYRkNpoWobGyPqY9Ix7rkU+s23m3/TNb7PaQ4/ZcJCyikqfBvk0ApC6+SKNOiK8QFzbZbX2tAbSp9kx3k5sJdMXt1mJ4 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3002001)(6041248)(20161123555025)(20161123558100)(20161123562025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(6072148);SRVR:MWHPR15MB1200;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1200; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1200;4:1HczbWjcvpun1Qu2W/DC1zTRHgfifWszUZClWdrZpOAb7Jp92tdEAOn6Z502lAfJD/jg0jrOw6bkxN7rWheaKZKdYVJCSlIjXkP+6l6dts3WryG0HyMwTH/o/p6ee9O9FPFkBT5iuXukeaSYkSMCifrflBmH8QoEmS19yfqrMy8nuqtbs16YLF07RDN+RYn7Zu6hTMCPwnCf4UC1bjJRW+nk/HZ2DdYWd2vUIIREGhhrbQh9zHNV6OPGekmq9Z41meP422rDzWM/4qdjkc7SRBLbmQoohtlm8i1fo1eOZmhzuGbUJbvEavcKsfB+u8RF5L4ojwnn7kVDaSQUsT/uw6n0Wmdd9J81FoIhpcSLkY3f2CK2GtfeMQg3K/AWl30boZqPqkBzpjEL69DSOAV3UyaK3yETWCF+5ab3qC5TjDpLPzRYYw38ZKBaKOzGQ2inygumV3o2YYit1WNDzZuLZ6ypbETFFUT63+NsFKykJsTOCdOcz0tyW2tvwTvZTypIbxG48XCU/Ci3msVU/IDlM6XSpwMrNv7Ep8g3ZNn7rYdyODt+3FFKtdwk0ZIha4xH0VRKLkUNBBRCSCcEXF3oi1gxd+gI5JnDytlvayl12zzgpJtM7RhvveDYx+mFC1EprSivEBLCWCFINhYZD0NcbVspo1T5D7SXRPlIx6P7nNzEkhbo+Qx/3Rh9RgInl4op5me7KmqzmeeTYbmwmNWHYMFnF+0lUFN0YswfVSEWkpt8hXNYS98vDUT/wEhKaF0ZaS7gybEHE65wRlRlCtZjHoKvU2A+S06GQDPT4s7Ar9Q= X-Forefront-PRVS: 0302D4F392 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(39840400002)(39850400002)(39410400002)(39450400003)(39400400002)(24454002)(377424004)(377454003)(81166006)(305945005)(5660300001)(7736002)(6116002)(8676002)(83506001)(230700001)(229853002)(47776003)(478600001)(65956001)(50986999)(54356999)(76176999)(93886004)(50466002)(6666003)(6916009)(2906002)(8666007)(36756003)(23676002)(33646002)(4001350100001)(38730400002)(6246003)(86362001)(31696002)(4326008)(110136004)(25786009)(53546009)(53936002)(2950100002)(6486002)(42186005)(189998001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1200;H:[IPv6:2620:10d:c081:1110:8000::20fa];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtNV0hQUjE1TUIxMjAwOzIzOnBiUm5pTjg2d293UmZ6Q3ppaFYzM3NqZ2ZV?= =?utf-8?B?ay83bkEyd2VQbnI0RTFhY2RYQkoyWGY2Q0t3MzNKeVc1R2Y5R1hjWXFEZ2Fr?= =?utf-8?B?OTBDa1EyMmQ2cEcvNUNBdmVxSVdpeTJWVzZkSjgvT0pjNjhCRFRnbWlOMUV6?= =?utf-8?B?Sm10RmkyblJpaVpqY0p4Qysrb2Q1OXgwcDJ2S1JoUVRzRjN0M2NHZHZQUjg0?= =?utf-8?B?a1Y5UHdtcUNFLzRvWkpsNHpKWTVvOVI1Yy8vMEhnS2xldms2S0g1MzYzWDVk?= =?utf-8?B?Tnp3T05BSWNlOFpSV2RUZjNuekw0UWFadW0xUnZGb3N4RjdlNFdNUmxnVTFu?= =?utf-8?B?TzV2S0g5eDFrR2FvNjlZK1dVVjd2NHQyblpnVUJVTDhXSFhlQk5obmFaaDRO?= =?utf-8?B?TU5OZFlZbWl5Z1VMOTh1bHlKMnplclhsSmkvbGljQTZJOG5UU1BxS0oyb0Js?= =?utf-8?B?M0E1ODFlL1NqVVVoeFlVMnZnVDFDL0VWS2doZlhlTUdUT1owelhGT3VtWUdv?= =?utf-8?B?UmpRNlBvYlVqRjJsZFBpVDFxcDVWYlF5bTBJRDZINHFVOFpWcXlqcnh3L0k5?= =?utf-8?B?Mm00ZTF0SkNyTGpzQTFsS012SUVXbWYxN25qSEdONXR5dE9CdU8yd3RObktV?= =?utf-8?B?WVZuM0ZsbkwxRXZ4aHJOZmpZYWZHQlF5RzVrRjBLNWZma2M2Mlh3MVo1cnFF?= =?utf-8?B?ZXlKL01meDFhUjM1aEMvcWJvNHNlaHpxRzJNZklSNjZkKzJVM1BIR3QxNGVV?= =?utf-8?B?LytSdDFxT0pQZjErS05HTUxRdFBOUzlvN2pLUW5GRzVKWHRFU0xDemp4UHR1?= =?utf-8?B?QWNYdVV1RTNiWEdzYVU4SGdoMGkvN29WSmNPeGpCWkFTM1owNmxvak9vL1dx?= =?utf-8?B?UlZNNUFhcFZoQ3FjZ01IaGdrRS9jdVIvTUVkRnVhOXJFYldjUXdGNWQ0d3Zp?= =?utf-8?B?T2dMZWlCRTNYTU9QMGdVS09oR1d1ZldrN2ZFVE5lQStOdTVxY0J5SkllT3hG?= =?utf-8?B?eUprU29aNVlNZnNmUGd5NEdJMEkxWWhJdy9NQWhleU9KZEpvZVl4anBSR1RE?= =?utf-8?B?eWFyMFVRWDhEYit3TUhxcUx2VkpnODl5d2hQdG9EeTAvZG41YVlVYktEM3Ft?= =?utf-8?B?MHRHT0xiejdOa1kvYXlvM3U3cWk0UWNldWFnQWNENzRtUW1RTXBVRVlpYmwr?= =?utf-8?B?TnY1aVhXSHdIYzlLUVJPN1UvWVU1MWRXNXRiYmRQZUZCUFg4UE9sQjkwUHZv?= =?utf-8?B?NnVpclYyM0tiMllpa0tickczZVlPRWp0azFpZEgxdnhSUlRCbTNDbnp4eGYz?= =?utf-8?B?TTVhN3l5aEQ4RGJhTDhDWGpKam43SmNSZWk1OGU5Sk5aRzdQZGpuN29sUGk3?= =?utf-8?B?S1dWNFJraTVvMkoxeFQ3L2NpMVFLOHNmcG9rbjArN1pjTE9SN2tFV3RzRDRI?= =?utf-8?B?NDBHNEhSb2pqR3p1Q296U0UzVVljWmZvM0Z3MjlyclA2eTBLR1RWbktSb3Vl?= =?utf-8?B?Q3h6WFIyZ0Qyb2xyYktHbS9EUWpDK1ZLbm1aNFNCbC9rNUVVYkI0RmpRZXcz?= =?utf-8?B?OCt2ZHBNRGxaMHBiQ2xlditKVy9HSzJLQ0ZzQUphaTZOVGlma0xrQ2VTUzZX?= =?utf-8?Q?xpsBmwsJY7zim7m2EL6B?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1200;6:jM//qpxrlWqsM5pbf0sZlOqqR0uQ0tuw8TS0h8l2f0H3qmxenjBEu0/rGcauRMEiMMph+rb+c3kYmHwZFULuJDl1/od/C/gRKYYcwLTMaf3pjVxjIptdXNd0TonzOxGcoNVXfjj0iDyiOPxj7JkIim33iBzkYCUzFmlp4L5WlLm1Xy/FRQicNKVhBHwr0FLM0rRJtQkj7ZMsAyPHgq1IC9Po3oBk8UQD9qgQjnIt+gIBhupRlaSQuH1T1GKh7c6tAS0oGvpUHyNspsul5Z5tnayCT4a+0y/uk7AioMSBvq8z1i/fk4elnj4zMJ54D7feXmrxwJUBCNGwx2pW1Y3sts7QOP5SXrq9Ty14S+tr3rDLP2pt8x56wZFvoVHRdLjd22QCMZSsPCLYeq+sTe0iHGrCIJg5NK9jBmnUbx2RWV3wXPKsD3lR3bBd20kr7fWM8Tlmqs7Ms7yN+O5EcR2aIF4VQbe2ZNJ/xwu0qHDBnIh3UFyxYdD1kktRvT0LQmSf0dLUb/MoU9GnbIgRFQzswA==;5:AjP2afLWYCnxHDqOEhJkAa6efxlTyHOhVvkbKT2EBLKsPsZVDJFqqHq6KSWeYVo0KjYrZTjH8TVExg0PWgOlW4B655Bcivc3Wl3zxc7acPb6+JgrY+OCfQnzKbmZVInbmvl/4XjmNQE6i1COjg5LjQ==;24:g7B0HZUx3ST4pC1spi/oQHWs/VY+/OS1ikh/YPSl5CxQufuY8qSVWfiUQ9JB5tfGyqXbo7f5nlaPl+JdfYHyEgRflbJzVhQ18F8F6vqn5n8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1200;7:ntQ+XOfUYgwalS+SRZrbUVaPjl8ggm9Aw4ynSo6vIQntCxRWGBWgW9ObU36tKKVnneWQmz6/eLg5KefHbtHqW0UPI/UZ3tiDaWit6Ji9ucgm9IDbvDazgRPY/2/itGKeLuxlyMVFe6DPHwkaa1/0ZRaiGSeUoZevEw0P3U8A5XNkFG+skucivIpBPEnSdKN+Ao2QmbIXjQ1sE0/kKmTno5D2jGKOxKp4cWoK9WOdGWPrBxeDSn1/CVsNNqfyCBNNxu14oRU1CpEb+QsEdzUVqzCwQq2NtAqAvaFQrR0+N48PeLyJmhQq3NXfU/WNUdq56qCiarWWNllGUEaEctXZdg==;20:7KyuLfQ1AlLcHlLC9PKqRKMhXEKrXB0pQs7raIuihNAHHyMD7hn2EuX88jaagv6NNARcLO1rpf93v/Cr/6vdgIDHPxy/idWLVGeqJHV67ykMwg9JCiSSZRrj79Fz6bV+h216OVUFIlEflURQpcK8Fx0tVdc+SfRKFWwNryPlwXE= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2017 15:15:37.2232 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1200 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-09_12:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/09/2017 09:13 AM, Jens Axboe wrote: > On 05/09/2017 09:04 AM, Mike Galbraith wrote: >> On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: >>> On 05/09/2017 12:07 AM, Mike Galbraith wrote: >>>> Hi Jens, >>>> >>>> I was about to fix up this splat.. >>>> >>>> [ 445.022141] loop: module loaded >>>> [ 445.078116] BUG: using smp_processor_id() in preemptible [00000000] code: loop0/3801 >>>> [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 >>>> [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G E 4.12.0-default #40 >>>> [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010 >>>> [ 445.108564] Call Trace: >>>> [ 445.111016] dump_stack+0x65/0x89 >>>> [ 445.114330] check_preemption_disabled+0xde/0xf0 >>>> [ 445.118945] debug_smp_processor_id+0x17/0x20 >>>> [ 445.123300] blk_stat_add+0xb0/0x130 >>>> [ 445.126876] __blk_mq_complete_request+0xb5/0x150 >>>> [ 445.131575] blk_mq_complete_request+0x16/0x20 >>>> [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] >>>> [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 >>>> [ 445.144816] ? finish_task_switch+0x85/0x270 >>>> [ 445.149085] ? __schedule+0x291/0x8c0 >>>> [ 445.152747] kthread_worker_fn+0xc2/0x1d0 >>>> [ 445.156754] kthread+0x114/0x150 >>>> [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 >>>> [ 445.164424] ? kthread_park+0x60/0x60 >>>> [ 445.168085] ret_from_fork+0x2c/0x40 >>> >>> Is it from this_cpu_ptr() in blk_stat_add()? >> >> Yeah. > > So why is this complaining, doesn't rcu_read_lock() disable > preemption? Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below? diff --git a/block/blk-stat.c b/block/blk-stat.c index 6c2f40940439..022f5925a427 100644 --- a/block/blk-stat.c +++ b/block/blk-stat.c @@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq) rcu_read_lock(); list_for_each_entry_rcu(cb, &q->stats->callbacks, list) { - if (blk_stat_is_active(cb)) { - bucket = cb->bucket_fn(rq); - if (bucket < 0) - continue; - stat = &this_cpu_ptr(cb->cpu_stat)[bucket]; - __blk_stat_add(stat, value); - } + if (!blk_stat_is_active(cb)) + continue; + + bucket = cb->bucket_fn(rq); + if (bucket < 0) + continue; + + stat = &get_cpu_var(cb->cpu_stat)[bucket]; + __blk_stat_add(stat, value); + put_cpu_var(cb->cpu_stat); } rcu_read_unlock(); } -- Jens Axboe