From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932126AbcELNG4 (ORCPT ); Thu, 12 May 2016 09:06:56 -0400 Received: from forward-corp1m.cmail.yandex.net ([5.255.216.100]:52175 "EHLO forward-corp1m.cmail.yandex.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751706AbcELNGz (ORCPT ); Thu, 12 May 2016 09:06:55 -0400 Authentication-Results: smtpcorp1m.mail.yandex.net; dkim=pass header.i=@yandex-team.ru Subject: Re: workqueue: race in mod_delayed_work_on? To: Tejun Heo References: <57319A12.2020403@yandex-team.ru> <57320C18.3000909@yandex-team.ru> <20160510163625.GM7110@mtj.duckdns.org> <57321852.80908@yandex-team.ru> Cc: "linux-kernel@vger.kernel.org" , Sasha Levin From: Konstantin Khlebnikov Message-ID: <57347FE9.4070709@yandex-team.ru> Date: Thu, 12 May 2016 16:06:49 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <57321852.80908@yandex-team.ru> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10.05.2016 20:20, Konstantin Khlebnikov wrote: > On 10.05.2016 19:36, Tejun Heo wrote: >> Hello, >> >> On Tue, May 10, 2016 at 07:28:08PM +0300, Konstantin Khlebnikov wrote: >>> On 10.05.2016 11:21, Konstantin Khlebnikov wrote: >>>> I've got plenty warnings, bugs and oops around trivial use of mod_delayed_work in drivers/infiniband/core/addr.c >>> >>> Looks like problem in mod_delayed_work_on was hidden because add_timer is equal to mod_timer >> >> The timer usages are gated behind PENDING bit, so whether add_timer() >> is equal to mod_timer() shouldn't matter. > > Hmm... this looks little bit more complicated than one bit. Yep, problem was here - both timer and work can be active at the same time. So try_to_grab_pending can return success for two concurrent callers: first get del_timer, second removes work from workqueue. After that both call add timer and one of them either catch BUG_ON or corrupt timer list. I see two possible fixes: always remove timer and work in try_to_grab_pending but this must be carefully synchronized. This will make it slower for sure. Or always use mod_timer in __queue_delayed_work() - both callers will modify timer, but here is no mod_timer_on(). > >> >>> but Sasha accidentally backported 874bbfe600a660cba9c776b3957b1ce393151b76 >>> (workqueue: make sure delayed work run in local cpu) into 3.18.25 >>> >>> I don't see reason why that commit could break delayed work, >>> most likely it highlighted some other problem. >> >> What are you running? Can you reproduce the issue on upstream kernel? >> > > This is slight patched 3.18.y. Looks like this started when we upgraded kernel to 3.18.25 and > somebody have loaded module ib_addr (ip in infiniband or something) which actually unused > because these machines have no infiniband at all. But this code is poked from ethernet arp > sometimes. So, it crashes somewhere from time to time. I'll try to stresstest this piece. > -- Konstantin