From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S932491AbdHWRr0 (ORCPT <rfc822;w@1wt.eu>);
        Wed, 23 Aug 2017 13:47:26 -0400
Received: from merlin.infradead.org ([205.233.59.134]:41162 "EHLO
        merlin.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S932111AbdHWRrZ (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Wed, 23 Aug 2017 13:47:25 -0400
Date: Wed, 23 Aug 2017 19:47:14 +0200
From: Peter Zijlstra <peterz@infradead.org>
To: Oleg Nesterov <oleg@redhat.com>
Cc: Byungchul Park <byungchul.park@lge.com>, mingo@kernel.org,
        linux-kernel@vger.kernel.org, kernel-team@lge.com,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Dave Chinner <david@fromorbit.com>, Tejun Heo <tj@kernel.org>,
        johannes@sipsolutions.net
Subject: Re: [PATCH v3 1/3] lockdep: Make LOCKDEP_CROSSRELEASE configs all
 part of PROVE_LOCKING
Message-ID: <20170823174714.in4mv7uc3rdheygg@hirez.programming.kicks-ass.net>
References: <20170821154600.asyzqs2zg6w6o4pg@hirez.programming.kicks-ass.net>
 <20170822051438.GD20323@X58A-UD3R>
 <20170822075238.uyfmhgxeal2bwcdg@hirez.programming.kicks-ass.net>
 <20170822085100.GH20323@X58A-UD3R>
 <20170822092141.fjmr74xhfid7vu7h@hirez.programming.kicks-ass.net>
 <20170822093337.GJ20323@X58A-UD3R>
 <20170822100840.eababgjcu76iois5@hirez.programming.kicks-ass.net>
 <20170822134922.m2g6kqsqo2eojrg7@hirez.programming.kicks-ass.net>
 <20170822144602.uh5jzkkchvdgzs3s@hirez.programming.kicks-ass.net>
 <20170823163903.GA12973@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170823163903.GA12973@redhat.com>
User-Agent: NeoMutt/20170609 (1.8.3)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Aug 23, 2017 at 06:39:03PM +0200, Oleg Nesterov wrote:
> Peter, I am all confused and I am still trying to understand your email.
> In particular, because I no longer understand the lockdep annotations in
> workqueue.c, it turns out I forgot everything...

Yeah, that happens :/

> On 08/22, Peter Zijlstra wrote:
> >
> > I am however slightly puzzled by the need of flush_work() to take Q,
> > what deadlock potential is there?
> 
> Do you really mean flush_work()? Or start_flush_work() ?

Same thing, start_flush_work() has exactly one caller: flush_work().

> > It was added by Oleg in commit:
> >
> >   a67da70dc095 ("workqueues: lockdep annotations for flush_work()")
> 
> No, these annotations were moved later into start_flush, iiuc...
> 
> This
> 
> 	lock_map_acquire(&work->lockdep_map);
> 	lock_map_release(&work->lockdep_map);
> 
> was added by another commit 0976dfc1d0cd80a4e9dfaf87bd8744612bde475a
> "workqueue: Catch more locking problems with flush_work()", and at
> first glance it is fine.

Those are fine and are indeed the flush_work() vs work inversion.

The two straight forward annotations are:

flush_work(work)	process_one_work(wq, work)
  A(work)		  A(work)
  R(work)		  work->func(work);
			  R(work)

Which catches:

Task-1:			work:

  mutex_lock(&A);	mutex_lock(&A);
  flush_work(work);


And the analogous:

flush_workqueue(wq)	process_one_work(wq, work)
  A(wq)			  A(wq)
  R(wq)			  work->func(work);
			  R(wq)


The thing I puzzled over was flush_work() (really start_flush_work())
doing:

        if (pwq->wq->saved_max_active == 1 || pwq->wq->rescuer)
                lock_map_acquire(&pwq->wq->lockdep_map);
        else
                lock_map_acquire_read(&pwq->wq->lockdep_map);
        lock_map_release(&pwq->wq->lockdep_map);

Why does flush_work() care about the wq->lockdep_map?

The answer is because, for single-threaded workqueues, doing
flush_work() from a work is a potential deadlock:

workqueue-thread:

	work-n:
	  flush_work(work-n+1);

	work-n+1:


Will not be going anywhere fast..

And by taking the wq->lockdep_map from flush_work(), but _only_ when
single-threaded (or rescuer, see other emails), and by doing that, it
forces a recursive lock deadlock message like:

  process_one_work(wq, work)
   A(wq)
   A(work)

   work->func(work)
     flush_work(work2)
       A(work2)
       R(work2)

       A(wq) <-- recursive lock deadlock




Make sense?