From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FF3C258EFF for ; Fri, 22 May 2026 02:09:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779415747; cv=none; b=WmBbmX+bIuobw++LAAehg24Ed8KsH+o559zgj1uBFNEsViPdqMDdz36BCi8PNuqV21QNuLHQyAk7o+KNiLisREJh8FmCNWAFW0w4CmpOEkl3PIlyXfC8Fp0YygnIqWaTRwyB5TL646OqC6+GumNO1DMIz4aBTXBq99MhbROf85Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779415747; c=relaxed/simple; bh=d/bl2vNDWJxyz2BCeaKg9nIx2Ty9+D11QH3ZHUWwq0Y=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=q2D+FXuAxSpWu54RD5RHTdu2a5lPrWlGTfIbXJKBtENJEEB5M9PmpZcZmJ/ojrRgM84IY4cVX4/Co3LDJcT/ruiNYM4/26p6/BQ7AKTFTXwdsQjwN8mAZMHFgkHgBiXoCqrV1KPHmjRSZcH1KYyXTLKyX8GF42RqPaMw+OqH2XA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=hao6+yA3; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hao6+yA3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1779415744; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U5nkuokd90UzjF4L/RiSZbmMH6uwJiJrisA648Cii7c=; b=hao6+yA3/kqTAEUdrSwcwLVKjErSAhWslEATTm9Ii2QSW5NduaYzAs+3uQiHhOd2Haq1/d dXfeSr1UPwKubqTJJtBrLgV32nprFbBgbEncYOtJK/m3V4WSnv/JWk5sh/xmK3cxQ0xc15 HgzBFwuvGz3AkIivHA3CPvjNpn9YH6g= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-126-5kFFpdDDNCWUYxiRkDg_4A-1; Thu, 21 May 2026 22:09:02 -0400 X-MC-Unique: 5kFFpdDDNCWUYxiRkDg_4A-1 X-Mimecast-MFC-AGG-ID: 5kFFpdDDNCWUYxiRkDg_4A_1779415741 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1BB1318002C9; Fri, 22 May 2026 02:09:01 +0000 (UTC) Received: from [10.22.80.218] (unknown [10.22.80.218]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 988DE180034E; Fri, 22 May 2026 02:08:59 +0000 (UTC) Message-ID: Date: Thu, 21 May 2026 22:08:58 -0400 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] locking/rwsem: Remove reader optimistic lock stealing To: Peng Wang , peterz@infradead.org, mingo@redhat.com, will@kernel.org, boqun@kernel.org, dbueso@suse.de Cc: linux-kernel@vger.kernel.org References: <20260521095926.29363-1-peng_wang@linux.alibaba.com> Content-Language: en-US From: Waiman Long In-Reply-To: <20260521095926.29363-1-peng_wang@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 On 5/21/26 5:59 AM, Peng Wang wrote: > Reader optimistic lock stealing, introduced by commit 1a728dff855a > ("locking/rwsem: Enable reader optimistic lock stealing") and made more > aggressive by commit 617f3ef95177 ("locking/rwsem: Remove reader > optimistic spinning"), allows a reader entering the slowpath to bypass > the wait queue and acquire the lock directly when WRITER_LOCKED and > HANDOFF bits are not set. > > This causes severe writer starvation in workloads where readers hold > the lock for extended periods, such as Direct I/O operations which > hold inode->i_rwsem for the entire duration of iomap_dio_rw(). A > common example is log-structured storage where one thread appends via > DIO writes while another thread tails the log via DIO reads -- a > pattern seen in database redo-log replay and shared-storage > replication. It is generally assume that reader lock critical section is shorter than that of writer. In this particular case, does the reader critical section run longer than the writer's one? Reader lock stealing should only happen if the previous lock owner is a writer. So readers and writer should at most alternately own the lock if there are many readers waiting. Of course, if a reader own the lock, it will wake up the remaining readers in the wait queue. > > The problem occurs because: > > 1. A reader entering the slowpath (due to RWSEM_FLAG_WAITERS being set) > can still steal the lock as the steal condition only checks > WRITER_LOCKED and HANDOFF, not WAITERS. This is inconsistent with > the fast path which already blocks readers when WAITERS is set (via > RWSEM_READ_FAILED_MASK). > > 2. In the window between the last reader releasing the lock and the > waiting writer being scheduled (~10-100us), a new reader can steal > the lock in ~50-100ns. This race is structurally inevitable due to > the 1000x speed difference between atomic operations vs context > switching. > > 3. Each stolen read lock is held for the full DIO duration (potentially > milliseconds), and the pattern repeats until the 4ms HANDOFF timeout. > This effectively taxes every write operation with a ~4ms penalty. > > Performance impact measured with a DIO mixed read/write workload > (1 writer + 1 reader, O_DIRECT, ext4): > > NVMe SSD: Before After > Write-only baseline: 397 MB/s 397 MB/s (no change) > Mixed write throughput: 11 MB/s 350 MB/s (+31x) > Mixed write latency: 880 us 23 us (-38x) > Mixed read throughput: 95 MB/s 95 MB/s (no change) Does the reader wait for something else and taking other locks after acquiring the read lock? Can you point me of the reader critical section for this particular workload? Cheers, Longman > > Fixes: 617f3ef95177 ("locking/rwsem: Remove reader optimistic spinning") > Signed-off-by: Peng Wang > --- > kernel/locking/lock_events_list.h | 1 - > kernel/locking/rwsem.c | 33 --------------------------------- > 2 files changed, 34 deletions(-) > > diff --git a/kernel/locking/lock_events_list.h b/kernel/locking/lock_events_list.h > index 97fb6f3f840a..35b45576bee4 100644 > --- a/kernel/locking/lock_events_list.h > +++ b/kernel/locking/lock_events_list.h > @@ -65,7 +65,6 @@ LOCK_EVENT(rwsem_opt_lock) /* # of opt-acquired write locks */ > LOCK_EVENT(rwsem_opt_fail) /* # of failed optspins */ > LOCK_EVENT(rwsem_opt_nospin) /* # of disabled optspins */ > LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ > -LOCK_EVENT(rwsem_rlock_steal) /* # of read locks by lock stealing */ > LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ > LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */ > LOCK_EVENT(rwsem_rlock_handoff) /* # of read lock handoffs */ > diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c > index bda5577339c0..40b141c5765f 100644 > --- a/kernel/locking/rwsem.c > +++ b/kernel/locking/rwsem.c > @@ -1017,42 +1017,9 @@ static struct rw_semaphore __sched * > rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int state) > { > long adjustment = -RWSEM_READER_BIAS; > - long rcnt = (count >> RWSEM_READER_SHIFT); > struct rwsem_waiter waiter, *first; > DEFINE_WAKE_Q(wake_q); > > - /* > - * To prevent a constant stream of readers from starving a sleeping > - * writer, don't attempt optimistic lock stealing if the lock is > - * very likely owned by readers. > - */ > - if ((atomic_long_read(&sem->owner) & RWSEM_READER_OWNED) && > - (rcnt > 1) && !(count & RWSEM_WRITER_LOCKED)) > - goto queue; > - > - /* > - * Reader optimistic lock stealing. > - */ > - if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF))) { > - rwsem_set_reader_owned(sem); > - lockevent_inc(rwsem_rlock_steal); > - > - /* > - * Wake up other readers in the wait queue if it is > - * the first reader. > - */ > - if ((rcnt == 1) && (count & RWSEM_FLAG_WAITERS)) { > - raw_spin_lock_irq(&sem->wait_lock); > - if (sem->first_waiter) > - rwsem_mark_wake(sem, RWSEM_WAKE_READ_OWNED, > - &wake_q); > - raw_spin_unlock_irq(&sem->wait_lock); > - wake_up_q(&wake_q); > - } > - return sem; > - } > - > -queue: > waiter.task = current; > waiter.type = RWSEM_WAITING_FOR_READ; > waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT; > -- > 2.39.3 >