From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753141Ab2G1UoU (ORCPT ); Sat, 28 Jul 2012 16:44:20 -0400 Received: from mail-we0-f174.google.com ([74.125.82.174]:58128 "EHLO mail-we0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753107Ab2G1UoS (ORCPT ); Sat, 28 Jul 2012 16:44:18 -0400 Subject: Re: [PATCH 2/3] Introduce percpu rw semaphores From: Eric Dumazet To: Mikulas Patocka Cc: Jeff Moyer , Jan Kara , Alexander Viro , Jens Axboe , "Alasdair G. Kergon" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, dm-devel@redhat.com, lwoodman@redhat.com, Andrea Arcangeli , kosaki.motohiro@jp.fujitsu.com In-Reply-To: References: <20120628111541.GB17515@quack.suse.cz> Content-Type: text/plain; charset="UTF-8" Date: Sat, 28 Jul 2012 22:44:12 +0200 Message-ID: <1343508252.2626.13184.camel@edumazet-glaptop> Mime-Version: 1.0 X-Mailer: Evolution 2.28.3 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, 2012-07-28 at 12:41 -0400, Mikulas Patocka wrote: > Introduce percpu rw semaphores > > When many CPUs are locking a rw semaphore for read concurrently, cache > line bouncing occurs. When a CPU acquires rw semaphore for read, the > CPU writes to the cache line holding the semaphore. Consequently, the > cache line is being moved between CPUs and this slows down semaphore > acquisition. > > This patch introduces new percpu rw semaphores. They are functionally > identical to existing rw semaphores, but locking the percpu rw semaphore > for read is faster and locking for write is slower. > > The percpu rw semaphore is implemented as a percpu array of rw > semaphores, each semaphore for one CPU. When some thread needs to lock > the semaphore for read, only semaphore on the current CPU is locked for > read. When some thread needs to lock the semaphore for write, semaphores > for all CPUs are locked for write. This avoids cache line bouncing. > > Note that the thread that is locking percpu rw semaphore may be > rescheduled, it doesn't cause bug, but cache line bouncing occurs in > this case. > > Signed-off-by: Mikulas Patocka I am curious to see how this performs with 4096 cpus ? Really you shouldnt use rwlock in a path if this might hurt performance. RCU is probably a better answer. (bdev->bd_block_size should be read exactly once )