From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1753683AbbBRT5Q (ORCPT <rfc822;w@1wt.eu>);
	Wed, 18 Feb 2015 14:57:16 -0500
Received: from www.linutronix.de ([62.245.132.108]:49914 "EHLO
	Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1752324AbbBRT5O (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Wed, 18 Feb 2015 14:57:14 -0500
Date: Wed, 18 Feb 2015 20:57:10 +0100
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: linux-kernel@vger.kernel.org,
        linux-rt-users <linux-rt-users@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>, Carsten Emde <C.Emde@osadl.org>,
        John Kacur <jkacur@redhat.com>,
        Paul Gortmaker <paul.gortmaker@windriver.com>,
        Peter Zijlstra <a.p.zijlstra@chello.nl>,
        "H. Peter Anvin" <hpa@zytor.com>
Subject: Re: [PATCH RT 1/2] rwsem-rt: Do not allow readers to nest
Message-ID: <20150218195710.GH28763@linutronix.de>
References: <20140409024700.702797305@goodmis.org>
 <20140409025231.998774075@goodmis.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20140409025231.998774075@goodmis.org>
X-Key-Id: 97C4700B
X-Key-Fingerprint: 09E2 D1F3 9A3A FF13 C3D3  961C 0688 1C1E 97C4 700B
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

* Steven Rostedt | 2014-04-08 22:47:01 [-0400]:

>From: "Steven Rostedt (Red Hat)" <rostedt@goodmis.org>
>
>The readers of mainline rwsems are not allowed to nest, the rwsems in the
>PREEMPT_RT kernel should not nest either.

I applied this and this is the reason why cpufreq isn't working. What I
see in cpufreq is:
|         test.sh-788   [004] .......    61.416288: store: down_read_try
|         test.sh-788   [004] .......    61.416296: cpufreq_cpu_get: down_read_try
|         test.sh-788   [004] .......    61.416301: cpufreq_cpu_put.part.6: up_read
|         test.sh-788   [004] .......    61.416332: store: up_read

as you see, one code path takes the read path of rw_sema twice.

Looking at the generic implementation, we have:
|#define RWSEM_UNLOCKED_VALUE            0x00000000L
|#define RWSEM_ACTIVE_BIAS               0x00000001L
|#define RWSEM_WAITING_BIAS              (-RWSEM_ACTIVE_MASK-1)

| static inline int __down_read_trylock(struct rw_semaphore *sem)
| {
|         long tmp;
| 
|         while ((tmp = sem->count) >= 0) {
|                 if (tmp == cmpxchg(&sem->count, tmp,
|                                    tmp + RWSEM_ACTIVE_READ_BIAS)) {
|                         return 1;
|                 }
|         }
|         return 0;
| }

While sem->count is >= 0 we loop and take the semaphore. So we can have
five readers at once. The first writer would set count to a negative
value resulting in trylock failure.

|static inline void __down_read(struct rw_semaphore *sem)
|{
|        if (unlikely(atomic_long_inc_return((atomic_long_t*)&sem->count) <= 0))
|                rwsem_down_read_failed(sem);
|}

Here the same thing but without cmpxchg(). _If_ after an increment the
value is negative then we take slowpath. Otherwise we have the lock.

I think I'm going to revert this patch. Where is it written that
multiple readers of a RW-semaphore can not nest? According to the code
we can even have multiple readers without nesting (two+ processes may
take a reader lock).

Sebastian