From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B71A1EEA43 for ; Sat, 25 Jan 2025 02:11:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737771097; cv=none; b=OF7jb7E6upDNerjvqHuE6QgO8PfOcNzfJlUuO9El0sPYmO6m1k1H7kVecichTBzOnAmka37HpnXfLOqdhaNrtPz/bv6Rd1FNMCYxkHEkNrNr2ENU9iAnKrjFYZQ+rH1PBHwBKZU7nSdYRy/rRUVU8fJ9SVRw2BaLMGgGJisOJN8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737771097; c=relaxed/simple; bh=cW5ZleZ3ZcP2MFmjDo1lqMOYTP8PZkxzfXP5zozKGu8=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=YC1LQfQwiS4wskr+1cCaGVDd3Kn9RnTLp9R0UvW20XROGA0MF7Pdd4zNqoQBUC/87FDWi2XHbMZ4/h1MvnD9xo+wJuy+iXHXLMZJpWhm5j9dxU5+Ylm110BVxqf50VHxRx5HAs9NJAjz4Oc3BFq4xwWvDvxGBDqC9Z1QX6WN0e4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=cCptbxI0; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="cCptbxI0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737771093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D6fDd2hy1gyD9MyTWY3pTGcfQVcmQI8cg8mKBZz1NrU=; b=cCptbxI0uRPsnP7xQt8chuYEKx31+jrVbjZYPCbMyIvDREVUPMockmNXJ/m9RLxwzQ9o9Q qtPdQwlVLTIZsAmJfyAv4upkcPCHUTiV5rNgeXAGrGHREDypdziyhoTqFd9+5xs/gbYuCD mLwtu4FlM/pY5YuklS+MHKD+Dua8IuI= Received: from mail-yb1-f197.google.com (mail-yb1-f197.google.com [209.85.219.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-373-Rq0TZ7KaOaGUGcNiY0HbTw-1; Fri, 24 Jan 2025 21:11:29 -0500 X-MC-Unique: Rq0TZ7KaOaGUGcNiY0HbTw-1 X-Mimecast-MFC-AGG-ID: Rq0TZ7KaOaGUGcNiY0HbTw Received: by mail-yb1-f197.google.com with SMTP id 3f1490d57ef6-e3fea893dc5so9606118276.1 for ; Fri, 24 Jan 2025 18:11:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737771089; x=1738375889; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D6fDd2hy1gyD9MyTWY3pTGcfQVcmQI8cg8mKBZz1NrU=; b=Cd7IwbjpASFEMn3EbVeOD5VOGbKYibyEqYJurSI43NywtCD/BxJGjC1pw89+WZS7ap Hvtl1nOJ/CP3uwI4Tn9L3mfHgcdRcU1ryyXPv0v0+EexAFN01X9KhrQWJf2BTGh2qFyw /QFHTUCpP7/lWeQxgH9R2lR12D5/m1evqz3+vrCsNiE5u8KMx/tTF2mWfYOMrIrDtNmK u5AN4Gp1WpMLSZXeqlzk0ieqN8CvzodS1OX4RtBXy5sC1jHytHKhNmXuyyzzv2txOPt/ c3VNJdYOv7oB6RnmEWpWNXhziL6xond4F8iduDC9BtueAu/uFE4a5E4pl/W3935jBtKZ ErLA== X-Gm-Message-State: AOJu0YwKUNCzfWHY/1YIj44qoZm6au52om1rwlv/oM8e8AeJXwifQuc4 jhfnxJo9W6MpRJa3LhGl7VJVUvUpMyASl/LzjP3hh6j33coQ0CpKXi+B9TaEHiK7+02sAHPYS8z dZV8DbHjhoFhP5/8+yJukvWCWlgOO7UyCt+TwuaqONmZWketehYfcQagRBHXq5Q== X-Gm-Gg: ASbGncsx1B0Si9H4jEi88wsKhOirxecgdRkojOo2pqFrJvZA0BXWdX9JQ4IwhCJjEMq nhf9mSoOJVU44BnAB/ql8d8ah1ob5uhjeHLG2vvQ0iR9KrKr4Y51WmaF0q1YNQyIP2vxg0F70nl w4fNASyPUg/u3S54sqTqAeIsQk4SHl/HNn6SAjh8ObBjbytS2NH6mkNcbRqvWMIosHPXFuzSA+2 EkoVbfe+1D8/Ey5ABNhI9NPBQRQqzslfldnXD+JTSpwfC9qA5fDjmZgDDtAitNZv2WCjI2F6qfU up17gE0PNdNy5+oKj74ToWKbNCrQqaJgN1GT55f9CXNNWiFHoZM= X-Received: by 2002:a05:690c:6204:b0:6ee:6a2a:a517 with SMTP id 00721157ae682-6f75855473dmr54188697b3.18.1737771089312; Fri, 24 Jan 2025 18:11:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IGlSNzA7NHejkrJHCqs8KBHv/2LfSlzLBJfGt6fGK5BU5IkTRO27YQkgJGsxYw966ZgFyBJkA== X-Received: by 2002:a05:690c:6204:b0:6ee:6a2a:a517 with SMTP id 00721157ae682-6f75855473dmr54188557b3.18.1737771089025; Fri, 24 Jan 2025 18:11:29 -0800 (PST) Received: from ?IPV6:2601:188:c100:5710:315f:57b3:b997:5fca? ([2601:188:c100:5710:315f:57b3:b997:5fca]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6f7578770bcsm5909497b3.13.2025.01.24.18.11.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Jan 2025 18:11:28 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: <7532b16e-2a1d-42ee-af1c-dfb91a55d741@redhat.com> Date: Fri, 24 Jan 2025 21:11:27 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 2/2] clocksource: Use get_random_bytes() in clocksource_verify_choose_cpus() To: John Stultz , Thomas Gleixner , Stephen Boyd , Feng Tang , "Paul E. McKenney" , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt Cc: linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev References: <20250125015442.3740588-1-longman@redhat.com> <20250125015442.3740588-2-longman@redhat.com> Content-Language: en-US In-Reply-To: <20250125015442.3740588-2-longman@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/24/25 8:54 PM, Waiman Long wrote: > The following bug report happened in a PREEMPT_RT kernel. > > [ 30.957705] BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48 > [ 30.957711] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2012, name: kwatchdog > [ 30.962673] preempt_count: 1, expected: 0 > [ 30.962676] RCU nest depth: 0, expected: 0 > [ 30.962680] 3 locks held by kwatchdog/2012: > [ 30.962684] #0: ffffffff8af2da60 (clocksource_mutex){+.+.}-{3:3}, at: clocksource_watchdog_kthread+0x13/0x50 > [ 30.967703] #1: ffffffff8aa8d4d0 (cpu_hotplug_lock){++++}-{0:0}, at: clocksource_verify_percpu.part.0+0x5c/0x330 > [ 30.972774] #2: ffff9fe02f5f33e0 ((batched_entropy_u32.lock)){+.+.}-{2:2}, at: get_random_u32+0x4f/0x110 > [ 30.977827] Preemption disabled at: > [ 30.977830] [] clocksource_verify_percpu.part.0+0x66/0x330 > [ 30.982837] CPU: 33 PID: 2012 Comm: kwatchdog Not tainted 5.14.0-503.23.1.el9_5.x86_64+rt-debug #1 > [ 30.982843] Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 04/29/2021 > [ 30.982846] Call Trace: > [ 30.982850] > [ 30.983821] dump_stack_lvl+0x57/0x81 > [ 30.983821] __might_resched.cold+0xf4/0x12f > [ 30.983824] rt_spin_lock+0x4c/0x100 > [ 30.988833] get_random_u32+0x4f/0x110 > [ 30.988833] clocksource_verify_choose_cpus+0xab/0x1a0 > [ 30.988833] clocksource_verify_percpu.part.0+0x6b/0x330 > [ 30.993894] __clocksource_watchdog_kthread+0x193/0x1a0 > [ 30.993898] clocksource_watchdog_kthread+0x18/0x50 > [ 30.993898] kthread+0x114/0x140 > [ 30.993898] ret_from_fork+0x2c/0x50 > [ 31.002864] > > It is due to the fact that get_random_u32() is called in > clocksource_verify_choose_cpus() with preemption disabled. > If crng_ready() is true by the time get_random_u32() is called, The > batched_entropy_32 local lock will be acquired. In PREEMPT_RT kernel, > it is a rtmutex and we can't acquire it with preemption disabled. > > Fix this problem by using the less random get_random_bytes() function > which will not take any lock. In fact, it has the same random-ness as > get_random_u32_below() when crng_ready() is false. > > Fixes: 7560c02bdffb ("clocksource: Check per-CPU clock synchronization when marked unstable") Oh, I should have added Suggested-by: Paul E. McKenney > Signed-off-by: Waiman Long > --- > kernel/time/clocksource.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c > index 77d9566d3aa6..659c4b79119c 100644 > --- a/kernel/time/clocksource.c > +++ b/kernel/time/clocksource.c > @@ -340,9 +340,13 @@ static void clocksource_verify_choose_cpus(void) > * and no replacement CPU is selected. This gracefully handles > * situations where verify_n_cpus is greater than the number of > * CPUs that are currently online. > + * > + * The get_random_bytes() is used here to avoid taking lock with > + * preemption disabled. > */ > for (i = 1; i < n; i++) { > - cpu = get_random_u32_below(nr_cpu_ids); > + get_random_bytes(&cpu, sizeof(cpu)); > + cpu %= nr_cpu_ids; > cpu = cpumask_next(cpu - 1, cpu_online_mask); > if (cpu >= nr_cpu_ids) > cpu = cpumask_first(cpu_online_mask);