From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDA887080A for ; Fri, 24 Jan 2025 20:36:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737751018; cv=none; b=t8cdEJ3CeHoeGRxjB3BQ3cNK9+FaicZO9CklwsajzpHHW93acJGGhRbrPSyKtK/gQp1eZYKzMVmEnBbpZuY8Miin7GBGiH5AeF7HzNcLb0P//hT+6ZGysuZqVztSjD9f3iJ6pHlL/2kUyrQqXSfuW2tEVaJWMH/2HT6bc5+5vM8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737751018; c=relaxed/simple; bh=oT/I0HTK1NGcwl7T9NpHluv2cwWMvqfm3nX16zXk8h8=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=kh1ajk+Hxe+Y81Pxe5MVvdcdRD/F6H4TYMnEBmNphk2F++iB+vurzCK8onLhpPbihW+ysSATqELEE7RLp05h9+zjBqxU4xFQ3zqDDoHLhFKMA2tAsRRA1I1sYPcdi1ZXciq28mGFNUwxK61v+i0fafHZFLwtnyQX7rjIruqN2iU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=iSfDqcHn; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iSfDqcHn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737751015; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FLs21lZZ8aY4y/zUCxxiI+oFbe6UlNBhVMGVIgTnCD4=; b=iSfDqcHnTIYxK3LkT/wm/6jr0BuiKg5LGxRXWxnRRFSsIdsL5j7jt1aWdqUU3nC7moAbbh fX8hROfH2AOrVzAJOdcmp2z67Pxkp1wM1wPfcdxKqVXjW+em1fN+Zx8iWCDzmeX4xrZmyq vuVt5ybhhlxPcWpSKszaDDH5mYb57z0= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-692-EcDY4rVgNOynVSL9R51CSg-1; Fri, 24 Jan 2025 15:36:23 -0500 X-MC-Unique: EcDY4rVgNOynVSL9R51CSg-1 X-Mimecast-MFC-AGG-ID: EcDY4rVgNOynVSL9R51CSg Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7b6d0be4fb8so402814985a.2 for ; Fri, 24 Jan 2025 12:36:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737750983; x=1738355783; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FLs21lZZ8aY4y/zUCxxiI+oFbe6UlNBhVMGVIgTnCD4=; b=LNp4jIoOU+kxekfv7/STvELdKcJ/8DGBka5U7WEQnx9G9hmbIOE+wz8QXFFUjbpapU ogwlQ2LMgC5DtsbpdwwfXcYTOVov2S0qzMUxEEYdFpv7SieEehdlQ7aWUemZSDSznXSi /UgW/dbAIuTE0juHuSYS9MzBbjWpZyxROyW6gDAAABA3PMQLZwFbvBgx6XEKcuWlo08l IBTfenXice2hn9sRvbSvcxf0J2AdaSORFsumnoIpmUzMPVSU9E7ghuiFCe3BCVGVBgPK mT3B7g6mhZ1DXRtR9gzJMGzj4OGMbAvO0Iq3Wgs7U0t1vuLPowlANwrQnrN9Av7Esc0G ubKg== X-Forwarded-Encrypted: i=1; AJvYcCXH9OAVcZSwTVKl/ZoBE+SCgCXXrdsqqF3ZOlEUB5qM6V8aYbuk5V6XXQXF+5QyQsqCEnKeUo0dR6l0DyQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9tNjovrWw3T+adZKBUpL5MjOo3pShpdrTam5fgY/Rx+P/McO7 +K4ymE2UANslNG9I5PbzMjq+zrfcEzDEi/wYs1ExKGVY1q0xw87CknDv5ja/2eGWMPa8kh3i8bK nWYAIwpTYZbDn4xMhDRSQ1SOd0TTTZFnBfceMA7Oitvu7NGBhDGaz2Bj2asPzkA== X-Gm-Gg: ASbGncvvjSoUuLvHfIcH1eX52YEd+db7vOlmV7sDQhL5Q3gfDxWt7mo1uieEzb6d1go YvDLsTLv3ND4gNVCY+UWqkyaAQFbOgowD6EqXhXTydsAq6Pzt7Xc0MX6ZrUOd09CZDHHNW+qsD7 uw6wjWjF2/rCcltbpTxwgcMT0iGaXJTRREhBqJwvzB5mRgGpN8Ms5ln4VSva/TbYM2CldCccrxy Sn24ENAUj2fY6guHTR8Wxp0meEWdxAsIwRnjEzkNfrDsgFPUmzGld7HcyVMhVLs3PvLESgu0iCv XfrWDGIbrJbZeecVmaRREyC8+OuueiApxMInJ+Yg X-Received: by 2002:a05:620a:1790:b0:7b6:d089:274c with SMTP id af79cd13be357-7be6320bb6dmr4766307485a.34.1737750982990; Fri, 24 Jan 2025 12:36:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IGKfqgIyKNMUv1Jn9Z0ULjM0OcUzrITU84FYxPBzFidccy5R3c5ppRNBE5sbKwfbh8JuSMSVw== X-Received: by 2002:a05:620a:1790:b0:7b6:d089:274c with SMTP id af79cd13be357-7be6320bb6dmr4766304285a.34.1737750982609; Fri, 24 Jan 2025 12:36:22 -0800 (PST) Received: from ?IPV6:2601:188:c100:5710:315f:57b3:b997:5fca? ([2601:188:c100:5710:315f:57b3:b997:5fca]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7be9af03b3dsm127803685a.90.2025.01.24.12.36.21 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Jan 2025 12:36:22 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: <901c9b9f-d393-4b84-9b6e-fe79e316ee2d@redhat.com> Date: Fri, 24 Jan 2025 15:36:21 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/2] clocksource: Defer preempt_disable() after clocksource_verify_choose_cpus() To: paulmck@kernel.org Cc: John Stultz , Thomas Gleixner , Stephen Boyd , Feng Tang , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev References: <20250124185923.3610964-1-longman@redhat.com> <20250124185923.3610964-2-longman@redhat.com> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/24/25 3:25 PM, Paul E. McKenney wrote: > On Fri, Jan 24, 2025 at 01:59:23PM -0500, Waiman Long wrote: >> The following bug report happened in a PREEMPT_RT kernel. >> >> [ 30.957705] BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48 >> [ 30.957711] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2012, name: kwatchdog >> [ 30.962673] preempt_count: 1, expected: 0 >> [ 30.962676] RCU nest depth: 0, expected: 0 >> [ 30.962680] 3 locks held by kwatchdog/2012: >> [ 30.962684] #0: ffffffff8af2da60 (clocksource_mutex){+.+.}-{3:3}, at: clocksource_watchdog_kthread+0x13/0x50 >> [ 30.967703] #1: ffffffff8aa8d4d0 (cpu_hotplug_lock){++++}-{0:0}, at: clocksource_verify_percpu.part.0+0x5c/0x330 >> [ 30.972774] #2: ffff9fe02f5f33e0 ((batched_entropy_u32.lock)){+.+.}-{2:2}, at: get_random_u32+0x4f/0x110 >> [ 30.977827] Preemption disabled at: >> [ 30.977830] [] clocksource_verify_percpu.part.0+0x66/0x330 >> [ 30.982837] CPU: 33 PID: 2012 Comm: kwatchdog Not tainted 5.14.0-503.23.1.el9_5.x86_64+rt-debug #1 >> [ 30.982843] Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 04/29/2021 >> [ 30.982846] Call Trace: >> [ 30.982850] >> [ 30.983821] dump_stack_lvl+0x57/0x81 >> [ 30.983821] __might_resched.cold+0xf4/0x12f >> [ 30.983824] rt_spin_lock+0x4c/0x100 >> [ 30.988833] get_random_u32+0x4f/0x110 >> [ 30.988833] clocksource_verify_choose_cpus+0xab/0x1a0 >> [ 30.988833] clocksource_verify_percpu.part.0+0x6b/0x330 >> [ 30.993894] __clocksource_watchdog_kthread+0x193/0x1a0 >> [ 30.993898] clocksource_watchdog_kthread+0x18/0x50 >> [ 30.993898] kthread+0x114/0x140 >> [ 30.993898] ret_from_fork+0x2c/0x50 >> [ 31.002864] >> >> It is due to the fact that get_random_u32() is called in >> clocksource_verify_choose_cpus() with preemption disabled. >> If crng_ready() is true by the time get_random_u32() is called, The >> batched_entropy_32 local lock will be acquired. In PREEMPT_RT kernel, >> it is a rtmutex and we can't acquire it with preemption disabled. >> >> To avoid this problem, we can't call get_random_u32() with preemption >> disabled. However, smp_processor_id() has to be called with preemption >> disabled though and we have to exclude the current CPU from the >> cpus_chosen list to be tested. >> >> Extract current CPU removal code out from >> clocksource_verify_choose_cpus() and defer the preempt_disable() >> call to after clocksource_verify_choose_cpus() and before >> current CPU removal. Also use raw_smp_processor_id() in >> clocksource_verify_choose_cpus(). >> >> Fixes: 7560c02bdffb ("clocksource: Check per-CPU clock synchronization when marked unstable") >> Signed-off-by: Waiman Long > Good catch! > > But we don't need cryptographically secure random numbers (or blistering > speed) here. Substituting something like torture_random()%nr_cpu_ids > for get_random_u32_below() work? > > Thanx, Paul That should work too. I can update the patch to use a less random generator if others have no objection. Cheers, Longman > >> --- >> kernel/time/clocksource.c | 19 +++++++++++-------- >> 1 file changed, 11 insertions(+), 8 deletions(-) >> >> diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c >> index 77d9566d3aa6..8ffec49971ac 100644 >> --- a/kernel/time/clocksource.c >> +++ b/kernel/time/clocksource.c >> @@ -313,7 +313,6 @@ static void clocksource_verify_choose_cpus(void) >> if (n < 0) { >> /* Check all of the CPUs. */ >> cpumask_copy(&cpus_chosen, cpu_online_mask); >> - cpumask_clear_cpu(smp_processor_id(), &cpus_chosen); >> return; >> } >> >> @@ -322,9 +321,13 @@ static void clocksource_verify_choose_cpus(void) >> if (n == 0 || num_online_cpus() <= 1) >> return; >> >> - /* Make sure to select at least one CPU other than the current CPU. */ >> + /* >> + * Make sure to select at least one CPU other than the current CPU. >> + * raw_smp_processor_id() is used opportunistically here as >> + * preemption isn't disabled. >> + */ >> cpu = cpumask_first(cpu_online_mask); >> - if (cpu == smp_processor_id()) >> + if (cpu == raw_smp_processor_id()) >> cpu = cpumask_next(cpu, cpu_online_mask); >> if (WARN_ON_ONCE(cpu >= nr_cpu_ids)) >> return; >> @@ -349,9 +352,6 @@ static void clocksource_verify_choose_cpus(void) >> if (!WARN_ON_ONCE(cpu >= nr_cpu_ids)) >> cpumask_set_cpu(cpu, &cpus_chosen); >> } >> - >> - /* Don't verify ourselves. */ >> - cpumask_clear_cpu(smp_processor_id(), &cpus_chosen); >> } >> >> static void clocksource_verify_one_cpu(void *csin) >> @@ -373,15 +373,18 @@ void clocksource_verify_percpu(struct clocksource *cs) >> cpumask_clear(&cpus_ahead); >> cpumask_clear(&cpus_behind); >> cpus_read_lock(); >> - preempt_disable(); >> clocksource_verify_choose_cpus(); >> + preempt_disable(); >> + >> + /* Don't verify ourselves. */ >> + testcpu = smp_processor_id(); >> + cpumask_clear_cpu(testcpu, &cpus_chosen); >> if (cpumask_empty(&cpus_chosen)) { >> preempt_enable(); >> cpus_read_unlock(); >> pr_warn("Not enough CPUs to check clocksource '%s'.\n", cs->name); >> return; >> } >> - testcpu = smp_processor_id(); >> pr_info("Checking clocksource %s synchronization from CPU %d to CPUs %*pbl.\n", >> cs->name, testcpu, cpumask_pr_args(&cpus_chosen)); >> for_each_cpu(cpu, &cpus_chosen) { >> -- >> 2.47.1 >>