From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B45715CD41 for ; Fri, 24 Jan 2025 20:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737751311; cv=none; b=RtgDpj695KUfAECUD+p7gCduUtaQgwLlj16ia4cbb2Np9UUrXbUSajyNzoAlDmnN1obX4De/fLwvPmFmoNpbvNh/AnLgslB2xbE3QfbDlkhAqKh5Q72ygFT+Cy9wbpNU//nopG6Fv/aE2DkFNsRY7+h7HWXApINF8Kb2CKYAHxU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737751311; c=relaxed/simple; bh=cdGFXcsehHzjccMA0addH9qHb6t8Bq8DG7VzEUYUbzk=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=dH1s85dHF6fCQrNOpvPVHD45M1UEvpCBsm0+YSXTpDU4VQ9NqUWcwZ3Vecs6KakqdOYDSvzRS+DUIZR3Qaq3YTX14nypa+IB1kaKOKtHFOMoDpfFw7Ad4yOwcaUQDQScCBo24IGXfROMkJEYxv6wAKV1BM6+GRPtrxi7IF7Zy2I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=BpAYu+7l; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BpAYu+7l" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737751309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z06IwRsUK7sR4WjL2LvTrEnN/ElRx2oGu5/nibKY9WI=; b=BpAYu+7lGR/XjpLrmBOTT0iXUxFryUFNvBXyaQJDUfITFhXlRksSJK/1cW8E4pyEn9UgI1 kR/sC4ULCwHR7EdfhX1Dh2fNVazmv5FTByRAr6+yqgThiJYGVMslU5i1JIPH4jBT251E3a b+BilZkQHQV/wa76F2h2VCNUNwYSCis= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-671-3a1ipDjhOoKSF7J3stnupQ-1; Fri, 24 Jan 2025 15:41:47 -0500 X-MC-Unique: 3a1ipDjhOoKSF7J3stnupQ-1 X-Mimecast-MFC-AGG-ID: 3a1ipDjhOoKSF7J3stnupQ Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-4679becb47eso59043441cf.3 for ; Fri, 24 Jan 2025 12:41:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737751307; x=1738356107; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Z06IwRsUK7sR4WjL2LvTrEnN/ElRx2oGu5/nibKY9WI=; b=Pf2kWFmYdcNqSYmIGaUQOlVnZSyQAzF1Y20j1XPiMDKgIwGPMLlCCDtV0B+czTA2oE FizULgZRqAJ06fu6LA0RXXIRTUlr2s8UCrSUhbInvaJVDPXFHe/12wtdKZUdFNikGOUJ 9hj0REA8gyYNyS0fJb7Zdi4Co+qtkgui0fQPHJ/cSoxmcrl2tbYYAhBl7s5W+U4FrE3q ymV/d0UViaPxtjVv3Pl08CP4HAAYSb8x7gJPiEGZl1puIQRMfThiGnPLbZoEXrqAjmav 3T4r8dP5/LsqTs1uM/pIJOV3y18yVmDn8MYyf6diiX53OGUCPMIs5QtClA5KNYXBTxVT qxmw== X-Forwarded-Encrypted: i=1; AJvYcCUK5s3BOIFVicONIPbbjrdnIjMVbTBDHR7Hu+3elh/YLRkRvTes2z1IHQhvHseR+ng1+VpxenowPZtY0Zk=@vger.kernel.org X-Gm-Message-State: AOJu0YxF3DW8PjD//+M9SsSzZCYb9Jj8Y94pNQzvHO9eyO2I8SqiIUjy YW24dKGGUortrn7ppEFk8uEmTYHjDgUWz/pD9IkMbvNM/kp7xr6tWMS+fi79qZKm9UXK/XWr3+J hnowzeSMv6BTMhnCzULZhYr85BNz/7PEW0j+SS77VVA/Jm/NHMozih+uVrkF0Pw== X-Gm-Gg: ASbGncs5aFIuY7G+gbHE7s0j/wEq68vnPZUDRFKjShznyzPDCjb4oF1fm/621bOm71E DhvUk78dTF9bey6cdJYJWTbyynTwtZ9L+OYdwh4MBt2s8X96Jzp7X5ej4or1CS1U39kYOB2WtK5 B9UcsAxFTpfc2q2hgRd6z5kboAV2IHsM1StE/mEYTBS2q93nWWa4AyR48QAg84lbG7R1wGNmvrX bnLeDzbtDD4ayi7ggIxr01q9z3HUwkIYQWP8FJTQ4TvqRPEhFO2QbAHFvfQXmRQGNG2Y2pQW31U xFkP+O/JUHnC+PzHByRjcH7Ufs7cPld7UdqOZxyl X-Received: by 2002:ac8:5753:0:b0:467:5926:fce8 with SMTP id d75a77b69052e-46e12a2c49fmr357524491cf.9.1737751307153; Fri, 24 Jan 2025 12:41:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IFZ9H3z92UfAxjZzKeIrLn3vB87WeMWbf3eunuDpEgOe+BYKFxFhOw5HvqNI6y4Ac/30fUvTA== X-Received: by 2002:ac8:5753:0:b0:467:5926:fce8 with SMTP id d75a77b69052e-46e12a2c49fmr357524261cf.9.1737751306809; Fri, 24 Jan 2025 12:41:46 -0800 (PST) Received: from ?IPV6:2601:188:c100:5710:315f:57b3:b997:5fca? ([2601:188:c100:5710:315f:57b3:b997:5fca]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-46e66b676b7sm13421501cf.61.2025.01.24.12.41.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Jan 2025 12:41:46 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: <33340159-3bee-48cc-85a0-830305e736aa@redhat.com> Date: Fri, 24 Jan 2025 15:41:45 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/2] clocksource: Defer preempt_disable() after clocksource_verify_choose_cpus() To: paulmck@kernel.org Cc: John Stultz , Thomas Gleixner , Stephen Boyd , Feng Tang , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev References: <20250124185923.3610964-1-longman@redhat.com> <20250124185923.3610964-2-longman@redhat.com> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/24/25 3:37 PM, Paul E. McKenney wrote: > On Fri, Jan 24, 2025 at 12:25:19PM -0800, Paul E. McKenney wrote: >> On Fri, Jan 24, 2025 at 01:59:23PM -0500, Waiman Long wrote: >>> The following bug report happened in a PREEMPT_RT kernel. >>> >>> [ 30.957705] BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48 >>> [ 30.957711] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2012, name: kwatchdog >>> [ 30.962673] preempt_count: 1, expected: 0 >>> [ 30.962676] RCU nest depth: 0, expected: 0 >>> [ 30.962680] 3 locks held by kwatchdog/2012: >>> [ 30.962684] #0: ffffffff8af2da60 (clocksource_mutex){+.+.}-{3:3}, at: clocksource_watchdog_kthread+0x13/0x50 >>> [ 30.967703] #1: ffffffff8aa8d4d0 (cpu_hotplug_lock){++++}-{0:0}, at: clocksource_verify_percpu.part.0+0x5c/0x330 >>> [ 30.972774] #2: ffff9fe02f5f33e0 ((batched_entropy_u32.lock)){+.+.}-{2:2}, at: get_random_u32+0x4f/0x110 >>> [ 30.977827] Preemption disabled at: >>> [ 30.977830] [] clocksource_verify_percpu.part.0+0x66/0x330 >>> [ 30.982837] CPU: 33 PID: 2012 Comm: kwatchdog Not tainted 5.14.0-503.23.1.el9_5.x86_64+rt-debug #1 >>> [ 30.982843] Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 04/29/2021 >>> [ 30.982846] Call Trace: >>> [ 30.982850] >>> [ 30.983821] dump_stack_lvl+0x57/0x81 >>> [ 30.983821] __might_resched.cold+0xf4/0x12f >>> [ 30.983824] rt_spin_lock+0x4c/0x100 >>> [ 30.988833] get_random_u32+0x4f/0x110 >>> [ 30.988833] clocksource_verify_choose_cpus+0xab/0x1a0 >>> [ 30.988833] clocksource_verify_percpu.part.0+0x6b/0x330 >>> [ 30.993894] __clocksource_watchdog_kthread+0x193/0x1a0 >>> [ 30.993898] clocksource_watchdog_kthread+0x18/0x50 >>> [ 30.993898] kthread+0x114/0x140 >>> [ 30.993898] ret_from_fork+0x2c/0x50 >>> [ 31.002864] >>> >>> It is due to the fact that get_random_u32() is called in >>> clocksource_verify_choose_cpus() with preemption disabled. >>> If crng_ready() is true by the time get_random_u32() is called, The >>> batched_entropy_32 local lock will be acquired. In PREEMPT_RT kernel, >>> it is a rtmutex and we can't acquire it with preemption disabled. >>> >>> To avoid this problem, we can't call get_random_u32() with preemption >>> disabled. However, smp_processor_id() has to be called with preemption >>> disabled though and we have to exclude the current CPU from the >>> cpus_chosen list to be tested. >>> >>> Extract current CPU removal code out from >>> clocksource_verify_choose_cpus() and defer the preempt_disable() >>> call to after clocksource_verify_choose_cpus() and before >>> current CPU removal. Also use raw_smp_processor_id() in >>> clocksource_verify_choose_cpus(). >>> >>> Fixes: 7560c02bdffb ("clocksource: Check per-CPU clock synchronization when marked unstable") >>> Signed-off-by: Waiman Long >> Good catch! >> >> But we don't need cryptographically secure random numbers (or blistering >> speed) here. Substituting something like torture_random()%nr_cpu_ids >> for get_random_u32_below() work? > I suppose I should add my concern... If we don't have preemption disabled > across this code, we cannot reliably avoid attempting to IPI ourselves. Does the CPU choosing process itsself needs to have preemption disabled? I thought it was because of need to use smp_processor_id() and have the current CPU excluded. Preemption is still disabled after that. Cheers, Longman