From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E0D342A8B for ; Wed, 29 Jan 2025 17:03:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738170221; cv=none; b=DVO1mHHKbsPZR3I8YvhkhLI74e7FH9IieBOp1n35j2CATsB2g85purQvCVtXV/BbC54rcdPM4Mn/hKA2oEDTHvUCVgEzgdhtnWsof7w68d+dt+k/Z/z6vt5WoQ5VYNhrVLkHUJ9GL6ldpLZRNgbY46bYio9cy3nfgtbLbZmaUHQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738170221; c=relaxed/simple; bh=2TTwdKklraHYomZrGXBGqZjTgxzSjLWVPQYctmw1ZHY=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=LmdfHv14gPsUcrXH6qtdWqnu+pnpl+WLfdJqvNmvrKPgqvJXxDuIqfFS9dmSR146J455RVdU41ejDbTMBayWk4C0F1iPf0CvfxdZHfb5dxpsgrrxP85EottydrGqueqahqsmwCqphXDcOGFW8290wuyoWu2LpowGh+Bz16zVN1A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Ly3SfGXM; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ly3SfGXM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738170218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9ye1xyBra/0rih/flelrQKuAGjII/XW7CPUWlmNVnDc=; b=Ly3SfGXMMq9HjMgnFJl20kPT2bP0HY0B8xJ8VfeelzFHkeBX3q+6aRe0h1Cm7DrTpVsw+a 35fb4CdlSPKBHtoTqRb3nc+vUtnDJoHT+hD7KJ4qQjXt0bEIsHZc3MROyCk1zcCW1Ktisz QxOj3hHAOeg5XQAoD29pPLt1NspYl+E= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-664-5Nfg0qGhOtuVM7q5hIDB2Q-1; Wed, 29 Jan 2025 12:03:36 -0500 X-MC-Unique: 5Nfg0qGhOtuVM7q5hIDB2Q-1 X-Mimecast-MFC-AGG-ID: 5Nfg0qGhOtuVM7q5hIDB2Q Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7b6e1b0373bso1101036185a.2 for ; Wed, 29 Jan 2025 09:03:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738170216; x=1738775016; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9ye1xyBra/0rih/flelrQKuAGjII/XW7CPUWlmNVnDc=; b=gdTKvf/7a8dCfbjEN2lcKP3fdVq9YGjJ4b6jUKtXb5SpyOGPrk5YzbpHWfyvpQiZgo E7vYAQEdrlPwx88SGAcZxl9wCvlbMZtZkQOWIFNw2FTNW67t67ZvTJu5db01xwwM9Wk7 OF1JEtmQxyDUunChkCAqqXCaujbosL/qwoOJRSALyil9tIEdVojhXY0YKK5EMCecN99y kaBeHBkfbg8TIA5v87rifxJJ5vbYca5Q9SZAUauHk2csqypv95+xZlz9Q2YkRj4a8tMA iYSPswAx3P/oFwacB2YakHl2urMzixFHDK/QokPC0TLsGrlQ3rEJeH+dMl55v62DgFZy cNCg== X-Forwarded-Encrypted: i=1; AJvYcCUxcF1T1Jfnzn2zzlYKe0Fbo4wK+dCO3mGPVy9qjWqgEeRdDTMgNgqM2zpso9jjj6xA0etQ1Fpnt2u/ERA=@vger.kernel.org X-Gm-Message-State: AOJu0YwCnvrKwf1AzE6UPwLHZNW/bBXGdvD6Q7UZT5SKmBrpkMyPoRlN MC0/GdRvBImnCAQKtwFF9E7aDv0TNPH5vQ0NQzJsvzXEpARkLmvyyYu/Va4TzMHbQ3leeKLulg3 59Fcy1PXdN3pbTTBUMkXjHUJvsDHNxHkf/wXMcEw0K6n4jjI8RvapE0gRBRwWng== X-Gm-Gg: ASbGncuZuSbDeLOFP/0FQixre9IHTSuFqOc1vHBMSPcTK8gxhi9wHMe9aW1Aq8j5cn+ vqNxGlbm0wF3WrA+PGNNuKnElCGqzUcXUkNqMacWOHI0zyK7zgWH4vQFhbFG24QPCP8da8SuPK9 x4fsE6bkRWIxZQradPomMWdjrmA1c8PaSAjsfiTdOhAMEcAlRI56NY3wH8dnIJlnA5sUJRTWJid 1UFMYPyx0y6bPKdIjTJSJZnPV82p/T9RCTlXGaA8H8UORGW+KiXnBIuAwUu/NmtXxnFdiodsrNa iSnRDLulz1DM5e6BwfjOdb+jXbSOo4zlZnWt9hE90ga22pPXAR4= X-Received: by 2002:a05:620a:278f:b0:7b6:d58b:9de5 with SMTP id af79cd13be357-7bffcd14cc8mr650202285a.13.1738170216337; Wed, 29 Jan 2025 09:03:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IFrWE1/4Zj1nnLdqik4mpC4pfeNi38xUT7n99j0anMC/IUemwTH7eEG2cIlKpp32x7hF96Trw== X-Received: by 2002:a05:620a:278f:b0:7b6:d58b:9de5 with SMTP id af79cd13be357-7bffcd14cc8mr650195385a.13.1738170215802; Wed, 29 Jan 2025 09:03:35 -0800 (PST) Received: from ?IPV6:2601:408:c101:1d00:6621:a07c:fed4:cbba? ([2601:408:c101:1d00:6621:a07c:fed4:cbba]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7be9ae7f879sm637015485a.14.2025.01.29.09.03.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 09:03:35 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: <5abcb42b-91be-4043-a138-5d97cbcb5378@redhat.com> Date: Wed, 29 Jan 2025 12:03:34 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 2/2] clocksource: Use get_random_bytes() in clocksource_verify_choose_cpus() To: Sebastian Andrzej Siewior Cc: John Stultz , Thomas Gleixner , Stephen Boyd , Feng Tang , "Paul E. McKenney" , Clark Williams , Steven Rostedt , linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev References: <20250125015442.3740588-1-longman@redhat.com> <20250125015442.3740588-2-longman@redhat.com> <20250129163442.vczSGi63@linutronix.de> Content-Language: en-US In-Reply-To: <20250129163442.vczSGi63@linutronix.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/29/25 11:34 AM, Sebastian Andrzej Siewior wrote: > On 2025-01-24 20:54:42 [-0500], Waiman Long wrote: >> The following bug report happened in a PREEMPT_RT kernel. >> >> [ 30.957705] BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:48 >> [ 30.957711] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2012, name: kwatchdog >> [ 30.962673] preempt_count: 1, expected: 0 >> [ 30.962676] RCU nest depth: 0, expected: 0 >> [ 30.962680] 3 locks held by kwatchdog/2012: >> [ 30.962684] #0: ffffffff8af2da60 (clocksource_mutex){+.+.}-{3:3}, at: clocksource_watchdog_kthread+0x13/0x50 >> [ 30.967703] #1: ffffffff8aa8d4d0 (cpu_hotplug_lock){++++}-{0:0}, at: clocksource_verify_percpu.part.0+0x5c/0x330 >> [ 30.972774] #2: ffff9fe02f5f33e0 ((batched_entropy_u32.lock)){+.+.}-{2:2}, at: get_random_u32+0x4f/0x110 >> [ 30.977827] Preemption disabled at: >> [ 30.977830] [] clocksource_verify_percpu.part.0+0x66/0x330 >> [ 30.982837] CPU: 33 PID: 2012 Comm: kwatchdog Not tainted 5.14.0-503.23.1.el9_5.x86_64+rt-debug #1 >> [ 30.982843] Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 04/29/2021 >> [ 30.982846] Call Trace: >> [ 30.982850] >> [ 30.983821] dump_stack_lvl+0x57/0x81 >> [ 30.983821] __might_resched.cold+0xf4/0x12f >> [ 30.983824] rt_spin_lock+0x4c/0x100 >> [ 30.988833] get_random_u32+0x4f/0x110 >> [ 30.988833] clocksource_verify_choose_cpus+0xab/0x1a0 >> [ 30.988833] clocksource_verify_percpu.part.0+0x6b/0x330 >> [ 30.993894] __clocksource_watchdog_kthread+0x193/0x1a0 >> [ 30.993898] clocksource_watchdog_kthread+0x18/0x50 >> [ 30.993898] kthread+0x114/0x140 >> [ 30.993898] ret_from_fork+0x2c/0x50 >> [ 31.002864] >> >> It is due to the fact that get_random_u32() is called in >> clocksource_verify_choose_cpus() with preemption disabled. >> If crng_ready() is true by the time get_random_u32() is called, The >> batched_entropy_32 local lock will be acquired. In PREEMPT_RT kernel, >> it is a rtmutex and we can't acquire it with preemption disabled. >> >> Fix this problem by using the less random get_random_bytes() function >> which will not take any lock. In fact, it has the same random-ness as >> get_random_u32_below() when crng_ready() is false. > So how does get_random_bytes() not take any locks? It takes locks in my > tree. You two have a lock less tree? You are right. I forgot to check the crng_make_state() call in _get_random_bytes() which does take lock. > > In case your tree is not lock less yet, couldn't we perform the loop > verify_n_cpus+1 times without disabled preemption? Then disable > preemption after return from clocksource_verify_choose_cpus() and then > either remove current CPU from the list if it is or remove a random one > so that we get back to verify_n_cpus CPUs set. > > Alternatively, (and this might be easier) use migrate_disable() instead > of preempt_disable() and only use preempt_disable() within the > for_each_cpu() loop if delta is important (which I assume it is). > > But all this would avoid having to run with disabled preemption within > clocksource_verify_choose_cpus() while having the guarantees you need. I guess we will have to break clocksource_verify_choose_cpus() into 2 separate parts, one without preemption disabled and other one with preemption disabled. I don't think it is a good idea to just use migrate_disable() as we may get too much latency that can affect the test result. I will send out a v3 patch to fix that. Thanks, Longman