From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD8CA19C56C for ; Wed, 22 Jan 2025 14:55:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737557756; cv=none; b=deib5aduMsVRpinKMC7NkX9ToKucxpBTZpLN0Je8rAUqfn7A+deSmw2PWh/WQQWOpwL+MCDsU3Km3utCnjasw3QPWbm995ljmWKWG1tf0TZ6ATeQTdEmswymmNk8ZvkOo2ljOnmjuOSVJWOStM/FpINkkZ5BWhBYOcMWqhpFCcQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737557756; c=relaxed/simple; bh=7x0d6t93GpRsTvIfLTGewde0T+d1BiGxzLT2GLpiQkc=; h=From:Message-ID:Date:MIME-Version:Subject:To:Cc:References: In-Reply-To:Content-Type; b=kNh2QjCoQJvTRIA14Mq2PwnugEGqYSCNhgA3368bTOsNwt6roBhSm1ttxfTVPKD3NJ6Ykb0/oac8yYUjVCGe69GFeknRv1Z93Jj/8Nx7L1+ud36SR6fbz4LaU2nLpjFF6cJFWecCWdKHOMaHeNYrlbdTV/rVucyNNUF9UBrHLsw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LrMghD6u; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LrMghD6u" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737557753; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Zb23txu9/vz6Skm0yBi8X+lcXxL9jBlY5FEbSEQNdI=; b=LrMghD6u1ItnamwF8Has/R3iZmTO6M2mrREpTPXG+ud4FFBRYvQvIL5t/htVnAsz1r7A9w HcFgMXjt37k+WfF3tYy8AQrlghthHhaoethPLu9dFWKTV9cZU9yJ4fsT+OyFBNbBKWVPmv mAJtHdnQE9e/nwyWkRa/BdKISHdhazk= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-213-rdsbvvphPAu-uPNp_1Ypyw-1; Wed, 22 Jan 2025 09:55:52 -0500 X-MC-Unique: rdsbvvphPAu-uPNp_1Ypyw-1 X-Mimecast-MFC-AGG-ID: rdsbvvphPAu-uPNp_1Ypyw Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7b93c7ffaeeso138074985a.1 for ; Wed, 22 Jan 2025 06:55:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737557752; x=1738162552; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/Zb23txu9/vz6Skm0yBi8X+lcXxL9jBlY5FEbSEQNdI=; b=HKl0DnRN14x6xD7D5jf5Ng4ExyXlb/bNI8rMTiqHwF4BCNFNLShHXhkwIC4cCWd/x/ VIPLCLxAbVQkt1jUlTA3u50vRGRlSqoHOp29kFxj4LIHAG30SU0hBC4hYU7YtbEHJNo2 sVVEHfUfhzVwfhwSIrwY74xsU6TuzO0SxyZXyJzEqpfEbwEBSE4Lq5LjHhZy/Sx0Uh+5 KzHPxAFnUpHsbgtuxb/FhrA7MOiWKmRq6YO36C2tJVDNIddqoGVhj3PfjwX2oacxto0J 3Bssn9VX9R19bNb1FOds0x5iiH2rui8J8TKlEl5OYsf55TwTTiTY8eCQwtVxgwS8VG/+ VICQ== X-Forwarded-Encrypted: i=1; AJvYcCU2L2Ut3FOUkTkaPuOM0XsNBSbeJwMKZQxW1XprUxcdFrPavWsdmWOXAMb0vYza1jlSiJy3tUxerG9f9vc=@vger.kernel.org X-Gm-Message-State: AOJu0YzVsvd96RzOTppNQRasPNdKeZKH9127abyV/CvtQdx9/ZiVW4jR VNX3goZWeCSNNLk/9+xL9STDS7vcD51dvQvfcPjkDAdZv9UEBszK6xaVc3hOvckqRNsqKKqmhdV G4VBZAkB6/VFmpleaDe5ckdG70UOivvxqHNwOa2PXcHdYqWxcImb0kLwbpuaYSA== X-Gm-Gg: ASbGncsv2Xd/bEaXqIBoqvGw8cy5JijLDVe29LSbmpB99ViN9jch8mJlv5e6JxP7ppy JEEw/72gukKZTA5ud0kSAg9LCHjWjMzfuMt/JJjCKBHwOCa0wGiszktHALm9So/fG3O1IpIEurf F157l8qsPRnNdOmbY2uAfDURAivIh11FgldSXXi32mAa0sRyeQ/oVoesU6RuRvd27q94eegni2q +WkJcfawaDJTVayJ3fDGjpAykHiXf+Het0g2letvuC5BXWo4CaVYerrxFOmdw7tmDoqy7u3LePe nlf9BTt8B4jIVqQzF9I9YZnUq05LjmT6JREgGLse X-Received: by 2002:a05:620a:40ce:b0:7b6:72fc:b046 with SMTP id af79cd13be357-7be638a2d06mr3305529885a.7.1737557751985; Wed, 22 Jan 2025 06:55:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IGkoAd/q/adJ55ZeO5lKruxZBufWvv1uIDk1mxu5KngDCQYAka/jNVkOsKjBqSVCtqsY8eOSg== X-Received: by 2002:a05:620a:40ce:b0:7b6:72fc:b046 with SMTP id af79cd13be357-7be638a2d06mr3305527185a.7.1737557751664; Wed, 22 Jan 2025 06:55:51 -0800 (PST) Received: from ?IPV6:2601:188:c100:5710:315f:57b3:b997:5fca? ([2601:188:c100:5710:315f:57b3:b997:5fca]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7be61484989sm668822685a.52.2025.01.22.06.55.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 22 Jan 2025 06:55:51 -0800 (PST) From: Waiman Long X-Google-Original-From: Waiman Long Message-ID: Date: Wed, 22 Jan 2025 09:55:50 -0500 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] locking/semaphore: Use wake_q to wake up processes outside lock critical section To: Peter Zijlstra Cc: Ingo Molnar , Will Deacon , Boqun Feng , linux-kernel@vger.kernel.org References: <20250122011314.2869715-1-longman@redhat.com> <20250122103914.GI7145@noisy.programming.kicks-ass.net> Content-Language: en-US In-Reply-To: <20250122103914.GI7145@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 1/22/25 5:39 AM, Peter Zijlstra wrote: > On Tue, Jan 21, 2025 at 08:13:14PM -0500, Waiman Long wrote: >> A circular lock dependency splat has been seen involving down_trylock(). >> >> [ 4011.795602] ====================================================== >> [ 4011.795603] WARNING: possible circular locking dependency detected >> [ 4011.795607] 6.12.0-41.el10.s390x+debug >> [ 4011.795612] ------------------------------------------------------ >> [ 4011.795613] dd/32479 is trying to acquire lock: >> [ 4011.795617] 0015a20accd0d4f8 ((console_sem).lock){-.-.}-{2:2}, at: down_trylock+0x26/0x90 >> [ 4011.795636] >> [ 4011.795636] but task is already holding lock: >> [ 4011.795637] 000000017e461698 (&zone->lock){-.-.}-{2:2}, at: rmqueue_bulk+0xac/0x8f0 >> >> the existing dependency chain (in reverse order) is: >> -> #4 (&zone->lock){-.-.}-{2:2}: >> -> #3 (hrtimer_bases.lock){-.-.}-{2:2}: >> -> #2 (&rq->__lock){-.-.}-{2:2}: >> -> #1 (&p->pi_lock){-.-.}-{2:2}: >> -> #0 ((console_sem).lock){-.-.}-{2:2}: > The whole #3->#4 thing seems dodgy, where is that? Specifically > hrtimer_bases.lock is a raw_spinlock, while zone->lock is a spinlock, > this is not a valid nesting. Ah, you are right. That is another raw_spinlock to spinlock nesting issue that needs to be addressed. [ 4011.795646] -> #4 (&zone->lock){-.-.}-{2:2}: [ 4011.795650] __lock_acquire+0xe86/0x1cc0 [ 4011.795655] lock_acquire.part.0+0x258/0x630 [ 4011.795657] lock_acquire+0xb8/0xe0 [ 4011.795659] _raw_spin_lock_irqsave+0xb4/0x120 [ 4011.795663] rmqueue_bulk+0xac/0x8f0 [ 4011.795665] __rmqueue_pcplist+0x580/0x830 [ 4011.795667] rmqueue_pcplist+0xfc/0x470 [ 4011.795669] rmqueue.isra.0+0xdec/0x11b0 [ 4011.795671] get_page_from_freelist+0x2ee/0xeb0 [ 4011.795673] __alloc_pages_noprof+0x2c2/0x520 [ 4011.795676] alloc_pages_mpol_noprof+0x1fc/0x4d0 [ 4011.795681] alloc_pages_noprof+0x8c/0xe0 [ 4011.795684] allocate_slab+0x320/0x460 [ 4011.795686] ___slab_alloc+0xa58/0x12b0 [ 4011.795688] __slab_alloc.isra.0+0x42/0x60 [ 4011.795690] kmem_cache_alloc_noprof+0x304/0x350 [ 4011.795692] fill_pool+0xf6/0x450 [ 4011.795697] debug_object_activate+0xfe/0x360 [ 4011.795700] enqueue_hrtimer+0x34/0x190 [ 4011.795703] __run_hrtimer+0x3c8/0x4c0 [ 4011.795705] __hrtimer_run_queues+0x1b2/0x260 [ 4011.795707] hrtimer_interrupt+0x316/0x760 [ 4011.795709] do_IRQ+0x9a/0xe0 [ 4011.795712] do_irq_async+0xf6/0x160 We probably need to look at debug object code to avoid doing memory allocation under some circumstances. PROVE_RAW_LOCK_NESTING was not enabled and so this case wasn't caught. Will enable that in the next minor release to find out more instances of such bugs. Anyway, do you think this patch is worth taking? There are may be other cases like this that are hiding somewhere and showing up from time to time. Cheers, Longman