From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 491FD20E6F0 for ; Thu, 23 Jan 2025 12:55:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737636915; cv=none; b=N5ikIIj9BO7xDIfQ51nfyEZxu2yZqyhLowpCc/UnJeZsY8B8AXFz3+MubViVIpx/KunL0UaLMWFboNdMJ/UInPStlT+DoT5EBRyE2DIRIrKZoBNqXWfJ6uVBzn0bmY8kVepMDGqBDykD5qY4i48yOcWVQC0ucBR06CaO99CcAcU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737636915; c=relaxed/simple; bh=PK0hKNQx2gi6TUd2uw/SLKXm5WefN2PigFGxcvQibio=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ZI31uXJXYtLXAZXq8UuQD8vNMANlGCNHhjTf1ielGrQY+SkN5YnGTz2gtc0xM9+HFh1QjQe0DMeHBUmks8GaHwxNjtDrB9EARPc2XaOuI6VPuo2GWWn6toyTa9TLM3ID1bIxgwRtA1v60/gXGb2y3GAImPY5QOhE4nwFGFK/WSg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=yQjLmFsz; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=b/YopV1o; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=yQjLmFsz; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=b/YopV1o; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="yQjLmFsz"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="b/YopV1o"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="yQjLmFsz"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="b/YopV1o" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 46BB01F365; Thu, 23 Jan 2025 12:55:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737636910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dj7zuGN2Lh05CmjVYI8s2Zi2Euw7raMe3WLPq7IXEt8=; b=yQjLmFszWhTOfsgcuh8fwFh908PxXwuJaFFY0jgvK5xL/bJUVOsFdjDKV1KDbLh1D8roVz BO7DvkfQ+vbtFb3QEJmXTFcYS78OeNjES8dsjdEodekVSVKtCmm1xi6QzmqjEQ21loh5NQ HgyNEt6ooMihDAKbDNt9odxPju5nyiQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737636910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dj7zuGN2Lh05CmjVYI8s2Zi2Euw7raMe3WLPq7IXEt8=; b=b/YopV1ow2jefE4I2LV1kK6Le6rdjRHRwVBvc3n08Mw05oo3Ha9fHIAkbRKdSoA5+a/+qf 42zrfBB5z5UlmjDg== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1737636910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dj7zuGN2Lh05CmjVYI8s2Zi2Euw7raMe3WLPq7IXEt8=; b=yQjLmFszWhTOfsgcuh8fwFh908PxXwuJaFFY0jgvK5xL/bJUVOsFdjDKV1KDbLh1D8roVz BO7DvkfQ+vbtFb3QEJmXTFcYS78OeNjES8dsjdEodekVSVKtCmm1xi6QzmqjEQ21loh5NQ HgyNEt6ooMihDAKbDNt9odxPju5nyiQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1737636910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dj7zuGN2Lh05CmjVYI8s2Zi2Euw7raMe3WLPq7IXEt8=; b=b/YopV1ow2jefE4I2LV1kK6Le6rdjRHRwVBvc3n08Mw05oo3Ha9fHIAkbRKdSoA5+a/+qf 42zrfBB5z5UlmjDg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 38301136A1; Thu, 23 Jan 2025 12:55:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6zjADC48kmfoKwAAD6G6ig (envelope-from ); Thu, 23 Jan 2025 12:55:10 +0000 Date: Thu, 23 Jan 2025 13:54:54 +0100 From: Daniel Wagner To: Steven Rostedt Cc: Christoph Hellwig , LKML , Linus Torvalds , Daniel Wagner , Hannes Reinecke , Ming Lei , John Garry , Jens Axboe Subject: Re: WARNING: CPU: 3 PID: 1 at block/blk-mq-cpumap.c:90 blk_mq_map_hw_queues+0xf3/0x100 Message-ID: References: <20250122093020.6e8a4e5b@gandalf.local.home> <0ec86878-d3c4-400c-a312-9452bf229d45@flourine.local> <20250122125445.37db5c38@gandalf.local.home> <20250122175817.3eac488f@gandalf.local.home> <9434298a-6756-4fdb-a412-ec34994e8376@flourine.local> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9434298a-6756-4fdb-a412-ec34994e8376@flourine.local> X-Spam-Level: X-Spamd-Result: default: False [-4.30 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_SEVEN(0.00)[9]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+] X-Spam-Score: -4.30 X-Spam-Flag: NO On Thu, Jan 23, 2025 at 08:59:57AM +0100, Daniel Wagner wrote: > On Wed, Jan 22, 2025 at 05:58:17PM -0500, Steven Rostedt wrote: > > On Wed, 22 Jan 2025 12:54:45 -0500 > > Steven Rostedt wrote: > > > > > Not sure its related. I can see how reproducible this is, and if it is, I > > > can try to bisect it. > > > > I bisected it down to: a5665c3d150c98 ("virtio: blk/scsi: replace > > blk_mq_virtio_map_queues with blk_mq_map_hw_queues") > > > > And reverting that as well as: > > > > 9bc1e897a821f ("blk-mq: remove unused queue mapping helpers") > > > > It booted fine. > > In the previous tests you just comment out the WARN_ON_ONCE or did you > also replace blk_mq_clear_mq_map with blk_mq_map_queues? The > blk_mq_clear_mq_map will map all queues to CPU 0 and if you offline CPU > 0, there is nothing left to serve the hctx. I'll try to reproduce your > test and see if I my idea works. I've reproduced the second crash as well using your good old stress-cpu-hotplug script. blk_mq_clear_mq_map the CPUs are mapped to the first hctx and when offline a CPU blk_mq_hctx_notify_offline is not happy about not finding any CPU mapped to the hctx. The patch below should fix your problem. I've tested the different setups and all looked good to me. >From ad6e1bd1705e127c191caf2d3becc6ce989e8d96 Mon Sep 17 00:00:00 2001 From: Daniel Wagner Date: Thu, 23 Jan 2025 12:59:58 +0100 Subject: [PATCH] blk-mq: create correct map for fallback case The fallback code in blk_mq_map_hw_queues is original from blk_mq_pci_map_queues and was added to handle the case where pci_irq_get_affinity will return NULL for !SMP configuration. blk_mq_map_hw_queues replaces besides blk_mq_pci_map_queues also blk_mq_virtio_map_queues which used to use blk_mq_map_queues for the fallback. It's possible to use blk_mq_map_queues for both cases though. blk_mq_map_queues creates the same map as blk_mq_clear_mq_map for !SMP that is CPU 0 will be mapped to hctx 0. The WARN_ON_ONCE has to be dropped for virtio as the fallback is also taken for certain configuration on default. Though there is still a WARN_ON_ONCE check in lib/group_cpus.c: WARN_ON(nr_present + nr_others < numgrps); which will trigger if the caller tries to create more hardware queues than CPUs. It tests the same as the WARN_ON_ONCE in blk_mq_pci_map_queues did. Reported-by: Steven Rostedt Signed-off-by: Daniel Wagner --- block/blk-mq-cpumap.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c index ad8d6a363f24..444798c5374f 100644 --- a/block/blk-mq-cpumap.c +++ b/block/blk-mq-cpumap.c @@ -87,7 +87,6 @@ void blk_mq_map_hw_queues(struct blk_mq_queue_map *qmap, return; fallback: - WARN_ON_ONCE(qmap->nr_queues > 1); - blk_mq_clear_mq_map(qmap); + blk_mq_map_queues(qmap); } EXPORT_SYMBOL_GPL(blk_mq_map_hw_queues); -- 2.48.1