From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from s1-ba86.socketlabs.email-od.com (s1-ba86.socketlabs.email-od.com [142.0.186.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0FCC3AEF3D for ; Fri, 26 Jun 2026 18:02:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=142.0.186.134 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782496950; cv=none; b=fGwFPYN36KCDY6Nw2ocN1c2A2ak5Xt50Gohc9fBzrVMZIxEthKp6F9TROmdRs6M+4fii7AH024PldwRJTYh72VZcgZ6vPvWj5X6Vb0SLbz9cMoISkIDVYNHHGo8sbB4QUDwMvZXxVCiXwa5JBqrq0i/QNiwI2lxhwQPluxNhJNY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782496950; c=relaxed/simple; bh=+0nIDz5/3v4pUWM78ysJXZEn71a7R2sVTE9U1GyN5Qo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=QujeZZkX2GSscSq1wDwE7jh1gqjn1UL0tTMHa4nHmfN5NKTBoZzxN/8o3u4FG/YlHx7Mg+PyJQ7jQss9mzKCvJ/U4yOH7AQZhdDFrMeKGLYdZwyaPgpFCDbdz6+TOno2B6lbfWks40xv0jPV/MHh85+Q65580lO6kx69I+aHfNU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=nalramli.com; spf=pass smtp.mailfrom=email-od.com; dkim=pass (1024-bit key) header.d=email-od.com header.i=@email-od.com header.b=donSYGZR; arc=none smtp.client-ip=142.0.186.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=nalramli.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=email-od.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=email-od.com header.i=@email-od.com header.b="donSYGZR" DKIM-Signature: v=1; a=rsa-sha256; d=email-od.com;i=@email-od.com;s=dkim; c=relaxed/relaxed; q=dns/txt; t=1782496949; x=1785088949; h=content-transfer-encoding:content-type:in-reply-to:from:content-language:references:cc:to:subject:mime-version:date:message-id:x-thread-info:subject:to:from:cc:reply-to; bh=54noMUafHcisrmJLQeLAAZeIsyBBva5/RhjHOCmMG0g=; b=donSYGZRzW6w83DmOP0UY2whmSuvlu8i5UcGGpqtVdEvhS7nXVcLh+QxHUAjub7pGrt3PdiyFJ83+9Njgsg3tmphlhUSPuy5UwzASW6PqvnjxMpisC+wfptlM5k53Ce5c0o4tdavA+vlVswMAA73rvAH4+jJGnF6u7FEjY+NGKM= X-Thread-Info: NDUwNC4xMi41ZjNlMTAwMDA0M2UzZTQubGludXgta2VybmVsPXZnZXIua2VybmVsLm9yZw== x-xsSpam: eyJTY29yZSI6MCwiRGV0YWlscyI6bnVsbH0= Received: from [192.168.0.207] (d4-50-191-215.clv.wideopenwest.com [50.4.215.191]) by nalramli.com (Postfix) with ESMTPSA id 3A4FF2CE04D9; Fri, 26 Jun 2026 14:02:19 -0400 (EDT) Message-ID: Date: Fri, 26 Jun 2026 14:02:18 -0400 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [mellanox/mlx5-next RFC 1/1] net/mlx5: RX, Fix refcount warning on frag page release To: Dragos Tatulea , saeedm@nvidia.com, tariqt@nvidia.com, mbloch@nvidia.com Cc: nalramli@fastly.com, leon@kernel.org, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org References: <20260625174059.2879717-1-dev@nalramli.com> <20260625174059.2879717-2-dev@nalramli.com> <9f150145-d95c-4a90-a358-5b33ab78a8ef@nvidia.com> Content-Language: en-US From: "Nabil S. Alramli" In-Reply-To: <9f150145-d95c-4a90-a358-5b33ab78a8ef@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 6/26/26 09:12, Dragos Tatulea wrote: > > > On 25.06.26 19:40, Nabil S. Alramli wrote: >> Under memory pressure, mlx5 driver has WARNING during fragmented page >> release. This happens because there is a discrepency between what mlx5 >> thinks the page fragment counter is vs what the page_pool actually says it >> is. >> > The mlx5 frag counter is not the same as pp_ref_count. The page gets > split into 64 parts during page allocation. The frag counter tracks how > many of those frags have been used. > Thank you for explaining that to me. I thought it was the same because in mlx5e_page_release_fragmented, drain_count is computed from frag_page->frags and then passed into page_pool_unref_page as the nr parameter which is then compared to the refcount in the page_pool. >> The cause of the issue is page allocations on concurrent cpus, which >> increment the non-atomic u16 page counter mlx5e_frag_page.frags, while at >> the same time the page reference counter net_iov.pp_ref_count is atomically >> incremented. That sometimes leads to a difference in the counts and >> therefore triggers the warning in page_pool_unref_netmem: >> > page_pool page allocations must not happen in parallel on different CPUs. > Each queue has its own page_pool and allocation happens within the NAPI of > that queue which sticks to a single CPU. The release path does support > releasing on another CPU (release to ring). > > How did you encounter this scenario of having parallel allocations on > different CPUs from the same page_pool? > Perhaps we didn't, I had assumed that the numbers must match. What we did encounter is these WARNINGs, and they seemed to go away with this patch but maybe it is a coincidence. >> ``` >> ret = atomic_long_sub_return(nr, pp_ref_count); >> WARN_ON(ret < 0); >> ``` >> >> The actual stack trace looks like this: >> >> ``` >> WARNING: CPU: 37 PID: 447795 at include/net/page_pool/helpers.h:277 mlx5e_page_release_fragmented.isra.0+0x51/0x60 [mlx5_core] >> Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE >> Hardware name: * >> RIP: 0010:mlx5e_page_release_fragmented.isra.0+0x51/0x60 [mlx5_core] >> RSP: 0018:ffffc90019814d98 EFLAGS: 00010293 >> RAX: 000000000000003f RBX: ffff88c0993d0a10 RCX: ffffea02424592c0 >> RDX: 0000000000000001 RSI: ffffea02424592c0 RDI: ffff88c090e20000 >> RBP: 000000000000000a R08: 0000000000001409 R09: 0000000000000006 >> R10: 0000000000000000 R11: ffff88c095fbc040 R12: 000000000000141f >> R13: 0000000000000009 R14: ffff88c090e20000 R15: 0000000000000001 >> FS: 00007f34149fa6c0(0000) GS:ffff89200fa40000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 00007ed0265eb000 CR3: 0000005091cbe000 CR4: 0000000000350ef0 >> Call Trace: >> >> mlx5e_free_rx_wqes+0x7b/0xa0 [mlx5_core] >> mlx5e_post_rx_wqes+0x1ac/0x5a0 [mlx5_core] >> mlx5e_napi_poll+0x5e5/0x6f0 [mlx5_core] >> __napi_poll+0x2b/0x1a0 >> net_rx_action+0x30e/0x370 >> ? sched_clock+0x9/0x10 >> ? sched_clock_cpu+0xf/0x170 >> handle_softirqs+0xe2/0x2a0 >> common_interrupt+0x85/0xa0 >> >> >> asm_common_interrupt+0x26/0x40 >> RIP: 0010:page_counter_uncharge+0x34/0x90 >> RSP: 0018:ffffc900e728bb00 EFLAGS: 00000213 >> RAX: ffff88aff4762000 RBX: ffff88aff4762100 RCX: 0000000000000304 >> RDX: 0000000000000001 RSI: 00000000004e9e1a RDI: ffff88aff4762100 >> RBP: 0000000000000001 R08: ffff891ea0560048 R09: 00007ffffffff000 >> R10: 0000000000001000 R11: ffff891ae8061b00 R12: ffffffffffffffff >> R13: ffff89107fcfd4c0 R14: ffff891ae8061b00 R15: ffff892002fe1400 >> uncharge_batch+0x40/0xd0 >> ``` >> > Can you provide more data on how you reproduced this? This helps to > narrow down the bug. Reproduction steps would be ideal. > I don't have clear steps to reproduce it, we just have seen it randomly on some servers that were under memory pressure. I will try to look into it more and find a way to reliably reproduce it. I agree that would be ideal to find a proper fix. >> The fix is to use an atomic page fragment counter, so it will always match >> the number of references held in the page_pool. >> > This is not the right fix. The mlx5 page frag counter is not atomic > on purpose because all changes to it happen only within the NAPI > context. > That was a question that I had, is it ever possible for frag_page->frags to be incremented / set outside of NAPI context? I tried to answer that by looking at code and by tracing it but could not get a clear picture. If it's not possible then I agree, this is not the right fix. > Thanks, > Dragos Thanks again for your guidance. Nabil S. Alramli