mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ian Rogers <irogers@google.com>
To: Johannes Berg <johannes@sipsolutions.net>
Cc: linux-wireless@vger.kernel.org, b43-dev@lists.infradead.org,
	 linux-kernel@vger.kernel.org, llvm@lists.linux.dev,
	 Elkin Cruz <elkin@google.com>, Ian Rogers <irogers@google.com>
Subject: [PATCH] wifi: b43: Use 32-bit signed integers for N-PHY RX IQ compensation math
Date: Thu, 19 Nov 2026 09:04:37 -0800	[thread overview]
Message-ID: <20261119170437.1512945-1-irogers@google.com> (raw)

In b43_nphy_calc_rx_iq_comp(), the intermediate variables tmp, a, and b
are declared as u16, even though the fixed-point calibration arithmetic
shifts 32-bit IQ correlation and power estimates up to bit 30 before
dividing:

  a = -((iq << (30 - iq_nbits)) + (ii >> (1 + arsh)));
  tmp = ii >> arsh;
  a /= tmp;
  ...
  b = (qq << (31 - qq_nbits));
  tmp = ii >> brsh;
  b = int_sqrt(b / tmp - a * a) - (1 << 10);

Truncating these 30/31-bit numerators and ~20-bit denominators to u16
discards their upper bits (frequently collapsing b = qq << (31 -
qq_nbits) to 0 and corrupting the signed division a /= tmp), producing
bogus 10-bit RX IQ calibration coefficients compared to the 32-bit int
implementation in lpphy_calc_rx_iq_comp().

Change tmp, a, and b to s32 so the fixed-point divisions and square-root
input retain full 32-bit signed precision.

Found by enabling Clang's -Wimplicit-int-conversion-on-negation on an
x86_64 allyesconfig build, with AI coding assistant help to audit the
warning and prepare the fix. Verified with a standalone C fixed-point
test and Clang build; not runtime-tested on Broadcom 802.11n hardware.

Fixes: 34a56f2cae86 ("b43: N-PHY: implement calculating RX IQ comp")
Signed-off-by: Ian Rogers <irogers@google.com>
---
 drivers/net/wireless/broadcom/b43/phy_n.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/wireless/broadcom/b43/phy_n.c b/drivers/net/wireless/broadcom/b43/phy_n.c
index 007940f3996e..7b020e0fcd93 100644
--- a/drivers/net/wireless/broadcom/b43/phy_n.c
+++ b/drivers/net/wireless/broadcom/b43/phy_n.c
@@ -4482,7 +4482,7 @@ static void b43_nphy_calc_rx_iq_comp(struct b43_wldev *dev, u8 mask)
 	u32 qq;
 	int iq_nbits, qq_nbits;
 	int arsh, brsh;
-	u16 tmp, a, b;
+	s32 tmp, a, b;
 
 	struct nphy_iq_est est;
 	struct b43_phy_n_iq_comp old;
-- 
2.56.1.708.g77570a6a1d-goog


                 reply	other threads:[~2026-11-19 17:04 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119170437.1512945-1-irogers@google.com \
    --to=irogers@google.com \
    --cc=b43-dev@lists.infradead.org \
    --cc=elkin@google.com \
    --cc=johannes@sipsolutions.net \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-wireless@vger.kernel.org \
    --cc=llvm@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®