[PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[Haoran Wang]

ppm_get_cpu_entries() disables preemption with get_cpu(), reads the
current CPU's pool with per_cpu_ptr(), then acquires spin_lock_bh()
while preemption is still disabled, and finally put_cpu().

    cpu = get_cpu();
    pool = per_cpu_ptr(ppm->pool, cpu);
    spin_lock_bh(&pool->lock);
    put_cpu();

On PREEMPT_RT kernels spin_lock_bh() becomes a sleeping lock
(rt_spin_lock).  get_cpu() raises preempt_count, so acquiring the
lock right after sleeps with preempt_count still positive.  Sleeping
in that state is the scheduling-while-atomic case, which under
CONFIG_DEBUG_ATOMIC_SLEEP reports a BUG and, if the lock is contended,
deadlocks on the spot.  The trigger path is any iSCSI offload I/O
through a Chelsio NIC that reaches ppm_get_cpu_entries().

The preemption disable exists only to stabilize the per-CPU accessor,
the pool is serialized by pool->lock, not by preemption.  Replace
get_cpu() + per_cpu_ptr() + put_cpu() with raw_smp_processor_id() +
raw_cpu_ptr(), which read the current CPU without disabling preemption.

On non-RT kernels spin_lock_bh() does not sleep, so the BUG does not
fire there; the defect is PREEMPT_RT-specific.

With the fix, preempt_count is 0 at spin_lock_bh(), so rt_spin_lock()'s
might_sleep() no longer triggers.

Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>
---
 drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
index 2e3973a32..5f68c5d57 100644
--- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
+++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
@@ -125,10 +125,9 @@ static int ppm_get_cpu_entries(struct cxgbi_ppm *ppm, unsigned int count,
 	if (!ppm->pool)
 		return -EINVAL;
 
-	cpu = get_cpu();
-	pool = per_cpu_ptr(ppm->pool, cpu);
+	cpu = raw_smp_processor_id();
+	pool = raw_cpu_ptr(ppm->pool);
 	spin_lock_bh(&pool->lock);
-	put_cpu();
 
 	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
 				    pool->next, count, 0);

Re: [PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - How the issue was discovered, e.g. hit in production, hit during
   development, syzbot report, manual code inspection, LLM or static
   analysis tool scan.

 - Whether the issue was actually triggered, or is only theoretical
   (e.g. found by code inspection). If it was triggered please include
   the symptoms, like the stack trace or error messages.

 - What hardware the change was tested on. For driver fixes please
   mention the device (and if relevant firmware version) used for
   testing, or say that the change was not tested on real hardware.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.

Re: [PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[Sebastian Andrzej Siewior]

On 2026-10-19 21:22:27 [+0800], Haoran Wang wrote:
> diff --git a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> index 2e3973a32..5f68c5d57 100644
> --- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> +++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> @@ -125,10 +125,9 @@ static int ppm_get_cpu_entries(struct cxgbi_ppm *ppm, unsigned int count,
>  	if (!ppm->pool)
>  		return -EINVAL;
>  
> -	cpu = get_cpu();
> -	pool = per_cpu_ptr(ppm->pool, cpu);
> +	cpu = raw_smp_processor_id();
> +	pool = raw_cpu_ptr(ppm->pool);

cpu needs too be obtained after the lock has been acquired.
Please also explain why it is okay to obtain pool belonging to CPU1 on
CPU0.

>  	spin_lock_bh(&pool->lock);
> -	put_cpu();
>  
>  	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
>  				    pool->next, count, 0);


Sebastian

Re: [PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] libcxgb: CPU migration between raw_smp_processor_id() and raw_cpu_ptr()
--

commit 45e692c0f47c6b069f27d74df45dc3a4a6360a3d
Author: Haoran Wang <haoranwangsec@gmail.com>

libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

This commit replaces get_cpu(), per_cpu_ptr(), and put_cpu() in
ppm_get_cpu_entries() with raw_smp_processor_id() and raw_cpu_ptr() to avoid
acquiring spin_lock_bh() with preemption disabled on PREEMPT_RT kernels.

> diff --git a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> index 2e3973a32d9d1..5f68c5d5766e5 100644
> --- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> +++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> @@ -125,10 +125,9 @@ static int ppm_get_cpu_entries(struct cxgbi_ppm *ppm, unsigned int count,
>  	if (!ppm->pool)
>  		return -EINVAL;
>  
> -	cpu = get_cpu();
> -	pool = per_cpu_ptr(ppm->pool, cpu);
> +	cpu = raw_smp_processor_id();
> +	pool = raw_cpu_ptr(ppm->pool);
>  	spin_lock_bh(&pool->lock);
> -	put_cpu();
>  
>  	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
>  				    pool->next, count, 0);

[Severity: Critical]
Can the task be preempted and migrated to another CPU between
raw_smp_processor_id() and raw_cpu_ptr(ppm->pool)?

Because preemption is enabled here, if a task in a preemptible iSCSI offload
path (such as cxgbit_ddp_reserve() -> cxgbi_ppm_ppods_reserve() ->
ppm_get_cpu_entries()) is migrated from CPU A to CPU B between the two
calls, cpu will hold CPU A while pool will point to CPU B's per-CPU pool.

ppm_get_cpu_entries() will then allocate bits in CPU B's pool->bmap, but
compute the global pagepod index using CPU A's ID:

drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c:ppm_get_cpu_entries() {
	...
	i += cpu * ppm->pool_index_max;
	ppm_mark_entries(ppm, i, count, caller_data);

	return i;
}

Because those bits were never marked in CPU A's pool->bmap, a subsequent
allocation on CPU A can hand out the same global index while it is still
active, overwriting ppm->ppod_data[i] and corrupting in-flight iSCSI DDP
DMA transfers.

Later, when cxgbi_ppm_ppod_release() -> ppm_unmark_entries() releases the
entries, it derives the pool from the global index:

drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c:ppm_unmark_entries() {
	...
	if (i < ppm->pool_rsvd) {
		unsigned int cpu;
		struct cxgbi_ppm_pool *pool;

		cpu = i / ppm->pool_index_max;
		i %= ppm->pool_index_max;

		pool = per_cpu_ptr(ppm->pool, cpu);
		spin_lock_bh(&pool->lock);
		bitmap_clear(pool->bmap, i, count);
	...
}

This clears the bits in CPU A's pool->bmap while permanently leaking the
bits that were set in CPU B's pool->bmap.

Would keeping per_cpu_ptr(ppm->pool, cpu) after raw_smp_processor_id() avoid
this mismatch by ensuring cpu and pool always refer to the same per-CPU pool
even if migration occurs?

([
---
])

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/179241614797.5916.13928836073297916808@whr?part=1

Re: [PATCH] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.

Critical: 0 · High: 1 · Medium: 0 · Low: 1

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [High] The patch introduces a cpu/pool mismatch race in
  ppm_get_cpu_entries()…
- [Low] The commit is a bug fix for a user-visible 'BUG: sleeping
  function called from invalid context' on PREEMPT_RT.

--

libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

This patch changes ppm_get_cpu_entries(). It drops get_cpu(),
per_cpu_ptr() and put_cpu() and uses raw_smp_processor_id() and
raw_cpu_ptr() instead. The aim is to stop spin_lock_bh() from being taken
with preemption disabled, which triggers a sleeping-in-atomic BUG on
PREEMPT_RT.

> Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>

[Severity: Low]
This fixes a user-visible BUG on PREEMPT_RT, but the commit has no Fixes:
tag. The get_cpu()/per_cpu_ptr()/spin_lock_bh()/put_cpu() sequence has
not changed since it was added in:

Fixes: 47da7e62b391 ("cxgb4: add iSCSI DDP page pod manager")

Should that tag be added, possibly with a Cc: stable?

> diff --git a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> index 2e3973a32d9d..5f68c5d5766e 100644
> --- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> +++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> @@ -125,10 +125,9 @@ static int ppm_get_cpu_entries(struct cxgbi_ppm *ppm, unsigned int count,
>  	if (!ppm->pool)
>  		return -EINVAL;
>  
> -	cpu = get_cpu();
> -	pool = per_cpu_ptr(ppm->pool, cpu);
> +	cpu = raw_smp_processor_id();
> +	pool = raw_cpu_ptr(ppm->pool);

[Severity: High]
Can cpu and pool refer to different CPUs here?

The new code reads the current CPU twice, with preemption and migration
both enabled. Say the task moves from CPU A to CPU B between
raw_smp_processor_id() and raw_cpu_ptr(). The bit is then reserved in CPU
B's bitmap under CPU B's lock, but the returned index is built from the
old cpu value:

drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c:ppm_get_cpu_entries() {
    ...
	i += cpu * ppm->pool_index_max;
	ppm_mark_entries(ppm, i, count, caller_data);
    ...
}

The index and the ppod_data[] entry now point into CPU A's slice, while
bit i in CPU A's bitmap is still clear.

Could a task on CPU A then get the same index, hardware page pod and
ppod_data slot while the first transfer is still in flight?

cxgbi_ppm_ppods_reserve() turns this index into hwidx and the DDP tag.
cxgbit_ddp_reserve() and cxgbi_ddp_reserve() then program that tag into
the adapter. The caller_data in ppod_data[] is also what
cxgbi_parse_pdu_itt() uses later to match responses to tasks.

On release, ppm_unmark_entries() gets the pool back from the index:

	cpu = i / ppm->pool_index_max;
	i %= ppm->pool_index_max;

	pool = per_cpu_ptr(ppm->pool, cpu);
	spin_lock_bh(&pool->lock);
	bitmap_clear(pool->bmap, i, count);

That would clear the bit in CPU A's pool, which may free another
transfer's live entry. The bit that was really set in CPU B's pool would
never be cleared.

One preemptible caller that holds no spinlock is the iSCSI target TX
path:

iscsit_send_r2t()
  cxgbit_get_r2t_ttt()
    cxgbit_ddp_reserve()
      cxgbi_ppm_ppods_reserve()
        ppm_get_cpu_entries()

iscsit_get_r2t_from_list() drops r2t_lock before it returns, so this
whole chain runs unlocked. A remote initiator can drive it simply by
sending WRITE commands to a Chelsio-offloaded target.

The raw_ accessors are not checked by CONFIG_DEBUG_PREEMPT, so nothing
would warn. The race exists on any SMP kernel built with CONFIG_PREEMPT,
PREEMPT_LAZY or PREEMPT_RT.

Given that, do these parts of the commit message still hold?

    The preemption disable exists only to stabilize the per-CPU accessor,
    the pool is serialized by pool->lock, not by preemption.

    ...the defect is PREEMPT_RT-specific.

The returned index encodes cpu, so cpu and pool still have to match even
though pool->lock serializes the bitmap.

Would it work to keep the saved cpu and derive the pool from it? For
example:

	cpu = raw_smp_processor_id();
	pool = per_cpu_ptr(ppm->pool, cpu);

Another option would be to wrap the lookup in
migrate_disable()/migrate_enable().

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/179241614797.5916.13928836073297916808%40whr

[PATCH v3] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[Haoran Wang]

ppm_get_cpu_entries() disables preemption with get_cpu(), reads the
current CPU's pool with per_cpu_ptr(), then acquires the pool's
spin_lock_bh() while preemption is still disabled, and finally put_cpu().

On PREEMPT_RT kernels spin_lock_bh() becomes a sleeping lock
(rt_spin_lock), so the task may sleep with preempt_count elevated:
scheduling while atomic, reported as a BUG by CONFIG_DEBUG_ATOMIC_SLEEP
and a deadlock once the lock is contended.  Any iSCSI offload I/O
through a Chelsio NIC that reaches ppm_get_cpu_entries() triggers it.

Read the current CPU with raw_cpu_ptr() without raising preempt_count,
and derive the global index base from pool->base (initialised when the
pools are allocated) instead of cpu * pool_index_max, so the base
always matches the pool that was locked.  The owning CPU remains in the
debug output, now derived from pool->base.

Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>

---
(Resent with the complete Cc list; the earlier copy reached only you.)

Hi Sebastian,

thanks for the review.  v3 below addresses your comments:

- The commit message now describes what the patch does; the review
  rationale moved into the changelog below the --- separator.
- PREEMPT_RT spelled out.
- pr_debug() keeps printing the owning CPU of the pool, derived as
  pool->base / ppm->pool_index_max; the global index stays
  i + pool->base.
- Tested on: x86_64, compile-tested with CONFIG_PREEMPT_RT=y and
  CONFIG_DEBUG_ATOMIC_SLEEP=y; no Chelsio hardware at hand, so the
  change is not exercised on real offload I/O.

Changes since v2:
- Keep the owning CPU in the pr_debug() output, derived from
  pool->base / ppm->pool_index_max (suggested by Sebastian).
- Reword the commit message to describe what the patch does and move
  the review discussion into this changelog; spell PREEMPT_RT.

Changes since v1:
- Drop the cpu variable: read the global-index base from pool->base
  (initialised at pool allocation) instead of a cpu captured before
  the lock, so the base is always consistent with the pool that was
  locked.  The per-CPU pools are shards of the global ppod index
  space and not tied to the running CPU (ppm_unmark_entries() derives
  the owning pool from the index itself), so locking another CPU's
  pool after a migration is a cache-locality matter, not a
  correctness one.

 drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

--- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
+++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
@@ -119,16 +119,13 @@
 			       unsigned long caller_data)
 {
 	struct cxgbi_ppm_pool *pool;
-	unsigned int cpu;
 	int i;
 
 	if (!ppm->pool)
 		return -EINVAL;
 
-	cpu = get_cpu();
-	pool = per_cpu_ptr(ppm->pool, cpu);
+	pool = raw_cpu_ptr(ppm->pool);
 	spin_lock_bh(&pool->lock);
-	put_cpu();
 
 	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
 				    pool->next, count, 0);
@@ -145,10 +142,10 @@
 	spin_unlock_bh(&pool->lock);
 
 	pr_debug("%s: cpu %u, idx %d + %d (%d), next %u.\n",
-		 __func__, cpu, i, count, i + cpu * ppm->pool_index_max,
-		pool->next);
+		 __func__, pool->base / ppm->pool_index_max, i, count,
+		i + pool->base, pool->next);
 
-	i += cpu * ppm->pool_index_max;
+	i += pool->base;
 	ppm_mark_entries(ppm, i, count, caller_data);
 
 	return i;
@@ -375,6 +372,7 @@
 
 		memset(ppool, 0, alloc_sz);
 		spin_lock_init(&ppool->lock);
+		ppool->base = count;
 		count += ppmax;
 	}
 

Re: [PATCH v3] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - How the issue was discovered, e.g. hit in production, hit during
   development, syzbot report, manual code inspection, LLM or static
   analysis tool scan.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.

Re: [PATCH v3] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[David Laight]

On Mon, 09 Nov 2026 18:07:48 +0800
Haoran Wang <haoranwangsec@gmail.com> wrote:

> ppm_get_cpu_entries() disables preemption with get_cpu(), reads the
> current CPU's pool with per_cpu_ptr(), then acquires the pool's
> spin_lock_bh() while preemption is still disabled, and finally put_cpu().
> 
> On PREEMPT_RT kernels spin_lock_bh() becomes a sleeping lock
> (rt_spin_lock), so the task may sleep with preempt_count elevated:
> scheduling while atomic, reported as a BUG by CONFIG_DEBUG_ATOMIC_SLEEP
> and a deadlock once the lock is contended.  Any iSCSI offload I/O
> through a Chelsio NIC that reaches ppm_get_cpu_entries() triggers it.
> 
> Read the current CPU with raw_cpu_ptr() without raising preempt_count,
> and derive the global index base from pool->base (initialised when the
> pools are allocated) instead of cpu * pool_index_max, so the base
> always matches the pool that was locked.  The owning CPU remains in the
> debug output, now derived from pool->base.

I think you really need to explain why it doesn't matter that the pool
for the 'wrong' cpu is used.

You don't need a big explanation of why you can't call spin_lock_bh()
with preemption disabled, it is probably enough just to say that you
can't (the most reason would be 'can sleep on PREEMPT_RT kernels).

This is separate from any issues of acquiring the wrong lock.
I'm not sure about the trace, if you are tracing the cpu the data belongs
to - then that should be what you call it.
Once you hold the lock I think the cpu is fixed (even under PREEMPT_RT)
so you could trace the actual cpu as well - especially if anything 'funny'
might happen because the data wor the wrong cpu is being used.

Of course, it the code is just trying to avoid lock contention (and maybe
reduce cache line bouncing) it may not matter at all which copy of the
data you get.
In fact one for each cpu may be excessive - think of systems with 1000s
of cpu not single digits.

David


> 
> Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>
> 
> ---
> (Resent with the complete Cc list; the earlier copy reached only you.)
> 
> Hi Sebastian,
> 
> thanks for the review.  v3 below addresses your comments:
> 
> - The commit message now describes what the patch does; the review
>   rationale moved into the changelog below the --- separator.
> - PREEMPT_RT spelled out.
> - pr_debug() keeps printing the owning CPU of the pool, derived as
>   pool->base / ppm->pool_index_max; the global index stays
>   i + pool->base.
> - Tested on: x86_64, compile-tested with CONFIG_PREEMPT_RT=y and
>   CONFIG_DEBUG_ATOMIC_SLEEP=y; no Chelsio hardware at hand, so the
>   change is not exercised on real offload I/O.
> 
> Changes since v2:
> - Keep the owning CPU in the pr_debug() output, derived from
>   pool->base / ppm->pool_index_max (suggested by Sebastian).
> - Reword the commit message to describe what the patch does and move
>   the review discussion into this changelog; spell PREEMPT_RT.
> 
> Changes since v1:
> - Drop the cpu variable: read the global-index base from pool->base
>   (initialised at pool allocation) instead of a cpu captured before
>   the lock, so the base is always consistent with the pool that was
>   locked.  The per-CPU pools are shards of the global ppod index
>   space and not tied to the running CPU (ppm_unmark_entries() derives
>   the owning pool from the index itself), so locking another CPU's
>   pool after a migration is a cache-locality matter, not a
>   correctness one.
> 
>  drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c |   12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
> 
> --- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> +++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
> @@ -119,16 +119,13 @@
>  			       unsigned long caller_data)
>  {
>  	struct cxgbi_ppm_pool *pool;
> -	unsigned int cpu;
>  	int i;
>  
>  	if (!ppm->pool)
>  		return -EINVAL;
>  
> -	cpu = get_cpu();
> -	pool = per_cpu_ptr(ppm->pool, cpu);
> +	pool = raw_cpu_ptr(ppm->pool);
>  	spin_lock_bh(&pool->lock);
> -	put_cpu();
>  
>  	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
>  				    pool->next, count, 0);
> @@ -145,10 +142,10 @@
>  	spin_unlock_bh(&pool->lock);
>  
>  	pr_debug("%s: cpu %u, idx %d + %d (%d), next %u.\n",
> -		 __func__, cpu, i, count, i + cpu * ppm->pool_index_max,
> -		pool->next);
> +		 __func__, pool->base / ppm->pool_index_max, i, count,
> +		i + pool->base, pool->next);
>  
> -	i += cpu * ppm->pool_index_max;
> +	i += pool->base;
>  	ppm_mark_entries(ppm, i, count, caller_data);
>  
>  	return i;
> @@ -375,6 +372,7 @@
>  
>  		memset(ppool, 0, alloc_sz);
>  		spin_lock_init(&ppool->lock);
> +		ppool->base = count;
>  		count += ppmax;
>  	}
>  
> 

[PATCH v4] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[Haoran Wang]

[-- Attachment #1: Type: text/plain, Size: 4308 bytes --]

ppm_get_cpu_entries() must not acquire a sleeping lock while preemption
is disabled: get_cpu() raises preempt_count and spin_lock_bh() may sleep
on PREEMPT_RT kernels.  Read the current CPU with raw_cpu_ptr() instead,
which keeps the per-CPU accessor stable without disabling preemption.

Using the pool of a CPU the thread has since migrated away from is
harmless: the per-CPU pools are shards of the global ppod index space,
and the free path ppm_unmark_entries() derives the owning pool from the
index instead of from the running CPU, so any thread can service any
shard.  Derive the global index base from pool->base so it always
matches the pool that was locked.

Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>

---
Hi David,

thanks for the comments.  v4 below:

- The commit message now explains why using the pool of another CPU is
  harmless, and the PREEMPT_RT problem statement is trimmed to two
  sentences.
- pr_debug() now labels the pool's owner as "pool cpu" and also prints
  the running CPU, so a thread servicing a foreign shard is visible in
  the log.
- I also stress-tested the patched allocator itself: the unmodified
  libcxgb_ppm.c with this patch applied, compiled into an out-of-tree
  module and driven through the real cxgbi_ppm_ppods_reserve() /
  cxgbi_ppm_ppod_release() API by 8 unbound kernel threads on a 2-CPU
  QEMU x86_64 guest, with about half of the releases executed by
  another thread (and so, usually, another CPU).  Across 7 runs
  totalling 1.12M allocations there was no double-issued index, no bad
  free, and every pool drained back to empty each run; 4 allocations
  were naturally served by another CPU's pool because the caller
  migrated inside the allocator, and the accounting stayed exact.  The
  test harness is AI-generated; module and logs are attached.
- On the sharding granularity: agreed, the per-pool split only spreads
  lock contention and cache-line bouncing; on systems with thousands of
  CPUs a coarser sharding would serve the same purpose.  That is a
  separate optimisation and does not affect this fix, so I left it
  untouched here.

Changes since v3:
- Explain in the commit message why a foreign CPU's pool is harmless;
  trim the PREEMPT_RT problem description (review).
- pr_debug(): print the running CPU and the pool-owning CPU separately
  and label them accordingly (review).
- pr_debug(): the owning CPU keeps being derived from pool->base, so it
  is always consistent with the locked pool.
- Attach stress-test evidence for the foreign-pool argument.

Changes since v2:
- Keep the owning CPU in the pr_debug() output, derived from
  pool->base / ppm->pool_index_max (suggested by Sebastian).
- Reword the commit message to describe what the patch does and move
  the review discussion into this changelog; spell PREEMPT_RT.

Changes since v1:
- Drop the cpu variable: read the global-index base from pool->base
  (initialised at pool allocation) instead of a cpu captured before
  the lock, so the base is always consistent with the pool that was
  locked.

 drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c |   15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

--- a/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
+++ b/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c
@@ -119,16 +119,13 @@
 			       unsigned long caller_data)
 {
 	struct cxgbi_ppm_pool *pool;
-	unsigned int cpu;
 	int i;
 
 	if (!ppm->pool)
 		return -EINVAL;
 
-	cpu = get_cpu();
-	pool = per_cpu_ptr(ppm->pool, cpu);
+	pool = raw_cpu_ptr(ppm->pool);
 	spin_lock_bh(&pool->lock);
-	put_cpu();
 
 	i = ppm_find_unused_entries(pool->bmap, ppm->pool_index_max,
 				    pool->next, count, 0);
@@ -144,11 +141,12 @@
 
 	spin_unlock_bh(&pool->lock);
 
-	pr_debug("%s: cpu %u, idx %d + %d (%d), next %u.\n",
-		 __func__, cpu, i, count, i + cpu * ppm->pool_index_max,
-		pool->next);
+	pr_debug("%s: cpu %u, pool cpu %u, idx %d + %d (%d), next %u.\n",
+		 __func__, raw_smp_processor_id(),
+		 pool->base / ppm->pool_index_max, i, count, i + pool->base,
+		 pool->next);
 
-	i += cpu * ppm->pool_index_max;
+	i += pool->base;
 	ppm_mark_entries(ppm, i, count, caller_data);
 
 	return i;
@@ -375,6 +373,7 @@
 
 		memset(ppool, 0, alloc_sz);
 		spin_lock_init(&ppool->lock);
+		ppool->base = count;
 		count += ppmax;
 	}
 

[-- Attachment #2: ppm_v4_stresstest.txt --]
[-- Type: text/plain, Size: 14776 bytes --]

libcxgb ppm v4 -- stress test for the "wrong CPU's pool" question
=================================================================

DISCLOSURE
----------
The test harness below (pooltest_real.c, Makefile and the run recipe)
is AI-generated code (Claude).  The code under test is the unmodified
drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c (Linux 7.3), with
the proposed v4 patch applied verbatim via `patch -p1`, compiled into
the test module.

WHAT IT TESTS
-------------
The question from review: why is it harmless when ppm_get_cpu_entries()
ends up using the pool of a CPU the caller has migrated away from?

The module includes the patched libcxgb_ppm.c and exercises the real
API only:

  - cxgbi_ppm_init()          real ppm setup (ppmax 16384, reserve
                              factor 2: 4096 ppods per CPU pool plus
                              8192 in the general pool),
  - cxgbi_ppm_ppods_reserve() allocates via ppm_get_cpu_entries() --
                              the function this patch touches -- with
                              the general pool as fallback, exactly as
                              the drivers use it,
  - cxgbi_ppm_ppod_release()  frees via ppm_unmark_entries(), which
                              derives the owning pool from the index.

8 unbound kernel threads over-subscribe 2 CPUs so the scheduler
migrates them, and about half of the ppods are handed to another
thread (and so, usually, another CPU) for release.  A global bitmap
oracle flags any index that is issued twice or freed twice / never
issued; at the end every pool bitmap must be empty and issued must
equal freed.  Each allocation also records whether the pool that
served it belongs to the CPU the caller is on when the call returns
("foreign pool observation").

RESULTS (Linux 7.3, x86_64, QEMU q35, 2 CPUs, pristine kernel; the
module is the only out-of-tree artifact)
-----------------------------------------------------------------
Last boot, 3 rounds of 8 threads x 20000 reservations each:

libcxgb: ippm lo: ppm 0x000000001a2cc2f8, 0x000000000787b5ae, base 0/16384, pg 4096,0, rsvd 8192,4096.
pooltest_real: real ppm: ppmax 16384, pool_rsvd 8192, pool_index_max 4096, bmap_index_max 8192, possible cpus 2
pooltest_real: cpu0 pool base 0 (set by v4 init hunk)
pooltest_real: cpu1 pool base 4096 (set by v4 init hunk)
pooltest_real: RESULT: allocs=160000 (own-free=79897 handoff-free=79894) migrations=0 foreign_pool_obs=0 enospc=0
pooltest_real: ORACLE: double_alloc=0 bad_free=0 issued=399501 freed=399501 left_issued=0 pool_leak=0 general_leak=0
pooltest_real: REAL CODE (v4-patched) HOLDS ALL INVARIANTS
libcxgb: ippm lo: ppm 0x000000001a2cc2f8, 0x000000000787b5ae, base 0/16384, pg 4096,0, rsvd 8192,4096.
pooltest_real: real ppm: ppmax 16384, pool_rsvd 8192, pool_index_max 4096, bmap_index_max 8192, possible cpus 2
pooltest_real: cpu0 pool base 0 (set by v4 init hunk)
pooltest_real: cpu1 pool base 4096 (set by v4 init hunk)
pooltest_real: RESULT: allocs=160000 (own-free=80407 handoff-free=79488) migrations=1 foreign_pool_obs=0 enospc=0
pooltest_real: ORACLE: double_alloc=0 bad_free=0 issued=399483 freed=399483 left_issued=0 pool_leak=0 general_leak=0
pooltest_real: REAL CODE (v4-patched) HOLDS ALL INVARIANTS
libcxgb: ippm lo: ppm 0x000000001a2cc2f8, 0x000000000787b5ae, base 0/16384, pg 4096,0, rsvd 8192,4096.
pooltest_real: real ppm: ppmax 16384, pool_rsvd 8192, pool_index_max 4096, bmap_index_max 8192, possible cpus 2
pooltest_real: cpu0 pool base 0 (set by v4 init hunk)
pooltest_real: cpu1 pool base 4096 (set by v4 init hunk)
pooltest_real: RESULT: allocs=160000 (own-free=80175 handoff-free=79621) migrations=0 foreign_pool_obs=0 enospc=0
pooltest_real: ORACLE: double_alloc=0 bad_free=0 issued=400107 freed=400107 left_issued=0 pool_leak=0 general_leak=0
pooltest_real: REAL CODE (v4-patched) HOLDS ALL INVARIANTS

Cumulative over 7 runs (rounds) of the same load:

  allocations                    1.12M  (7 x 160k)
  thread migrations observed        14
  allocations served by a foreign
  CPU's pool (naturally occurring)   4
  double-issued index                0
  bad free                           0
  leaked / lost entries              0   (all pools drained to empty
                                          every run, issued == freed)

So the "wrong CPU's pool" case does occur on the real, patched code,
and the accounting stays exact.  This matches the argument in the
commit message: the pool that is locked always owns the index range it
hands out, and the free path derives the pool from the index itself,
never from the running CPU.

HOW TO REPRODUCE
----------------
    # 1. patched copy of the real source (single source of truth is
    #    the v4 patch itself)
    mkdir -p r/drivers/net/ethernet/chelsio/libcxgb
    cp <kernel>/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.{c,h} \
       r/drivers/net/ethernet/chelsio/libcxgb/
    ( cd r && patch -p1 < libcxgb_v4.patch )
    sed '/^MODULE_AUTHOR/,$d' \
        r/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.c \
        > libcxgb_ppm_patched.c
    cp r/drivers/net/ethernet/chelsio/libcxgb/libcxgb_ppm.h .

    # 2. build as an out-of-tree module (kernel stays pristine)
    make -C <kernel-src> O=<build-dir> M=$PWD KBUILD_MODPOST_WARN=1 modules

    # 3. run on any x86_64 guest with >= 2 CPUs
    insmod pooltest_real.ko threads=8 ops=20000
    rmmod pooltest_real.ko        # results in dmesg

======================================================================
pooltest_real.c
======================================================================
// pooltest_real.c — stress the REAL libcxgb_ppm.c code, v4 patch applied,
// compiled into this out-of-tree module (kernel stays pristine).
//
// Everything below the API line is the untouched driver code path:
//   cxgbi_ppm_init()          — real init: creates the per-cpu pools
//                               (ppm_alloc_cpu_pool, incl. v4's base init)
//   cxgbi_ppm_ppods_reserve() — real alloc: per-cpu pool first — the very
//                               function the patch touches — then the
//                               general pool as fallback
//   cxgbi_ppm_ppod_release()  — real free: ppm_unmark_entries() derives the
//                               owning pool from the index, never the CPU
//
// 8 UNBOUND kernel threads over-subscribe 2 CPUs, so the scheduler migrates
// them constantly, and tickets are handed between threads, so frees land on
// foreign CPUs at a high rate.  Oracles:
//   - global issued bitmap: an index handed out twice (double_alloc) or
//     freed twice / never allocated (bad_free) is caught regardless of
//     which pool/CPU was involved;
//   - every per-cpu pool and the general pool must drain to empty;
//   - issued == freed;
//   - each per-cpu-pool allocation is checked: owning pool vs running CPU
//     at return (foreign-pool observation).
#include "libcxgb_ppm_patched.c"

#undef pr_fmt
#define pr_fmt(fmt) "pooltest_real: " fmt

#include <linux/moduleparam.h>
#include <linux/kthread.h>
#include <linux/completion.h>
#include <linux/random.h>
#include <linux/sched.h>
#include <linux/smp.h>

static struct cxgbi_ppm *g_ppm;
static void *g_ppm_pp;
static struct net_device *g_ndev;
static unsigned long *issued;

static atomic_t n_issued   = ATOMIC_INIT(0);
static atomic_t n_freed    = ATOMIC_INIT(0);
static atomic_t n_double   = ATOMIC_INIT(0);
static atomic_t n_badclear = ATOMIC_INIT(0);
static atomic_t n_enospc   = ATOMIC_INIT(0);
static atomic_t n_foreign  = ATOMIC_INIT(0);

static int ops = 20000;
static int threads = 8;
module_param(ops, int, 0444);
module_param(threads, int, 0444);

struct tick {
	u32 idx;
	unsigned int npods;
};

#define RHOLD 16

/* FIFO ring: tickets handed to OTHER threads, freed on whatever CPU */
struct hq {
	spinlock_t lock;
	int head, tail, cap;
	struct tick *ring;
};
static struct hq handoff;

static int handoff_push(u32 idx, unsigned int npods)
{
	unsigned long flags;
	int r = -1;

	spin_lock_irqsave(&handoff.lock, flags);
	if (handoff.tail - handoff.head < handoff.cap) {
		handoff.ring[handoff.tail % handoff.cap].idx = idx;
		handoff.ring[handoff.tail % handoff.cap].npods = npods;
		handoff.tail++;
		r = 0;
	}
	spin_unlock_irqrestore(&handoff.lock, flags);
	return r;
}

static int handoff_pop(struct tick *t)
{
	unsigned long flags;
	int r = -1;

	spin_lock_irqsave(&handoff.lock, flags);
	if (handoff.tail > handoff.head) {
		*t = handoff.ring[handoff.head % handoff.cap];
		handoff.head++;
		r = 0;
	}
	spin_unlock_irqrestore(&handoff.lock, flags);
	return r;
}

/* free through the REAL release path + oracle bookkeeping.
 * Oracle bits must be cleared BEFORE the real release: after the real
 * free another thread may legitimately re-allocate the same range, and
 * its test_and_set would collide with our not-yet-cleared bit (false
 * double).  While our bit is cleared but the real allocator still owns
 * the range, nobody can obtain it, so there is no window in this order.
 */
static void release_ticket(struct tick *t)
{
	unsigned int k;

	for (k = 0; k < t->npods; k++) {
		if (!test_and_clear_bit(t->idx + k, issued))
			atomic_inc(&n_badclear);
		else
			atomic_inc(&n_freed);
	}
	cxgbi_ppm_ppod_release(g_ppm, t->idx);
}

struct rarg {
	struct completion done;
	struct tick hold[RHOLD];
	long n_alloc, n_hofree, n_minefree, migrations, foreign;
};

static int rworker(void *data)
{
	struct rarg *a = data;
	int i, last = raw_smp_processor_id();

	for (i = 0; i < ops; i++) {
		int slot = get_random_u32() % RHOLD;
		int c = raw_smp_processor_id();
		u32 idx, tag;
		int rc;

		if (c != last) {
			a->migrations++;
			last = c;
		}
		if ((i & 0xf) == 0)
			cond_resched();	/* let the scheduler move us around */

		/* make room: give the ticket away, or free it right here */
		if (a->hold[slot].npods) {
			if ((get_random_u32() & 1) &&
			    !handoff_push(a->hold[slot].idx, a->hold[slot].npods)) {
				a->hold[slot].npods = 0;
			} else {
				release_ticket(&a->hold[slot]);
				a->hold[slot].npods = 0;
				a->n_minefree++;
			}
		}
		/* free a ticket produced on some other CPU/thread */
		if (get_random_u32() & 1) {
			struct tick t;

			if (!handoff_pop(&t)) {
				release_ticket(&t);
				a->n_hofree++;
			}
		}

		/* the real alloc path */
		rc = cxgbi_ppm_ppods_reserve(g_ppm, 1 + get_random_u32() % 16,
					     0, &idx, &tag, 0);
		if (rc > 0) {
			unsigned int k;

			for (k = 0; k < rc; k++) {
				if (test_and_set_bit(idx + k, issued))
					atomic_inc(&n_double);
				atomic_inc(&n_issued);
			}
			/* foreign-pool observation: pool that served us vs
			 * CPU we are on when the call returns */
			if (idx < g_ppm->pool_rsvd &&
			    idx / g_ppm->pool_index_max !=
			    raw_smp_processor_id()) {
				atomic_inc(&n_foreign);
				a->foreign++;
			}
			a->hold[slot].idx = idx;
			a->hold[slot].npods = rc;
			a->n_alloc++;
		} else {
			int s2 = get_random_u32() % RHOLD;

			atomic_inc(&n_enospc);
			if (a->hold[s2].npods) {
				release_ticket(&a->hold[s2]);
				a->hold[s2].npods = 0;
			}
		}
	}
	for (i = 0; i < RHOLD; i++) {
		if (a->hold[i].npods) {
			release_ticket(&a->hold[i]);
			a->hold[i].npods = 0;
			a->n_minefree++;
		}
	}
	complete(&a->done);
	return 0;
}

static int __init pooltest_real_init(void)
{
	struct cxgbi_tag_format tformat = {0};
	struct task_struct **tsk;
	struct rarg *wa;
	long n_alloc = 0, n_hofree = 0, n_minefree = 0, migr = 0, foreign = 0;
	int cpu, i, rc, leak_pool = 0, leak_gen, issued_left;
	struct tick t;

	g_ndev = dev_get_by_name(&init_net, "lo");
	if (!g_ndev) {
		pr_err("no loopback netdev\n");
		return -ENODEV;
	}

	tformat.pgsz_order[0] = 0;
	tformat.pgsz_order[1] = 1;
	tformat.pgsz_order[2] = 2;
	tformat.pgsz_order[3] = 3;

	/* ppmax 16384, reserve_factor 2: half in per-cpu pools, half general */
	rc = cxgbi_ppm_init(&g_ppm_pp, g_ndev, NULL, NULL, &tformat,
			    16384 << PPOD_SIZE_SHIFT, 0, 0, 2, 0, 0);
	if (rc < 0) {
		pr_err("cxgbi_ppm_init failed: %d\n", rc);
		dev_put(g_ndev);
		return rc;
	}
	g_ppm = g_ppm_pp;

	pr_info("real ppm: ppmax %u, pool_rsvd %u, pool_index_max %u, bmap_index_max %u, possible cpus %d\n",
		g_ppm->ppmax, g_ppm->pool_rsvd, g_ppm->pool_index_max,
		g_ppm->bmap_index_max, num_possible_cpus());
	for_each_possible_cpu(cpu) {
		struct cxgbi_ppm_pool *pp = per_cpu_ptr(g_ppm->pool, cpu);

		pr_info("cpu%d pool base %u (set by v4 init hunk)\n",
			cpu, pp->base);
	}

	issued = bitmap_zalloc(g_ppm->ppmax, GFP_KERNEL);
	handoff.cap = 8192;
	handoff.ring = kcalloc(handoff.cap, sizeof(*handoff.ring), GFP_KERNEL);
	spin_lock_init(&handoff.lock);

	wa = kcalloc(threads, sizeof(*wa), GFP_KERNEL);
	tsk = kcalloc(threads, sizeof(*tsk), GFP_KERNEL);
	for (i = 0; i < threads; i++) {
		init_completion(&wa[i].done);
		tsk[i] = kthread_run(rworker, &wa[i], "ppmr%d", i);
		if (IS_ERR(tsk[i]))
			complete(&wa[i].done);
	}
	for (i = 0; i < threads; i++) {
		wait_for_completion(&wa[i].done);
		n_alloc += wa[i].n_alloc;
		n_hofree += wa[i].n_hofree;
		n_minefree += wa[i].n_minefree;
		migr += wa[i].migrations;
		foreign += wa[i].foreign;
	}
	while (!handoff_pop(&t))
		release_ticket(&t);

	for_each_possible_cpu(cpu)
		leak_pool += bitmap_weight(
			per_cpu_ptr(g_ppm->pool, cpu)->bmap,
			g_ppm->pool_index_max);
	leak_gen = bitmap_weight(g_ppm->ppod_bmap, g_ppm->bmap_index_max);
	issued_left = bitmap_weight(issued, g_ppm->ppmax);

	pr_info("RESULT: allocs=%ld (own-free=%ld handoff-free=%ld) migrations=%ld foreign_pool_obs=%d enospc=%d\n",
		n_alloc, n_minefree, n_hofree, migr,
		atomic_read(&n_foreign), atomic_read(&n_enospc));
	pr_info("ORACLE: double_alloc=%d bad_free=%d issued=%d freed=%d left_issued=%d pool_leak=%d general_leak=%d\n",
		atomic_read(&n_double), atomic_read(&n_badclear),
		atomic_read(&n_issued), atomic_read(&n_freed), issued_left,
		leak_pool, leak_gen);
	if (!atomic_read(&n_double) && !atomic_read(&n_badclear) &&
	    !issued_left && !leak_pool && !leak_gen &&
	    atomic_read(&n_issued) == atomic_read(&n_freed))
		pr_info("REAL CODE (v4-patched) HOLDS ALL INVARIANTS\n");
	else
		pr_info("INVARIANT VIOLATION in real code run\n");

	bitmap_free(issued);
	kfree(handoff.ring);
	kfree(wa);
	kfree(tsk);
	cxgbi_ppm_release(g_ppm);
	dev_put(g_ndev);
	return 0;
}

static void __exit pooltest_real_exit(void)
{
}

module_init(pooltest_real_init);
module_exit(pooltest_real_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("stress the real (v4-patched) libcxgb_ppm.c alloc paths");

======================================================================
Makefile
======================================================================
obj-m := pooltest.o pooltest_real.o

KDIR  := /root/linux-7.3
BUILD := /root/lkbuild73

all:
	make -C $(KDIR) O=$(BUILD) M=$(CURDIR) KBUILD_MODPOST_WARN=1 modules

Re: [PATCH v4] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[netdev-bot+sinfo]

Hi!

This is an automated message. This series looks like a fix, but its
commit messages seem to be missing some information:

 - How the issue was discovered, e.g. hit in production, hit during
   development, syzbot report, manual code inspection, LLM or static
   analysis tool scan.

 - Whether the issue was actually triggered, or is only theoretical
   (e.g. found by code inspection). If it was triggered please include
   the symptoms, like the stack trace or error messages.

Please do not repost the series just to address the above. Instead,
reply to this email with the missing information, so that reviewers
can take it into account. If the series needs another revision for
other reasons, please include the information in the commit messages
then.

The evaluation is done by an LLM so it may be wrong, if you think
that is the case please reply and explain.

Re: [PATCH v3] libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] libcxgb: missing Fixes: trailer on bug-fix commit for
  ppm_get_cpu_entries()
--

commit f26f870fe2fc150e7c6210fe2807b65ce57e879d
Author: Haoran Wang <haoranwangsec@gmail.com>

> libcxgb: use raw_cpu_ptr() in ppm_get_cpu_entries()
>
> ppm_get_cpu_entries() disables preemption with get_cpu(), reads the
> current CPU's pool with per_cpu_ptr(), then acquires the pool's
> spin_lock_bh() while preemption is still disabled, and finally put_cpu().
>
> On PREEMPT_RT kernels spin_lock_bh() becomes a sleeping lock
> (rt_spin_lock), so the task may sleep with preempt_count elevated:
> scheduling while atomic, reported as a BUG by CONFIG_DEBUG_ATOMIC_SLEEP
> and a deadlock once the lock is contended.  Any iSCSI offload I/O
> through a Chelsio NIC that reaches ppm_get_cpu_entries() triggers it.
>
> Read the current CPU with raw_cpu_ptr() without raising preempt_count,
> and derive the global index base from pool->base (initialised when the
> pools are allocated) instead of cpu * pool_index_max, so the base
> always matches the pool that was locked.  The owning CPU remains in the
> debug output, now derived from pool->base.
>
> Signed-off-by: Haoran Wang <haoranwangsec@gmail.com>

[Severity: Low]
Since this fixes a sleeping-while-atomic bug in ppm_get_cpu_entries() on
PREEMPT_RT kernels, should this include a Fixes: tag?

  Fixes: 47da7e62b391 ("cxgb4: add iSCSI DDP page pod manager")

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261109180748.33682-1-haoranwangsec@gmail.com?part=1