From: "Lin Ma" <linma@zju.edu.cn>
To: "Leon Romanovsky" <leon@kernel.org>
Cc: jgg@ziepe.ca, cmeiohas@nvidia.com, michaelgur@nvidia.com,
linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [bug report] RDMA/iwpm: reentrant iwpm hello message
Date: Wed, 25 Dec 2024 00:16:45 +0800 (GMT+08:00) [thread overview]
Message-ID: <48307bf.eecb.193f974dadf.Coremail.linma@zju.edu.cn> (raw)
In-Reply-To: <20241224141127.GH171473@unreal>
Hello Leon,
> > Please let me know if I understand this correctly or incorrectly?
>
> The thing is that down_write() is called when we unregistering module
> which sent netlink messages. It shouldn't happen.
>
I acknowledge that this is a low-probability event. However, the race
condition still exists; otherwise, these read and write semaphores
would not be necessary. Why not just remove all of them?
Moreover, I find that even without the deadlock, this reentrant message
would hang the kernel and cannot be killed, with logs like below:
(after disabling locking sanitizer, tested in latest ubuntu)
[2187983.899998] INFO: task poc.elf:1717021 blocked for more than 122 seconds.
[2187983.900049] Not tainted 6.8.0-49-generic #49~22.04.1-Ubuntu
[2187983.900057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2187983.900063] task:poc.elf state:D stack:0 pid:1717021 tgid:1717021 ppid:1716834 flags:0x00004006
[2187983.900087] Call Trace:
[2187983.900094] <TASK>
[2187983.900355] __schedule+0x27c/0x6a0
[2187983.900430] schedule+0x33/0x110
[2187983.900442] schedule_preempt_disabled+0x15/0x30
[2187983.900454] __mutex_lock.constprop.0+0x3f8/0x7a0
[2187983.900476] __mutex_lock_slowpath+0x13/0x20
[2187983.900486] mutex_lock+0x3c/0x50
[2187983.900493] __netlink_dump_start+0x76/0x2a0
[2187983.900552] rdma_nl_rcv_msg+0x24c/0x310 [ib_core]
[2187983.900673] ? __pfx_iwpm_hello_cb+0x10/0x10 [iw_cm]
[2187983.900699] rdma_nl_rcv_skb.constprop.0.isra.0+0xbb/0x120 [ib_core]
[2187983.900802] rdma_nl_rcv+0xe/0x20 [ib_core]
[2187983.900898] netlink_unicast+0x1b0/0x2a0
[2187983.900911] rdma_nl_unicast+0x49/0x70 [ib_core]
[2187983.901005] iwpm_send_hello+0xfd/0x150 [iw_cm]
[2187983.901030] iwpm_hello_cb+0xb9/0x130 [iw_cm]
[2187983.901052] netlink_dump+0x1c0/0x340
[2187983.901065] __netlink_dump_start+0x1ef/0x2a0
[2187983.901077] rdma_nl_rcv_msg+0x24c/0x310 [ib_core]
[2187983.901219] ? __pfx_iwpm_hello_cb+0x10/0x10 [iw_cm]
[2187983.901245] rdma_nl_rcv_skb.constprop.0.isra.0+0xbb/0x120 [ib_core]
[2187983.901344] rdma_nl_rcv+0xe/0x20 [ib_core]
[2187983.901437] netlink_unicast+0x1b0/0x2a0
[2187983.901449] rdma_nl_unicast+0x49/0x70 [ib_core]
[2187983.901544] iwpm_send_hello+0xfd/0x150 [iw_cm]
[2187983.901567] iwpm_hello_cb+0xb9/0x130 [iw_cm]
[2187983.901589] netlink_dump+0x1c0/0x340
[2187983.901602] __netlink_dump_start+0x1ef/0x2a0
[2187983.901613] rdma_nl_rcv_msg+0x24c/0x310 [ib_core]
[2187983.901707] ? __pfx_iwpm_hello_cb+0x10/0x10 [iw_cm]
[2187983.901731] rdma_nl_rcv_skb.constprop.0.isra.0+0xbb/0x120 [ib_core]
[2187983.901830] rdma_nl_rcv+0xe/0x20 [ib_core]
[2187983.901922] netlink_unicast+0x1b0/0x2a0
[2187983.901933] netlink_sendmsg+0x214/0x470
[2187983.901946] __sys_sendto+0x21b/0x230
[2187983.901992] __x64_sys_sendto+0x24/0x40
[2187983.902002] x64_sys_call+0x1fc0/0x24b0
[2187983.902023] do_syscall_64+0x81/0x170
[2187983.902059] ? security_file_alloc+0x5f/0xf0
[2187983.902079] ? alloc_empty_file+0x85/0x130
[2187983.902140] ? alloc_file+0x9b/0x170
[2187983.902150] ? alloc_file_pseudo+0x9e/0x100
[2187983.902163] ? restore_fpregs_from_fpstate+0x3d/0xd0
[2187983.902197] ? switch_fpu_return+0x55/0xf0
[2187983.902208] ? syscall_exit_to_user_mode+0x83/0x260
[2187983.902229] ? do_syscall_64+0x8d/0x170
[2187983.902240] ? irqentry_exit+0x43/0x50
[2187983.902249] ? clear_bhb_loop+0x15/0x70
[2187983.902293] ? clear_bhb_loop+0x15/0x70
[2187983.902302] ? clear_bhb_loop+0x15/0x70
[2187983.902311] entry_SYSCALL_64_after_hwframe+0x78/0x80
[2187983.902319] RIP: 0033:0x440624
[2187983.902582] RSP: 002b:00007ffcfa4b29f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[2187983.902592] RAX: ffffffffffffffda RBX: 0000000000400400 RCX: 0000000000440624
[2187983.902598] RDX: 0000000000000018 RSI: 00007ffcfa4b2a30 RDI: 0000000000000003
[2187983.902604] RBP: 00007ffcfa4b3a40 R08: 000000000047df08 R09: 000000000000000c
[2187983.902609] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000403990
[2187983.902614] R13: 0000000000000000 R14: 00000000006a6018 R15: 0000000000000000
That's why I'm quite sure this is a bug and requires fixing.
Thanks
Lin
next prev parent reply other threads:[~2024-12-24 16:16 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-20 15:32 Lin Ma
2024-12-24 9:29 ` Leon Romanovsky
2024-12-24 10:51 ` Lin Ma
2024-12-24 14:11 ` Leon Romanovsky
2024-12-24 16:16 ` Lin Ma [this message]
2024-12-24 19:26 ` Leon Romanovsky
2024-12-25 1:58 ` Lin Ma
2024-12-30 18:28 ` Leon Romanovsky
2025-01-08 15:14 ` Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=48307bf.eecb.193f974dadf.Coremail.linma@zju.edu.cn \
--to=linma@zju.edu.cn \
--cc=cmeiohas@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=michaelgur@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®