From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Chuck Lever <chuck.lever@oracle.com>
Subject: [PATCH 4.19 16/46] svcrdma: Fix trace point use-after-free race
Date: Fri, 1 May 2020 15:22:41 +0200 [thread overview]
Message-ID: <20200501131504.178553519@linuxfoundation.org> (raw)
In-Reply-To: <20200501131457.023036302@linuxfoundation.org>
From: Chuck Lever <chuck.lever@oracle.com>
commit e28b4fc652c1830796a4d3e09565f30c20f9a2cf upstream.
I hit this while testing nfsd-5.7 with kernel memory debugging
enabled on my server:
Mar 30 13:21:45 klimt kernel: BUG: unable to handle page fault for address: ffff8887e6c279a8
Mar 30 13:21:45 klimt kernel: #PF: supervisor read access in kernel mode
Mar 30 13:21:45 klimt kernel: #PF: error_code(0x0000) - not-present page
Mar 30 13:21:45 klimt kernel: PGD 3601067 P4D 3601067 PUD 87c519067 PMD 87c3e2067 PTE 800ffff8193d8060
Mar 30 13:21:45 klimt kernel: Oops: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
Mar 30 13:21:45 klimt kernel: CPU: 2 PID: 1933 Comm: nfsd Not tainted 5.6.0-rc6-00040-g881e87a3c6f9 #1591
Mar 30 13:21:45 klimt kernel: Hardware name: Supermicro Super Server/X10SRL-F, BIOS 1.0c 09/09/2015
Mar 30 13:21:45 klimt kernel: RIP: 0010:svc_rdma_post_chunk_ctxt+0xab/0x284 [rpcrdma]
Mar 30 13:21:45 klimt kernel: Code: c1 83 34 02 00 00 29 d0 85 c0 7e 72 48 8b bb a0 02 00 00 48 8d 54 24 08 4c 89 e6 48 8b 07 48 8b 40 20 e8 5a 5c 2b e1 41 89 c6 <8b> 45 20 89 44 24 04 8b 05 02 e9 01 00 85 c0 7e 33 e9 5e 01 00 00
Mar 30 13:21:45 klimt kernel: RSP: 0018:ffffc90000dfbdd8 EFLAGS: 00010286
Mar 30 13:21:45 klimt kernel: RAX: 0000000000000000 RBX: ffff8887db8db400 RCX: 0000000000000030
Mar 30 13:21:45 klimt kernel: RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000246
Mar 30 13:21:45 klimt kernel: RBP: ffff8887e6c27988 R08: 0000000000000000 R09: 0000000000000004
Mar 30 13:21:45 klimt kernel: R10: ffffc90000dfbdd8 R11: 00c068ef00000000 R12: ffff8887eb4e4a80
Mar 30 13:21:45 klimt kernel: R13: ffff8887db8db634 R14: 0000000000000000 R15: ffff8887fc931000
Mar 30 13:21:45 klimt kernel: FS: 0000000000000000(0000) GS:ffff88885bd00000(0000) knlGS:0000000000000000
Mar 30 13:21:45 klimt kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 30 13:21:45 klimt kernel: CR2: ffff8887e6c279a8 CR3: 000000081b72e002 CR4: 00000000001606e0
Mar 30 13:21:45 klimt kernel: Call Trace:
Mar 30 13:21:45 klimt kernel: ? svc_rdma_vec_to_sg+0x7f/0x7f [rpcrdma]
Mar 30 13:21:45 klimt kernel: svc_rdma_send_write_chunk+0x59/0xce [rpcrdma]
Mar 30 13:21:45 klimt kernel: svc_rdma_sendto+0xf9/0x3ae [rpcrdma]
Mar 30 13:21:45 klimt kernel: ? nfsd_destroy+0x51/0x51 [nfsd]
Mar 30 13:21:45 klimt kernel: svc_send+0x105/0x1e3 [sunrpc]
Mar 30 13:21:45 klimt kernel: nfsd+0xf2/0x149 [nfsd]
Mar 30 13:21:45 klimt kernel: kthread+0xf6/0xfb
Mar 30 13:21:45 klimt kernel: ? kthread_queue_delayed_work+0x74/0x74
Mar 30 13:21:45 klimt kernel: ret_from_fork+0x3a/0x50
Mar 30 13:21:45 klimt kernel: Modules linked in: ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue ib_umad ib_ipoib mlx4_ib sb_edac x86_pkg_temp_thermal iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper crypto_simd cryptd pcspkr rpcrdma i2c_i801 rdma_ucm lpc_ich mfd_core ib_iser rdma_cm iw_cm ib_cm mei_me raid0 libiscsi mei sg scsi_transport_iscsi ioatdma wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod sr_mod cdrom mlx4_core crc32c_intel igb nvme i2c_algo_bit ahci i2c_core libahci nvme_core dca libata t10_pi qedr dm_mirror dm_region_hash dm_log dm_mod dax qede qed crc8 ib_uverbs ib_core
Mar 30 13:21:45 klimt kernel: CR2: ffff8887e6c279a8
Mar 30 13:21:45 klimt kernel: ---[ end trace 87971d2ad3429424 ]---
It's absolutely not safe to use resources pointed to by the @send_wr
argument of ib_post_send() _after_ that function returns. Those
resources are typically freed by the Send completion handler, which
can run before ib_post_send() returns.
Thus the trace points currently around ib_post_send() in the
server's RPC/RDMA transport are a hazard, even when they are
disabled. Rearrange them so that they touch the Work Request only
_before_ ib_post_send() is invoked.
Fixes: bd2abef33394 ("svcrdma: Trace key RDMA API events")
Fixes: 4201c7464753 ("svcrdma: Introduce svc_rdma_send_ctxt")
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
include/trace/events/rpcrdma.h | 50 ++++++++++++++++++++++++----------
net/sunrpc/xprtrdma/svc_rdma_rw.c | 3 --
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 16 ++++++----
3 files changed, 46 insertions(+), 23 deletions(-)
--- a/include/trace/events/rpcrdma.h
+++ b/include/trace/events/rpcrdma.h
@@ -1322,17 +1322,15 @@ DECLARE_EVENT_CLASS(svcrdma_sendcomp_eve
TRACE_EVENT(svcrdma_post_send,
TP_PROTO(
- const struct ib_send_wr *wr,
- int status
+ const struct ib_send_wr *wr
),
- TP_ARGS(wr, status),
+ TP_ARGS(wr),
TP_STRUCT__entry(
__field(const void *, cqe)
__field(unsigned int, num_sge)
__field(u32, inv_rkey)
- __field(int, status)
),
TP_fast_assign(
@@ -1340,12 +1338,11 @@ TRACE_EVENT(svcrdma_post_send,
__entry->num_sge = wr->num_sge;
__entry->inv_rkey = (wr->opcode == IB_WR_SEND_WITH_INV) ?
wr->ex.invalidate_rkey : 0;
- __entry->status = status;
),
- TP_printk("cqe=%p num_sge=%u inv_rkey=0x%08x status=%d",
+ TP_printk("cqe=%p num_sge=%u inv_rkey=0x%08x",
__entry->cqe, __entry->num_sge,
- __entry->inv_rkey, __entry->status
+ __entry->inv_rkey
)
);
@@ -1410,26 +1407,23 @@ TRACE_EVENT(svcrdma_wc_receive,
TRACE_EVENT(svcrdma_post_rw,
TP_PROTO(
const void *cqe,
- int sqecount,
- int status
+ int sqecount
),
- TP_ARGS(cqe, sqecount, status),
+ TP_ARGS(cqe, sqecount),
TP_STRUCT__entry(
__field(const void *, cqe)
__field(int, sqecount)
- __field(int, status)
),
TP_fast_assign(
__entry->cqe = cqe;
__entry->sqecount = sqecount;
- __entry->status = status;
),
- TP_printk("cqe=%p sqecount=%d status=%d",
- __entry->cqe, __entry->sqecount, __entry->status
+ TP_printk("cqe=%p sqecount=%d",
+ __entry->cqe, __entry->sqecount
)
);
@@ -1525,6 +1519,34 @@ DECLARE_EVENT_CLASS(svcrdma_sendqueue_ev
DEFINE_SQ_EVENT(full);
DEFINE_SQ_EVENT(retry);
+TRACE_EVENT(svcrdma_sq_post_err,
+ TP_PROTO(
+ const struct svcxprt_rdma *rdma,
+ int status
+ ),
+
+ TP_ARGS(rdma, status),
+
+ TP_STRUCT__entry(
+ __field(int, avail)
+ __field(int, depth)
+ __field(int, status)
+ __string(addr, rdma->sc_xprt.xpt_remotebuf)
+ ),
+
+ TP_fast_assign(
+ __entry->avail = atomic_read(&rdma->sc_sq_avail);
+ __entry->depth = rdma->sc_sq_depth;
+ __entry->status = status;
+ __assign_str(addr, rdma->sc_xprt.xpt_remotebuf);
+ ),
+
+ TP_printk("addr=%s sc_sq_avail=%d/%d status=%d",
+ __get_str(addr), __entry->avail, __entry->depth,
+ __entry->status
+ )
+);
+
#endif /* _TRACE_RPCRDMA_H */
#include <trace/define_trace.h>
--- a/net/sunrpc/xprtrdma/svc_rdma_rw.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c
@@ -331,8 +331,6 @@ static int svc_rdma_post_chunk_ctxt(stru
if (atomic_sub_return(cc->cc_sqecount,
&rdma->sc_sq_avail) > 0) {
ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
- trace_svcrdma_post_rw(&cc->cc_cqe,
- cc->cc_sqecount, ret);
if (ret)
break;
return 0;
@@ -345,6 +343,7 @@ static int svc_rdma_post_chunk_ctxt(stru
trace_svcrdma_sq_retry(rdma);
} while (1);
+ trace_svcrdma_sq_post_err(rdma, ret);
set_bit(XPT_CLOSE, &xprt->xpt_flags);
/* If even one was posted, there will be a completion. */
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -310,15 +310,17 @@ int svc_rdma_send(struct svcxprt_rdma *r
}
svc_xprt_get(&rdma->sc_xprt);
+ trace_svcrdma_post_send(wr);
ret = ib_post_send(rdma->sc_qp, wr, NULL);
- trace_svcrdma_post_send(wr, ret);
- if (ret) {
- set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags);
- svc_xprt_put(&rdma->sc_xprt);
- wake_up(&rdma->sc_send_wait);
- }
- break;
+ if (ret)
+ break;
+ return 0;
}
+
+ trace_svcrdma_sq_post_err(rdma, ret);
+ set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags);
+ svc_xprt_put(&rdma->sc_xprt);
+ wake_up(&rdma->sc_send_wait);
return ret;
}
next prev parent reply other threads:[~2020-05-01 13:53 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-01 13:22 [PATCH 4.19 00/46] 4.19.120-rc1 review Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 01/46] remoteproc: Fix wrong rvring index computation Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 02/46] mtd: cfi: fix deadloop in cfi_cmdset_0002.c do_write_buffer Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 03/46] include/uapi/linux/swab.h: fix userspace breakage, use __BITS_PER_LONG for swap Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 04/46] binder: take read mode of mmap_sem in binder_alloc_free_page() Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 05/46] usb: dwc3: gadget: Do link recovery for SS and SSP Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 06/46] usb: gadget: udc: bdc: Remove unnecessary NULL checks in bdc_req_complete Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 07/46] iio:ad7797: Use correct attribute_group Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 08/46] ASoC: q6dsp6: q6afe-dai: add missing channels to MI2S DAIs Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 09/46] ASoC: tas571x: disable regulators on failed probe Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 10/46] ASoC: wm8960: Fix wrong clock after suspend & resume Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 11/46] nfsd: memory corruption in nfsd4_lock() Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 12/46] i2c: altera: use proper variable to hold errno Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 13/46] rxrpc: Fix DATA Tx to disable nofrag for UDP on AF_INET6 socket Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 14/46] net/cxgb4: Check the return from t4_query_params properly Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 15/46] xfs: acquire superblock freeze protection on eofblocks scans Greg Kroah-Hartman
2020-05-01 13:22 ` Greg Kroah-Hartman [this message]
2020-05-01 13:22 ` [PATCH 4.19 17/46] svcrdma: Fix leak of svc_rdma_recv_ctxt objects Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 18/46] PCI: Avoid ASMedia XHCI USB PME# from D0 defect Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 19/46] PCI: Move Apex Edge TPU class quirk to fix BAR assignment Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 20/46] ARM: dts: bcm283x: Disable dsi0 node Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 21/46] cpumap: Avoid warning when CONFIG_DEBUG_PER_CPU_MAPS is enabled Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 22/46] net/mlx5: Fix failing fw tracer allocation on s390 Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 23/46] perf/core: fix parent pid/tid in task exit events Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 24/46] bpf, x86_32: Fix incorrect encoding in BPF_LDX zero-extension Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 25/46] mm: shmem: disable interrupt when acquiring info->lock in userfaultfd_copy path Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 26/46] xfs: clear PF_MEMALLOC before exiting xfsaild thread Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 27/46] bpf, x86: Fix encoding for lower 8-bit registers in BPF_STX BPF_B Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 28/46] net: fec: set GPR bit on suspend by DT configuration Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 29/46] x86: hyperv: report value of misc_features Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 30/46] xfs: fix partially uninitialized structure in xfs_reflink_remap_extent Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 31/46] ALSA: hda: Keep the controller initialization even if no codecs found Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 32/46] ALSA: hda: Explicitly permit using autosuspend if runtime PM is supported Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 33/46] scsi: target: fix PR IN / READ FULL STATUS for FC Greg Kroah-Hartman
2020-05-01 13:22 ` [PATCH 4.19 34/46] scsi: target: tcmu: reset_ring should reset TCMU_DEV_BIT_BROKEN Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 35/46] objtool: Fix CONFIG_UBSAN_TRAP unreachable warnings Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 36/46] objtool: Support Clang non-section symbols in ORC dump Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 37/46] xen/xenbus: ensure xenbus_map_ring_valloc() returns proper grant status Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 38/46] ALSA: hda: call runtime_allow() for all hda controllers Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 39/46] arm64: Delete the space separator in __emit_inst Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 40/46] ext4: use matching invalidatepage in ext4_writepage Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 41/46] ext4: increase wait time needed before reuse of deleted inode numbers Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 42/46] ext4: convert BUG_ONs to WARN_ONs in mballoc.c Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 43/46] hwmon: (jc42) Fix name to have no illegal characters Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 44/46] bpf, x86_32: Fix clobbering of dst for BPF_JSET Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 45/46] qed: Fix use after free in qed_chain_free Greg Kroah-Hartman
2020-05-01 13:23 ` [PATCH 4.19 46/46] ext4: check for non-zero journal inum in ext4_calculate_overhead Greg Kroah-Hartman
2020-05-01 15:16 ` [PATCH 4.19 00/46] 4.19.120-rc1 review Jon Hunter
2020-05-01 22:05 ` Naresh Kamboju
2020-05-01 22:11 ` Guenter Roeck
2020-05-02 23:17 ` shuah
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200501131504.178553519@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=chuck.lever@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®