From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A7E2280CF6 for ; Wed, 28 Jan 2026 17:23:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769621028; cv=none; b=HXaEmdXWfdRMi74C37rZSWUM69+FkHpu2FrblEYPImvRla5QCeMW0jrc8h8n85YnP3okka6c4+/tueGDZXZ+RNG2+WSY7zJ7QmQCObPvRanjP7Y97rlwhpRXG5sO3sPWiS2scVmm+kVR9NNWRa3rLPVQtVWd3XllWta8AU4qKso= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769621028; c=relaxed/simple; bh=qbbTNigXkPLBi/kTiMQHzZE3hHFs36knucV3j8O5E5c=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=i/i5HeEWsEa23+V+UuFJSZCgbsf5eIPVvzdimwisIDUUzeqaV5L4ZGXHUPjZzQCp8Ntu5sbHccTibWU8zyvKnRE7Z/foPnjt4Sbo77PGAGmpcqqVMYdL0ihl81+MnWjmUpjGYwcEil9VMy4WMNGqfmGrb7eReoELf9L0WlZYMlM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=iiXWvfny; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=njUzzOvK; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iiXWvfny"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="njUzzOvK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1769621026; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=gPnWFJelX/wlQBx2ivYdH45+n52GlMlmx8tnvEyIbww=; b=iiXWvfnyPJnKmz6ewgRG/hC7NU9+/KShOznFiWjYMgHUs697hSRaUvnA10ql2AiNctT/kL k+DEjGCZXh4uy+4HxR5wj1pGiToLmWXu8gbE6uxvbazKBwl1rR2swgZuhT8Q1i+5oyCaZS WOXUBS/Q1w4Zi6nmaIUhASvOfjwiqs4= Received: from mail-lj1-f197.google.com (mail-lj1-f197.google.com [209.85.208.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-94-9KfeRWpoOCSkWJyXVyowSQ-1; Wed, 28 Jan 2026 12:23:41 -0500 X-MC-Unique: 9KfeRWpoOCSkWJyXVyowSQ-1 X-Mimecast-MFC-AGG-ID: 9KfeRWpoOCSkWJyXVyowSQ_1769621020 Received: by mail-lj1-f197.google.com with SMTP id 38308e7fff4ca-385ba7ace13so238571fa.1 for ; Wed, 28 Jan 2026 09:23:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1769621020; x=1770225820; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=gPnWFJelX/wlQBx2ivYdH45+n52GlMlmx8tnvEyIbww=; b=njUzzOvKxuPPRa09w+SwKqwekCnzxfUwApSBpt1VIZc3ZCt/t1JX96Se99cc4P1uOl VY7zzgCxYMtcYsWQT2B0VB0z80QWPlz+hiTybhGC9G1TC5ekgbEz/pjCd33lwuo+MeQ+ u6fmzEWW9GfJBZPdXdmFMY/Qa0ZG1iQRAHnoIztUnV8bSycUPRsNwb4/k67T343d89Xt DNAqh9MWYM5+jK0H6YdXXt27AqQoc57dtDa73EPhGnEDBpk96AL9y+/GVIlU3HxU3rnO P9HAjS9bMaqTzT60jjFyaOqVh7o5/WdF4VN2lzqVT70W2jtv8lywHFItpFVxQNomzZVv K8rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769621020; x=1770225820; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gPnWFJelX/wlQBx2ivYdH45+n52GlMlmx8tnvEyIbww=; b=Sl90gl5OtdaRP8PYtBQT7VAosL5X0MumjH5dDhB8oo0UOSGDd94NHL4dkEUppPBbs8 gHsQK9JBKdnyMtbi2gSjYtU3OBwXJaNtZW/zzjkimCACynJt6stplh/2QmAHHSWNLhxR +zm6h8EUrduyxB9NV083ntrrXkn3eFT9vk5rLGDNe23BJcuie0tmEizFOFw9DKGPDYWU rO1flgV4VO+pPGku+e6xt8WTT+B6ZcOjRcPkrCQbrVON231+AiZ6xslyjgLTSZb7R6Kw NhtBoLUmgj23BoddahiqVqAC9HPwteHOYqFfwdFwJXkjWKXJIEX7ZesCMx2LXMyqPKO5 LEjA== X-Forwarded-Encrypted: i=1; AJvYcCW7UNSGsbtIqtthns8pTUTafcAHoqeDqG1YI+SMXLiGz6Qg1oqPWoc1zKGj8GqRTdWRJafE94BJA9KLJf0=@vger.kernel.org X-Gm-Message-State: AOJu0YxSAdb+zeVL6FXgBIMu2mq+MknfplEfU1ZtgP3qN1lE3SSy3i89 BdAd3v6knZldOtRoxhQxwdIptW3ztmJw0DvZlFMNUFPnDTAgioaxlRsRxVrOVMSfi1ycbsnhPrR p3r2t+fc3lkwJnGby6qXt4KSiRP0et8rIrb7pKO7S2Uk+O7bfij63tyOJvi0kgkulBw== X-Gm-Gg: AZuq6aKxBKxvpb4QoDQVMs1MHSSpnzbgKp6HTCBBLo8O5QUDaT5FF0BhXIPorYWaipu m9+ZEGx6LmwlZXwE4pbc65UEd/OWlyNQZfAVNEmxv+KkGR/33y55OAvqu8M5LmDmO8dHcxD8nUy D8R6eRoRK+6q57gX4+6LzZEgJSpgj8wwuN14Wq7UDCL+vacGy9bv559JeqJ1Yea8PP3RJfU0jk+ FmpVxsISLsumusG7cxV3hRwuxd27r3xDlr8GuQvOojvlYHf7n5PbY3ZTQGw9y/YSVpnlH86VlYS 4aVPsDvEZNWT9jJv0J0M28iuNUhZY/tUhcyvqeCi7JlJ63iOovAojOElRz7I2lJMHCooTAjgTQq YnUWo3d6hCfsbkNwaU+1fvy3o6Mwxcm2TRg== X-Received: by 2002:a05:6512:3b99:b0:59d:4a12:5f7 with SMTP id 2adb3069b0e04-59e04025374mr2793092e87.31.1769621019806; Wed, 28 Jan 2026 09:23:39 -0800 (PST) X-Received: by 2002:a05:6512:3b99:b0:59d:4a12:5f7 with SMTP id 2adb3069b0e04-59e04025374mr2793078e87.31.1769621019241; Wed, 28 Jan 2026 09:23:39 -0800 (PST) Received: from redhat.com (IGLD-80-230-34-155.inter.net.il. [80.230.34.155]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-59e074b7a8csm643655e87.77.2026.01.28.09.23.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jan 2026 09:23:38 -0800 (PST) Date: Wed, 28 Jan 2026 12:23:35 -0500 From: "Michael S. Tsirkin" To: Johannes Thumshirn Cc: Jason Wang , Xuan Zhuo , Eugenio =?iso-8859-1?Q?P=E9rez?= , "open list:VIRTIO CORE" , open list , Alexander Graf Subject: Re: [PATCH] virtio_ring: Add READ_ONCE annotations for device-writable fields Message-ID: <20260128121309-mutt-send-email-mst@kernel.org> References: <20260128135947.455686-1-johannes.thumshirn@wdc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260128135947.455686-1-johannes.thumshirn@wdc.com> On Wed, Jan 28, 2026 at 02:59:46PM +0100, Johannes Thumshirn wrote: > From: Alexander Graf > > KCSAN reports data races when accessing virtio ring fields that are > concurrently written by the device (host). These are legitimate > concurrent accesses where the CPU reads fields that the device updates > via DMA-like mechanisms. > > Add accessor functions that use READ_ONCE() to properly annotate these > device-writable fields and prevent compiler optimizations that could let's add "in theory" here > break the code. This also serves as documentation showing which fields > are shared with the device. > > The affected fields are: > - Split ring: used->idx, used->ring[].id, used->ring[].len > - Packed ring: desc[].flags, desc[].id, desc[].len I would add here: using WRITE_ONCE for driver writes into fields isn't done here as it does not currently trigger warnings. > > Reported-by: Kernel Concurrency Sanitizer (KCSAN) > Signed-off-by: Alexander Graf > [jth: Add READ_ONCE in virtqueue_kick_prepare_split ] > Signed-off-by: Johannes Thumshirn Looks good to me. Yet something to improve: > --- > drivers/virtio/virtio_ring.c | 88 ++++++++++++++++++++++++++++++------ > 1 file changed, 73 insertions(+), 15 deletions(-) > > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c > index ddab68959671..74957c83e138 100644 > --- a/drivers/virtio/virtio_ring.c > +++ b/drivers/virtio/virtio_ring.c > @@ -222,6 +222,63 @@ struct vring_virtqueue { > #endif > }; > > +/* > + * Accessors for device-writable fields in virtio rings. > + * These fields are concurrently written by the device and read by the driver. > + * Use READ_ONCE() to prevent compiler optimizations and document the > + * intentional data race. ... and prevent KCSAN warnings > + */ > + > +/* Split ring: read device-written fields from used ring */ > +static inline u16 vring_used_idx_read(const struct vring_virtqueue *vq) I'd prefer a consistent prefix. Maybe: vring_read_used_idx and so on. > +{ > + return virtio16_to_cpu(vq->vq.vdev, > + READ_ONCE(vq->split.vring.used->idx)); > +} > + > +static inline u32 vring_used_id_read(const struct vring_virtqueue *vq, > + u16 idx) > +{ > + return virtio32_to_cpu(vq->vq.vdev, > + READ_ONCE(vq->split.vring.used->ring[idx].id)); > +} > + > +static inline u32 vring_used_len_read(const struct vring_virtqueue *vq, > + u16 idx) > +{ > + return virtio32_to_cpu(vq->vq.vdev, > + READ_ONCE(vq->split.vring.used->ring[idx].len)); > +} > + > +/* Packed ring: read device-written fields from descriptors */ > +static inline u16 vring_packed_desc_flags_read(const struct vring_virtqueue *vq, > + u16 idx) > +{ > + return le16_to_cpu(READ_ONCE(vq->packed.vring.desc[idx].flags)); > +} > + > +static inline u16 vring_packed_desc_id_read(const struct vring_virtqueue *vq, > + u16 idx) > +{ > + return le16_to_cpu(READ_ONCE(vq->packed.vring.desc[idx].id)); > +} > + > +static inline u32 vring_packed_desc_len_read(const struct vring_virtqueue *vq, > + u16 idx) > +{ > + return le32_to_cpu(READ_ONCE(vq->packed.vring.desc[idx].len)); > +} > + > +/* > + * Note: We don't need READ_ONCE for driver->device fields like: > + * - split.vring.avail->idx (driver writes, device reads) > + * - packed.vring.desc[].addr (driver writes, device reads) > + * These are written by the driver and only read by the device, so the > + * driver can safely access them read them, really. > without READ_ONCE. ... and without triggering KCSAM warnings. > The device must use > + * appropriate barriers on its side. > + */ > + > + extra empty line here. > static struct vring_desc_extra *vring_alloc_desc_extra(unsigned int num); > static void vring_free(struct virtqueue *_vq); > > @@ -736,9 +793,10 @@ static bool virtqueue_kick_prepare_split(struct virtqueue *_vq) > LAST_ADD_TIME_INVALID(vq); > > if (vq->event) { > - needs_kick = vring_need_event(virtio16_to_cpu(_vq->vdev, > - vring_avail_event(&vq->split.vring)), > - new, old); > + u16 event = virtio16_to_cpu(_vq->vdev, > + READ_ONCE(vring_avail_event(&vq->split.vring))); why not wrap this one, too? > + > + needs_kick = vring_need_event(event, new, old); > } else { > needs_kick = !(vq->split.vring.used->flags & > cpu_to_virtio16(_vq->vdev, > @@ -808,8 +866,7 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, > > static bool more_used_split(const struct vring_virtqueue *vq) > { > - return vq->last_used_idx != virtio16_to_cpu(vq->vq.vdev, > - vq->split.vring.used->idx); > + return vq->last_used_idx != vring_used_idx_read(vq); > } > > static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > @@ -838,10 +895,8 @@ static void *virtqueue_get_buf_ctx_split(struct virtqueue *_vq, > virtio_rmb(vq->weak_barriers); > > last_used = (vq->last_used_idx & (vq->split.vring.num - 1)); > - i = virtio32_to_cpu(_vq->vdev, > - vq->split.vring.used->ring[last_used].id); > - *len = virtio32_to_cpu(_vq->vdev, > - vq->split.vring.used->ring[last_used].len); > + i = vring_used_id_read(vq, last_used); > + *len = vring_used_len_read(vq, last_used); > > if (unlikely(i >= vq->split.vring.num)) { > BAD_RING(vq, "id %u out of range\n", i); > @@ -923,8 +978,7 @@ static bool virtqueue_poll_split(struct virtqueue *_vq, unsigned int last_used_i > { > struct vring_virtqueue *vq = to_vvq(_vq); > > - return (u16)last_used_idx != virtio16_to_cpu(_vq->vdev, > - vq->split.vring.used->idx); > + return (u16)last_used_idx != vring_used_idx_read(vq); > } > > static bool virtqueue_enable_cb_delayed_split(struct virtqueue *_vq) > @@ -1701,10 +1755,10 @@ static void detach_buf_packed(struct vring_virtqueue *vq, > static inline bool is_used_desc_packed(const struct vring_virtqueue *vq, > u16 idx, bool used_wrap_counter) > { > - bool avail, used; > u16 flags; > + bool avail, used; > > - flags = le16_to_cpu(vq->packed.vring.desc[idx].flags); > + flags = vring_packed_desc_flags_read(vq, idx); > avail = !!(flags & (1 << VRING_PACKED_DESC_F_AVAIL)); > used = !!(flags & (1 << VRING_PACKED_DESC_F_USED)); > > @@ -1751,8 +1805,8 @@ static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq, > last_used_idx = READ_ONCE(vq->last_used_idx); > used_wrap_counter = packed_used_wrap_counter(last_used_idx); > last_used = packed_last_used(last_used_idx); > - id = le16_to_cpu(vq->packed.vring.desc[last_used].id); > - *len = le32_to_cpu(vq->packed.vring.desc[last_used].len); > + id = vring_packed_desc_id_read(vq, last_used); > + *len = vring_packed_desc_len_read(vq, last_used); > > if (unlikely(id >= vq->packed.vring.num)) { > BAD_RING(vq, "id %u out of range\n", id); > @@ -1850,6 +1904,10 @@ static bool virtqueue_poll_packed(struct virtqueue *_vq, u16 off_wrap) > bool wrap_counter; > u16 used_idx; > > + /* > + * Note: off_wrap is from virtqueue_enable_cb_prepare_packed() which > + * already used READ_ONCE on vq->last_used_idx, so we don't need it again. we don't need what again? off_wrap is a local variable. > + */ > wrap_counter = off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR; > used_idx = off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR); > > -- > 2.52.0