mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Dragos Tatulea <dtatulea@nvidia.com>
To: "Michael S. Tsirkin" <mst@redhat.com>,
	Vladimir Petrzhikovskii <vladimir@dfb.sh>
Cc: "Jason Wang" <jasowangio@gmail.com>,
	"Eugenio Pérez" <eperezma@redhat.com>,
	"Xuan Zhuo" <xuanzhuo@linux.alibaba.com>,
	virtualization@lists.linux.dev, linux-kernel@vger.kernel.org,
	stable@vger.kernel.org, "Nikolai Kalinin" <nikolai@kalinidze.me>
Subject: Re: [PATCH] vdpa/mlx5: Bound the KLM count of the user MR
Date: Thu, 19 Nov 2026 19:12:52 +0100	[thread overview]
Message-ID: <d4f67696-5095-4948-9029-f657058431b0@nvidia.com> (raw)
In-Reply-To: <20261119101523-mutt-send-email-mst@kernel.org>

On 19.11.26 16:15, Michael S. Tsirkin wrote:
> On Mon, Nov 09, 2026 at 08:21:28PM +0100, Vladimir Petrzhikovskii wrote:
>> create_user_mr() builds one indirect MKey that covers the whole vhost
>> IOTLB. It fills each gap between two mappings with null KLMs, one per
>> GiB, and never checks the total number of KLMs against the device
>> limit, MLX5_CAP_GEN(log_max_klm_list_size).
>>
>> With a virtio-IOMMU the guest chooses the IOVAs: the VMM (QEMU)
>> forwards each map the guest makes to the vhost-vDPA device. If the
>> guest maps one page at IOVA 0 and one at IOVA 2^58, the driver needs
>> more than 2^28 KLMs. create_indirect_key() computes the buffer size in
>> an int, which overflows: the driver allocates 336 bytes, and
>> fill_indir() then writes 16 bytes per KLM, 4 GiB in total, far past
>> the end of the buffer.
>>
>> nnuls is an int, so a gap of 2^61 bytes or more is miscounted on every
>> architecture, and on 32-bit the unsigned long num_klms wraps. The gap
>> is rounded up to whole GiB as (size + 2^30 - 1) >> 30. For a gap
>> larger than 2^64 - 2^30, for example below a page mapped in the top
>> GiB, that addition wraps and the driver counts no null KLMs for the
>> gap, while fill_indir() still writes one for every GiB of it.
>>
Please make this paragraph more human friendly. Enough to say that
the type doesn't fit the value or something like that.

>> Count KLMs in u64 and round the gap up without the overflowing
>> addition. Before creating any direct key, return -E2BIG with a
>> rate-limited warning if the MR needs more KLMs than the device limit,
>> or more than fit in a CREATE_MKEY command whose length is an int.
>> Allocate the command with kvzalloc(), because a list within the device
>> limit can be too large for kzalloc(). A normal guest is unaffected:
>> memory at 0-3 GiB and 4-68 GiB needs 68 KLMs.
>>
No need to say what the code does. Focus on the why.

>> Found by an LLM code review agent auditing vhost-vDPA backends. We have
>> no mlx5 hardware, so the fix is untested on it; it builds with W=1 on
>> x86_64 and i386.
>>
>> Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration code")
>> Cc: stable@vger.kernel.org
>> Co-developed-by: Nikolai Kalinin <nikolai@kalinidze.me>
>> Signed-off-by: Nikolai Kalinin <nikolai@kalinidze.me>
>> Signed-off-by: Vladimir Petrzhikovskii <vladimir@dfb.sh>
>> ---
>> Dragos, one question on the limit: the patch takes
>> MLX5_CAP_GEN(log_max_klm_list_size) as the largest KLM list an indirect
>> CREATE_MKEY accepts, as mlx5_ib does for fast-reg lists. If firmware
>> applies a different limit to CREATE_MKEY, please say so and I will
>> change the bound.
>>
>> The guest kernel change used for the reachability check is available
>> on request.
> 
Yes The limit is right.

> 
> Dragos, care to answer?
> 
Yes. Done.

>> ---
>>  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
>>  drivers/vdpa/mlx5/core/mr.c        | 30 +++++++++++++++++++++++++-----
>>  2 files changed, 26 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> index 42f2f44b383c..1d84824ca429 100644
>> --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> @@ -32,7 +32,7 @@ struct mlx5_vdpa_mr {
>>  	/* list of direct MRs descendants of this indirect mr */
>>  	struct list_head head;
>>  	unsigned long num_directs;
>> -	unsigned long num_klms;
>> +	u64 num_klms;
>>  
>>  	struct vhost_iotlb *iotlb;
>>  
>> diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
>> index b0c5ff23d022..03db8345c1bb 100644
>> --- a/drivers/vdpa/mlx5/core/mr.c
>> +++ b/drivers/vdpa/mlx5/core/mr.c
>> @@ -206,6 +206,16 @@ static int klm_byte_size(int nklms)
>>  	return 16 * ALIGN(nklms, 4);
>>  }
>>  
>> +/* Largest KLM count whose CREATE_MKEY input length still fits in an int. */
Comment not needed.

>> +#define MLX5_VDPA_MAX_KLMS_INLEN \
>> +	ALIGN_DOWN((INT_MAX - MLX5_ST_SZ_BYTES(create_mkey_in)) / 16, 4)
>> +
>> +static u64 max_klms(struct mlx5_vdpa_dev *mvdev)
>> +{
>> +	return min_t(u64, BIT_ULL(MLX5_CAP_GEN(mvdev->mdev, log_max_klm_list_size)),
>> +		     MLX5_VDPA_MAX_KLMS_INLEN);
>> +}
>> +
>>  #define MLX5_VDPA_MTT_ALIGN 16
>>  
>>  static int create_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr)
>> @@ -338,7 +348,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>>  		return -EINVAL;
>>  
>>  	inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + klm_byte_size(mr->num_klms);
>> -	in = kzalloc(inlen, GFP_KERNEL);
>> +	in = kvzalloc(inlen, GFP_KERNEL);
Unrelated change for fix. Drop please.

>>  	if (!in)
>>  		return -ENOMEM;
>>  
>> @@ -355,7 +365,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>>  	MLX5_SET(create_mkey_in, in, translations_octword_actual_size, mr->num_klms);
>>  	fill_indir(mvdev, mr, in);
>>  	err = mlx5_vdpa_create_mkey(mvdev, &mr->mkey, in, inlen);
>> -	kfree(in);
>> +	kvfree(in);
>>  	return err;
>>  }
>>  
>> @@ -501,6 +511,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>>  {
>>  	struct mlx5_vdpa_direct_mr *dmr;
>>  	struct mlx5_vdpa_direct_mr *n;
>> +	u64 klm_limit = max_klms(mvdev);
>>  	struct vhost_iotlb_map *map;
>>  	u32 pperm = U16_MAX;
>>  	u64 last = U64_MAX;
>> @@ -508,7 +519,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>>  	u64 pe = U64_MAX;
>>  	u64 start = 0;
>>  	int err = 0;
>> -	int nnuls;
>> +	u64 nnuls;
>>  
>>  	INIT_LIST_HEAD(&mr->head);
>>  	for (map = vhost_iotlb_itree_first(iotlb, start, last); map;
>> @@ -521,9 +532,10 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>>  				if (pe < map->start) {
>>  					/* We have a hole in the map. Check how
>>  					 * many null keys are required to fill it.
>> +					 * The hole can be nearly 2^64 bytes, so
>> +					 * round up without adding to its size.
>>  					 */
>> -					nnuls = MLX5_DIV_ROUND_UP_POW2(map->start - pe,
>> -								       LOG_MAX_KLM_SIZE);
>> +					nnuls = ((map->start - pe - 1) >> LOG_MAX_KLM_SIZE) + 1;
Why not DIV_ROUND_UP_POW2(map->start - pe, MAX_KLM_SIZE)?

>>  					mr->num_klms += nnuls;
>>  				}
>>  				err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb);
>> @@ -539,6 +551,14 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>>  	if (err)
>>  		goto err_chain;
>>  
>> +	if (mr->num_klms > klm_limit) {
>> +		dev_warn_ratelimited(mvdev->mdev->device,
>> +				     "map needs %llu KLMs, limit is %llu\n",
>> +				     mr->num_klms, klm_limit);
Use mlx5_vdpa_warn(). Rate limiting here won't help: the other prints
will trigger.

Thanks,
Dragos

  reply	other threads:[~2026-11-19 18:13 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-09 19:21 Vladimir Petrzhikovskii
2026-11-19 15:15 ` Michael S. Tsirkin
2026-11-19 18:12   ` Dragos Tatulea [this message]
2026-11-20 13:20     ` Vladimir Petrzhikovskii

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d4f67696-5095-4948-9029-f657058431b0@nvidia.com \
    --to=dtatulea@nvidia.com \
    --cc=eperezma@redhat.com \
    --cc=jasowangio@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mst@redhat.com \
    --cc=nikolai@kalinidze.me \
    --cc=stable@vger.kernel.org \
    --cc=virtualization@lists.linux.dev \
    --cc=vladimir@dfb.sh \
    --cc=xuanzhuo@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®