From: Dragos Tatulea <dtatulea@nvidia.com>
To: "Michael S. Tsirkin" <mst@redhat.com>,
Vladimir Petrzhikovskii <vladimir@dfb.sh>
Cc: "Jason Wang" <jasowangio@gmail.com>,
"Eugenio Pérez" <eperezma@redhat.com>,
"Xuan Zhuo" <xuanzhuo@linux.alibaba.com>,
virtualization@lists.linux.dev, linux-kernel@vger.kernel.org,
stable@vger.kernel.org, "Nikolai Kalinin" <nikolai@kalinidze.me>
Subject: Re: [PATCH] vdpa/mlx5: Bound the KLM count of the user MR
Date: Thu, 19 Nov 2026 19:12:52 +0100 [thread overview]
Message-ID: <d4f67696-5095-4948-9029-f657058431b0@nvidia.com> (raw)
In-Reply-To: <20261119101523-mutt-send-email-mst@kernel.org>
On 19.11.26 16:15, Michael S. Tsirkin wrote:
> On Mon, Nov 09, 2026 at 08:21:28PM +0100, Vladimir Petrzhikovskii wrote:
>> create_user_mr() builds one indirect MKey that covers the whole vhost
>> IOTLB. It fills each gap between two mappings with null KLMs, one per
>> GiB, and never checks the total number of KLMs against the device
>> limit, MLX5_CAP_GEN(log_max_klm_list_size).
>>
>> With a virtio-IOMMU the guest chooses the IOVAs: the VMM (QEMU)
>> forwards each map the guest makes to the vhost-vDPA device. If the
>> guest maps one page at IOVA 0 and one at IOVA 2^58, the driver needs
>> more than 2^28 KLMs. create_indirect_key() computes the buffer size in
>> an int, which overflows: the driver allocates 336 bytes, and
>> fill_indir() then writes 16 bytes per KLM, 4 GiB in total, far past
>> the end of the buffer.
>>
>> nnuls is an int, so a gap of 2^61 bytes or more is miscounted on every
>> architecture, and on 32-bit the unsigned long num_klms wraps. The gap
>> is rounded up to whole GiB as (size + 2^30 - 1) >> 30. For a gap
>> larger than 2^64 - 2^30, for example below a page mapped in the top
>> GiB, that addition wraps and the driver counts no null KLMs for the
>> gap, while fill_indir() still writes one for every GiB of it.
>>
Please make this paragraph more human friendly. Enough to say that
the type doesn't fit the value or something like that.
>> Count KLMs in u64 and round the gap up without the overflowing
>> addition. Before creating any direct key, return -E2BIG with a
>> rate-limited warning if the MR needs more KLMs than the device limit,
>> or more than fit in a CREATE_MKEY command whose length is an int.
>> Allocate the command with kvzalloc(), because a list within the device
>> limit can be too large for kzalloc(). A normal guest is unaffected:
>> memory at 0-3 GiB and 4-68 GiB needs 68 KLMs.
>>
No need to say what the code does. Focus on the why.
>> Found by an LLM code review agent auditing vhost-vDPA backends. We have
>> no mlx5 hardware, so the fix is untested on it; it builds with W=1 on
>> x86_64 and i386.
>>
>> Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration code")
>> Cc: stable@vger.kernel.org
>> Co-developed-by: Nikolai Kalinin <nikolai@kalinidze.me>
>> Signed-off-by: Nikolai Kalinin <nikolai@kalinidze.me>
>> Signed-off-by: Vladimir Petrzhikovskii <vladimir@dfb.sh>
>> ---
>> Dragos, one question on the limit: the patch takes
>> MLX5_CAP_GEN(log_max_klm_list_size) as the largest KLM list an indirect
>> CREATE_MKEY accepts, as mlx5_ib does for fast-reg lists. If firmware
>> applies a different limit to CREATE_MKEY, please say so and I will
>> change the bound.
>>
>> The guest kernel change used for the reachability check is available
>> on request.
>
Yes The limit is right.
>
> Dragos, care to answer?
>
Yes. Done.
>> ---
>> drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 +-
>> drivers/vdpa/mlx5/core/mr.c | 30 +++++++++++++++++++++++++-----
>> 2 files changed, 26 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> index 42f2f44b383c..1d84824ca429 100644
>> --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
>> @@ -32,7 +32,7 @@ struct mlx5_vdpa_mr {
>> /* list of direct MRs descendants of this indirect mr */
>> struct list_head head;
>> unsigned long num_directs;
>> - unsigned long num_klms;
>> + u64 num_klms;
>>
>> struct vhost_iotlb *iotlb;
>>
>> diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
>> index b0c5ff23d022..03db8345c1bb 100644
>> --- a/drivers/vdpa/mlx5/core/mr.c
>> +++ b/drivers/vdpa/mlx5/core/mr.c
>> @@ -206,6 +206,16 @@ static int klm_byte_size(int nklms)
>> return 16 * ALIGN(nklms, 4);
>> }
>>
>> +/* Largest KLM count whose CREATE_MKEY input length still fits in an int. */
Comment not needed.
>> +#define MLX5_VDPA_MAX_KLMS_INLEN \
>> + ALIGN_DOWN((INT_MAX - MLX5_ST_SZ_BYTES(create_mkey_in)) / 16, 4)
>> +
>> +static u64 max_klms(struct mlx5_vdpa_dev *mvdev)
>> +{
>> + return min_t(u64, BIT_ULL(MLX5_CAP_GEN(mvdev->mdev, log_max_klm_list_size)),
>> + MLX5_VDPA_MAX_KLMS_INLEN);
>> +}
>> +
>> #define MLX5_VDPA_MTT_ALIGN 16
>>
>> static int create_direct_keys(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr)
>> @@ -338,7 +348,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>> return -EINVAL;
>>
>> inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + klm_byte_size(mr->num_klms);
>> - in = kzalloc(inlen, GFP_KERNEL);
>> + in = kvzalloc(inlen, GFP_KERNEL);
Unrelated change for fix. Drop please.
>> if (!in)
>> return -ENOMEM;
>>
>> @@ -355,7 +365,7 @@ static int create_indirect_key(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr
>> MLX5_SET(create_mkey_in, in, translations_octword_actual_size, mr->num_klms);
>> fill_indir(mvdev, mr, in);
>> err = mlx5_vdpa_create_mkey(mvdev, &mr->mkey, in, inlen);
>> - kfree(in);
>> + kvfree(in);
>> return err;
>> }
>>
>> @@ -501,6 +511,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>> {
>> struct mlx5_vdpa_direct_mr *dmr;
>> struct mlx5_vdpa_direct_mr *n;
>> + u64 klm_limit = max_klms(mvdev);
>> struct vhost_iotlb_map *map;
>> u32 pperm = U16_MAX;
>> u64 last = U64_MAX;
>> @@ -508,7 +519,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>> u64 pe = U64_MAX;
>> u64 start = 0;
>> int err = 0;
>> - int nnuls;
>> + u64 nnuls;
>>
>> INIT_LIST_HEAD(&mr->head);
>> for (map = vhost_iotlb_itree_first(iotlb, start, last); map;
>> @@ -521,9 +532,10 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>> if (pe < map->start) {
>> /* We have a hole in the map. Check how
>> * many null keys are required to fill it.
>> + * The hole can be nearly 2^64 bytes, so
>> + * round up without adding to its size.
>> */
>> - nnuls = MLX5_DIV_ROUND_UP_POW2(map->start - pe,
>> - LOG_MAX_KLM_SIZE);
>> + nnuls = ((map->start - pe - 1) >> LOG_MAX_KLM_SIZE) + 1;
Why not DIV_ROUND_UP_POW2(map->start - pe, MAX_KLM_SIZE)?
>> mr->num_klms += nnuls;
>> }
>> err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb);
>> @@ -539,6 +551,14 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev,
>> if (err)
>> goto err_chain;
>>
>> + if (mr->num_klms > klm_limit) {
>> + dev_warn_ratelimited(mvdev->mdev->device,
>> + "map needs %llu KLMs, limit is %llu\n",
>> + mr->num_klms, klm_limit);
Use mlx5_vdpa_warn(). Rate limiting here won't help: the other prints
will trigger.
Thanks,
Dragos
next prev parent reply other threads:[~2026-11-19 18:13 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-09 19:21 Vladimir Petrzhikovskii
2026-11-19 15:15 ` Michael S. Tsirkin
2026-11-19 18:12 ` Dragos Tatulea [this message]
2026-11-20 13:20 ` Vladimir Petrzhikovskii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d4f67696-5095-4948-9029-f657058431b0@nvidia.com \
--to=dtatulea@nvidia.com \
--cc=eperezma@redhat.com \
--cc=jasowangio@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mst@redhat.com \
--cc=nikolai@kalinidze.me \
--cc=stable@vger.kernel.org \
--cc=virtualization@lists.linux.dev \
--cc=vladimir@dfb.sh \
--cc=xuanzhuo@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®