From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3CB8C4167B for ; Tue, 31 Oct 2023 13:19:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344490AbjJaNTj (ORCPT ); Tue, 31 Oct 2023 09:19:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344449AbjJaNTh (ORCPT ); Tue, 31 Oct 2023 09:19:37 -0400 Received: from mail-oa1-x2e.google.com (mail-oa1-x2e.google.com [IPv6:2001:4860:4864:20::2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE9D7E6 for ; Tue, 31 Oct 2023 06:19:31 -0700 (PDT) Received: by mail-oa1-x2e.google.com with SMTP id 586e51a60fabf-1e0ee4e777bso3714672fac.3 for ; Tue, 31 Oct 2023 06:19:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1698758371; x=1699363171; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=BtWJYaqzVCuAB1FYlsIzdtoi66Ke7zjmsu4PHOZOvs0=; b=IHbMVC6ElDtO7R2fV01G91I7ZOlm1MSIfM4+uEkHkvFEvLgAoen2hOtnOsV3mg8uKN xYR/22ygfO93Ir4T+qMle+MQ0V7SIR6Etdv4STyBXVlcRqHgTxlwa4wqM1yUi7UlmrGo iNkTEmR2QgKtsDEO/VdQZVbkyTvroF+wtI/V4TPUB1PKFGpsZAzyCKVYD0E2TucMox6v tUhXMe6WBUfCX5VyPo4/vdwHQkC0xVR/epOS7VbkLGoVu9mXZ9S0cJ/ySoiU6KNxDaFf a+Jlowhcf4ixBjzEXOgAR1xfAw86f8L9LbVQa8oyPj+sdykC9K45IlIW+UIzljZFiFOx Uk+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698758371; x=1699363171; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BtWJYaqzVCuAB1FYlsIzdtoi66Ke7zjmsu4PHOZOvs0=; b=YFXzeXHLukKGRBH2Sv6V3qXzK0BK6AwFnaNO60xObKYsIfHNcx4677nW8Iwkcg+Qli xV3phS7Wmw+LNiOUEFeK6IFut8S3+1J55qLqyZC8oNmJ5Gx5BW3XXIdjuNcU8MNab4bE Ik/Umox+sljdqjrOgqCIzpzzDzmyGsHXaPXuXr9nCbsgF5Yqo3TvR+c6QMyWsWDIS+nj 55wE1maNCeYn4cKJQcVR8vq7rzMIaD6wzGc8ycs4qCF41G5lesSI/eU/Ynqbfz8VS4ZY NHv9cEsDYfMGP1Xla76kqxAcoGafIONa/lpkwp7mhWyEbfOE9mgZ9riLEQEWWlkGdEuO T+0A== X-Gm-Message-State: AOJu0Yz/QWmQV//P7kM4FgbJh7UILxMEZKalYAQFmDhKHN6m5YQAqDhI BXbcJiN9tvHRcslwHIFZMVWlxg== X-Google-Smtp-Source: AGHT+IGobMzcpmUVEatZpnPLLSMhma3hu4rk+HEtm3dT5rrB5fQjHHicvBPXnBa6d9YXWfbq6fOjUQ== X-Received: by 2002:a05:6870:78b:b0:1d6:439d:d03e with SMTP id en11-20020a056870078b00b001d6439dd03emr15262386oab.18.1698758371098; Tue, 31 Oct 2023 06:19:31 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-26-201.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.26.201]) by smtp.gmail.com with ESMTPSA id ml19-20020a056214585300b00671b009412asm500861qvb.141.2023.10.31.06.19.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 06:19:29 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1qxoeW-007cYl-IU; Tue, 31 Oct 2023 10:19:28 -0300 Date: Tue, 31 Oct 2023 10:19:28 -0300 From: Jason Gunthorpe To: Zhu Yanjun Cc: "Zhijian Li (Fujitsu)" , "zyjzyj2000@gmail.com" , "leon@kernel.org" , "linux-rdma@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "rpearsonhpe@gmail.com" , "Daisuke Matsuda (Fujitsu)" , "bvanassche@acm.org" Subject: Re: [PATCH RFC 1/2] RDMA/rxe: don't allow registering !PAGE_SIZE mr Message-ID: <20231031131928.GH691768@ziepe.ca> References: <20231027054154.2935054-1-lizhijian@fujitsu.com> <4da48f85-a72f-4f6b-900f-fc293d63b5ae@fujitsu.com> <20231030124041.GE691768@ziepe.ca> <7c5dd149-395a-46a2-96a0-89c182105eaf@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <7c5dd149-395a-46a2-96a0-89c182105eaf@linux.dev> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Oct 31, 2023 at 04:52:23PM +0800, Zhu Yanjun wrote: > 在 2023/10/30 20:40, Jason Gunthorpe 写道: > > On Mon, Oct 30, 2023 at 07:51:41AM +0000, Zhijian Li (Fujitsu) wrote: > > > > > > > > > On 27/10/2023 13:41, Li Zhijian wrote: > > > > mr->page_list only encodes *page without page offset, when > > > > page_size != PAGE_SIZE, we cannot restore the address with a wrong > > > > page_offset. > > > > > > > > Note that this patch will break some ULPs that try to register 4K > > > > MR when PAGE_SIZE is not 4K. > > > > SRP and nvme over RXE is known to be impacted. > > > > > > > > Signed-off-by: Li Zhijian > > > > --- > > > > drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c > > > > index f54042e9aeb2..61a136ea1d91 100644 > > > > --- a/drivers/infiniband/sw/rxe/rxe_mr.c > > > > +++ b/drivers/infiniband/sw/rxe/rxe_mr.c > > > > @@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, > > > > struct rxe_mr *mr = to_rmr(ibmr); > > > > unsigned int page_size = mr_page_size(mr); > > > > + if (page_size != PAGE_SIZE) { > > > > > > It seems this condition is too strict, it should be: > > > if (!IS_ALIGNED(page_size, PAGE_SIZE)) > > > > > > So that, page_size with (N * PAGE_SIZE) can work as previously. > > > Because the offset(mr.iova & page_mask) will get lost only when !IS_ALIGNED(page_size, PAGE_SIZE) > > > > That makes sense > > I read all the discussions very carefully. > > Thanks, Greg. > > Because RXE only supports PAGE_SIZE, when CONFIG_ARM64_64K_PAGES is enabled, > the PAGE_SIZE is 64K, when CONFIG_ARM64_64K_PAGES is disabled, PAGE_SIZE is > 4K. > > But NVMe calls ib_map_mr_sg with a fixed size SZ_4K. When > CONFIG_ARM64_64K_PAGES is enabled, it is still 4K. This is not a problem in > RXE. This problem is in NVMe. Maybe, but no real RDMA devices don't support 4K. The xarray conversion may need revision to use physical addresses instead of storing struct pages so it can handle this kind of segmentation. Certainly in the mean time it should be rejected. Jason