From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1594FC4167D for ; Thu, 9 Nov 2023 06:36:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232543AbjKIGgS (ORCPT ); Thu, 9 Nov 2023 01:36:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230300AbjKIGgR (ORCPT ); Thu, 9 Nov 2023 01:36:17 -0500 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [IPv6:2001:41d0:1004:224b::aa]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C765426B0 for ; Wed, 8 Nov 2023 22:36:14 -0800 (PST) Message-ID: <8d78d1d8-85e7-41b6-9ba0-f18c2cb94427@linux.dev> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1699511772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dQeTuG/d1jTRxBIJaHmdqmi6OeHpWS7JPZsZS9fBp/4=; b=aa8BWohLbAhw02n9QYFHUOC2+sWcbckgNiTM55rtVutnZ6wDkp2y3YVJKFSONtJhGgQSXs pPsCJkYRFLaLxdSaDIo4eRqOEXYdOqI1BdzJ35IRIx6baOz6ltiALON/BlVwZkZ/9nmycc Ggx+y0ZifeeDEreqv3S4Y30W13cZWdQ= Date: Thu, 9 Nov 2023 14:36:02 +0800 MIME-Version: 1.0 Subject: Re: [PATCH RFC V2 0/6] rxe_map_mr_sg() fix cleanup and refactor To: "Zhijian Li (Fujitsu)" , "zyjzyj2000@gmail.com" , "jgg@ziepe.ca" , "leon@kernel.org" , "linux-rdma@vger.kernel.org" Cc: "linux-kernel@vger.kernel.org" , "rpearsonhpe@gmail.com" , "Daisuke Matsuda (Fujitsu)" , "bvanassche@acm.org" , "yi.zhang@redhat.com" References: <20231103095549.490744-1-lizhijian@fujitsu.com> <037148c3-c15b-4859-9b82-8349fcb54d0a@fujitsu.com> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Zhu Yanjun In-Reply-To: <037148c3-c15b-4859-9b82-8349fcb54d0a@fujitsu.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 在 2023/11/9 10:24, Zhijian Li (Fujitsu) 写道: > > On 06/11/2023 21:58, Zhu Yanjun wrote: >> 在 2023/11/6 12:07, Zhijian Li (Fujitsu) 写道: >>> >>> On 03/11/2023 21:00, Zhu Yanjun wrote: >>>> 在 2023/11/3 17:55, Li Zhijian 写道: >>>>> I don't collect the Reviewed-by to the patch1-2 this time, since i >>>>> think we can make it better. >>>>> >>>>> Patch1-2: Fix kernel panic[1] and benifit to make srp work again. >>>>>             Almost nothing change from V1. >>>>> Patch3-5: cleanups # newly add >>>>> Patch6: make RXE support PAGE_SIZE aligned mr # newly add, but not fully tested >>>>> >>>>> My bad arm64 mechine offten hangs when doing blktests even though i use the >>>>> default siw driver. >>>>> >>>>> - nvme and ULPs(rtrs, iser) always registers 4K mr still don't supported yet. >>>> Zhijian >>>> >>>> Please read carefully the whole discussion about this problem. You will find a lot of valuable suggestions, especially suggestions from Jason. >>> Okay, i will read it again. If you can tell me which thread, that would be better. >>> >>> >>>>   From the whole discussion, it seems that the root cause is very clear. >>>> We need to fix this prolem. Please do not send this kind of commits again. >>>> >>> Let's think about what's our goal first. >>> >>> - 1) Fix the panic[1] and only support PAGE_SIZE MR >>> - 2) support PAGE_SIZE aligned MR >>> - 3) support any page_size MR. >>> >>> I'm sorry i'm not familiar with the linux MM subsystem. It seem it's safe/correct to access >>> address/memory across pages start from the return of kmap_loca_page(page). >>> In other words, 2) is already native supported, right? >> Yes. Please read the comments from Jason, Leon and Bart. They shared a lot of good advice. > I read the whole discussion again, but I believed i still missed a lost. > > >> From them, we can know the root cause and how to fix this problem. > I don't think i misunderstood the root cause: > RXE splits memory into PAGE_SIZE units in the xarray. As a result, when we extract an address from the xarray, > we should not access address beyond a PAGE_SIZE window. This is a complicated problem and it is deeply involved with memory management. A guy who is very familiar with linux MM is to provide a better solution to this problem. I expect a whole perfect solution to this problem. Zhu Yanjun > > IIUC, then how to fix it? > - I'm not going to "removing page_size set", it's out of this patch scope. > Feel free to do the cleanup separately. > - I'm not going to fix the NVMe/rtrs etc problems in this patch set when 64K page is enabled. > But RXE will tell its callers explicitly "RXE don't don't support such page_size" > - I didn't state RXE supports PAGE_SIZE aligned page_size MR before refactoring rxe_map_mr_sg(), > because I worry about it was not correct to access address beyond the PAGE_SIZE window. > > What I should do next? > Just state "RXE support PAGE_SIZE aligned MR" ? Then patches become > RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE aligned MR > RDMA/rxe: set RXE_PAGE_SIZE_CAP to starting from PAGE_SIZE > > Or just keep we have done in the V1 > > Thanks > > >> Good Luck. >> >> Zhu Yanjun >> >>> I get totally confused now. >>> >>> >>> >>>> Zhu Yanjun >>>> >>>>> [1] https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@mail.gmail.com/T/ >>>>> >>>>> Li Zhijian (6): >>>>>     RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr >>>>>     RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE >>>>>     RDMA/rxe: remove unused rxe_mr.page_shift >>>>>     RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from >>>>>       page_list >>>>>     RDMA/rxe: cleanup rxe_mr.{page_size,page_shift} >>>>>     RDMA/rxe: Support PAGE_SIZE aligned MR >>>>> >>>>>    drivers/infiniband/sw/rxe/rxe_mr.c    | 80 ++++++++++++++++----------- >>>>>    drivers/infiniband/sw/rxe/rxe_param.h |  2 +- >>>>>    drivers/infiniband/sw/rxe/rxe_verbs.h |  9 --- >>>>>    3 files changed, 48 insertions(+), 43 deletions(-) >>>>>