From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18027C54EE9 for ; Wed, 7 Sep 2022 19:15:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229771AbiIGTPD (ORCPT ); Wed, 7 Sep 2022 15:15:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229472AbiIGTPA (ORCPT ); Wed, 7 Sep 2022 15:15:00 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 66CD129CAF for ; Wed, 7 Sep 2022 12:14:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662578098; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F2hl65/5BHZV+h8ft7hHxLgHBBHzGCSnWLOyF+9Pr+8=; b=U3NwU6PSfplSlk43GsylcPwMz//4YVhFZoPxSiVvV1Y4hhtYDayBkFieYq0bwk6P0SmEtw nXYy8dkRSeLi9CQ0clqq3+7LzCFjgrFJ/1LWIn1Xal9ooLfwjWD82phZdRS62BmJ8+Hbwi TcTkFPQqolDfYK5D2P/Doaro4WTQZjQ= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-533-cWG5nOZEOFCZq60BCbbSdg-1; Wed, 07 Sep 2022 15:14:57 -0400 X-MC-Unique: cWG5nOZEOFCZq60BCbbSdg-1 Received: by mail-wr1-f69.google.com with SMTP id j9-20020a5d4649000000b00226d830857cso3981261wrs.12 for ; Wed, 07 Sep 2022 12:14:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date; bh=F2hl65/5BHZV+h8ft7hHxLgHBBHzGCSnWLOyF+9Pr+8=; b=Iv8+vvZ5+/7IYK58BsLXUObxPjFUSKbexUFhzcAd54mDiIQR/5CM0DB3iiio2LL66Y PmWMrrGv37SIwi/cr12IoFcu6C3bM0vhSod5Z/bFg3piDPP7D93QyIgfW/Zlk0KAtL7V rqWHw4X6eq5TSoOIPyzHPcZfBAN9bwuxncTC6M/k8OvB6t9EbSv2v9dKw0jKHhP8Snqy 3QPlzLdG7NJ31HZ+v+Bz6skAMhxBtmLYJlC8anND4VdJYNiSHe38Jl+YezbPrhgUVvjY hCNAH1m3BS91Xi/BGI9muFm8Qdcp4pwn2e54TV/tl4hBygObeXlrvcGSN2xcmW0q8kB7 aM5Q== X-Gm-Message-State: ACgBeo1D9kHEvyC+QeYs31KFmnwPNL4SeWmJEd58nux00TgtFzV0ci/Q E3BVB6qkvqPwTA7uYmjHmhnndOMBGJFShagVfsBNxgEGg9BWGYqC4Ia/OFH1zauM638biaK4qZj T8pgMyv5BkQrcPRXj3XpIXNco X-Received: by 2002:a05:6000:cf:b0:228:e37b:361b with SMTP id q15-20020a05600000cf00b00228e37b361bmr2599742wrx.374.1662578096102; Wed, 07 Sep 2022 12:14:56 -0700 (PDT) X-Google-Smtp-Source: AA6agR5Mej1JLF8Idc5qNLyIzkfO5zK2VWFm2c7LZq5/eJiJrZnnv9hvkXiOQaSwnEjKeK//Gm5oog== X-Received: by 2002:a05:6000:cf:b0:228:e37b:361b with SMTP id q15-20020a05600000cf00b00228e37b361bmr2599728wrx.374.1662578095776; Wed, 07 Sep 2022 12:14:55 -0700 (PDT) Received: from ?IPV6:2003:d8:2f0d:ba00:c951:31d7:b2b0:8ba0? (p200300d82f0dba00c95131d7b2b08ba0.dip0.t-ipconnect.de. [2003:d8:2f0d:ba00:c951:31d7:b2b0:8ba0]) by smtp.gmail.com with ESMTPSA id m18-20020adff392000000b00228b3ff1f5dsm13553530wro.117.2022.09.07.12.14.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 07 Sep 2022 12:14:54 -0700 (PDT) Message-ID: <7ed97088-b4f0-e36f-c935-87cd1e94c574@redhat.com> Date: Wed, 7 Sep 2022 21:14:54 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.0 Subject: Re: [PATCH] vfio/type1: Unpin zero pages Content-Language: en-US To: Alex Williamson , Jason Gunthorpe Cc: "Tian, Kevin" , "kvm@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "lpivarc@redhat.com" , "Liu, Jingqi" , "Lu, Baolu" References: <166182871735.3518559.8884121293045337358.stgit@omen> <20220907095552.336c8f34.alex.williamson@redhat.com> <20220907125627.0579e592.alex.williamson@redhat.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <20220907125627.0579e592.alex.williamson@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07.09.22 20:56, Alex Williamson wrote: > On Wed, 7 Sep 2022 13:40:52 -0300 > Jason Gunthorpe wrote: > >> On Wed, Sep 07, 2022 at 09:55:52AM -0600, Alex Williamson wrote: >> >>>> So, if we go back far enough in the git history we will find a case >>>> where PUP is returning something for the zero page, and that something >>>> caused is_invalid_reserved_pfn() == false since VFIO did work at some >>>> point. >>> >>> Can we assume that? It takes a while for a refcount leak on the zero >>> page to cause an overflow. My assumption is that it's never worked, we >>> pin zero pages, don't account them against the locked memory limits >>> because our is_invalid_reserved_pfn() test returns true, and therefore >>> we don't unpin them. >> >> Oh, you think it has been buggy forever? That is not great.. >> >>>> IHMO we should simply go back to the historical behavior - make >>>> is_invalid_reserved_pfn() check for the zero_pfn and return >>>> false. Meaning that PUP returned it. >>> >>> We've never explicitly tested for zero_pfn and as David notes, >>> accounting the zero page against the user's locked memory limits has >>> user visible consequences. VMs that worked with a specific locked >>> memory limit may no longer work. >> >> Yes, but the question is if that is a strict ABI we have to preserve, >> because if you take that view it also means because VFIO has this >> historical bug that David can't fix the FOLL_FORCE issue either. >> >> If the view holds for memlock then it should hold for cgroups >> also. This means the kernel can never change anything about >> GFP_KERNEL_ACCOUNT allocations because it might impact userspace >> having set a tight limit there. >> >> It means we can't fix the bug that VFIO is using the wrong storage for >> memlock. >> >> It means qemu can't change anything about how it sets up this memory, >> ie Kevin's idea to change the ordering. >> >> On the other hand the "abi break" we are talking about is that a user >> might have to increase a configured limit in a config file after a >> kernel upgrade. >> >> IDK what consensus exists here, I've never heard of anyone saying >> these limits are a strict ABI like this.. I think at least for cgroup >> that would be so invasive as to immediately be off the table. > > I thought we'd already agreed that we were stuck with locked_vm for > type1 and any compatibility mode of type1 due to this. Native iommufd > support can do the right thing since userspace will need to account for > various new usage models anyway. > > I've raised the issue with David for the zero page accounting, but I > don't know what the solution is. libvirt automatically adds a 1GB > fudge factor to the VM locked memory limits to account for things like > ROM mappings, or at least the non-zeropage backed portion of those > ROMs. I think that most management tools have adopted similar, so the > majority of users shouldn't notice. However this won't cover all > users, so we certainly risk breaking userspace if we introduce hard > page accounting of zero pages. > > I think David suggested possibly allowing some degree of exceeding > locked memory limits for zero page COWs. Potentially type1 could do > this as well; special case handling with some heuristically determined, > module parameter defined limit. We might also consider whether we > could just ignore zero page mappings, maybe with a optional "strict" > mode module option to generate an errno on such mappings. Thanks, So far I played with the ideas a) allow slightly exceeding the limit and warn b) weird vfio kernel parameter to control the zeropage behavior (old vs. new) I certainly have in mind that we might need some toggle for vfio. -- Thanks, David / dhildenb