From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F626EB64DD for ; Fri, 11 Aug 2023 17:26:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236101AbjHKR02 (ORCPT ); Fri, 11 Aug 2023 13:26:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234215AbjHKR01 (ORCPT ); Fri, 11 Aug 2023 13:26:27 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D059030C1 for ; Fri, 11 Aug 2023 10:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691774736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X6sQ8Q6WelCDCKZ8Z8ftpDHYvOx3deYiCFU+E3Ltf9Y=; b=cqK9XUGuRGxftkPmSYSWt794Re7Z7lbP/5CgJ4BA2KkYI4h7hDCoQ01InY/7I4vs5S9rjl WxKw7XM0q1G5CkGyR1Fmskj3tOVieQLKcB9m6qcCigJJPk2r2b0VVfolFyuxAcXDLH75Ar PGCFeQsO4kG1EOTBevEe3hlfoHVRhk4= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-284-KIcQXnVbOZiHRT-HPWkHdQ-1; Fri, 11 Aug 2023 13:25:33 -0400 X-MC-Unique: KIcQXnVbOZiHRT-HPWkHdQ-1 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-30e4943ca7fso1227185f8f.3 for ; Fri, 11 Aug 2023 10:25:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691774732; x=1692379532; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=X6sQ8Q6WelCDCKZ8Z8ftpDHYvOx3deYiCFU+E3Ltf9Y=; b=Iko0rGcEQBL2bRxYhJ8RK2dminqXFTprz3zRWxokoyn6qwFm9dP5CbaE2mQCCKKvqF XWD8tPrc3+NV3luIbsMXWEW5ZeM1DqcfwBSJ4yGGxH2WOEVGg3pAkUH4boRAC7rLoNR3 g8+UtzQtqnw3ZX0n8m9mgilVXxl6XmXZYdhSSVmJqAvFHNSAug3ukK8ubOCRKWRqp/hG 6Z/9dKF5uBe1tZgA6m/gtqq4ls6sqib+y8VbI4ZKpEdCXSxuLzfYnOxAjKXFVn3DW99h xBR4SFKvQKLvdu1own2UHyTVrAQ+PeT4KAZfI4rRzsHS0Xe4G5Uw7Klp8NlnpN5HXzRC b3xw== X-Gm-Message-State: AOJu0Yy8Nt26xYfo/V6rkXC1r2Wp4HXwHVBVNyPseVrfP6y+oaGsF+nW /XEih6PhAk1ft0nUhVPdQ89AhqojkxHyc/XofkygXsadMgBaNPwjQ98S6WPEZDaxkuW9h8XCnU/ aHuCLsPRRn3ipXVsNEVFUGYtG X-Received: by 2002:adf:f511:0:b0:317:6b92:26b5 with SMTP id q17-20020adff511000000b003176b9226b5mr1820010wro.23.1691774732465; Fri, 11 Aug 2023 10:25:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IESQhR+T4R/2hhEkOkTbdXkRMH8nFN557GFq/6P+WCs1OfxoSnCzpP/VC0zaopv5X94Y+Tueg== X-Received: by 2002:adf:f511:0:b0:317:6b92:26b5 with SMTP id q17-20020adff511000000b003176b9226b5mr1820002wro.23.1691774732111; Fri, 11 Aug 2023 10:25:32 -0700 (PDT) Received: from ?IPV6:2003:cb:c71a:3000:973c:c367:3012:8b20? (p200300cbc71a3000973cc36730128b20.dip0.t-ipconnect.de. [2003:cb:c71a:3000:973c:c367:3012:8b20]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c378d00b003fe2de3f94fsm5803925wmr.12.2023.08.11.10.25.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 11 Aug 2023 10:25:31 -0700 (PDT) Message-ID: <6b48a161-257b-a02b-c483-87c04b655635@redhat.com> Date: Fri, 11 Aug 2023 19:25:30 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in a VM Content-Language: en-US To: Yan Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, pbonzini@redhat.com, seanjc@google.com, mike.kravetz@oracle.com, apopple@nvidia.com, jgg@nvidia.com, rppt@kernel.org, akpm@linux-foundation.org, kevin.tian@intel.com, John Hubbard References: <20230810085636.25914-1-yan.y.zhao@intel.com> <41a893e1-f2e7-23f4-cad2-d5c353a336a3@redhat.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10.08.23 11:50, Yan Zhao wrote: > On Thu, Aug 10, 2023 at 11:34:07AM +0200, David Hildenbrand wrote: >>> This series first introduces a new flag MMU_NOTIFIER_RANGE_NUMA in patch 1 >>> to work with mmu notifier event type MMU_NOTIFY_PROTECTION_VMA, so that >>> the subscriber (e.g.KVM) of the mmu notifier can know that an invalidation >>> event is sent for NUMA migration purpose in specific. >>> >>> Patch 2 skips setting PROT_NONE to long-term pinned pages in the primary >>> MMU to avoid NUMA protection introduced page faults and restoration of old >>> huge PMDs/PTEs in primary MMU. >>> >>> Patch 3 introduces a new mmu notifier callback .numa_protect(), which >>> will be called in patch 4 when a page is ensured to be PROT_NONE protected. >>> >>> Then in patch 5, KVM can recognize a .invalidate_range_start() notification >>> is for NUMA balancing specific and do not do the page unmap in secondary >>> MMU until .numa_protect() comes. >>> >> >> Why do we need all that, when we should simply not be applying PROT_NONE to >> pinned pages? >> >> In change_pte_range() we already have: >> >> if (is_cow_mapping(vma->vm_flags) && >> page_count(page) != 1) >> >> Which includes both, shared and pinned pages. > Ah, right, currently in my side, I don't see any pinned pages are > outside of this condition. > But I have a question regarding to is_cow_mapping(vma->vm_flags), do we > need to allow pinned pages in !is_cow_mapping(vma->vm_flags)? One issue is that folio_maybe_pinned...() ... is unreliable as soon as your page is mapped more than 1024 times. One might argue that we also want to exclude pages that are mapped that often. That might possibly work. > >> Staring at page #2, are we still missing something similar for THPs? > Yes. > >> Why is that MMU notifier thingy and touching KVM code required? > Because NUMA balancing code will firstly send .invalidate_range_start() with > event type MMU_NOTIFY_PROTECTION_VMA to KVM in change_pmd_range() > unconditionally, before it goes down into change_pte_range() and > change_huge_pmd() to check each page count and apply PROT_NONE. Ah, okay I see, thanks. That's indeed unfortunate. > > Then current KVM will unmap all notified pages from secondary MMU > in .invalidate_range_start(), which could include pages that finally not > set to PROT_NONE in primary MMU. > > For VMs with pass-through devices, though all guest pages are pinned, > KVM still periodically unmap pages in response to the > .invalidate_range_start() notification from auto NUMA balancing, which > is a waste. Should we want to disable NUMA hinting for such VMAs instead (for example, by QEMU/hypervisor) that knows that any NUMA hinting activity on these ranges would be a complete waste of time? I recall that John H. once mentioned that there are similar issues with GPU memory: NUMA hinting is actually counter-productive and they end up disabling it. -- Cheers, David / dhildenb