From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF29A3B38B9 for ; Thu, 14 May 2026 09:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778749808; cv=none; b=V6ICd0uPxEB5dLJPJ7rnZ7Pf9v0e08ldKytUEyZFwPhELqgWKiyrX2lkZN5d7Em5z7PdDmcXmbb9pcFSDZnnL07x3FgWZSDt7HDbpmx0lepq4PoUvkQSaAKDrYJZB05YWBQ3yJSGKcAmLtPbvfqMEi6KvjbR/yuZExE810MoBzg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778749808; c=relaxed/simple; bh=C3Lb5cvFAMNlyCIVl8ipWcR3gjPB2vdBwhDFk2jz4H0=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=SXl8Mc83TPpN8HaK1Emz7oKSLYp79lto0RhcGNTelQvmzq3+aBDpeEcAeITFCkJQBOlh9rcG7YgC0xes3Eq+rh7/s3zKBi/dEDsT5pD6P24yu6DZNMBM4FUgMyKd18wbcGXiAGFUC9e83HDA8bzJNwsrRD6hAxXBMb9PVb1j3sw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=NUaaeNab; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="NUaaeNab" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-488b0e1b870so128513015e9.2 for ; Thu, 14 May 2026 02:10:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1778749804; x=1779354604; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=/zJBoYCXvN5LO7DBit1Ku7VesIuEtdWFZWxNt6CCQH0=; b=NUaaeNab7oLY0hdE+In7wT4Mp52ctILg34IevAQUIqU+F5Bn5Kp2Y5M8YFecG+dywY OHQuIdMqUpv3MTgXjrCcTxBwgl92Z4G7XKUFkRxUFrdhsahSDqixjTDjbDboOOwy0Fzu 0/Ld+P7GUDmBnQGRO6L0x3yNE8oBcqB/iJ9+c+EWt5E3a1ROGvmyYLekT3YTbw7kt8d6 S1H4gQdfZiA17GOBdeVJYRyBVYam19M3PBXSDwzBc11fPdBQPbsHd+ytklHMoalPGEQo gdgOoBZt7FCpoAVjhfKrgu1/nwolVDgoExTNfgxcgkBA4m81UIcyuzXkSHVo2gFvW9mV faSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778749804; x=1779354604; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/zJBoYCXvN5LO7DBit1Ku7VesIuEtdWFZWxNt6CCQH0=; b=TkYNytuwnjmx2HhEr10EOra9ph+1ViSZlxkZyP2TTQHNz2nKyyKIpCGw7hg50wzeU7 w6ik0a5Pbn4AfKoJGF0vohsinR55f2oiucBbbxayw240X+R2GNctZYU35CZmGC7pt47h NRvCl8g1Ldb0c5jkiFkqsfknIw/0hKRu9mynQ5zqsNIRoAu4mFUEWQ1yobwFjSMwcBmy j+w3T1auSXpyLe+1WLpxPGgnxtX34s7Wto8l7KDeRSwtoKcSTnFUGpKwoc/zjqStscsA Vjudo+8PSII8Bzv5j647Ny9F9QASdjQz1xW9IhU9PBplGWEiMCu+g/6IHa8r0MwW1sXJ RxXQ== X-Forwarded-Encrypted: i=1; AFNElJ/y9cJyNLunL5rgA881iFEROA7yLlHOiDkWCkbSf6sX5+NK9dV6owsjghFZHWXykusx1YAoNR+kBtkWr/8=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2AB/NRbOaahHuHVFfHF9WpkL3u5o3EPcmUIrdI7dXRPOUMDvj YX659MfX+hAAnimRQTgIt2z2HkcP4vuH9R6Tsx6HnB0E0BlmaumdvxHYsZoVuDh/Oag= X-Gm-Gg: Acq92OHkG1nweRSOrvgGkdASbq1u7Rwne9+8VN91jR0o+TVAx1O2cBb0gpZ36/dSzz1 nf9iexN/wJVFOQByB/OecJcEejx/bqQ4GA9hCpsDupRgSG8xrN9zM4ecVmkUDhKl8uWEyboxooB L1eNdIUj4fv0qwelf0xHaFD0Zr47doT/p07G2mtlQAUgMTzqpzTnWUc3UYf2ZHANe8y9rLVe/bH Z7e2BlliJUs0c1XQP0coCNUjCJTXm7qsnZ5ad7BbE7PZ60L0n2MLxFZ/jtle/NuSDbXnFCh0ZLN uoq93I4qMtKVeYvEnmYxggAC8QTeHkr/fxxgb3CVUIIlmuK2qNnVx0TMHTru96kPWqH3U3cwK+m /kzOMqS19YkXRX0xFTrNd0TSHCwZv/iB9ff6Tftje5/J/qGZVAEE4l7neQ/RDJbF2YTx7Y8uO4l GfrmxO0qnQilh9EPG//B6WpuRecEez X-Received: by 2002:a05:600c:19d1:b0:48e:8499:4be0 with SMTP id 5b1f17b1804b1-48fdc714b8amr23783895e9.15.1778749804059; Thu, 14 May 2026 02:10:04 -0700 (PDT) Received: from [192.168.1.3] ([185.48.77.170]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-48fdb2788eesm34097965e9.7.2026.05.14.02.10.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 14 May 2026 02:10:03 -0700 (PDT) Message-ID: <66797cb2-18f7-4782-9370-68d0c10c35f4@linaro.org> Date: Thu, 14 May 2026 10:10:02 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7 13/20] KVM: arm64: Apply dynamic guest counter reservations To: Colton Lewis Cc: alexandru.elisei@arm.com, pbonzini@redhat.com, corbet@lwn.net, linux@armlinux.org.uk, catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, mizhang@google.com, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, mark.rutland@arm.com, shuah@kernel.org, gankulkarni@os.amperecomputing.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, kvm@vger.kernel.org References: Content-Language: en-US From: James Clark In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 13/05/2026 5:45 pm, Colton Lewis wrote: > James Clark writes: > >> On 04/05/2026 10:18 pm, Colton Lewis wrote: >>> Apply dynamic guest counter reservations by checking if the requested >>> guest mask collides with any events the host has scheduled and calling >>> pmu_perf_resched_update() with a hook that updates the mask of >>> available counters in between schedule out and schedule in. > >>> Signed-off-by: Colton Lewis >>> --- >>>    arch/arm64/kvm/pmu-direct.c  | 69 ++++++++++++++++++++++++++++++++ >>> ++++ >>>    include/linux/perf/arm_pmu.h |  1 + >>>    2 files changed, 70 insertions(+) > >>> diff --git a/arch/arm64/kvm/pmu-direct.c b/arch/arm64/kvm/pmu-direct.c >>> index 2252d3b905db9..14cc419dbafad 100644 >>> --- a/arch/arm64/kvm/pmu-direct.c >>> +++ b/arch/arm64/kvm/pmu-direct.c >>> @@ -100,6 +100,73 @@ u8 kvm_pmu_hpmn(struct kvm_vcpu *vcpu) >>>        return *host_data_ptr(nr_event_counters); >>>    } > >>> +/* Callback to update counter mask between perf scheduling */ >>> +static void kvm_pmu_update_mask(struct pmu *pmu, void *data) >>> +{ >>> +    struct arm_pmu *arm_pmu = to_arm_pmu(pmu); >>> +    unsigned long *new_mask = data; >>> + >>> +    bitmap_copy(arm_pmu->cntr_mask, new_mask, ARMPMU_MAX_HWEVENTS); >>> +} >>> + >>> +/** >>> + * kvm_pmu_set_guest_counters() - Handle dynamic counter reservations >>> + * @cpu_pmu: struct arm_pmu to potentially modify >>> + * @guest_mask: new guest mask for the pmu >>> + * >>> + * Check if guest counters will interfere with current host events and >>> + * call into perf_pmu_resched_update if a reschedule is required. >>> + */ >>> +static void kvm_pmu_set_guest_counters(struct arm_pmu *cpu_pmu, u64 >>> guest_mask) >>> +{ >>> +    struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events); >>> +    DECLARE_BITMAP(guest_bitmap, ARMPMU_MAX_HWEVENTS); >>> +    DECLARE_BITMAP(new_mask, ARMPMU_MAX_HWEVENTS); >>> +    bool need_resched = false; >>> + >>> +    bitmap_from_arr64(guest_bitmap, &guest_mask, ARMPMU_MAX_HWEVENTS); >>> +    bitmap_copy(new_mask, cpu_pmu->hw_cntr_mask, ARMPMU_MAX_HWEVENTS); >>> + >>> +    if (guest_mask) { >>> +        /* Subtract guest counters from available host mask */ >>> +        bitmap_andnot(new_mask, new_mask, guest_bitmap, >>> ARMPMU_MAX_HWEVENTS); >>> + >>> +        /* Did we collide with an active host event? */ >>> +        if (bitmap_intersects(cpuc->used_mask, guest_bitmap, >>> ARMPMU_MAX_HWEVENTS)) { >>> +            int idx; >>> + >>> +            need_resched = true; >>> +            cpuc->host_squeezed = true; >>> + >>> +            /* Look for pinned events that are about to be preempted */ >>> +            for_each_set_bit(idx, guest_bitmap, ARMPMU_MAX_HWEVENTS) { >>> +                if (test_bit(idx, cpuc->used_mask) && cpuc- >>> >events[idx] && >>> +                    cpuc->events[idx]->attr.pinned) { >>> +                    pr_warn_ratelimited("perf: Pinned host event >>> squeezed out by KVM guest PMU partition\n"); > >> Hi Colton, > >> I get "perf: Pinned host event squeezed out by KVM guest PMU partition" >> even with arm_pmuv3.reserved_host_counters=3 for example. I would have >> expected any non zero value to stop the warning. > >> I think armv8pmu_get_single_idx() needs to be changed to allocate from >> the high end host counters first. A more complicated option would be >> checking to see if there are any non-pinned counters in the host >> reserved half when a new pinned counter is opened, then swapping the >> places of the new pinned and existing non-pinned counters so pinned >> always prefer being put into the host half. But it's probably not worth >> doing that. > >> James > > > I agree it makes the most sense to allocate from the top, but I'm happy > the basic idea works. > Another thing I forgot to mention is that even with the ratelimited warning, this spams the logs any time the host and guest are both using the PMU and I'm not sure how useful that is. >>> +                    break; >>> +                } >>> +            } >>> +        } >>> +    } else { >>> +        /* >>> +         * Restoring to hw_cntr_mask. >>> +         * Only resched if we previously squeezed an event. >>> +         */ >>> +        if (cpuc->host_squeezed) { >>> +            need_resched = true; >>> +            cpuc->host_squeezed = false; >>> +        } >>> +    } >>> + >>> +    if (need_resched) { >>> +        /* Collision: run full perf reschedule */ >>> +        perf_pmu_resched_update(&cpu_pmu->pmu, kvm_pmu_update_mask, >>> new_mask); >>> +    } else { >>> +        /* Host was never using guest counters anyway */ >>> +        bitmap_copy(cpu_pmu->cntr_mask, new_mask, ARMPMU_MAX_HWEVENTS); >>> +    } >>> +} >>> + >>>    /** >>>     * kvm_pmu_host_counter_mask() - Compute bitmask of host-reserved >>> counters >>>     * @pmu: Pointer to arm_pmu struct >>> @@ -218,6 +285,7 @@ void kvm_pmu_load(struct kvm_vcpu *vcpu) > >>>        pmu = vcpu->kvm->arch.arm_pmu; >>>        guest_counters = kvm_pmu_guest_counter_mask(pmu); >>> +    kvm_pmu_set_guest_counters(pmu, guest_counters); >>>        kvm_pmu_apply_event_filter(vcpu); > >>>        for_each_set_bit(i, &guest_counters, ARMPMU_MAX_HWEVENTS) { >>> @@ -319,5 +387,6 @@ void kvm_pmu_put(struct kvm_vcpu *vcpu) >>>        val = read_sysreg(pmintenset_el1); >>>        __vcpu_assign_sys_reg(vcpu, PMINTENSET_EL1, val & mask); > >>> +    kvm_pmu_set_guest_counters(pmu, 0); >>>        preempt_enable(); >>>    } >>> diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h >>> index f7b000bb3eca8..63f88fec5e80f 100644 >>> --- a/include/linux/perf/arm_pmu.h >>> +++ b/include/linux/perf/arm_pmu.h >>> @@ -75,6 +75,7 @@ struct pmu_hw_events { > >>>        /* Active events requesting branch records */ >>>        unsigned int        branch_users; >>> +    bool host_squeezed; >>>    }; > >>>    enum armpmu_attr_groups {