From: Auger Eric <eric.auger@redhat.com>
To: Marc Zyngier <marc.zyngier@arm.com>,
linux-arm-kernel@lists.infradead.org,
kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: Mark Rutland <mark.rutland@arm.com>,
Andre Przywara <Andre.Przywara@arm.com>,
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
Christoffer Dall <christoffer.dall@linaro.org>,
Shanker Donthineni <shankerd@codeaurora.org>
Subject: Re: [PATCH v5 26/26] KVM: arm/arm64: GICv4: Theory of operations
Date: Wed, 8 Nov 2017 10:13:16 +0100 [thread overview]
Message-ID: <98265bff-57ca-a31e-f41c-75b41a4e34f7@redhat.com> (raw)
In-Reply-To: <20171027142855.21584-27-marc.zyngier@arm.com>
Hi Marc,
On 27/10/2017 16:28, Marc Zyngier wrote:
> Yet another braindump so I can free some cells...
>
> Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
> virt/kvm/arm/vgic/vgic-v4.c | 67 +++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 67 insertions(+)
>
> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
> index d10e18eabd3b..e367d65a0ebe 100644
> --- a/virt/kvm/arm/vgic/vgic-v4.c
> +++ b/virt/kvm/arm/vgic/vgic-v4.c
> @@ -23,6 +23,73 @@
>
> #include "vgic.h"
>
> +/*
> + * How KVM uses GICv4 (insert rude comments here):
> + *
> + * The vgic-v4 layer acts as a bridge between several entities:
> + * - The GICv4 ITS representation offered by the ITS driver
> + * - VFIO, which is in charge of the PCI endpoint
> + * - The virtual ITS, which is the only thing the guest sees
> + *
> + * The configuration of VLPIs is triggered by a callback from VFIO,
> + * instructing KVM that a PCI device has been configured to deliver
> + * MSIs to a vITS.
We actually have a negotiation protocol between VFIO PCI (irq bypass
producer) and KVM irqfd (IRQ bypass consumer). When both recognize they
are tied together, handling an MSI tunneling, they initiate the
forwarding setup.
> + *
> + * kvm_vgic_v4_set_forwarding() is thus called with the routing entry,
> + * and this is used to find the corresponding vITS data structures
> + * (ITS instance, device, event and irq) using a process that is
> + * extremely similar to the injection of an MSI.
Is it correct to say we replace the following injection chain:
pEventID|
(pITS) |-> pLPIID -> VFIO PCI IRQ handler -> KVM irqfd ...
pDevID |
vEventID|
... inject (vITS) | -> vLPIID
vDevID |
by
pEventID|
(pITS) | -> vLPIID
pDevID |
Thanks
Eric
> + *
> + * At this stage, we can link the guest's view of an LPI (uniquely
> + * identified by the routing entry) and the host irq, using the GICv4
> + * driver mapping operation. Should the mapping succeed, we've then
> + * successfully upgraded the guest's LPI to a VLPI. We can then start
> + * with updating GICv4's view of the property table and generating an
> + * INValidation in order to kickstart the delivery of this VLPI to the
> + * guest directly, without software intervention. Well, almost.
> + *
> + * When the PCI endpoint is deconfigured, this operation is reversed
> + * with VFIO calling kvm_vgic_v4_unset_forwarding().
> + *
> + * Once the VLPI has been mapped, it needs to follow any change the
> + * guest performs on its LPI through the vITS. For that, a number of
> + * command handlers have hooks to communicate these changes to the HW:
> + * - Any invalidation triggers a call to its_prop_update_vlpi()
> + * - The INT command results in a irq_set_irqchip_state(), which
> + * generates an INT on the corresponding VLPI.
> + * - The CLEAR command results in a irq_set_irqchip_state(), which
> + * generates an CLEAR on the corresponding VLPI.
> + * - DISCARD translates into an unmap, similar to a call to
> + * kvm_vgic_v4_unset_forwarding().
> + * - MOVI is translated by an update of the existing mapping, changing
> + * the target vcpu, resulting in a VMOVI being generated.
> + * - MOVALL is translated by a string of mapping updates (similar to
> + * the handling of MOVI). MOVALL is horrible.
> + *
> + * Note that a DISCARD/MAPTI sequence emitted from the guest without
> + * reprogramming the PCI endpoint after MAPTI does not result in a
> + * VLPI being mapped, as there is no callback from VFIO (the guest
> + * will get the interrupt via the normal SW injection). Fixing this is
> + * not trivial, and requires some horrible messing with the VFIO
> + * internals. Not fun. Don't do that.
> + *
> + * Then there is the scheduling. Each time a vcpu is about to run on a
> + * physical CPU, KVM must tell the corresponding redistributor about
> + * it. And if we've migrated our vcpu from one CPU to another, we must
> + * tell the ITS (so that the messages reach the right redistributor).
> + * This is done in two steps: first issue a irq_set_affinity() on the
> + * irq corresponding to the vcpu, then call its_schedule_vpe(). You
> + * must be in a non-preemptible context. On exit, another call to
> + * its_schedule_vpe() tells the redistributor that we're done with the
> + * vcpu.
> + *
> + * Finally, the doorbell handling: Each vcpu is allocated an interrupt
> + * which will fire each time a VLPI is made pending whilst the vcpu is
> + * not running. Each time the vcpu gets blocked, the doorbell
> + * interrupt gets enabled. When the vcpu is unblocked (for whatever
> + * reason), the doorbell interrupt is disabled.
> + */
> +
> #define DB_IRQ_FLAGS (IRQ_NOAUTOEN | IRQ_DISABLE_UNLAZY | IRQ_NO_BALANCING)
>
> static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
>
next prev parent reply other threads:[~2017-11-08 9:13 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-27 14:28 [PATCH v5 00/26] KVM/ARM: Add support for GICv4 Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 01/26] irqchip/gic-v3-its: Setup VLPI properties at map time Marc Zyngier
2017-10-30 6:46 ` Christoffer Dall
2017-10-27 14:28 ` [PATCH v5 02/26] KVM: arm/arm64: register irq bypass consumer on ARM/ARM64 Marc Zyngier
2017-10-30 6:47 ` Christoffer Dall
2017-10-27 14:28 ` [PATCH v5 03/26] KVM: arm/arm64: vgic: restructure kvm_vgic_(un)map_phys_irq Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 04/26] KVM: arm: Select ARM_GIC_V3 and ARM_GIC_V3_ITS Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 05/26] KVM: arm/arm64: vgic: Move kvm_vgic_destroy call around Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 06/26] KVM: arm/arm64: vITS: Add MSI translation helpers Marc Zyngier
2017-11-07 20:34 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 07/26] KVM: arm/arm64: vITS: Add a helper to update the affinity of an LPI Marc Zyngier
2017-11-07 13:44 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 08/26] KVM: arm/arm64: GICv4: Add property field and per-VM predicate Marc Zyngier
2017-11-07 20:30 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 09/26] KVM: arm/arm64: GICv4: Add init/teardown of the per-VM vPE irq domain Marc Zyngier
2017-11-07 13:08 ` Auger Eric
2017-11-10 8:20 ` Christoffer Dall
2017-11-10 8:55 ` Marc Zyngier
2017-11-07 13:09 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 10/26] KVM: arm/arm64: GICv4: Wire mapping/unmapping of VLPIs in VFIO irq bypass Marc Zyngier
2017-11-07 13:06 ` Auger Eric
2017-11-07 14:42 ` Marc Zyngier
2017-11-07 15:59 ` Auger Eric
2017-11-08 11:30 ` Marc Zyngier
2017-11-10 8:28 ` Christoffer Dall
2017-11-10 9:05 ` Marc Zyngier
2017-11-10 9:41 ` Christoffer Dall
2017-10-27 14:28 ` [PATCH v5 11/26] KVM: arm/arm64: GICv4: Handle INT command applied to a VLPI Marc Zyngier
2017-11-07 20:15 ` Auger Eric
2017-11-08 11:40 ` Marc Zyngier
2017-11-08 14:14 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 12/26] KVM: arm/arm64: GICv4: Unmap VLPI when freeing an LPI Marc Zyngier
2017-11-07 20:28 ` Auger Eric
2017-11-08 11:52 ` Marc Zyngier
2017-11-08 14:14 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 13/26] KVM: arm/arm64: GICv4: Propagate affinity changes to the physical ITS Marc Zyngier
2017-11-07 21:01 ` Auger Eric
2017-11-08 12:05 ` Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 14/26] KVM: arm/arm64: GICv4: Handle CLEAR applied to a VLPI Marc Zyngier
2017-11-07 21:04 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 15/26] KVM: arm/arm64: GICv4: Handle MOVALL applied to a vPE Marc Zyngier
2017-11-07 21:06 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 16/26] KVM: arm/arm64: GICv4: Propagate property updates to VLPIs Marc Zyngier
2017-11-07 21:28 ` Auger Eric
2017-11-08 15:08 ` Marc Zyngier
2017-11-10 8:37 ` Christoffer Dall
2017-11-10 8:58 ` Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 17/26] KVM: arm/arm64: GICv4: Handle INVALL applied to a vPE Marc Zyngier
2017-11-07 21:23 ` Auger Eric
2017-11-10 8:41 ` Christoffer Dall
2017-11-10 8:56 ` Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 18/26] KVM: arm/arm64: GICv4: Use pending_last as a scheduling hint Marc Zyngier
2017-11-07 21:38 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 19/26] KVM: arm/arm64: GICv4: Add doorbell interrupt handling Marc Zyngier
2017-11-07 21:43 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 20/26] KVM: arm/arm64: GICv4: Use the doorbell interrupt as an unblocking source Marc Zyngier
2017-11-07 21:45 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 21/26] KVM: arm/arm64: GICv4: Hook vPE scheduling into vgic flush/sync Marc Zyngier
2017-11-07 21:54 ` Auger Eric
2017-11-07 22:14 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 22/26] KVM: arm/arm64: GICv4: Enable virtual cpuif if VLPIs can be delivered Marc Zyngier
2017-11-08 8:46 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 23/26] KVM: arm/arm64: GICv4: Prevent a VM using GICv4 from being saved Marc Zyngier
2017-11-07 15:24 ` Auger Eric
2017-11-07 15:38 ` Marc Zyngier
2017-11-07 16:12 ` Auger Eric
2017-11-07 16:34 ` Marc Zyngier
2017-11-07 22:24 ` Auger Eric
2017-11-08 9:35 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 24/26] KVM: arm/arm64: GICv4: Prevent userspace from changing doorbell affinity Marc Zyngier
2017-10-30 6:51 ` Christoffer Dall
2017-11-07 22:17 ` Auger Eric
2017-10-27 14:28 ` [PATCH v5 25/26] KVM: arm/arm64: GICv4: Enable VLPI support Marc Zyngier
2017-11-08 8:44 ` Auger Eric
2017-11-08 15:14 ` Marc Zyngier
2017-10-27 14:28 ` [PATCH v5 26/26] KVM: arm/arm64: GICv4: Theory of operations Marc Zyngier
2017-11-08 9:13 ` Auger Eric [this message]
2017-11-08 15:19 ` Marc Zyngier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98265bff-57ca-a31e-f41c-75b41a4e34f7@redhat.com \
--to=eric.auger@redhat.com \
--cc=Andre.Przywara@arm.com \
--cc=christoffer.dall@linaro.org \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=marc.zyngier@arm.com \
--cc=mark.rutland@arm.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=shankerd@codeaurora.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®