From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=35.83.148.184 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794438399; b=ZusOVnS9k0BuBKgsLrK/kFp+RWpwMBxUiRMrjEy3zj5jKmDvnG9Fe6c97O8RwGMKelqm N6+VcA916UmU6m3ld+dDlraFUlJau6s1oGDcEvqub/X/KxXfiN7l2ozf9ZU7uQuuUdhJZ xxsIIVRSba+mWtZUjGjS2uS7iYiJQexBi8LrB1OV5lTpU8Jid9s9pZIXrzgAzNxjWhpgE 1tbyAEnfMOzuUPP+fYPjsh7OkdFLs2zK570HAlEVdR5uGbUHYISYYhrsDXX2kzPxq/bcr ei+kk9KS2ajIBq9pcfKizI4JKpYsghGi+M9ez50yADvAPuklW6o3EpIRE217MCxDmkw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794438399; h=DMARC-Filter:DKIM-Signature:X-CSE-ConnectionGUID:X-CSE-MsgGUID: X-IronPort-AV:Received:Received:X-Farcaster-Flow-ID:Received:Received: From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding:Content-Type:X-Originating-IP: X-ClientProxiedBy; bh=lMLQtoMK+FF9PMlISJsUmmPmSZAB3/0ZdR3/LW/bJ9I=; b=g+ehJY0dKBS3ZL707RBhEF+qZa4VE6McRTCVBYb6D34xNUrtQe7osO2JG89AIBv/7otu EFzi2OKeGr+iE2WDgzR/2brlMqV2CS/iqULvkEfKpTXpKf1WWwk9vlKYz2PwuMuWkpzw8 1OsTe3Oksm0VLs6TM1aiEIoRpfN2+vnQygBLPBYu0fu/1yS79LrEvCxYkuj1b1P34tHuM HMiw7ABZrcBMbRIN8Hp3btyL079IPJ/aIWN1Cx6m4Ajfn9dV9wV3ywXVKtrkgnRvkOO7O 2q14LRnA+/mdDVwZ7LqZXgR4dGszCUjbfHYaD7TRgRe18o1rkARFwuFybuY51buIrjg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=amazon.com; spf=pass smtp.mailfrom=amazon.de; dkim=pass header.d=amazon.com header.i=@amazon.com header.a=rsa-sha256 header.s=amazoncorp2 header.b=Jz0TnK1C; arc=none smtp.remote-ip=35.83.148.184 Received: from pdx-out-014.esa.us-west-2.outbound.mail-perimeter.amazon.com (pdx-out-014.esa.us-west-2.outbound.mail-perimeter.amazon.com [35.83.148.184]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3500C606A324; Wed, 11 Nov 2026 23:06:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 3500C606A324 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.a=rsa-sha256 header.s=amazoncorp2 header.b=Jz0TnK1C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1794438399; x=1825974399; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lMLQtoMK+FF9PMlISJsUmmPmSZAB3/0ZdR3/LW/bJ9I=; b=Jz0TnK1C0Q51eo60UsqVjcfXAPApyYvtZrrl8utk3mEax3Y1+zgyQZDM j7upeLysdKIIIR1NgsINyQeQ0zB9ecDp68R6kUk0zWzb3VHduIdAgUJvw dX62rCS7vExjBkcaOPFkZm5TZK+GcYSl7bBjKcCs/ktIfcDMblH4MSr6i rOpfJknF7yOEIwfEuSUaJpB4hW6S61tYr8rNeJpAldIOBpjzVjXnqJnfQ iBQy+UDp73N9mwcS84eP5/Q22qTUiQBJEWG9sKEM0vhyUSxxmqXb/i6iS 7liqWtY01APEH4yKh8q9hHc5lnGi4YgO+22symhb77oU8A33OhFYAu9A3 A==; X-CSE-ConnectionGUID: a0qphS5PSkW3vKQQb7o4Tw== X-CSE-MsgGUID: puhEAopkQb68Udz8yNQ+5w== X-IronPort-AV: E=Sophos;i="6.27,208,1787011200"; d="scan'208";a="33551130" Received: from ip-10-5-6-203.us-west-2.compute.internal (HELO smtpout.naws.us-west-2.prod.farcaster.email.amazon.dev) ([10.5.6.203]) by internal-pdx-out-014.esa.us-west-2.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2026 23:06:38 +0000 Received: from EX19MTAUWC001.ant.amazon.com [205.251.233.53:16609] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.34.255:2525] with esmtp (Farcaster) id 11ec9860-3c62-40c8-a31e-cc39e6372e83; Wed, 11 Nov 2026 23:06:38 +0000 (UTC) X-Farcaster-Flow-ID: 11ec9860-3c62-40c8-a31e-cc39e6372e83 Received: from EX19D001UWA001.ant.amazon.com (10.13.138.214) by EX19MTAUWC001.ant.amazon.com (10.250.64.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.56; Wed, 11 Nov 2026 23:06:38 +0000 Received: from ip-10-253-83-51.amazon.com (172.19.99.218) by EX19D001UWA001.ant.amazon.com (10.13.138.214) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.56; Wed, 11 Nov 2026 23:06:35 +0000 From: Alexander Graf To: , Greg Kroah-Hartman , Arnd Bergmann CC: The AWS Nitro Enclaves Team , "Shuah Khan" , Randy Dunlap , Marek Szyprowski , Robin Murphy , , , Andrew Morton , David Hildenbrand , Mike Rapoport , , , Jonathan Corbet Subject: [PATCH 27/40] nitro_enclaves: Dispatch multiplexed interrupts to per-queue bindings Date: Wed, 11 Nov 2026 22:59:31 +0000 Message-ID: <7cb23d5eb46abf8a1caa61b8887132837fe256c4.1794393490.git.graf@amazon.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: EX19D031UWC004.ant.amazon.com (10.13.139.246) To EX19D001UWA001.ant.amazon.com (10.13.138.214) Nitro Sandboxes provide a way for user space to receive per-device-virtqueue events. These are multiplexed through a nitro_enclaves device built-in interrupt multiplexer. Implement logic to support this multiplexer and distribute the respective interrupt events. Signed-off-by: Alexander Graf --- drivers/virt/nitro_enclaves/ne_pci_dev.c | 179 ++++++++++++++++++++++- drivers/virt/nitro_enclaves/ne_pci_dev.h | 47 ++++++ 2 files changed, 225 insertions(+), 1 deletion(-) diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c index 0fafd42162f3a..de96d68906bbf 100644 --- a/drivers/virt/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c @@ -7,6 +7,7 @@ * DOC: Nitro Enclaves (NE) PCI device driver. */ +#include #include #include #include @@ -27,6 +28,7 @@ #include #include #include +#include #include "ne_misc_dev.h" #include "ne_pci_dev.h" @@ -551,6 +553,129 @@ static void ne_teardown_msix(struct pci_dev *pdev) pci_free_irq_vectors(pdev); } +/** + * ne_mux_binding_rcu_free() - RCU callback freeing one enclave queue binding. + * @head: RCU head of the binding to free. + * + * Context: Softirq or process context. + */ +static void ne_mux_binding_rcu_free(struct rcu_head *head) +{ + struct ne_mux_binding *binding = container_of(head, + struct ne_mux_binding, + rcu); + + eventfd_ctx_put(binding->ctx); + + kfree(binding); +} + +void ne_mux_binding_free_rcu(struct ne_mux_binding *binding) +{ + call_rcu(&binding->rcu, ne_mux_binding_rcu_free); +} + +/** + * ne_mux_scan() - Deliver the bits set in the bitmap of a pooled vector that it + * has a binding for. + * @mux_vec: Multiplexer state of the pooled vector. + * + * A bit leaves the bitmap by the same atomic read-modify-write that decides to + * deliver it, so a bit this vector has no binding for stays set. It is left set + * on purpose: a queue can be kicked before user space has bound an eventfd to + * it, and a bit taken out of the bitmap on the strength of a binding that is not + * there yet is a kick nobody can ever be told about. The bit waits instead, and + * the bind of that queue takes it and delivers it. + * + * So a set bit is consumed at most once, and exactly once if a scan finds a + * binding for it or a bind for its index runs. A bit whose index is never bound + * is consumed never, and stays set until the page is reclaimed when the device + * unbinds. A bit whose bind is still coming looks the same to the driver, so it + * holds both. + * + * The bound only ever rises, so a scan reads every word a binding has reached + * into and not only the words the bindings of the moment need. + * + * Context: Interrupt context. Called with the RCU read-side held. + * Return: + * * true if the scan found no bit it could deliver, false if it found at least + * one. + */ +static bool ne_mux_scan(struct ne_mux_vec *mux_vec) +{ + struct ne_pci_dev *ne_pci_dev = mux_vec->ne_pci_dev; + u32 bits_per_word = BITS_PER_TYPE(*mux_vec->bitmap); + u32 nr_words = READ_ONCE(mux_vec->max_bit) / bits_per_word + 1; + bool empty = true; + u32 i = 0; + + for (i = 0; i < nr_words; i++) { + u64 word = READ_ONCE(mux_vec->bitmap[i]); + + while (word) { + u32 bit = i * bits_per_word + __ffs64(word); + unsigned long notify_index = bit * ne_pci_dev->mux_nr_vecs + + mux_vec->vec_rel; + struct ne_mux_binding *binding = NULL; + + word &= word - 1; + + binding = xa_load(&ne_pci_dev->notify_xa, notify_index); + if (!binding) + continue; + + empty = false; + + if (test_and_clear_bit(bit, (unsigned long *)mux_vec->bitmap)) + eventfd_signal(binding->ctx); + } + } + + return empty; +} + +/** + * ne_mux_handler() - Interrupt handler for one pooled MSI-X vector. + * @irq: Interrupt line the pooled vector is on. + * @data: Multiplexer state of the pooled vector. + * + * Context: Interrupt context. + * Return: + * * IRQ_HANDLED on interrupt handled. + */ +static irqreturn_t ne_mux_handler(int irq, void *data) +{ + struct ne_mux_vec *mux_vec = data; + u32 pass = 0; + + /* + * xa_load() drops its own RCU read-side before it returns, so take one + * here to keep a binding alive for as long as it is dereferenced. + */ + rcu_read_lock(); + + /* + * The device sets a bit and then raises the interrupt, and one interrupt + * can stand for several bits, so a scan can miss a bit set while it runs + * or one past the bound it read. The device raises for every bit it sets + * whatever the bitmap holds, so such a bit has an interrupt of its own + * still to come and reaches its eventfd on that one. Scanning again, + * reading the bound again, delivers it on this interrupt instead, so stop + * on the first scan that delivers nothing. Stopping on an empty bitmap + * instead would wait for a read a bit no bind is coming for never gives. + * NE_MUX_MAX_PASSES caps the scans either way, so a device that goes on + * setting bits cannot hold the handler. + */ + do { + if (ne_mux_scan(mux_vec)) + break; + } while (++pass < NE_MUX_MAX_PASSES); + + rcu_read_unlock(); + + return IRQ_HANDLED; +} + /** * ne_mux_teardown() - Take the interrupt multiplexer bitmaps back from the * device. @@ -561,6 +686,8 @@ static void ne_teardown_msix(struct pci_dev *pdev) static void ne_mux_teardown(struct pci_dev *pdev) { struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev); + struct ne_mux_binding *binding = NULL; + unsigned long notify_index = 0; u32 i = 0; if (!ne_pci_dev->mux) @@ -568,13 +695,40 @@ static void ne_mux_teardown(struct pci_dev *pdev) iowrite32(NE_MUX_SHUTDOWN_ON, ne_pci_dev->iomem_base + NE_MUX_SHUTDOWN); - for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++) + for (i = 0; i < ne_pci_dev->mux_nr_vecs; i++) { + if (!ne_pci_dev->mux[i].bitmap) + continue; + + if (ne_pci_dev->mux[i].irq_requested) + free_irq(pci_irq_vector(pdev, + ne_pci_dev->mux_vec_base + i), + &ne_pci_dev->mux[i]); + free_page((unsigned long)ne_pci_dev->mux[i].bitmap); + } kfree(ne_pci_dev->mux); ne_pci_dev->mux = NULL; ne_pci_dev->mux_nr_vecs = 0; + + xa_for_each(&ne_pci_dev->notify_xa, notify_index, binding) { + xa_erase(&ne_pci_dev->notify_xa, notify_index); + + eventfd_ctx_put(binding->ctx); + + kfree(binding); + } + + /* + * Freeing the handlers above quiesced the readers that hold no lock, + * and the loop above freed every binding that outlived them. Wait for + * the callbacks a replaced binding already queued, so that none of + * them runs once the module text is gone. + */ + rcu_barrier(); + + xa_destroy(&ne_pci_dev->notify_xa); } /** @@ -631,8 +785,11 @@ static int ne_mux_setup(struct pci_dev *pdev) ne_pci_dev->mux_nr_vecs = nr_vecs; ne_pci_dev->mux_vec_base = vec_base; + xa_init(&ne_pci_dev->notify_xa); + for (i = 0; i < nr_vecs; i++) { u16 donate_err = 0; + int irq = 0; ne_pci_dev->mux[i].bitmap = (u64 *)get_zeroed_page(GFP_KERNEL); if (!ne_pci_dev->mux[i].bitmap) { @@ -663,6 +820,26 @@ static int ne_mux_setup(struct pci_dev *pdev) goto teardown_mux; } + + ne_pci_dev->mux[i].ne_pci_dev = ne_pci_dev; + ne_pci_dev->mux[i].vec_rel = i; + + irq = pci_irq_vector(pdev, vec_base + i); + if (irq < 0) { + rc = irq; + + goto teardown_mux; + } + + rc = request_irq(irq, ne_mux_handler, 0, "enclave_mux", + &ne_pci_dev->mux[i]); + if (rc < 0) { + dev_err(&pdev->dev, "Error in request irq mux [rc=%d]\n", rc); + + goto teardown_mux; + } + + ne_pci_dev->mux[i].irq_requested = true; } return 0; diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h index 2cef40c627f9f..0dbce83cb06bd 100644 --- a/drivers/virt/nitro_enclaves/ne_pci_dev.h +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h @@ -9,12 +9,15 @@ #include #include #include +#include #include #include #include #include +#include #include #include +#include /** * DOC: Nitro Enclaves (NE) PCI device @@ -201,6 +204,12 @@ static_assert(NE_MUX_MAX_BITS / BITS_PER_TYPE(u64) * sizeof(u64) <= PAGE_SIZE, "the bitmap of a pooled vector has to fit its donated page"); +/** + * NE_MUX_MAX_PASSES - Largest number of times the handler of a pooled vector + * scans its bitmap in one interrupt. + */ +#define NE_MUX_MAX_PASSES (4) + /** * enum ne_pci_dev_cmd_type - Device command types. * @INVALID_CMD: Invalid command. @@ -648,9 +657,35 @@ struct ne_slot_pool_run { * @bitmap: Page-sized bitmap in which the device sets a bit * for every enclave queue that signalled through * this vector. + * @max_bit: Highest bitmap bit an enclave queue has been + * bound to, bounding the scan of @bitmap. It only + * rises, so a scan reads the words of bindings that + * are gone as well. + * @ne_pci_dev: NE PCI device this vector belongs to. + * @vec_rel: Index of this vector within the pool. + * @irq_requested: Whether the interrupt handler of this vector is + * installed, which is not implied by @bitmap: a + * vector whose bitmap the device already owns may + * have no handler yet. */ struct ne_mux_vec { u64 *bitmap; + u32 max_bit; + struct ne_pci_dev *ne_pci_dev; + u32 vec_rel; + bool irq_requested; +}; + +/** + * struct ne_mux_binding - Eventfd bound to one enclave queue. + * @ctx: Eventfd signalled when the queue is kicked. + * @rcu: Head used to free the binding after a grace + * period, so that an interrupt handler which + * already loaded it can still dereference it. + */ +struct ne_mux_binding { + struct eventfd_ctx *ctx; + struct rcu_head rcu; }; /** @@ -672,6 +707,8 @@ struct ne_mux_vec { * capability, and with it the multiplexer and slot * pool registers. * @iomem_base : MMIO region of the PCI device. + * @notify_xa: Enclave queue bindings, keyed by the notify index of + * the queue. * @mux: Per-pooled-vector interrupt multiplexer state. * @mux_nr_vecs: Number of MSI-X vectors in the interrupt * multiplexer pool. @@ -706,6 +743,7 @@ struct ne_pci_dev { struct workqueue_struct *event_wq; bool has_vendor_cap; void __iomem *iomem_base; + struct xarray notify_xa; struct ne_mux_vec *mux; u32 mux_nr_vecs; u32 mux_vec_base; @@ -750,6 +788,15 @@ int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type, struct ne_pci_dev_cmd_reply *cmd_reply, size_t cmd_reply_size); +/** + * ne_mux_binding_free_rcu() - Free an enclave queue binding after a grace + * period. + * @binding: Binding that has been removed from the queue bindings xarray. + * + * Context: Process context. + */ +void ne_mux_binding_free_rcu(struct ne_mux_binding *binding); + /* Nitro Enclaves (NE) PCI device driver */ extern struct pci_driver ne_pci_driver; -- 2.47.1