* [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
@ 2026-08-06 3:10 Fengnan Chang
2026-08-10 20:55 ` Keith Busch
2026-08-12 17:50 ` Anuj Gupta
0 siblings, 2 replies; 9+ messages in thread
From: Fengnan Chang @ 2026-08-06 3:10 UTC (permalink / raw)
To: kbusch, axboe, hch, sagi, linux-nvme
Cc: linux-kernel, Fengnan Chang, Guzebing
The idea behind this approach is: Let each I/O queue switch itself between
interrupt and poll mode based on its own recent completion rate.
This version is still in the testing phase, and there are still some issues
with the code implementation. I releasing it now to see if the approach
is generally acceptable. If the approach looks good, I’ll continue to
refine it and conduct more extensive testing. The main implementation
logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
focus on reviewing the implementation of these two functions.
Compared to the previous version, this represents a much smaller
performance regression while offering greater benefits.
In high-IOPS scenarios, relying on interrupts to handle I/O operations
can limit performance. This issue becomes particularly pronounced in
multi-disk environments, where performance is constrained by the CPU's
interrupt-handling capacity.
Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
4 KiB random-read IOPS. Four of them should be good for about 12.8M IOPS,
but interrupt-driven completion tops out at 5.59M, only about 44% of that.
Polling gets rid of that cost, but polling every queue all the time burns
CPU and hurts the sparse or bursty queues that interrupts handle just fine.
So instead of a global switch, let each queue make the call on its own,
from how fast it has been completing lately, and re-check often enough that
the decision tracks the workload rather than a fixed tunable.
Each queue runs a small loop with three stages. First it samples its
completion rate while still on interrupts. Only if that rate is high
enough to fill a small batch inside a bounded latency window does it mask
its own IRQ and start draining the CQ from a high-resolution timer, with
each wait sized to collect roughly one batch. It keeps polling as long as
it keeps up with that rate; the moment it stalls or slows down it turns the
IRQ back on and backs off, waiting longer the further behind it fell.
A queue that doesn't benefit drops back quickly and only gets retried once
in a while, so polling stays on the queues that are actually
interrupt-bound and everything else keeps running on the untouched IRQ
path.
Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
versus off:
QD32 QD64 QD128
one device, one job +18.44% +24.35% +26.38%
four devices, eight jobs +83.76% +99.02% +96.26%
The four-device eight-job aggregate goes from 5.59M IOPS (44% of the 12.8M
ceiling) to 10.89M IOPS (85%). Tail latency improves too: QD64 p99 drops
from 1073 us to 498 us (-54%) and p99.9 from 1909 us to 741 us (-61%).
Interrupts per I/O drop from about 0.91 to 0.08 (~12x fewer) in the
four-device case, and from 0.96 to 0.03 (~33x fewer) on a single high-QD
queue.
Link:https://lore.kernel.org/linux-nvme/d9210bcdf73fbe1ac8b6ec132865609a3ed68688.ff265e95.1296.491e.89f9.8ae888a03346@bytedance.com/T/#mea881a7898c85b73992f568864001913cb456d59
Signed-off-by: Guzebing <guzebing@bytedance.com>
Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
drivers/nvme/host/Kconfig | 1 +
drivers/nvme/host/pci.c | 468 ++++++++++++++++++++++++++++++++++++--
2 files changed, 452 insertions(+), 17 deletions(-)
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9..22164b901da85 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -5,6 +5,7 @@ config NVME_CORE
config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
+ select IRQ_POLL
select NVME_CORE
help
The NVM Express driver is for solid state drives directly
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b537..18559dd16b752 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -10,9 +10,13 @@
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
#include <linux/dmi.h>
+#include <linux/hrtimer.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/irq_poll.h>
+#include <linux/jump_label.h>
+#include <linux/ktime.h>
#include <linux/kstrtox.h>
#include <linux/memremap.h>
#include <linux/mm.h>
@@ -82,6 +86,57 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static DEFINE_STATIC_KEY_FALSE(nvme_adaptive_irq_polling_key);
+static bool use_adaptive_irq_polling;
+
+static int nvme_adaptive_irq_polling_set(const char *val,
+ const struct kernel_param *kp)
+{
+ int ret = param_set_bool(val, kp);
+
+ if (ret)
+ return ret;
+ if (use_adaptive_irq_polling)
+ static_branch_enable(&nvme_adaptive_irq_polling_key);
+ else
+ static_branch_disable(&nvme_adaptive_irq_polling_key);
+ return 0;
+}
+
+static const struct kernel_param_ops nvme_adaptive_irq_polling_ops = {
+ .set = nvme_adaptive_irq_polling_set,
+ .get = param_get_bool,
+};
+
+module_param_cb(use_adaptive_irq_polling, &nvme_adaptive_irq_polling_ops,
+ &use_adaptive_irq_polling, 0644);
+MODULE_PARM_DESC(use_adaptive_irq_polling,
+ "enable adaptive polling on non-threaded MSI-X I/O queues");
+
+/*
+ * Adaptive IRQ polling flips a busy interrupt-driven queue over to a
+ * timer-based poll path and back, based only on how fast that queue is
+ * completing. Each queue goes through three stages:
+ *
+ * 1. Sample (still in IRQ mode): count completions over
+ * NVME_ADAPTIVE_SAMPLE_CQES interrupts and work out the average gap
+ * between them. If the queue is too slow to fill a batch within
+ * NVME_ADAPTIVE_MAX_DELAY_NS, or already fast enough to batch on its own,
+ * leave it alone on the normal IRQ path.
+ * 2. Poll: mask the queue's IRQ and drain the CQ from an hrtimer, arming
+ * each wait for NVME_ADAPTIVE_TARGET_BATCH gaps (but never less than 2 us
+ * or more than NVME_ADAPTIVE_MAX_DELAY_NS). Keep polling as long as the
+ * queue keeps up, up to NVME_ADAPTIVE_EPISODE_CQES completions.
+ * 3. Back off: once a queue falls behind, go back to IRQ mode and skip the
+ * next deficit * NVME_ADAPTIVE_BACKOFF_MULT completions before sampling
+ * it again. Queues that don't benefit get retried only now and then.
+ */
+#define NVME_ADAPTIVE_TARGET_BATCH 5U
+#define NVME_ADAPTIVE_SAMPLE_CQES 256U
+#define NVME_ADAPTIVE_MAX_DELAY_NS (10U * NSEC_PER_USEC)
+#define NVME_ADAPTIVE_EPISODE_CQES (32U * NVME_ADAPTIVE_SAMPLE_CQES)
+#define NVME_ADAPTIVE_BACKOFF_MULT 20U
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -358,6 +413,25 @@ static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl)
return container_of(ctrl, struct nvme_dev, ctrl);
}
+/*
+ * Per-queue adaptive polling state. This sits outside struct nvme_queue on
+ * purpose, so the completion path's layout doesn't change when the feature is
+ * built in but not used. @lock covers every field below; it's separate from
+ * the legacy polling lock because the two paths never touch the same queue at
+ * the same time.
+ */
+struct nvme_adaptive_poll {
+ struct hrtimer timer; /* fires the next poll drain */
+ struct irq_poll iopoll; /* softirq context for the drain */
+ struct nvme_queue *nvmeq;
+ spinlock_t lock;
+ u64 start_ns; /* when the current sample/episode started */
+ u64 retry_completions; /* IRQ completions to skip before sampling again */
+ u32 interval_ns; /* average gap between completions, last sample */
+ u32 completions; /* completions seen so far this sample/episode */
+ int irq;
+};
+
/*
* An NVM Express queue. Each device has at least two (one for admin
* commands and one for I/O commands).
@@ -367,7 +441,8 @@ struct nvme_queue {
struct nvme_descriptor_pools descriptor_pools;
spinlock_t sq_lock;
void *sq_cmds;
- /* only used for poll queues: */
+ struct nvme_adaptive_poll *adaptive;
+ /* Only used for poll queues. */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
struct nvme_completion *cqes;
dma_addr_t sq_dma_addr;
@@ -386,6 +461,8 @@ struct nvme_queue {
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_ADAPTIVE_POLLING 4
+#define NVMEQ_ADAPTIVE_STALE_IRQ 5
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1606,13 +1683,12 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
{
- bool found = false;
+ unsigned int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1696,7 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ found++;
}
if (found)
@@ -1627,6 +1704,258 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+/* Keep the normal completion loop branch-free. */
+static unsigned int nvme_poll_cq_bounded(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob,
+ unsigned int limit)
+{
+ unsigned int found = 0;
+
+ while (found < limit && nvme_cqe_pending(nvmeq)) {
+ dma_rmb();
+ nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
+ nvme_update_cq_head(nvmeq);
+ found++;
+ }
+ if (found)
+ nvme_ring_cq_doorbell(nvmeq);
+ return found;
+}
+
+static irqreturn_t nvme_irq_check(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+
+ if (nvme_cqe_pending(nvmeq))
+ return IRQ_WAKE_THREAD;
+ return IRQ_NONE;
+}
+
+static bool nvme_adaptive_enabled(struct nvme_queue *nvmeq)
+{
+ return READ_ONCE(use_adaptive_irq_polling) &&
+ test_bit(NVMEQ_ENABLED, &nvmeq->flags);
+}
+
+/*
+ * Stop polling and turn the queue's IRQ back on. @elapsed is how long the
+ * episode ran after it started falling behind, or 0 if it ended cleanly.
+ * The bigger @elapsed is, the more completions we missed, and the longer we
+ * wait before sampling this queue again, so a queue that polling doesn't
+ * help is left alone most of the time.
+ */
+static void nvme_adaptive_poll_end(struct nvme_queue *nvmeq, u64 elapsed)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u64 deficit = 0;
+
+ if (elapsed) {
+ deficit = div64_u64(elapsed - 1, adaptive->interval_ns) + 1;
+ deficit -= min_t(u64, deficit, adaptive->completions);
+ }
+ adaptive->retry_completions =
+ deficit > U64_MAX / NVME_ADAPTIVE_BACKOFF_MULT ? U64_MAX :
+ deficit * NVME_ADAPTIVE_BACKOFF_MULT;
+ adaptive->start_ns = 0;
+ adaptive->completions = 0;
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ enable_irq(adaptive->irq);
+}
+
+/*
+ * Set the poll timer to fire one batch from now: how long the sampled
+ * rate needs to produce NVME_ADAPTIVE_TARGET_BATCH completions.
+ */
+static void nvme_adaptive_arm(struct nvme_adaptive_poll *adaptive, u64 now)
+{
+ u64 delay = clamp_t(u64,
+ (u64)adaptive->interval_ns * NVME_ADAPTIVE_TARGET_BATCH,
+ 2U * NSEC_PER_USEC, NVME_ADAPTIVE_MAX_DELAY_NS);
+
+ hrtimer_start(&adaptive->timer,
+ ns_to_ktime(now + delay), HRTIMER_MODE_ABS_PINNED_HARD);
+}
+
+static enum hrtimer_restart nvme_adaptive_poll_timer(struct hrtimer *timer)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(timer,
+ struct nvme_adaptive_poll, timer);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+
+ if (test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))
+ irq_poll_sched(&adaptive->iopoll);
+ return HRTIMER_NORESTART;
+}
+
+/*
+ * The poll drain, run from softirq when the timer fires. Reap some CQEs, then
+ * pick one of three things: stop if we hit the episode cap, wait again if the
+ * queue is keeping up, or go back to IRQ mode if it went idle or slowed down.
+ */
+static int nvme_adaptive_irq_poll(struct irq_poll *iop, int budget)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(iop,
+ struct nvme_adaptive_poll, iopoll);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+ unsigned int completions, limit;
+ unsigned long flags;
+ bool on_schedule;
+ u64 elapsed, now;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (unlikely(!test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ completions = 0;
+ irq_poll_complete(iop);
+ goto out;
+ }
+
+ limit = min_t(unsigned int,
+ budget - !nvme_adaptive_enabled(nvmeq),
+ NVME_ADAPTIVE_EPISODE_CQES - adaptive->completions);
+ completions = nvme_poll_cq_bounded(nvmeq, &iob, limit);
+ adaptive->completions += completions;
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+
+ if (completions >= budget)
+ goto out;
+ irq_poll_complete(iop);
+
+ if (!nvme_adaptive_enabled(nvmeq)) {
+ nvme_adaptive_poll_end(nvmeq, 0);
+ goto out;
+ }
+
+ /*
+ * Only keep polling if the queue is still hitting the sampled rate.
+ * MAX_DELAY leaves room for one empty wait, so a queue that's still
+ * completing keeps polling; one that stalled or slowed down goes back
+ * to IRQ mode and backs off.
+ */
+ now = ktime_get_ns();
+ elapsed = now - adaptive->start_ns;
+ on_schedule = elapsed <= (u64)adaptive->completions *
+ adaptive->interval_ns + NVME_ADAPTIVE_MAX_DELAY_NS;
+ if (adaptive->completions >= NVME_ADAPTIVE_EPISODE_CQES)
+ nvme_adaptive_poll_end(nvmeq, on_schedule ? 0 : elapsed);
+ else if (on_schedule)
+ nvme_adaptive_arm(adaptive, now);
+ else
+ nvme_adaptive_poll_end(nvmeq, elapsed);
+out:
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ return completions;
+}
+
+/*
+ * Called from the IRQ handler after a reap that found something. If we're
+ * still in backoff, just count it down. Otherwise time how long
+ * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
+ * them. If that looks worth polling (see the filter below) mask the IRQ and
+ * switch to poll mode; if not, leave the queue on interrupts.
+ */
+static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
+ unsigned int completions)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned int sample;
+ unsigned long flags;
+ u64 delta, interval, now;
+
+ if (adaptive->retry_completions) {
+ if (adaptive->retry_completions != U64_MAX)
+ adaptive->retry_completions -= min_t(u64, completions,
+ adaptive->retry_completions);
+ return;
+ }
+ if (!adaptive->start_ns) {
+ adaptive->start_ns = ktime_get_ns();
+ return;
+ }
+ adaptive->completions += completions;
+ if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
+ return;
+
+ now = ktime_get_ns();
+ delta = now - adaptive->start_ns;
+ sample = adaptive->completions;
+ adaptive->start_ns = now;
+ adaptive->completions = 0;
+ if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
+ NVME_ADAPTIVE_MAX_DELAY_NS)
+ return;
+ /*
+ * Is this rate worth polling? The 100/99 factor trims 1% off the gap so
+ * a queue sitting right on the threshold isn't pulled in. Skip it if
+ * it's too slow to fill a batch within MAX_DELAY, and also skip it if
+ * it's already fast enough to batch by itself. The hardware's own
+ * coalescing already handles that case, so leave it on interrupts.
+ */
+ interval = div64_u64(delta * 100, sample * 99);
+ if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||
+ (delta > NSEC_PER_MSEC &&
+ interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
+ NVME_ADAPTIVE_TARGET_BATCH))
+ return;
+
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (nvme_adaptive_enabled(nvmeq)) {
+ adaptive->interval_ns = interval;
+ set_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ disable_irq_nosync(adaptive->irq);
+ nvme_adaptive_arm(adaptive, now);
+ }
+
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+}
+
+static irqreturn_t nvme_irq(int irq, void *data);
+
+/*
+ * IRQ handler for queues that may switch to adaptive polling. It reaps the CQ
+ * like the normal handler, then feeds the count to the sampler, which may flip
+ * the queue into poll mode. An empty CQ right after an IRQ we masked ourselves
+ * (NVMEQ_ADAPTIVE_STALE_IRQ) is still ours to ack.
+ */
+static noinline irqreturn_t nvme_irq_adaptive_enabled(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ unsigned int completions;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ completions = nvme_poll_cq(nvmeq, &iob);
+ if (!completions)
+ return test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ,
+ &nvmeq->flags) ? IRQ_HANDLED : IRQ_NONE;
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ nvme_adaptive_sample(nvmeq, completions);
+ return IRQ_HANDLED;
+}
+
+static __always_inline bool nvme_adaptive_armed(void)
+{
+ return static_branch_unlikely(&nvme_adaptive_irq_polling_key) &&
+ READ_ONCE(use_adaptive_irq_polling);
+}
+
+static irqreturn_t nvme_irq_adaptive(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ irqreturn_t ret;
+
+ if (nvme_adaptive_armed())
+ return nvme_irq_adaptive_enabled(irq, data);
+
+ ret = nvme_irq(irq, data);
+ if (ret == IRQ_NONE &&
+ test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags))
+ return IRQ_HANDLED;
+ return ret;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
@@ -1640,13 +1969,20 @@ static irqreturn_t nvme_irq(int irq, void *data)
return IRQ_NONE;
}
-static irqreturn_t nvme_irq_check(int irq, void *data)
+static unsigned long nvme_adaptive_lock(struct nvme_queue *nvmeq)
{
- struct nvme_queue *nvmeq = data;
+ unsigned long flags = 0;
- if (nvme_cqe_pending(nvmeq))
- return IRQ_WAKE_THREAD;
- return IRQ_NONE;
+ if (nvmeq->adaptive)
+ spin_lock_irqsave(&nvmeq->adaptive->lock, flags);
+ return flags;
+}
+
+static void nvme_adaptive_unlock(struct nvme_queue *nvmeq,
+ unsigned long flags)
+{
+ if (nvmeq->adaptive)
+ spin_unlock_irqrestore(&nvmeq->adaptive->lock, flags);
}
/*
@@ -1656,15 +1992,18 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+ unsigned long flags;
int irq;
WARN_ON_ONCE(test_bit(NVMEQ_POLLED, &nvmeq->flags));
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
+ flags = nvme_adaptive_lock(nvmeq);
spin_lock(&nvmeq->cq_poll_lock);
nvme_poll_cq(nvmeq, NULL);
spin_unlock(&nvmeq->cq_poll_lock);
+ nvme_adaptive_unlock(nvmeq, flags);
enable_irq(irq);
}
@@ -2017,8 +2356,7 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
if (!nvmeq->sq_cmds)
- return;
-
+ goto free_adaptive;
if (test_and_clear_bit(NVMEQ_SQ_CMB, &nvmeq->flags)) {
pci_free_p2pmem(to_pci_dev(nvmeq->dev->dev),
nvmeq->sq_cmds, SQ_SIZE(nvmeq));
@@ -2026,6 +2364,9 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, SQ_SIZE(nvmeq),
nvmeq->sq_cmds, nvmeq->sq_dma_addr);
}
+free_adaptive:
+ kfree(nvmeq->adaptive);
+ nvmeq->adaptive = NULL;
}
static void nvme_free_queues(struct nvme_dev *dev, int lowest)
@@ -2038,9 +2379,42 @@ static void nvme_free_queues(struct nvme_dev *dev, int lowest)
}
}
+static int nvme_adaptive_suspend(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+ int irq;
+
+ if (!adaptive || adaptive->irq < 0)
+ return -1;
+ irq = adaptive->irq;
+ synchronize_irq(irq);
+ irq_poll_disable(&adaptive->iopoll);
+ spin_lock_irqsave(&adaptive->lock, flags);
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ hrtimer_cancel(&adaptive->timer);
+ spin_lock_irqsave(&adaptive->lock, flags);
+ if (test_and_clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags)) {
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(irq);
+ }
+ spin_unlock_irqrestore(&adaptive->lock, flags);
+ return irq;
+}
+
+static void nvme_adaptive_suspend_done(struct nvme_queue *nvmeq, int irq)
+{
+ if (irq < 0)
+ return;
+ nvmeq->adaptive->irq = -1;
+ irq_poll_enable(&nvmeq->adaptive->iopoll);
+}
+
static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
{
struct nvme_queue *nvmeq = &dev->queues[qid];
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ int irq;
if (!test_and_clear_bit(NVMEQ_ENABLED, &nvmeq->flags))
return;
@@ -2051,8 +2425,11 @@ static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
nvmeq->dev->online_queues--;
if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q)
nvme_quiesce_admin_queue(&nvmeq->dev->ctrl);
- if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags))
- pci_free_irq(to_pci_dev(dev->dev), nvmeq->cq_vector, nvmeq);
+ if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags)) {
+ irq = nvme_adaptive_suspend(nvmeq);
+ pci_free_irq(pdev, nvmeq->cq_vector, nvmeq);
+ nvme_adaptive_suspend_done(nvmeq, irq);
+ }
}
static void nvme_suspend_io_queues(struct nvme_dev *dev)
@@ -2166,18 +2543,74 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
return -ENOMEM;
}
+static bool nvme_adaptive_init(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ int irq = pci_irq_vector(to_pci_dev(nvmeq->dev->dev),
+ nvmeq->cq_vector);
+
+ if (irq < 0)
+ return false;
+ if (!adaptive) {
+ adaptive = kzalloc_node(sizeof(*adaptive), GFP_KERNEL,
+ dev_to_node(nvmeq->dev->dev));
+ if (!adaptive)
+ return false;
+ adaptive->nvmeq = nvmeq;
+ spin_lock_init(&adaptive->lock);
+ hrtimer_setup(&adaptive->timer, nvme_adaptive_poll_timer,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
+ irq_poll_init(&adaptive->iopoll, 64, nvme_adaptive_irq_poll);
+ adaptive->irq = irq;
+ WRITE_ONCE(nvmeq->adaptive, adaptive);
+ return true;
+ }
+ adaptive->irq = irq;
+ return true;
+}
+
static int queue_request_irq(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
+ bool adaptive_queue;
+ int ret;
if (use_threaded_interrupts) {
return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
+ /*
+ * Decide once, at setup, whether this queue can ever poll adaptively.
+ * nvme_irq_adaptive() is only installed when the adaptive state is
+ * allocated and armed here, so its fast path never has to re-check
+ * these static queue properties on every completion IRQ.
+ */
+ adaptive_queue = nvmeq->qid && nvmeq->dev->num_vecs > 1 &&
+ pdev->msix_enabled &&
+ nvmeq->q_depth >= NVME_ADAPTIVE_TARGET_BATCH;
+ if (adaptive_queue)
+ adaptive_queue = nvme_adaptive_init(nvmeq);
+ ret = pci_request_irq(pdev, nvmeq->cq_vector,
+ adaptive_queue ? nvme_irq_adaptive : nvme_irq,
+ NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+ if (ret && nvmeq->adaptive)
+ nvmeq->adaptive->irq = -1;
+ return ret;
+}
+
+static void nvme_adaptive_reset(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ if (!adaptive)
+ return;
+ adaptive->start_ns = 0;
+ adaptive->completions = 0;
+ adaptive->retry_completions = 0;
+ adaptive->irq = -1;
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2188,6 +2621,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvme_adaptive_reset(nvmeq);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
--
2.39.5 (Apple Git-154)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-06 3:10 [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling Fengnan Chang
@ 2026-08-10 20:55 ` Keith Busch
2026-08-11 2:32 ` changfengnan
2026-08-12 17:50 ` Anuj Gupta
1 sibling, 1 reply; 9+ messages in thread
From: Keith Busch @ 2026-08-10 20:55 UTC (permalink / raw)
To: Fengnan Chang; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> The idea behind this approach is: Let each I/O queue switch itself between
> interrupt and poll mode based on its own recent completion rate.
>
> This version is still in the testing phase, and there are still some issues
> with the code implementation. I releasing it now to see if the approach
> is generally acceptable. If the approach looks good, I´ll continue to
> refine it and conduct more extensive testing. The main implementation
> logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
> focus on reviewing the implementation of these two functions.
Can we subscribe to the dynamic interrupt moderation (dim) library? I
know it's generally used in conjuction with a hardware interrupt
coalescing feature, but we can just do pure software with it too. The
library provides a hill climb to adapt the policy at run time.
This is a quick PoC I put together. I haven't tested on fast devices, so
I'm not sure if I've dialed in the profiles, but it's start of what I
had in mind.
---
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9..9fad1c7b1f678 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -6,6 +6,7 @@ config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
select NVME_CORE
+ select DIMLIB
help
The NVM Express driver is for solid state drives directly
connected to the PCI or PCI Express bus. If you know you
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 8438c904ec496..ff0c8d74fae08 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -9,6 +9,8 @@
#include <linux/blkdev.h>
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
+#include <linux/delay.h>
+#include <linux/dim.h>
#include <linux/dmi.h>
#include <linux/init.h>
#include <linux/interrupt.h>
@@ -82,6 +84,45 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static unsigned int irq_poll_thresh = 7;
+module_param(irq_poll_thresh, uint, 0644);
+MODULE_PARM_DESC(irq_poll_thresh,
+ "outstanding depth after which a queue switches to threaded polling");
+
+static unsigned int irq_poll_spin = 7;
+module_param(irq_poll_spin, uint, 0644);
+MODULE_PARM_DESC(irq_poll_spin,
+ "poll iterations to spin (cpu_relax) before sleeping in the poll loop (zero based)");
+
+static unsigned int irq_poll_sleep_us = 20;
+static unsigned int irq_poll_idle_us = 80;
+static unsigned int irq_poll_idle_rounds = DIV_ROUND_UP(80, 20);
+
+static int irq_poll_us_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (ret)
+ return ret;
+
+ irq_poll_idle_rounds = irq_poll_sleep_us ?
+ DIV_ROUND_UP(irq_poll_idle_us, irq_poll_sleep_us) : 1;
+ return 0;
+}
+
+static const struct kernel_param_ops irq_poll_us_ops = {
+ .set = irq_poll_us_set,
+ .get = param_get_uint,
+};
+
+module_param_cb(irq_poll_sleep_us, &irq_poll_us_ops, &irq_poll_sleep_us, 0644);
+MODULE_PARM_DESC(irq_poll_sleep_us,
+ "microseconds to sleep between poll bursts (<=10 busy-delays, does not yield)");
+
+module_param_cb(irq_poll_idle_us, &irq_poll_us_ops, &irq_poll_idle_us, 0644);
+MODULE_PARM_DESC(irq_poll_idle_us,
+ "microseconds to wait for stragglers before handing a queue back to the IRQ path");
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -381,11 +422,21 @@ struct nvme_queue {
u16 qid;
u8 cq_phase;
u8 sqes;
+ /* Adaptive polling knobs, seeded from the irq_poll_* module params. */
+ unsigned int poll_thresh;
+ unsigned int poll_sleep_us;
+ unsigned int poll_idle_rounds;
+ unsigned int poll_budget;
+ /* Adaptive interrupt moderation (DIM) sampling state. */
+ struct dim dim;
+ u16 dim_events; /* cumulative interrupts (BIT_GAP-safe) */
+ u32 dim_comps; /* cumulative completions */
unsigned long flags;
#define NVMEQ_ENABLED 0
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_POLLING 4
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1606,13 +1657,13 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline int __nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob, int budget)
{
- bool found = false;
+ int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
+ found++;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1671,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ if (budget && found == budget)
+ break;
}
if (found)
@@ -1627,26 +1680,253 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
+/* budget == 0 means reap the whole queue. */
+static int nvme_poll_cq(struct nvme_queue *nvmeq, int budget)
+{
+ DEFINE_IO_COMP_BATCH(iob);
+ int found = __nvme_poll_cq(nvmeq, &iob, budget);
+
+ if (found && !rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return found;
+}
+
+static inline unsigned int nvmeq_outstanding(struct nvme_queue *nvmeq)
+{
+ u16 sq_tail = READ_ONCE(nvmeq->last_sq_tail);
+ u16 cq_head = nvmeq->cq_head;
+
+ if (sq_tail >= cq_head)
+ return sq_tail - cq_head;
+ return nvmeq->q_depth - cq_head + sq_tail;
+}
+
+static inline bool nvmeq_cq_continue(struct nvme_queue *nvmeq)
+{
+ return nvmeq_outstanding(nvmeq) > nvmeq->poll_thresh;
+}
+
+static inline bool nvmeq_cq_idle(struct nvme_queue *nvmeq)
+{
+ return nvmeq->cq_head == READ_ONCE(nvmeq->last_sq_tail);
+}
+
+/*
+ * Interrupt-moderation profiles, ordered from least moderation (index 0: enter
+ * polling late, short sleep, brief linger -> lowest latency, most interrupts)
+ * to most moderation (enter early, long sleep, long linger -> fewest
+ * interrupts). The hill-climb walks this table to keep completions-per-
+ * interrupt high, which is what bounds the interrupt rate.
+ */
+static const struct nvme_poll_prof {
+ u16 thresh;
+ u16 sleep_us;
+ u16 idle_rounds;
+ u16 budget; /* hard-IRQ initial poll cap; 0 == reap all */
+} nvme_poll_prof[] = {
+ { 31, 0, 1, 0 }, /* drain inline: lowest latency, most interrupts */
+ { 15, 10, 2, 16 },
+ { 7, 20, 4, 8 }, /* start profile; matches irq_poll_* defaults */
+ { 3, 40, 8, 4 },
+ { 1, 80, 16, 2 }, /* reap a little, hand the bulk to the thread */
+};
+
+#define NVME_DIM_START_PROFILE 2
+
+static void nvme_apply_profile(struct nvme_queue *nvmeq)
+{
+ const struct nvme_poll_prof *p = &nvme_poll_prof[nvmeq->dim.profile_ix];
+
+ nvmeq->poll_thresh = p->thresh;
+ nvmeq->poll_sleep_us = p->sleep_us;
+ nvmeq->poll_idle_rounds = p->idle_rounds;
+ nvmeq->poll_budget = p->budget;
+}
+
+/* Mirror of rdma_dim_step() bounded to our profile table. */
+static int nvme_dim_step(struct dim *dim)
+{
+ if (dim->tune_state == DIM_GOING_RIGHT) {
+ if (dim->profile_ix == ARRAY_SIZE(nvme_poll_prof) - 1)
+ return DIM_ON_EDGE;
+ dim->profile_ix++;
+ dim->steps_right++;
+ }
+ if (dim->tune_state == DIM_GOING_LEFT) {
+ if (dim->profile_ix == 0)
+ return DIM_ON_EDGE;
+ dim->profile_ix--;
+ dim->steps_left++;
+ }
+ return DIM_STEPPED;
+}
+
+/* Mirror of rdma_dim_stats_compare(): completion rate first, then batching. */
+static int nvme_dim_stats_compare(struct dim_stats *curr, struct dim_stats *prev)
+{
+ if (!prev->cpms)
+ return DIM_STATS_SAME;
+
+ if (IS_SIGNIFICANT_DIFF(curr->cpms, prev->cpms))
+ return curr->cpms > prev->cpms ? DIM_STATS_BETTER :
+ DIM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->cpe_ratio, prev->cpe_ratio))
+ return curr->cpe_ratio > prev->cpe_ratio ? DIM_STATS_BETTER :
+ DIM_STATS_WORSE;
+
+ return DIM_STATS_SAME;
+}
+
+/* Mirror of rdma_dim_decision(); returns true if the profile changed. */
+static bool nvme_dim_decision(struct dim_stats *curr, struct dim *dim)
+{
+ int prev_ix = dim->profile_ix;
+ int stats_res;
+
+ stats_res = nvme_dim_stats_compare(curr, &dim->prev_stats);
+ switch (stats_res) {
+ case DIM_STATS_SAME:
+ if (curr->cpe_ratio <= 50 * prev_ix)
+ dim->profile_ix = 0;
+ break;
+ case DIM_STATS_WORSE:
+ dim_turn(dim);
+ fallthrough;
+ case DIM_STATS_BETTER:
+ if (nvme_dim_step(dim) == DIM_ON_EDGE)
+ dim_turn(dim);
+ break;
+ }
+
+ dim->prev_stats = *curr;
+ return dim->profile_ix != prev_ix;
+}
+
+/*
+ * Sample the completion/interrupt counters and, once per DIM_NEVENTS
+ * interrupts, compute the load stats. Called only from the interrupt owner
+ * (never concurrently with the poll thread), so the counters have a single
+ * writer. ktime_get() is taken only at a window boundary, not per interrupt.
+ *
+ * On each completed window the hill-climb picks a profile and, if it changed,
+ * applies it to nvmeq->poll_*.
+ */
+static void nvme_dim(struct nvme_queue *nvmeq)
+{
+ struct dim *dim = &nvmeq->dim;
+ struct dim_sample end;
+ struct dim_stats stats;
+
+ if (dim->state == DIM_START_MEASURE) {
+ dim_update_sample_with_comps(nvmeq->dim_events, 0, 0,
+ nvmeq->dim_comps, &dim->start_sample);
+ dim->state = DIM_MEASURE_IN_PROGRESS;
+ return;
+ }
+
+ /* Cheap gate: only recompute once a full window of events accrues. */
+ if ((u16)(nvmeq->dim_events - dim->start_sample.event_ctr) < DIM_NEVENTS)
+ return;
+
+ dim_update_sample_with_comps(nvmeq->dim_events, 0, 0, nvmeq->dim_comps,
+ &end);
+ if (dim_calc_stats(&dim->start_sample, &end, &stats)) {
+ if (nvme_dim_decision(&stats, dim))
+ nvme_apply_profile(nvmeq);
+ trace_nvme_dim(nvmeq->qid, stats.cpms, stats.epms,
+ stats.cpe_ratio, dim->profile_ix);
+ }
+ dim->start_sample = end;
+}
+
static irqreturn_t nvme_irq(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
- DEFINE_IO_COMP_BATCH(iob);
+ struct nvme_dev *dev = nvmeq->dev;
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ irqreturn_t ret = IRQ_NONE;
+ unsigned int idle = 0;
- if (nvme_poll_cq(nvmeq, &iob)) {
- if (!rq_list_empty(&iob.req_list))
- nvme_pci_complete_batch(&iob);
- return IRQ_HANDLED;
+ for (;;) {
+ bool worked = false;
+ unsigned int i;
+
+ for (i = 0; i <= irq_poll_spin; i++) {
+ int n = nvme_poll_cq(nvmeq, 0);
+
+ if (n) {
+ nvmeq->dim_comps += n;
+ ret = IRQ_HANDLED;
+ worked = true;
+ idle = 0;
+ }
+
+ if (nvmeq_cq_idle(nvmeq))
+ goto done;
+ else if (need_resched())
+ cond_resched();
+ else
+ cpu_relax();
+ }
+
+ if (worked || ++idle < nvmeq->poll_idle_rounds) {
+ fsleep(nvmeq->poll_sleep_us);
+ continue;
+ }
+done:
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ if (!nvme_cqe_pending(nvmeq) ||
+ test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
+ break;
+
+ if (need_resched())
+ cond_resched();
+ else
+ cpu_relax();
+ idle = 0;
}
- return IRQ_NONE;
+
+ if (pdev->msi_enabled)
+ writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMC);
+ return ret;
}
static irqreturn_t nvme_irq_check(int irq, void *data)
{
struct nvme_queue *nvmeq = data;
+ int found;
+
+ nvmeq->dim_events++;
- if (nvme_cqe_pending(nvmeq))
+ if (test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
+ return IRQ_HANDLED;
+
+ found = nvme_poll_cq(nvmeq, nvmeq->poll_budget);
+ nvmeq->dim_comps += found;
+ nvme_dim(nvmeq);
+
+ if (!found) {
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ return IRQ_NONE;
+ }
+
+ /*
+ * Hand off to the poll thread when the queue is still deep, or when the
+ * budgeted initial poll left CQEs behind: their interrupt is already
+ * spent, so releasing here would strand them.
+ */
+ if (nvme_cqe_pending(nvmeq) || nvmeq_cq_continue(nvmeq)) {
+ struct nvme_dev *dev = nvmeq->dev;
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+
+ if (pdev->msi_enabled)
+ writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMS);
return IRQ_WAKE_THREAD;
- return IRQ_NONE;
+ }
+
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
+ return IRQ_HANDLED;
}
/*
@@ -1663,7 +1943,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
spin_lock(&nvmeq->cq_poll_lock);
- nvme_poll_cq(nvmeq, NULL);
+ __nvme_poll_cq(nvmeq, NULL, 0);
spin_unlock(&nvmeq->cq_poll_lock);
enable_irq(irq);
}
@@ -1671,14 +1951,14 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
{
struct nvme_queue *nvmeq = hctx->driver_data;
- bool found;
+ int found;
if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
!nvme_cqe_pending(nvmeq))
return 0;
spin_lock(&nvmeq->cq_poll_lock);
- found = nvme_poll_cq(nvmeq, iob);
+ found = __nvme_poll_cq(nvmeq, iob, 0);
spin_unlock(&nvmeq->cq_poll_lock);
return found;
@@ -2075,7 +2355,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)
for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
spin_lock(&dev->queues[i].cq_poll_lock);
- nvme_poll_cq(&dev->queues[i], NULL);
+ __nvme_poll_cq(&dev->queues[i], NULL, 0);
spin_unlock(&dev->queues[i].cq_poll_lock);
}
}
@@ -2171,13 +2451,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
- if (use_threaded_interrupts) {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
- nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- }
+ return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
+ nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2188,6 +2463,17 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvmeq->poll_thresh = irq_poll_thresh;
+ nvmeq->poll_sleep_us = irq_poll_sleep_us;
+ nvmeq->poll_idle_rounds = irq_poll_idle_rounds;
+ nvmeq->poll_budget = nvme_poll_prof[NVME_DIM_START_PROFILE].budget;
+ memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
+ nvmeq->dim.priv = nvmeq;
+ nvmeq->dim.profile_ix = NVME_DIM_START_PROFILE;
+ nvmeq->dim.tune_state = DIM_GOING_RIGHT;
+ nvmeq->dim_events = 0;
+ nvmeq->dim_comps = 0;
+ clear_bit(NVMEQ_POLLING, &nvmeq->flags);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
index ad25ad1e40412..4bdd5abc60880 100644
--- a/drivers/nvme/host/trace.c
+++ b/drivers/nvme/host/trace.c
@@ -498,3 +498,4 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)
}
EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_sq);
+EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_dim);
diff --git a/drivers/nvme/host/trace.h b/drivers/nvme/host/trace.h
index 4fb5922ffdac5..b7099b3026079 100644
--- a/drivers/nvme/host/trace.h
+++ b/drivers/nvme/host/trace.h
@@ -161,6 +161,29 @@ TRACE_EVENT(nvme_sq,
)
);
+TRACE_EVENT(nvme_dim,
+ TP_PROTO(u16 qid, int cpms, int epms, int cpe_ratio, u8 profile),
+ TP_ARGS(qid, cpms, epms, cpe_ratio, profile),
+ TP_STRUCT__entry(
+ __field(u16, qid)
+ __field(int, cpms)
+ __field(int, epms)
+ __field(int, cpe_ratio)
+ __field(u8, profile)
+ ),
+ TP_fast_assign(
+ __entry->qid = qid;
+ __entry->cpms = cpms;
+ __entry->epms = epms;
+ __entry->cpe_ratio = cpe_ratio;
+ __entry->profile = profile;
+ ),
+ TP_printk("qid=%u cpms=%d epms=%d cpe_ratio=%d profile=%u",
+ __entry->qid, __entry->cpms, __entry->epms,
+ __entry->cpe_ratio, __entry->profile
+ )
+);
+
#endif /* _TRACE_NVME_H */
#undef TRACE_INCLUDE_PATH
--
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-10 20:55 ` Keith Busch
@ 2026-08-11 2:32 ` changfengnan
2026-08-11 20:29 ` Keith Busch
0 siblings, 1 reply; 9+ messages in thread
From: changfengnan @ 2026-08-11 2:32 UTC (permalink / raw)
To: Keith Busch; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
> From: "Keith Busch"<kbusch@kernel.org>
> Date: Tue, Aug 11, 2026, 04:55
> Subject: Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> To: "Fengnan Chang"<changfengnan@bytedance.com>
> Cc: <axboe@kernel.dk>, <hch@lst.de>, <sagi@grimberg.me>, <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>, "Guzebing"<guzebing@bytedance.com>
> On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> > The idea behind this approach is: Let each I/O queue switch itself between
> > interrupt and poll mode based on its own recent completion rate.
> >
> > This version is still in the testing phase, and there are still some issues
> > with the code implementation. I releasing it now to see if the approach
> > is generally acceptable. If the approach looks good, I´ll continue to
> > refine it and conduct more extensive testing. The main implementation
> > logic is in `nvme_adaptive_sample` and `nvme_adaptive_irq_poll`; you should
> > focus on reviewing the implementation of these two functions.
>
> Can we subscribe to the dynamic interrupt moderation (dim) library? I
> know it's generally used in conjuction with a hardware interrupt
> coalescing feature, but we can just do pure software with it too. The
> library provides a hill climb to adapt the policy at run time.
>
> This is a quick PoC I put together. I haven't tested on fast devices, so
> I'm not sure if I've dialed in the profiles, but it's start of what I
> had in mind.
The code looks much cleaner when using dim, I'll see if I can replace the
sample-and-poll logic I wrote myself with the dim library.
I ran a quick test on the POC patch and didn't see any performance
improvements; in fact, there were quite a few regressions. Maybe some
parameters need to be adjusted.
>
> ---
> diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
> index 31974c7dd20c9..9fad1c7b1f678 100644
> --- a/drivers/nvme/host/Kconfig
> +++ b/drivers/nvme/host/Kconfig
> @@ -6,6 +6,7 @@ config BLK_DEV_NVME
> tristate "NVM Express block device"
> depends on PCI && BLOCK
> select NVME_CORE
> + select DIMLIB
> help
> The NVM Express driver is for solid state drives directly
> connected to the PCI or PCI Express bus. If you know you
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 8438c904ec496..ff0c8d74fae08 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -9,6 +9,8 @@
> #include <linux/blkdev.h>
> #include <linux/blk-mq-dma.h>
> #include <linux/blk-integrity.h>
> +#include <linux/delay.h>
> +#include <linux/dim.h>
> #include <linux/dmi.h>
> #include <linux/init.h>
> #include <linux/interrupt.h>
> @@ -82,6 +84,45 @@ struct quirk_entry {
> static int use_threaded_interrupts;
> module_param(use_threaded_interrupts, int, 0444);
>
> +static unsigned int irq_poll_thresh = 7;
> +module_param(irq_poll_thresh, uint, 0644);
> +MODULE_PARM_DESC(irq_poll_thresh,
> + "outstanding depth after which a queue switches to threaded polling");
> +
> +static unsigned int irq_poll_spin = 7;
> +module_param(irq_poll_spin, uint, 0644);
> +MODULE_PARM_DESC(irq_poll_spin,
> + "poll iterations to spin (cpu_relax) before sleeping in the poll loop (zero based)");
> +
> +static unsigned int irq_poll_sleep_us = 20;
> +static unsigned int irq_poll_idle_us = 80;
> +static unsigned int irq_poll_idle_rounds = DIV_ROUND_UP(80, 20);
> +
> +static int irq_poll_us_set(const char *val, const struct kernel_param *kp)
> +{
> + int ret = param_set_uint(val, kp);
> +
> + if (ret)
> + return ret;
> +
> + irq_poll_idle_rounds = irq_poll_sleep_us ?
> + DIV_ROUND_UP(irq_poll_idle_us, irq_poll_sleep_us) : 1;
> + return 0;
> +}
> +
> +static const struct kernel_param_ops irq_poll_us_ops = {
> + .set = irq_poll_us_set,
> + .get = param_get_uint,
> +};
> +
> +module_param_cb(irq_poll_sleep_us, &irq_poll_us_ops, &irq_poll_sleep_us, 0644);
> +MODULE_PARM_DESC(irq_poll_sleep_us,
> + "microseconds to sleep between poll bursts (<=10 busy-delays, does not yield)");
> +
> +module_param_cb(irq_poll_idle_us, &irq_poll_us_ops, &irq_poll_idle_us, 0644);
> +MODULE_PARM_DESC(irq_poll_idle_us,
> + "microseconds to wait for stragglers before handing a queue back to the IRQ path");
> +
> static bool use_cmb_sqes = true;
> module_param(use_cmb_sqes, bool, 0444);
> MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
> @@ -381,11 +422,21 @@ struct nvme_queue {
> u16 qid;
> u8 cq_phase;
> u8 sqes;
> + /* Adaptive polling knobs, seeded from the irq_poll_* module params. */
> + unsigned int poll_thresh;
> + unsigned int poll_sleep_us;
> + unsigned int poll_idle_rounds;
> + unsigned int poll_budget;
> + /* Adaptive interrupt moderation (DIM) sampling state. */
> + struct dim dim;
> + u16 dim_events; /* cumulative interrupts (BIT_GAP-safe) */
> + u32 dim_comps; /* cumulative completions */
> unsigned long flags;
> #define NVMEQ_ENABLED 0
> #define NVMEQ_SQ_CMB 1
> #define NVMEQ_DELETE_ERROR 2
> #define NVMEQ_POLLED 3
> +#define NVMEQ_POLLING 4
> __le32 *dbbuf_sq_db;
> __le32 *dbbuf_cq_db;
> __le32 *dbbuf_sq_ei;
> @@ -1606,13 +1657,13 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
> }
> }
>
> -static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> - struct io_comp_batch *iob)
> +static inline int __nvme_poll_cq(struct nvme_queue *nvmeq,
> + struct io_comp_batch *iob, int budget)
> {
> - bool found = false;
> + int found = 0;
>
> while (nvme_cqe_pending(nvmeq)) {
> - found = true;
> + found++;
> /*
> * load-load control dependency between phase and the rest of
> * the cqe requires a full read memory barrier
> @@ -1620,6 +1671,8 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> dma_rmb();
> nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
> nvme_update_cq_head(nvmeq);
> + if (budget && found == budget)
> + break;
> }
>
> if (found)
> @@ -1627,26 +1680,253 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
> return found;
> }
>
> +/* budget == 0 means reap the whole queue. */
> +static int nvme_poll_cq(struct nvme_queue *nvmeq, int budget)
> +{
> + DEFINE_IO_COMP_BATCH(iob);
> + int found = __nvme_poll_cq(nvmeq, &iob, budget);
> +
> + if (found && !rq_list_empty(&iob.req_list))
> + nvme_pci_complete_batch(&iob);
> + return found;
> +}
> +
> +static inline unsigned int nvmeq_outstanding(struct nvme_queue *nvmeq)
> +{
> + u16 sq_tail = READ_ONCE(nvmeq->last_sq_tail);
> + u16 cq_head = nvmeq->cq_head;
> +
> + if (sq_tail >= cq_head)
> + return sq_tail - cq_head;
> + return nvmeq->q_depth - cq_head + sq_tail;
> +}
> +
> +static inline bool nvmeq_cq_continue(struct nvme_queue *nvmeq)
> +{
> + return nvmeq_outstanding(nvmeq) > nvmeq->poll_thresh;
> +}
> +
> +static inline bool nvmeq_cq_idle(struct nvme_queue *nvmeq)
> +{
> + return nvmeq->cq_head == READ_ONCE(nvmeq->last_sq_tail);
> +}
> +
> +/*
> + * Interrupt-moderation profiles, ordered from least moderation (index 0: enter
> + * polling late, short sleep, brief linger -> lowest latency, most interrupts)
> + * to most moderation (enter early, long sleep, long linger -> fewest
> + * interrupts). The hill-climb walks this table to keep completions-per-
> + * interrupt high, which is what bounds the interrupt rate.
> + */
> +static const struct nvme_poll_prof {
> + u16 thresh;
> + u16 sleep_us;
> + u16 idle_rounds;
> + u16 budget; /* hard-IRQ initial poll cap; 0 == reap all */
> +} nvme_poll_prof[] = {
> + { 31, 0, 1, 0 }, /* drain inline: lowest latency, most interrupts */
> + { 15, 10, 2, 16 },
> + { 7, 20, 4, 8 }, /* start profile; matches irq_poll_* defaults */
> + { 3, 40, 8, 4 },
> + { 1, 80, 16, 2 }, /* reap a little, hand the bulk to the thread */
> +};
> +
> +#define NVME_DIM_START_PROFILE 2
> +
> +static void nvme_apply_profile(struct nvme_queue *nvmeq)
> +{
> + const struct nvme_poll_prof *p = &nvme_poll_prof[nvmeq->dim.profile_ix];
> +
> + nvmeq->poll_thresh = p->thresh;
> + nvmeq->poll_sleep_us = p->sleep_us;
> + nvmeq->poll_idle_rounds = p->idle_rounds;
> + nvmeq->poll_budget = p->budget;
> +}
> +
> +/* Mirror of rdma_dim_step() bounded to our profile table. */
> +static int nvme_dim_step(struct dim *dim)
> +{
> + if (dim->tune_state == DIM_GOING_RIGHT) {
> + if (dim->profile_ix == ARRAY_SIZE(nvme_poll_prof) - 1)
> + return DIM_ON_EDGE;
> + dim->profile_ix++;
> + dim->steps_right++;
> + }
> + if (dim->tune_state == DIM_GOING_LEFT) {
> + if (dim->profile_ix == 0)
> + return DIM_ON_EDGE;
> + dim->profile_ix--;
> + dim->steps_left++;
> + }
> + return DIM_STEPPED;
> +}
> +
> +/* Mirror of rdma_dim_stats_compare(): completion rate first, then batching. */
> +static int nvme_dim_stats_compare(struct dim_stats *curr, struct dim_stats *prev)
> +{
> + if (!prev->cpms)
> + return DIM_STATS_SAME;
> +
> + if (IS_SIGNIFICANT_DIFF(curr->cpms, prev->cpms))
> + return curr->cpms > prev->cpms ? DIM_STATS_BETTER :
> + DIM_STATS_WORSE;
> +
> + if (IS_SIGNIFICANT_DIFF(curr->cpe_ratio, prev->cpe_ratio))
> + return curr->cpe_ratio > prev->cpe_ratio ? DIM_STATS_BETTER :
> + DIM_STATS_WORSE;
> +
> + return DIM_STATS_SAME;
> +}
> +
> +/* Mirror of rdma_dim_decision(); returns true if the profile changed. */
> +static bool nvme_dim_decision(struct dim_stats *curr, struct dim *dim)
> +{
> + int prev_ix = dim->profile_ix;
> + int stats_res;
> +
> + stats_res = nvme_dim_stats_compare(curr, &dim->prev_stats);
> + switch (stats_res) {
> + case DIM_STATS_SAME:
> + if (curr->cpe_ratio <= 50 * prev_ix)
> + dim->profile_ix = 0;
> + break;
> + case DIM_STATS_WORSE:
> + dim_turn(dim);
> + fallthrough;
> + case DIM_STATS_BETTER:
> + if (nvme_dim_step(dim) == DIM_ON_EDGE)
> + dim_turn(dim);
> + break;
> + }
> +
> + dim->prev_stats = *curr;
> + return dim->profile_ix != prev_ix;
> +}
> +
> +/*
> + * Sample the completion/interrupt counters and, once per DIM_NEVENTS
> + * interrupts, compute the load stats. Called only from the interrupt owner
> + * (never concurrently with the poll thread), so the counters have a single
> + * writer. ktime_get() is taken only at a window boundary, not per interrupt.
> + *
> + * On each completed window the hill-climb picks a profile and, if it changed,
> + * applies it to nvmeq->poll_*.
> + */
> +static void nvme_dim(struct nvme_queue *nvmeq)
> +{
> + struct dim *dim = &nvmeq->dim;
> + struct dim_sample end;
> + struct dim_stats stats;
> +
> + if (dim->state == DIM_START_MEASURE) {
> + dim_update_sample_with_comps(nvmeq->dim_events, 0, 0,
> + nvmeq->dim_comps, &dim->start_sample);
> + dim->state = DIM_MEASURE_IN_PROGRESS;
> + return;
> + }
> +
> + /* Cheap gate: only recompute once a full window of events accrues. */
> + if ((u16)(nvmeq->dim_events - dim->start_sample.event_ctr) < DIM_NEVENTS)
> + return;
> +
> + dim_update_sample_with_comps(nvmeq->dim_events, 0, 0, nvmeq->dim_comps,
> + &end);
> + if (dim_calc_stats(&dim->start_sample, &end, &stats)) {
> + if (nvme_dim_decision(&stats, dim))
> + nvme_apply_profile(nvmeq);
> + trace_nvme_dim(nvmeq->qid, stats.cpms, stats.epms,
> + stats.cpe_ratio, dim->profile_ix);
> + }
> + dim->start_sample = end;
> +}
> +
> static irqreturn_t nvme_irq(int irq, void *data)
> {
> struct nvme_queue *nvmeq = data;
> - DEFINE_IO_COMP_BATCH(iob);
> + struct nvme_dev *dev = nvmeq->dev;
> + struct pci_dev *pdev = to_pci_dev(dev->dev);
> + irqreturn_t ret = IRQ_NONE;
> + unsigned int idle = 0;
>
> - if (nvme_poll_cq(nvmeq, &iob)) {
> - if (!rq_list_empty(&iob.req_list))
> - nvme_pci_complete_batch(&iob);
> - return IRQ_HANDLED;
> + for (;;) {
> + bool worked = false;
> + unsigned int i;
> +
> + for (i = 0; i <= irq_poll_spin; i++) {
> + int n = nvme_poll_cq(nvmeq, 0);
> +
> + if (n) {
> + nvmeq->dim_comps += n;
> + ret = IRQ_HANDLED;
> + worked = true;
> + idle = 0;
> + }
> +
> + if (nvmeq_cq_idle(nvmeq))
> + goto done;
> + else if (need_resched())
> + cond_resched();
> + else
> + cpu_relax();
> + }
> +
> + if (worked || ++idle < nvmeq->poll_idle_rounds) {
> + fsleep(nvmeq->poll_sleep_us);
> + continue;
> + }
> +done:
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + if (!nvme_cqe_pending(nvmeq) ||
> + test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
> + break;
> +
> + if (need_resched())
> + cond_resched();
> + else
> + cpu_relax();
> + idle = 0;
> }
> - return IRQ_NONE;
> +
> + if (pdev->msi_enabled)
> + writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMC);
> + return ret;
> }
>
> static irqreturn_t nvme_irq_check(int irq, void *data)
> {
> struct nvme_queue *nvmeq = data;
> + int found;
> +
> + nvmeq->dim_events++;
>
> - if (nvme_cqe_pending(nvmeq))
> + if (test_and_set_bit(NVMEQ_POLLING, &nvmeq->flags))
> + return IRQ_HANDLED;
> +
> + found = nvme_poll_cq(nvmeq, nvmeq->poll_budget);
> + nvmeq->dim_comps += found;
> + nvme_dim(nvmeq);
> +
> + if (!found) {
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + return IRQ_NONE;
> + }
> +
> + /*
> + * Hand off to the poll thread when the queue is still deep, or when the
> + * budgeted initial poll left CQEs behind: their interrupt is already
> + * spent, so releasing here would strand them.
> + */
> + if (nvme_cqe_pending(nvmeq) || nvmeq_cq_continue(nvmeq)) {
> + struct nvme_dev *dev = nvmeq->dev;
> + struct pci_dev *pdev = to_pci_dev(dev->dev);
> +
> + if (pdev->msi_enabled)
> + writel(BIT(nvmeq->cq_vector), dev->bar + NVME_REG_INTMS);
> return IRQ_WAKE_THREAD;
> - return IRQ_NONE;
> + }
> +
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> + return IRQ_HANDLED;
> }
>
> /*
> @@ -1663,7 +1943,7 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
> irq = pci_irq_vector(pdev, nvmeq->cq_vector);
> disable_irq(irq);
> spin_lock(&nvmeq->cq_poll_lock);
> - nvme_poll_cq(nvmeq, NULL);
> + __nvme_poll_cq(nvmeq, NULL, 0);
> spin_unlock(&nvmeq->cq_poll_lock);
> enable_irq(irq);
> }
> @@ -1671,14 +1951,14 @@ static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
> static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
> {
> struct nvme_queue *nvmeq = hctx->driver_data;
> - bool found;
> + int found;
>
> if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
> !nvme_cqe_pending(nvmeq))
> return 0;
>
> spin_lock(&nvmeq->cq_poll_lock);
> - found = nvme_poll_cq(nvmeq, iob);
> + found = __nvme_poll_cq(nvmeq, iob, 0);
> spin_unlock(&nvmeq->cq_poll_lock);
>
> return found;
> @@ -2075,7 +2355,7 @@ static void nvme_reap_pending_cqes(struct nvme_dev *dev)
>
> for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
> spin_lock(&dev->queues[i].cq_poll_lock);
> - nvme_poll_cq(&dev->queues[i], NULL);
> + __nvme_poll_cq(&dev->queues[i], NULL, 0);
> spin_unlock(&dev->queues[i].cq_poll_lock);
> }
> }
> @@ -2171,13 +2451,8 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
> struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
> int nr = nvmeq->dev->ctrl.instance;
>
> - if (use_threaded_interrupts) {
> - return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
> - nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> - } else {
> - return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
> - NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> - }
> + return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
> + nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
> }
>
> static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
> @@ -2188,6 +2463,17 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
> nvmeq->last_sq_tail = 0;
> nvmeq->cq_head = 0;
> nvmeq->cq_phase = 1;
> + nvmeq->poll_thresh = irq_poll_thresh;
> + nvmeq->poll_sleep_us = irq_poll_sleep_us;
> + nvmeq->poll_idle_rounds = irq_poll_idle_rounds;
> + nvmeq->poll_budget = nvme_poll_prof[NVME_DIM_START_PROFILE].budget;
> + memset(&nvmeq->dim, 0, sizeof(nvmeq->dim));
> + nvmeq->dim.priv = nvmeq;
> + nvmeq->dim.profile_ix = NVME_DIM_START_PROFILE;
> + nvmeq->dim.tune_state = DIM_GOING_RIGHT;
> + nvmeq->dim_events = 0;
> + nvmeq->dim_comps = 0;
> + clear_bit(NVMEQ_POLLING, &nvmeq->flags);
> nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
> memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
> nvme_dbbuf_init(dev, nvmeq, qid);
> diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c
> index ad25ad1e40412..4bdd5abc60880 100644
> --- a/drivers/nvme/host/trace.c
> +++ b/drivers/nvme/host/trace.c
> @@ -498,3 +498,4 @@ const char *nvme_trace_disk_name(struct trace_seq *p, char *name)
> }
>
> EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_sq);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(nvme_dim);
> diff --git a/drivers/nvme/host/trace.h b/drivers/nvme/host/trace.h
> index 4fb5922ffdac5..b7099b3026079 100644
> --- a/drivers/nvme/host/trace.h
> +++ b/drivers/nvme/host/trace.h
> @@ -161,6 +161,29 @@ TRACE_EVENT(nvme_sq,
> )
> );
>
> +TRACE_EVENT(nvme_dim,
> + TP_PROTO(u16 qid, int cpms, int epms, int cpe_ratio, u8 profile),
> + TP_ARGS(qid, cpms, epms, cpe_ratio, profile),
> + TP_STRUCT__entry(
> + __field(u16, qid)
> + __field(int, cpms)
> + __field(int, epms)
> + __field(int, cpe_ratio)
> + __field(u8, profile)
> + ),
> + TP_fast_assign(
> + __entry->qid = qid;
> + __entry->cpms = cpms;
> + __entry->epms = epms;
> + __entry->cpe_ratio = cpe_ratio;
> + __entry->profile = profile;
> + ),
> + TP_printk("qid=%u cpms=%d epms=%d cpe_ratio=%d profile=%u",
> + __entry->qid, __entry->cpms, __entry->epms,
> + __entry->cpe_ratio, __entry->profile
> + )
> +);
> +
> #endif /* _TRACE_NVME_H */
>
> #undef TRACE_INCLUDE_PATH
> --
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-11 2:32 ` changfengnan
@ 2026-08-11 20:29 ` Keith Busch
2026-08-13 2:11 ` changfengnan
0 siblings, 1 reply; 9+ messages in thread
From: Keith Busch @ 2026-08-11 20:29 UTC (permalink / raw)
To: changfengnan; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
On Tue, Aug 11, 2026 at 10:32:35AM +0800, changfengnan wrote:
> The code looks much cleaner when using dim, I'll see if I can replace the
> sample-and-poll logic I wrote myself with the dim library.
> I ran a quick test on the POC patch and didn't see any performance
> improvements; in fact, there were quite a few regressions. Maybe some
> parameters need to be adjusted.
Yeah, I'm just now testing mid-tier devices and it's also performing a
bit worse for high throughput workloads. But I hadn't really tried to
tune the settings here, and maybe my criteria is all wrong. I was mainly
trying to see if we can utilize the dim library before honing in on the
right implementation details.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-06 3:10 [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling Fengnan Chang
2026-08-10 20:55 ` Keith Busch
@ 2026-08-12 17:50 ` Anuj Gupta
2026-08-13 2:04 ` changfengnan
1 sibling, 1 reply; 9+ messages in thread
From: Anuj Gupta @ 2026-08-12 17:50 UTC (permalink / raw)
To: Fengnan Chang
Cc: kbusch, axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
> 4 KiB random-read IOPS. Four of them should be good for about 12.8M IOPS,
> but interrupt-driven completion tops out at 5.59M, only about 44% of that.
>
> Polling gets rid of that cost, but polling every queue all the time burns
> CPU and hurts the sparse or bursty queues that interrupts handle just fine.
> So instead of a global switch, let each queue make the call on its own,
> from how fast it has been completing lately, and re-check often enough that
> the decision tracks the workload rather than a fixed tunable.
>
> Each queue runs a small loop with three stages. First it samples its
> completion rate while still on interrupts. Only if that rate is high
> enough to fill a small batch inside a bounded latency window does it mask
> its own IRQ and start draining the CQ from a high-resolution timer, with
> each wait sized to collect roughly one batch. It keeps polling as long as
> it keeps up with that rate; the moment it stalls or slows down it turns the
> IRQ back on and backs off, waiting longer the further behind it fell.
> A queue that doesn't benefit drops back quickly and only gets retried once
> in a while, so polling stays on the queues that are actually
> interrupt-bound and everything else keeps running on the untouched IRQ
> path.
>
> Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
> versus off:
>
> QD32 QD64 QD128
> one device, one job +18.44% +24.35% +26.38%
> four devices, eight jobs +83.76% +99.02% +96.26%
>
Did a quick test on single NVMe with QD64, one job, 4K random reads via
io_uring, I see ~23% improvement: 569K -> 701K IOPS.
Do you expect the fixed batch size, sample size, maximum delay, episode
length, and backoff multiplier to work across devices with different
latency profiles and across different workloads, or should these values
somehow adapt to observed queue behavior?
> +/*
> + * Stop polling and turn the queue's IRQ back on. @elapsed is how long the
> + * episode ran after it started falling behind, or 0 if it ended cleanly.
> + * The bigger @elapsed is, the more completions we missed, and the longer we
> + * wait before sampling this queue again, so a queue that polling doesn't
> + * help is left alone most of the time.
> + */
@elapsed is the total polling-episode duration, not the time since the
queue fell behind. It is then used with the total completion count to
estimate cumulative deficit. Could the comment be reworded accordingly?
> +/*
> + * Called from the IRQ handler after a reap that found something. If we're
> + * still in backoff, just count it down. Otherwise time how long
> + * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
> + * them. If that looks worth polling (see the filter below) mask the IRQ and
> + * switch to poll mode; if not, leave the queue on interrupts.
> + */
> +static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
> + unsigned int completions)
> +{
> + struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
> + unsigned int sample;
> + unsigned long flags;
> + u64 delta, interval, now;
> +
> + if (adaptive->retry_completions) {
> + if (adaptive->retry_completions != U64_MAX)
> + adaptive->retry_completions -= min_t(u64, completions,
> + adaptive->retry_completions);
> + return;
> + }
> + if (!adaptive->start_ns) {
> + adaptive->start_ns = ktime_get_ns();
> + return;
> + }
> + adaptive->completions += completions;
> + if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
> + return;
> +
> + now = ktime_get_ns();
> + delta = now - adaptive->start_ns;
> + sample = adaptive->completions;
> + adaptive->start_ns = now;
> + adaptive->completions = 0;
> + if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
> + NVME_ADAPTIVE_MAX_DELAY_NS)
> + return;
> + /*
> + * Is this rate worth polling? The 100/99 factor trims 1% off the gap so
> + * a queue sitting right on the threshold isn't pulled in. Skip it if
> + * it's too slow to fill a batch within MAX_DELAY, and also skip it if
> + * it's already fast enough to batch by itself. The hardware's own
> + * coalescing already handles that case, so leave it on interrupts.
> + */
> + interval = div64_u64(delta * 100, sample * 99);
> + if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||
The comment says this rejects queues that cannot fill a target batch
within MAX_DELAY, but this condition only checks whether one completion
interval exceeds MAX_DELAY. Should this instead account for TARGET_BATCH
or this comment is describing a different policy?
> + (delta > NSEC_PER_MSEC &&
> + interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
> + NVME_ADAPTIVE_TARGET_BATCH))
> + return;
> +
For a normal 256-CQE sample with an interval of 2us or less, delta will
be about 512us or less, so the 1ms condition prevents this exclusion
from firing. Is this intended to detect only samples that substantially
overshoot 256 CQEs? If so, could the comment make that narrower intent
explicit?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-12 17:50 ` Anuj Gupta
@ 2026-08-13 2:04 ` changfengnan
0 siblings, 0 replies; 9+ messages in thread
From: changfengnan @ 2026-08-13 2:04 UTC (permalink / raw)
To: Anuj Gupta; +Cc: kbusch, axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
[-- Attachment #1: Type: text/plain, Size: 7524 bytes --]
> From: "Anuj Gupta"<anuj1072538@gmail.com>
> Date: Thu, Aug 13, 2026, 01:50
> Subject: Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> To: "Fengnan Chang"<changfengnan@bytedance.com>
> Cc: <kbusch@kernel.org>, <axboe@kernel.dk>, <hch@lst.de>, <sagi@grimberg.me>, <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>, "Guzebing"<guzebing@bytedance.com>
> On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> > Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
> > 4 KiB random-read IOPS. Four of them should be good for about 12.8M IOPS,
> > but interrupt-driven completion tops out at 5.59M, only about 44% of that.
> >
> > Polling gets rid of that cost, but polling every queue all the time burns
> > CPU and hurts the sparse or bursty queues that interrupts handle just fine.
> > So instead of a global switch, let each queue make the call on its own,
> > from how fast it has been completing lately, and re-check often enough that
> > the decision tracks the workload rather than a fixed tunable.
> >
> > Each queue runs a small loop with three stages. First it samples its
> > completion rate while still on interrupts. Only if that rate is high
> > enough to fill a small batch inside a bounded latency window does it mask
> > its own IRQ and start draining the CQ from a high-resolution timer, with
> > each wait sized to collect roughly one batch. It keeps polling as long as
> > it keeps up with that rate; the moment it stalls or slows down it turns the
> > IRQ back on and backs off, waiting longer the further behind it fell.
> > A queue that doesn't benefit drops back quickly and only gets retried once
> > in a while, so polling stays on the queues that are actually
> > interrupt-bound and everything else keeps running on the untouched IRQ
> > path.
> >
> > Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
> > versus off:
> >
> > QD32 QD64 QD128
> > one device, one job +18.44% +24.35% +26.38%
> > four devices, eight jobs +83.76% +99.02% +96.26%
> >
>
> Did a quick test on single NVMe with QD64, one job, 4K random reads via
> io_uring, I see ~23% improvement: 569K -> 701K IOPS.
>
> Do you expect the fixed batch size, sample size, maximum delay, episode
> length, and backoff multiplier to work across devices with different
> latency profiles and across different workloads, or should these values
> somehow adapt to observed queue behavior?
Thank you for sharing the data. My goal is for these parameters to be
applicable to all disk and workload scenarios, but more testing is needed
in practice.
I’ve made some adjustments to the parameters and logic in the version
I’m currently developing, and it looks pretty good so far.
Attached is the version I'm currently developing. If you have time to take
a look or test it and offer some feedback, I'd really appreciate it.
>
> > +/*
> > + * Stop polling and turn the queue's IRQ back on. @elapsed is how long the
> > + * episode ran after it started falling behind, or 0 if it ended cleanly.
> > + * The bigger @elapsed is, the more completions we missed, and the longer we
> > + * wait before sampling this queue again, so a queue that polling doesn't
> > + * help is left alone most of the time.
> > + */
>
> @elapsed is the total polling-episode duration, not the time since the
> queue fell behind. It is then used with the total completion count to
> estimate cumulative deficit. Could the comment be reworded accordingly?
Sorry, there are some issues with the comments in this version. I'll fix them
in the next version.
>
> > +/*
> > + * Called from the IRQ handler after a reap that found something. If we're
> > + * still in backoff, just count it down. Otherwise time how long
> > + * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
> > + * them. If that looks worth polling (see the filter below) mask the IRQ and
> > + * switch to poll mode; if not, leave the queue on interrupts.
> > + */
> > +static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
> > + unsigned int completions)
> > +{
> > + struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
> > + unsigned int sample;
> > + unsigned long flags;
> > + u64 delta, interval, now;
> > +
> > + if (adaptive->retry_completions) {
> > + if (adaptive->retry_completions != U64_MAX)
> > + adaptive->retry_completions -= min_t(u64, completions,
> > + adaptive->retry_completions);
> > + return;
> > + }
> > + if (!adaptive->start_ns) {
> > + adaptive->start_ns = ktime_get_ns();
> > + return;
> > + }
> > + adaptive->completions += completions;
> > + if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
> > + return;
> > +
> > + now = ktime_get_ns();
> > + delta = now - adaptive->start_ns;
> > + sample = adaptive->completions;
> > + adaptive->start_ns = now;
> > + adaptive->completions = 0;
> > + if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
> > + NVME_ADAPTIVE_MAX_DELAY_NS)
> > + return;
> > + /*
> > + * Is this rate worth polling? The 100/99 factor trims 1% off the gap so
> > + * a queue sitting right on the threshold isn't pulled in. Skip it if
> > + * it's too slow to fill a batch within MAX_DELAY, and also skip it if
> > + * it's already fast enough to batch by itself. The hardware's own
> > + * coalescing already handles that case, so leave it on interrupts.
> > + */
> > + interval = div64_u64(delta * 100, sample * 99);
> > + if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||
>
> The comment says this rejects queues that cannot fill a target batch
> within MAX_DELAY, but this condition only checks whether one completion
> interval exceeds MAX_DELAY. Should this instead account for TARGET_BATCH
> or this comment is describing a different policy?
TARGET_BATCH will no longer be used in the next version.
>
> > + (delta > NSEC_PER_MSEC &&
> > + interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
> > + NVME_ADAPTIVE_TARGET_BATCH))
> > + return;
> > +
>
> For a normal 256-CQE sample with an interval of 2us or less, delta will
> be about 512us or less, so the 1ms condition prevents this exclusion
> from firing. Is this intended to detect only samples that substantially
> overshoot 256 CQEs? If so, could the comment make that narrower intent
> explicit?
I did it that way before because I thought that if the interrupt mode could
process a CQE in 2 μs, that would mean the interrupt mode was already very fast.
I’ve removed that check from the version I’m currently developing.
>
[-- Attachment #2: 0001-nvme-pci-add-adaptive-interrupt-polling.patch --]
[-- Type: application/octet-stream, Size: 27757 bytes --]
From c3d9989d99c901abde1397651801309df3e11347 Mon Sep 17 00:00:00 2001
From: Fengnan Chang <changfengnan@bytedance.com>
Date: Tue, 11 Aug 2026 20:22:27 +0800
Subject: [PATCH] nvme-pci: add adaptive interrupt polling
Add opt-in adaptive polling for nvme, measure an IRQ
completion-rate baseline over 8192 CQEs, then admit a bounded
polling trial only when the observed rate can sustain the fixed
10 us polling period.
Signed-off-by: Fengnan Chang <changfengnan@bytedance.com>
---
Documentation/ABI/testing/sysfs-nvme | 15 +
drivers/nvme/host/Kconfig | 1 +
drivers/nvme/host/pci.c | 564 +++++++++++++++++++++++++--
3 files changed, 554 insertions(+), 26 deletions(-)
diff --git a/Documentation/ABI/testing/sysfs-nvme b/Documentation/ABI/testing/sysfs-nvme
index 499d5f843cd43..309d59dc6b872 100644
--- a/Documentation/ABI/testing/sysfs-nvme
+++ b/Documentation/ABI/testing/sysfs-nvme
@@ -11,3 +11,18 @@ Description:
(REPLACETLSPSK) with the target. After a reauthentication
the value returned by tls_configured_key will be the new
serial.
+
+What: /sys/class/nvme/nvmeX/adaptive_irq_polling
+Date: August 2026
+KernelVersion: 7.2
+Contact: Linux NVMe mailing list <linux-nvme@lists.infradead.org>
+Description:
+ Enable (1) or disable (0) adaptive IRQ polling on eligible I/O
+ queues of one PCI NVMe controller. Changing the mode freezes its
+ namespace request queues and waits for all outstanding namespace I/O
+ to complete before updating the queue IRQ state. Writes fail with
+ EBUSY unless the controller is live.
+
+ The attribute is not available with threaded NVMe interrupts. The
+ use_adaptive_irq_polling module parameter supplies only the initial
+ value for newly probed controllers.
diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig
index 31974c7dd20c9..22164b901da85 100644
--- a/drivers/nvme/host/Kconfig
+++ b/drivers/nvme/host/Kconfig
@@ -5,6 +5,7 @@ config NVME_CORE
config BLK_DEV_NVME
tristate "NVM Express block device"
depends on PCI && BLOCK
+ select IRQ_POLL
select NVME_CORE
help
The NVM Express driver is for solid state drives directly
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 69932d640b537..b560278247d1a 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -10,9 +10,12 @@
#include <linux/blk-mq-dma.h>
#include <linux/blk-integrity.h>
#include <linux/dmi.h>
+#include <linux/hrtimer.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/irq_poll.h>
+#include <linux/ktime.h>
#include <linux/kstrtox.h>
#include <linux/memremap.h>
#include <linux/mm.h>
@@ -82,6 +85,42 @@ struct quirk_entry {
static int use_threaded_interrupts;
module_param(use_threaded_interrupts, int, 0444);
+static bool use_adaptive_irq_polling;
+module_param(use_adaptive_irq_polling, bool, 0444);
+MODULE_PARM_DESC(use_adaptive_irq_polling,
+ "default adaptive polling mode for non-threaded MSI-X I/O queues");
+
+/*
+ * Adaptive IRQ polling moves an eligible interrupt-driven queue to a
+ * timer-based poll path according to its observed completion rate. Each
+ * queue goes through three stages:
+ *
+ * 1. Baseline (still in IRQ mode): measure completion throughput over one
+ * NVME_ADAPTIVE_EPISODE_CQES window. Trial polling only while the average
+ * completion gap is short enough that a poll fired once per
+ * NVME_ADAPTIVE_POLL_PERIOD_NS would still find a completion.
+ * 2. Trial: mask the queue's IRQ and drain the CQ from an hrtimer every
+ * NVME_ADAPTIVE_POLL_PERIOD_NS. Reject the trial immediately if its
+ * cumulative completion rate falls behind the IRQ baseline. At the end
+ * of a full completion window, accept polling only if its throughput is
+ * strictly higher than the IRQ baseline.
+ * 3. Park or back off: keep an accepted queue in polling mode, recheck each
+ * completion window, and periodically return to IRQ mode for a fresh
+ * baseline. A rejected or slowing queue may retry twice; three consecutive
+ * failures trigger a long, completion-counted backoff. This lets a
+ * profitable queue get past a noisy first trial without making failed
+ * trials a measurable part of workloads where IRQs are faster.
+ *
+ * The poll period bounds the extra completion latency polling may add and
+ * defines the admission rate above. The completion-counted windows make the
+ * comparison and retry duty cycle independent of wall-clock load changes.
+ */
+#define NVME_ADAPTIVE_TARGET_BATCH 5U
+#define NVME_ADAPTIVE_POLL_PERIOD_NS (10U * NSEC_PER_USEC)
+#define NVME_ADAPTIVE_EPISODE_CQES 8192U
+#define NVME_ADAPTIVE_REEVAL_CQES (64U * NVME_ADAPTIVE_EPISODE_CQES)
+#define NVME_ADAPTIVE_POLL_RETRIES 2U
+
static bool use_cmb_sqes = true;
module_param(use_cmb_sqes, bool, 0444);
MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
@@ -307,6 +346,7 @@ struct nvme_dev {
void __iomem *bar;
unsigned long bar_mapped_size;
struct mutex shutdown_lock;
+ bool adaptive_irq_polling;
bool subsystem;
u64 cmb_size;
bool cmb_use_sqes;
@@ -358,6 +398,22 @@ static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl)
return container_of(ctrl, struct nvme_dev, ctrl);
}
+/*
+ * Allocated only for interrupt queues that can poll adaptively. cq_poll_lock
+ * serializes this state and CQ access between the IRQ and timer poll paths.
+ */
+struct nvme_adaptive_poll {
+ struct hrtimer timer; /* fires the next poll drain */
+ struct irq_poll iopoll; /* softirq context for the drain */
+ struct nvme_queue *nvmeq;
+ u64 start_ns; /* when the current sample/episode started */
+ u32 retry_completions; /* completions until retry or IRQ rebaseline */
+ u32 interval_ns; /* sampled average gap between completions */
+ u32 completions; /* completions seen so far this sample/episode */
+ int irq;
+ u8 poll_failures; /* consecutive rejected polling trials */
+};
+
/*
* An NVM Express queue. Each device has at least two (one for admin
* commands and one for I/O commands).
@@ -367,7 +423,8 @@ struct nvme_queue {
struct nvme_descriptor_pools descriptor_pools;
spinlock_t sq_lock;
void *sq_cmds;
- /* only used for poll queues: */
+ struct nvme_adaptive_poll *adaptive;
+ /* Used for both poll queues and adaptive interrupt polling. */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
struct nvme_completion *cqes;
dma_addr_t sq_dma_addr;
@@ -386,6 +443,9 @@ struct nvme_queue {
#define NVMEQ_SQ_CMB 1
#define NVMEQ_DELETE_ERROR 2
#define NVMEQ_POLLED 3
+#define NVMEQ_ADAPTIVE_POLLING 4
+#define NVMEQ_ADAPTIVE_ENABLED 5
+#define NVMEQ_ADAPTIVE_STALE_IRQ 6
__le32 *dbbuf_sq_db;
__le32 *dbbuf_cq_db;
__le32 *dbbuf_sq_ei;
@@ -1606,13 +1666,12 @@ static inline void nvme_update_cq_head(struct nvme_queue *nvmeq)
}
}
-static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
- struct io_comp_batch *iob)
+static inline unsigned int nvme_poll_cq(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob)
{
- bool found = false;
+ unsigned int found = 0;
while (nvme_cqe_pending(nvmeq)) {
- found = true;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
@@ -1620,6 +1679,7 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
dma_rmb();
nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
nvme_update_cq_head(nvmeq);
+ found++;
}
if (found)
@@ -1627,17 +1687,22 @@ static inline bool nvme_poll_cq(struct nvme_queue *nvmeq,
return found;
}
-static irqreturn_t nvme_irq(int irq, void *data)
+/* Keep the normal completion loop branch-free. */
+static unsigned int nvme_poll_cq_bounded(struct nvme_queue *nvmeq,
+ struct io_comp_batch *iob,
+ unsigned int limit)
{
- struct nvme_queue *nvmeq = data;
- DEFINE_IO_COMP_BATCH(iob);
+ unsigned int found = 0;
- if (nvme_poll_cq(nvmeq, &iob)) {
- if (!rq_list_empty(&iob.req_list))
- nvme_pci_complete_batch(&iob);
- return IRQ_HANDLED;
+ while (found < limit && nvme_cqe_pending(nvmeq)) {
+ dma_rmb();
+ nvme_handle_cqe(nvmeq, iob, nvmeq->cq_head);
+ nvme_update_cq_head(nvmeq);
+ found++;
}
- return IRQ_NONE;
+ if (found)
+ nvme_ring_cq_doorbell(nvmeq);
+ return found;
}
static irqreturn_t nvme_irq_check(int irq, void *data)
@@ -1649,6 +1714,262 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
return IRQ_NONE;
}
+static void nvme_adaptive_state_reset(struct nvme_adaptive_poll *adaptive)
+{
+ adaptive->start_ns = 0;
+ adaptive->retry_completions = 0;
+ adaptive->interval_ns = 0;
+ adaptive->completions = 0;
+ adaptive->poll_failures = 0;
+}
+
+/*
+ * Stop polling and turn the queue's IRQ back on. Retry two rejected trials
+ * promptly so a noisy transition does not hide a profitable state; three
+ * consecutive failures use a long IRQ backoff before another baseline. The
+ * resulting low retry duty cycle prevents failed trials from reducing
+ * steady-state IRQ throughput. Called with cq_poll_lock held.
+ */
+static void nvme_adaptive_poll_end(struct nvme_queue *nvmeq, bool backoff)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u8 poll_failures = adaptive->poll_failures;
+
+ nvme_adaptive_state_reset(adaptive);
+ if (backoff) {
+ if (poll_failures < NVME_ADAPTIVE_POLL_RETRIES)
+ adaptive->poll_failures = poll_failures + 1;
+ else
+ adaptive->retry_completions =
+ NVME_ADAPTIVE_REEVAL_CQES;
+ }
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(adaptive->irq);
+}
+
+static void nvme_adaptive_poll_window_start(struct nvme_adaptive_poll *adaptive,
+ u64 now)
+{
+ adaptive->start_ns = now;
+ adaptive->completions = 0;
+}
+
+/*
+ * Fire the next poll one fixed period from now. A per-queue cadence is not
+ * needed: the cumulative-rate check below keeps a queue in poll mode only
+ * while it stays faster than its sampled IRQ rate, and a fixed period bounds
+ * the completion latency polling may add. Anchor the timer to the caller's
+ * "now" so per-poll processing time does not stretch the effective period.
+ */
+static void nvme_adaptive_arm(struct nvme_adaptive_poll *adaptive, u64 now)
+{
+ hrtimer_start(&adaptive->timer,
+ ns_to_ktime(now + NVME_ADAPTIVE_POLL_PERIOD_NS),
+ HRTIMER_MODE_ABS_PINNED_HARD);
+}
+
+static enum hrtimer_restart nvme_adaptive_poll_timer(struct hrtimer *timer)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(timer,
+ struct nvme_adaptive_poll, timer);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+
+ if (test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))
+ irq_poll_sched(&adaptive->iopoll);
+ return HRTIMER_NORESTART;
+}
+
+/*
+ * The poll drain, run from softirq when the timer fires. Reap some CQEs, then
+ * pick one of three things: stop if we hit the episode cap, wait again if the
+ * queue is keeping up, or go back to IRQ mode if it went idle or slowed down.
+ */
+static int nvme_adaptive_irq_poll(struct irq_poll *iop, int budget)
+{
+ struct nvme_adaptive_poll *adaptive = container_of(iop,
+ struct nvme_adaptive_poll, iopoll);
+ struct nvme_queue *nvmeq = adaptive->nvmeq;
+ unsigned int completions, limit;
+ unsigned long flags;
+ u64 deadline, elapsed, now;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (unlikely(!test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ completions = 0;
+ irq_poll_complete(iop);
+ goto out;
+ }
+ if (!test_bit(NVMEQ_ENABLED, &nvmeq->flags)) {
+ completions = 0;
+ irq_poll_complete(iop);
+ nvme_adaptive_poll_end(nvmeq, false);
+ goto out;
+ }
+
+ limit = min_t(unsigned int,
+ budget,
+ NVME_ADAPTIVE_EPISODE_CQES - adaptive->completions);
+ completions = nvme_poll_cq_bounded(nvmeq, &iob, limit);
+ adaptive->completions += completions;
+
+ if (completions >= budget &&
+ adaptive->completions < NVME_ADAPTIVE_EPISODE_CQES)
+ goto out;
+ irq_poll_complete(iop);
+
+ /*
+ * Before the window fills, tolerate two poll periods of cumulative lag
+ * to absorb timer jitter and bursty completions. At the window boundary,
+ * remove that slack and require a strictly shorter elapsed time than the
+ * sampled IRQ interval. interval_ns is rounded down, so this final test
+ * cannot accept polling that is equal to or slower than the IRQ baseline.
+ */
+ now = ktime_get_ns();
+ elapsed = now - adaptive->start_ns;
+ deadline = (u64)adaptive->completions * adaptive->interval_ns;
+ if (adaptive->completions < NVME_ADAPTIVE_EPISODE_CQES) {
+ if (elapsed > deadline +
+ 2U * NVME_ADAPTIVE_POLL_PERIOD_NS)
+ nvme_adaptive_poll_end(nvmeq, true);
+ else
+ nvme_adaptive_arm(adaptive, now);
+ goto out;
+ }
+ if (elapsed >= deadline) {
+ nvme_adaptive_poll_end(nvmeq, true);
+ goto out;
+ }
+
+ adaptive->poll_failures = 0;
+ /* Reuse the retry counter, updating it once per window. */
+ adaptive->retry_completions -= NVME_ADAPTIVE_EPISODE_CQES;
+ if (!adaptive->retry_completions) {
+ nvme_adaptive_poll_end(nvmeq, false);
+ goto out;
+ }
+ nvme_adaptive_poll_window_start(adaptive, now);
+ nvme_adaptive_arm(adaptive, now);
+out:
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return completions;
+}
+
+/*
+ * Called from the IRQ handler after a reap that found something. If we're
+ * still in backoff, just count it down. Otherwise establish an IRQ completion
+ * rate over one episode. If that rate is high enough that a poll every
+ * NVME_ADAPTIVE_POLL_PERIOD_NS would still find work, mask the IRQ and switch
+ * to a same-sized polling trial; if not, leave the queue on interrupts.
+ *
+ * Called with nvmeq->cq_poll_lock held.
+ */
+static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
+ unsigned int completions)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ u64 delta, interval, now;
+
+ if (adaptive->retry_completions) {
+ adaptive->retry_completions -= min(completions,
+ adaptive->retry_completions);
+ return;
+ }
+ if (!adaptive->start_ns) {
+ adaptive->start_ns = ktime_get_ns();
+ return;
+ }
+ adaptive->completions += completions;
+ if (adaptive->completions < NVME_ADAPTIVE_EPISODE_CQES)
+ return;
+
+ now = ktime_get_ns();
+ delta = now - adaptive->start_ns;
+ /*
+ * The completion-rate test is an admission gate, not the trial result.
+ * Comparing a full IRQ window against a full poll window below decides
+ * whether polling actually pays for itself on this queue.
+ */
+ interval = div64_u64(delta, adaptive->completions);
+ if (!interval || interval > NVME_ADAPTIVE_POLL_PERIOD_NS ||
+ !test_bit(NVMEQ_ENABLED, &nvmeq->flags)) {
+ nvme_adaptive_poll_window_start(adaptive, now);
+ return;
+ }
+
+ adaptive->interval_ns = interval;
+ adaptive->retry_completions = NVME_ADAPTIVE_REEVAL_CQES;
+ nvme_adaptive_poll_window_start(adaptive, now);
+ set_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ disable_irq_nosync(adaptive->irq);
+ nvme_adaptive_arm(adaptive, now);
+}
+
+static irqreturn_t nvme_irq(int irq, void *data);
+
+/*
+ * IRQ handler for queues that may switch to adaptive polling. It reaps the CQ
+ * like the normal handler, then feeds the count to the sampler, which may flip
+ * the queue into poll mode. The CQ lock makes an IRQ racing with the timer
+ * harmless: it either drains the CQ before the switch or observes poll mode
+ * and leaves the CQ to the timer. An empty IRQ after polling is the interrupt
+ * that was pending when the vector was masked, so consume it as handled.
+ */
+static noinline irqreturn_t nvme_irq_adaptive_enabled(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ unsigned int completions;
+ unsigned long flags;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (unlikely(test_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags))) {
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ return IRQ_HANDLED;
+ }
+ completions = nvme_poll_cq(nvmeq, &iob);
+ if (completions)
+ nvme_adaptive_sample(nvmeq, completions);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ if (!completions)
+ return test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ,
+ &nvmeq->flags) ? IRQ_HANDLED : IRQ_NONE;
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t nvme_irq_adaptive(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ irqreturn_t ret;
+
+ if (!test_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags)) {
+ ret = nvme_irq(irq, data);
+ if (ret == IRQ_NONE &&
+ test_and_clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags))
+ return IRQ_HANDLED;
+ return ret;
+ }
+ return nvme_irq_adaptive_enabled(irq, data);
+}
+
+static irqreturn_t nvme_irq(int irq, void *data)
+{
+ struct nvme_queue *nvmeq = data;
+ DEFINE_IO_COMP_BATCH(iob);
+
+ if (nvme_poll_cq(nvmeq, &iob)) {
+ if (!rq_list_empty(&iob.req_list))
+ nvme_pci_complete_batch(&iob);
+ return IRQ_HANDLED;
+ }
+ return IRQ_NONE;
+}
+
/*
* Poll for completions for any interrupt driven queue
* Can be called from any context.
@@ -1656,30 +1977,38 @@ static irqreturn_t nvme_irq_check(int irq, void *data)
static void nvme_poll_irqdisable(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+ unsigned long flags;
int irq;
WARN_ON_ONCE(test_bit(NVMEQ_POLLED, &nvmeq->flags));
irq = pci_irq_vector(pdev, nvmeq->cq_vector);
disable_irq(irq);
- spin_lock(&nvmeq->cq_poll_lock);
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
nvme_poll_cq(nvmeq, NULL);
- spin_unlock(&nvmeq->cq_poll_lock);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
enable_irq(irq);
}
static int nvme_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
{
struct nvme_queue *nvmeq = hctx->driver_data;
+ unsigned long flags;
bool found;
if (!test_bit(NVMEQ_POLLED, &nvmeq->flags) ||
!nvme_cqe_pending(nvmeq))
return 0;
- spin_lock(&nvmeq->cq_poll_lock);
+ /*
+ * cq_poll_lock is also acquired from hardirq context by adaptive IRQ
+ * polling on interrupt-driven queues. Disable IRQs here so all
+ * acquirers share a consistent context and lockdep cannot see an
+ * IRQ-safe class taken with IRQs enabled.
+ */
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
found = nvme_poll_cq(nvmeq, iob);
- spin_unlock(&nvmeq->cq_poll_lock);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
return found;
}
@@ -2017,8 +2346,7 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
if (!nvmeq->sq_cmds)
- return;
-
+ goto free_adaptive;
if (test_and_clear_bit(NVMEQ_SQ_CMB, &nvmeq->flags)) {
pci_free_p2pmem(to_pci_dev(nvmeq->dev->dev),
nvmeq->sq_cmds, SQ_SIZE(nvmeq));
@@ -2026,6 +2354,9 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
dma_free_coherent(nvmeq->dev->dev, SQ_SIZE(nvmeq),
nvmeq->sq_cmds, nvmeq->sq_dma_addr);
}
+free_adaptive:
+ kfree(nvmeq->adaptive);
+ nvmeq->adaptive = NULL;
}
static void nvme_free_queues(struct nvme_dev *dev, int lowest)
@@ -2038,9 +2369,97 @@ static void nvme_free_queues(struct nvme_dev *dev, int lowest)
}
}
+static int nvme_adaptive_suspend(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+ int irq;
+
+ if (!adaptive || adaptive->irq < 0)
+ return -1;
+ irq = adaptive->irq;
+ synchronize_irq(irq);
+ irq_poll_disable(&adaptive->iopoll);
+ /* irq_poll_complete() can run before the poll callback returns. */
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ if (test_and_clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags)) {
+ set_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ enable_irq(irq);
+ }
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ hrtimer_cancel(&adaptive->timer);
+ return irq;
+}
+
+static void nvme_adaptive_set_queue(struct nvme_queue *nvmeq, bool enable)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ unsigned long flags;
+
+ if (nvme_adaptive_suspend(nvmeq) < 0) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ return;
+ }
+
+ spin_lock_irqsave(&nvmeq->cq_poll_lock, flags);
+ nvme_adaptive_state_reset(adaptive);
+ if (enable)
+ set_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ else
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ spin_unlock_irqrestore(&nvmeq->cq_poll_lock, flags);
+ irq_poll_enable(&adaptive->iopoll);
+}
+
+/*
+ * Freeze and drain namespace I/O before changing the completion mode. The
+ * locks keep namespace and controller teardown paths from changing the queue
+ * set while its IRQ state is updated.
+ */
+static int nvme_adaptive_switch(struct nvme_dev *dev, bool enable)
+{
+ int qid, ret = 0;
+
+ mutex_lock(&dev->ctrl.scan_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ goto out_unlock;
+ }
+ if (enable == READ_ONCE(dev->adaptive_irq_polling))
+ goto out_unlock;
+
+ nvme_start_freeze(&dev->ctrl);
+ nvme_wait_freeze(&dev->ctrl);
+
+ mutex_lock(&dev->shutdown_lock);
+ if (nvme_ctrl_state(&dev->ctrl) != NVME_CTRL_LIVE) {
+ ret = -EBUSY;
+ } else {
+ for (qid = 1; qid < dev->ctrl.queue_count; qid++)
+ nvme_adaptive_set_queue(&dev->queues[qid], enable);
+ WRITE_ONCE(dev->adaptive_irq_polling, enable);
+ }
+ mutex_unlock(&dev->shutdown_lock);
+
+ nvme_unfreeze(&dev->ctrl);
+out_unlock:
+ mutex_unlock(&dev->ctrl.scan_lock);
+ return ret;
+}
+
+static void nvme_adaptive_suspend_done(struct nvme_queue *nvmeq, int irq)
+{
+ if (irq < 0)
+ return;
+ nvmeq->adaptive->irq = -1;
+ irq_poll_enable(&nvmeq->adaptive->iopoll);
+}
+
static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
{
struct nvme_queue *nvmeq = &dev->queues[qid];
+ struct pci_dev *pdev = to_pci_dev(dev->dev);
+ int irq;
if (!test_and_clear_bit(NVMEQ_ENABLED, &nvmeq->flags))
return;
@@ -2051,8 +2470,11 @@ static void nvme_suspend_queue(struct nvme_dev *dev, unsigned int qid)
nvmeq->dev->online_queues--;
if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q)
nvme_quiesce_admin_queue(&nvmeq->dev->ctrl);
- if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags))
- pci_free_irq(to_pci_dev(dev->dev), nvmeq->cq_vector, nvmeq);
+ if (!test_and_clear_bit(NVMEQ_POLLED, &nvmeq->flags)) {
+ irq = nvme_adaptive_suspend(nvmeq);
+ pci_free_irq(pdev, nvmeq->cq_vector, nvmeq);
+ nvme_adaptive_suspend_done(nvmeq, irq);
+ }
}
static void nvme_suspend_io_queues(struct nvme_dev *dev)
@@ -2071,12 +2493,13 @@ static void nvme_suspend_io_queues(struct nvme_dev *dev)
*/
static void nvme_reap_pending_cqes(struct nvme_dev *dev)
{
+ unsigned long flags;
int i;
for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
- spin_lock(&dev->queues[i].cq_poll_lock);
+ spin_lock_irqsave(&dev->queues[i].cq_poll_lock, flags);
nvme_poll_cq(&dev->queues[i], NULL);
- spin_unlock(&dev->queues[i].cq_poll_lock);
+ spin_unlock_irqrestore(&dev->queues[i].cq_poll_lock, flags);
}
}
@@ -2166,18 +2589,75 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
return -ENOMEM;
}
+static bool nvme_adaptive_init(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+ int irq = pci_irq_vector(to_pci_dev(nvmeq->dev->dev),
+ nvmeq->cq_vector);
+
+ if (irq < 0)
+ return false;
+ if (!adaptive) {
+ adaptive = kzalloc_node(sizeof(*adaptive), GFP_KERNEL,
+ dev_to_node(nvmeq->dev->dev));
+ if (!adaptive)
+ return false;
+ adaptive->nvmeq = nvmeq;
+ hrtimer_setup(&adaptive->timer, nvme_adaptive_poll_timer,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
+ irq_poll_init(&adaptive->iopoll, 64, nvme_adaptive_irq_poll);
+ adaptive->irq = irq;
+ WRITE_ONCE(nvmeq->adaptive, adaptive);
+ return true;
+ }
+ adaptive->irq = irq;
+ return true;
+}
+
static int queue_request_irq(struct nvme_queue *nvmeq)
{
struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
int nr = nvmeq->dev->ctrl.instance;
+ bool adaptive_queue;
+ int ret;
if (use_threaded_interrupts) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
- } else {
- return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq,
- NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
}
+ /* Decide static eligibility once, before installing the IRQ handler. */
+ adaptive_queue = nvmeq->qid && nvmeq->dev->num_vecs > 1 &&
+ pdev->msix_enabled &&
+ nvmeq->q_depth >= NVME_ADAPTIVE_TARGET_BATCH;
+ if (adaptive_queue)
+ adaptive_queue = nvme_adaptive_init(nvmeq);
+ if (adaptive_queue && READ_ONCE(nvmeq->dev->adaptive_irq_polling))
+ set_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ else
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ ret = pci_request_irq(pdev, nvmeq->cq_vector,
+ adaptive_queue ? nvme_irq_adaptive : nvme_irq,
+ NULL, nvmeq, "nvme%dq%d", nr, nvmeq->qid);
+ if (!adaptive_queue || ret) {
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ if (ret && nvmeq->adaptive)
+ nvmeq->adaptive->irq = -1;
+ }
+ return ret;
+}
+
+static void nvme_adaptive_reset(struct nvme_queue *nvmeq)
+{
+ struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
+
+ clear_bit(NVMEQ_ADAPTIVE_POLLING, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_ENABLED, &nvmeq->flags);
+ clear_bit(NVMEQ_ADAPTIVE_STALE_IRQ, &nvmeq->flags);
+ if (!adaptive)
+ return;
+ nvme_adaptive_state_reset(adaptive);
+ adaptive->irq = -1;
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
@@ -2188,6 +2668,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
nvmeq->last_sq_tail = 0;
nvmeq->cq_head = 0;
nvmeq->cq_phase = 1;
+ nvme_adaptive_reset(nvmeq);
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq));
nvme_dbbuf_init(dev, nvmeq, qid);
@@ -2808,6 +3289,33 @@ static ssize_t hmb_store(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RW(hmb);
+static ssize_t adaptive_irq_polling_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+
+ return sysfs_emit(buf, "%d\n", READ_ONCE(ndev->adaptive_irq_polling));
+}
+
+static ssize_t adaptive_irq_polling_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct nvme_dev *ndev = to_nvme_dev(dev_get_drvdata(dev));
+ bool enable;
+ int ret;
+
+ ret = kstrtobool(buf, &enable);
+ if (ret)
+ return ret;
+ ret = nvme_adaptive_switch(ndev, enable);
+ if (ret)
+ return ret;
+ return count;
+}
+static DEVICE_ATTR_RW(adaptive_irq_polling);
+
static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
struct attribute *a, int n)
{
@@ -2823,6 +3331,8 @@ static umode_t nvme_pci_attrs_are_visible(struct kobject *kobj,
}
if (a == &dev_attr_hmb.attr && !ctrl->hmpre)
return 0;
+ if (a == &dev_attr_adaptive_irq_polling.attr && use_threaded_interrupts)
+ return 0;
return a->mode;
}
@@ -2832,6 +3342,7 @@ static struct attribute *nvme_pci_attrs[] = {
&dev_attr_cmbloc.attr,
&dev_attr_cmbsz.attr,
&dev_attr_hmb.attr,
+ &dev_attr_adaptive_irq_polling.attr,
NULL,
};
@@ -3685,6 +4196,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
return ERR_PTR(-ENOMEM);
INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work);
mutex_init(&dev->shutdown_lock);
+ dev->adaptive_irq_polling = use_adaptive_irq_polling;
dev->nr_write_queues = write_queues;
dev->nr_poll_queues = poll_queues;
--
2.39.5 (Apple Git-154)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-11 20:29 ` Keith Busch
@ 2026-08-13 2:11 ` changfengnan
2026-08-13 3:45 ` Anuj Gupta
0 siblings, 1 reply; 9+ messages in thread
From: changfengnan @ 2026-08-13 2:11 UTC (permalink / raw)
To: Keith Busch; +Cc: axboe, hch, sagi, linux-nvme, linux-kernel, Guzebing
> From: "Keith Busch"<kbusch@kernel.org>
> Date: Wed, Aug 12, 2026, 04:30
> Subject: Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> To: "changfengnan"<changfengnan@bytedance.com>
> Cc: <axboe@kernel.dk>, <hch@lst.de>, <sagi@grimberg.me>, <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>, "Guzebing"<guzebing@bytedance.com>
> On Tue, Aug 11, 2026 at 10:32:35AM +0800, changfengnan wrote:
> > The code looks much cleaner when using dim, I'll see if I can replace the
> > sample-and-poll logic I wrote myself with the dim library.
> > I ran a quick test on the POC patch and didn't see any performance
> > improvements; in fact, there were quite a few regressions. Maybe some
> > parameters need to be adjusted.
>
> Yeah, I'm just now testing mid-tier devices and it's also performing a
> bit worse for high throughput workloads. But I hadn't really tried to
> tune the settings here, and maybe my criteria is all wrong. I was mainly
> trying to see if we can utilize the dim library before honing in on the
> right implementation details.
I tried modifying the original approach to use DIM, but the results were
consistently poor. Using the original approach, my current optimization
has managed to keep 4K random read backoff within 2% (tested on 10
different drive models), but when using DIM, the backoff rate is 10–15%.
I believe there are several reasons for this:
1. The core of NVMe adaptive polling is determining when polling is more
efficient, but DIM cannot answer this question. Although irq and poll can
be disguised as a DIM profile, this is essentially just borrowing the DIM
framework—the core decision-making still requires writing your own baseline,
thresholds, backoff, and re-baseline, so the code complexity and volume
haven’t actually decreased.
2. The DIM state machine is relatively heavy, which reduces performance
gains under I/O-intensive workloads.
So I tend to avoid using DIM. Please correct me if I'm wrong.
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-13 2:11 ` changfengnan
@ 2026-08-13 3:45 ` Anuj Gupta
2026-08-14 22:16 ` Keith Busch
0 siblings, 1 reply; 9+ messages in thread
From: Anuj Gupta @ 2026-08-13 3:45 UTC (permalink / raw)
To: changfengnan
Cc: Keith Busch, axboe, hch, sagi, linux-nvme, linux-kernel,
Guzebing, anuj20.g
On Thu, Aug 13, 2026 at 10:11:19AM +0800, changfengnan wrote:
>
> > From: "Keith Busch"<kbusch@kernel.org>
> > Date: Wed, Aug 12, 2026, 04:30
> > Subject: Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> > To: "changfengnan"<changfengnan@bytedance.com>
> > Cc: <axboe@kernel.dk>, <hch@lst.de>, <sagi@grimberg.me>, <linux-nvme@lists.infradead.org>, <linux-kernel@vger.kernel.org>, "Guzebing"<guzebing@bytedance.com>
> > On Tue, Aug 11, 2026 at 10:32:35AM +0800, changfengnan wrote:
> > > The code looks much cleaner when using dim, I'll see if I can replace the
> > > sample-and-poll logic I wrote myself with the dim library.
> > > I ran a quick test on the POC patch and didn't see any performance
> > > improvements; in fact, there were quite a few regressions. Maybe some
> > > parameters need to be adjusted.
> >
> > Yeah, I'm just now testing mid-tier devices and it's also performing a
> > bit worse for high throughput workloads. But I hadn't really tried to
> > tune the settings here, and maybe my criteria is all wrong. I was mainly
> > trying to see if we can utilize the dim library before honing in on the
> > right implementation details.
>
> I tried modifying the original approach to use DIM, but the results were
> consistently poor. Using the original approach, my current optimization
> has managed to keep 4K random read backoff within 2% (tested on 10
> different drive models), but when using DIM, the backoff rate is 10–15%.
> I believe there are several reasons for this:
> 1. The core of NVMe adaptive polling is determining when polling is more
> efficient, but DIM cannot answer this question. Although irq and poll can
> be disguised as a DIM profile, this is essentially just borrowing the DIM
> framework—the core decision-making still requires writing your own baseline,
> thresholds, backoff, and re-baseline, so the code complexity and volume
> haven’t actually decreased.
> 2. The DIM state machine is relatively heavy, which reduces performance
> gains under I/O-intensive workloads.
>
> So I tend to avoid using DIM. Please correct me if I'm wrong.
I also had a similar assesment on DIM. Emperically, with Keith's POC
interrupts per IO drop from ~1.0 to ~0.29 but with a small performance
regression (572K -> 557K IOPS). With your original approach they drop to
~0.03 with a 23% performance gain. The DIM machinery overhead maybe
outweighing the partial interrupt reduction.
>
> >
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
2026-08-13 3:45 ` Anuj Gupta
@ 2026-08-14 22:16 ` Keith Busch
0 siblings, 0 replies; 9+ messages in thread
From: Keith Busch @ 2026-08-14 22:16 UTC (permalink / raw)
To: Anuj Gupta
Cc: changfengnan, axboe, hch, sagi, linux-nvme, linux-kernel,
Guzebing, anuj20.g
On Thu, Aug 13, 2026 at 09:15:31AM +0530, Anuj Gupta wrote:
>
> I also had a similar assesment on DIM. Emperically, with Keith's POC
> interrupts per IO drop from ~1.0 to ~0.29 but with a small performance
> regression (572K -> 557K IOPS). With your original approach they drop to
> ~0.03 with a 23% performance gain. The DIM machinery overhead maybe
> outweighing the partial interrupt reduction.
I may ultimately adandon this suggestion, but it looks like a major
factor in performance loss was from using IRQ_WAKE_THREAD with sleep
cycles instead of the softirq hrtimer. Just changing that is getting me
back to performance parity, but since it's not any better, perhaps the
hill climb is a worse algorithm than trying to reason out the delayed
poll time directly.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-08-14 22:16 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-08-06 3:10 [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling Fengnan Chang
2026-08-10 20:55 ` Keith Busch
2026-08-11 2:32 ` changfengnan
2026-08-11 20:29 ` Keith Busch
2026-08-13 2:11 ` changfengnan
2026-08-13 3:45 ` Anuj Gupta
2026-08-14 22:16 ` Keith Busch
2026-08-12 17:50 ` Anuj Gupta
2026-08-13 2:04 ` changfengnan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®