Re: [PATCH v5 2/2] nvme-tcp: allow setting per-queue io_cpu through sysfs

[Hannes Reinecke <hare@suse.de>]

On 10/20/26 5:28 PM, Daniel Wagner wrote:
> On Sat, Oct 17, 2026 at 06:00:59PM -0700, Saravanan D wrote:
>> nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time
>> as the least loaded CPU in the queue's blk-mq map group, and all socket
>> work runs there for the connection's lifetime. This decision falls
>> short when the host partitions its CPUs after connect time. On a 384
>> cpu multi tenant host with 128 queue controllers, blk-mq folds three
>> CPUs into every map group, some groups straddle two tenants' cpusets,
>> and 9% of nvme_tcp_io_work executions ran outside the submitting VM's
>> cpuset, seen by the neighbor as steal time.
>>
>> Expose the io_cpu in each I/O queue's directory
>>
>>    /sys/class/nvme/nvmeX/queues/<qid>/io_cpu
>>
>> so a control plane that owns CPU placement can set it directly instead
>> of relying on the driver's heuristic. The attribute accepts a decimal
>> cpu number, -1 or "unbound". A user assignment, unbound included, is
>> marked in the queue flags, persists across reconnects and is never
>> reselected by the driver. The read-only managed attribute reports 1
>> while io_cpu is driver managed and 0 once the user assigned it. The
>> accounting of nvme_tcp_cpu_queues applies regardless of who set the
>> io_cpu.
>>
>> A queue's directory is added when the queue is allocated and removed
>> when it is freed, before the queue is torn down, so a sysfs store
>> never runs on a freed queue. The queues directory is created before
>> the I/O queues are set up in parallel. All io_cpu writes, from sysfs
>> and from the driver, serialize with the accounting under
>> nvme_tcp_cpu_queues_lock, while the I/O paths read io_cpu locklessly.
>> The queue stop now clears NVME_TCP_Q_IO_CPU_SET under that lock, since
>> a sysfs store may move io_cpu in between.
>>
>> nvme_tcp_free_queue() now clears NVME_TCP_Q_IO_CPU_SET and drops its
>> count, because a queue can still be counted when it is freed. A sysfs
>> store may have run after the queue stop, or the connect may have
>> failed, and the setup error path frees such a queue without calling
>> nvme_tcp_stop_queue(), where the count is normally dropped.
>   
> This looks good, just trying to wrap my head around with isolcpu (yeah,
> doing a bit of work in that area makes me a paranoid).
> 
> I think it make sense to allow that the user can change queue CPU
> pinning during runtime even with isolcpu enabled. And then the question
> is should this be connected to cgroups?

Not sure if cgroups is the right concept here; cgroups are user/
application centric whereas nvme-tcp is pretty much queue/cpu centric.
IE after all the cgroup calculations are done.
Don't think cgroups are a good fit here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich