Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup

[Hannes Reinecke <hare@suse.de>]

On 10/30/26 10:25 PM, Surabhi Gogte (she/her) wrote:
> On Mon, Oct 26, 2026 at 3:57 AM Martin Belanger
> <martin.nvme.dev@gmail.com> wrote:
>>
>> On 9/22/26 16:07, Surabhi Gogte wrote:
>>   > Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
>>   > allocation and startup"), refactor nvme tcp I/O queue setup to use async
>>   > API, combining allocation and startup into a single parallel operation
>>   > per queue. This reduces connection and reconnection setup time when
>>   > there are delays in establishing connections, which is especially
>>   > important for high-core-count hosts.
>> [...]
>>
>> Hi Surabhi,
>>
>> This patch breaks NVMe/TCP connects with TLS. This is a different bug
>> from the pf_cache_lock one that Shin'ichiro reported.
>>
>> Test: one VM with 4 CPUs, nvmet-tcp on 127.0.0.1 with tsas tls1.3, a
>> connect with --tls and 4 I/O queues, and tlshd at upstream commit
>> a0bd51d.
>>
>> - Linus master 6b4b543fa187: 14 of 20 connects fail.
>> - Same tree with this patch reverted: 20 of 20 connects pass.
>> - Same tree with Daniel's pf_cache_lock fix [1]: 22 of 40 and 17 of 40
>>     connects fail.
>>
>> On each failed connect, one I/O queue logs:
>>
>>     nvme nvme0: queue 4: TLS handshake complete, error -110
>>
>> The tlshd that runs for the target logs:
>>
>>     failed to search 'psk' key 'NVMe0R01 <hostnqn> <subnqn>'
>>     gnutls: Error in password/key file. (-31)
>>
>> and closes the connection. The tlshd that runs for the host then gets
>> GNUTLS_E_PREMATURE_TERMINATION, which it reports to the kernel as
>> ETIMEDOUT. The other queues use the same PSK, and their handshakes
>> succeed. blktests nvme/062 and nvme/063 fail the same way.
>>
>> I can share the reproducer script.
> 
> Hey Martin, can you try the tests once with tlshd using commit 3a6658e
> ("tlshd: Link the NFS keyrings per handshake instead of at startup")
> or ktls-utils 1.5. blktests nvme/062 and nvme/063 passed with this
> commit. I believe that change fixed the keyring race in tlshd
> independently. If the issue persists, let me know the reproducer
> script so I can also take a look. I saw the same errors while running
> blktests with an older tlshd version from ktls-utils 1.3.1 and
> updating tlshd worked.
> 
Ouch. That will be tricky for distros. If there is a specific commit
which requires an update to ktlshd we need to have a way to identify
that; otherwise we'll see lots of regressions where customers have
updated only one side.
Please modify the patch to work with older versions of ktlshd.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich