[PATCH 0/2] ksmbd: fix named-pipe reply loss and operation races

[Wang Zhaolong <wangzhaolong3@outlook.com>]

This series fixes two related problems in the ksmbd SMB2 named-pipe
implementation.

The first problem is that RPC IPC replies are matched only by the pipe
handle. READ, WRITE, FSCTL_PIPE_TRANSCEIVE and CLOSE on one pipe all use
that handle, and the session RPC lock is dropped before an IPC request
waits for its reply. There is no separate IPC correlation ID, so two
operations can overlap and interfere with reply delivery. CLOSE can also
release the handle while an earlier operation is still in flight.
Protecting only the handle table is not enough: an active pipe needs
per-pipe serialization and lifetime.

Patch 1 serializes operations per pipe, keeps the pipe state and handle
alive until all users finish, and rechecks cancellation, connection
release and session teardown after taking the pipe lock. It erases a
closed pipe before waiting for its active operation and releases the
handle id only after the last user. Session logoff now clears the RPC
pipe list after the other requests have drained. Failed pipe operations
use standard SMB2 error replies while preserving the selected status.

The second problem is reply loss with small output buffers.
FSCTL_PIPE_TRANSCEIVE copies at most MaxOutputResponse bytes and returns
STATUS_BUFFER_OVERFLOW when the RPC reply is larger, but then frees the
whole IPC reply. The unread bytes are no longer available to a follow-up
READ, so the RPC call can fail even though userspace produced a complete
reply. READ also ignores the requested Length when copying an IPC reply.

Patch 2 builds on the per-pipe lifetime from patch 1 and keeps the
original reply allocation and an offset on the pipe.
READ and TRANSCEIVE consume only the bytes that fit and retain the rest.
The retained reply is freed only when drained, or when the pipe is
closed, logged off or disconnected. A failed response preparation or
input request does not discard output the client has not received. An
IPC failure is reported as an I/O error instead of an empty read or a
successful write without a reply. No kernel RPC queue is added;
additional reply queueing remains userspace's responsibility.

Reproduction:

- A reproduced NetShareEnum case produced a 2632-byte RPC reply. With a
  1024-byte MaxOutputResponse the baseline returned 1024 bytes and
  discarded the remaining 1608 bytes; a follow-up READ returned
  STATUS_UNEXPECTED_IO_ERROR. With a 512-byte output limit, 2120 bytes
  were lost.
- The same overlap/close sequence was exercised with READ, WRITE and
  TRANSCEIVE on one handle, including CANCEL and LOGOFF.

Wang Zhaolong (2):
  ksmbd: avoid races between named-pipe operations
  ksmbd: fix named-pipe reply loss with small output buffers

 fs/smb/server/mgmt/user_session.c | 127 +++++++++-----
 fs/smb/server/mgmt/user_session.h |  18 ++
 fs/smb/server/smb2pdu.c           | 283 ++++++++++++++++++++----------
 fs/smb/server/transport_ipc.c     |  45 +++--
 fs/smb/server/transport_ipc.h     |  14 +-
 5 files changed, 323 insertions(+), 164 deletions(-)


base-commit: f47d223627f80cb7a1eb14c585fe6cb35de5e984
-- 
2.56.0