[RFC] ksmbd: Deprecate MD5 support and enhance AES-GCM for SMB 3.1.1 compliance

[Yunseong Kim <ysk@kzalloc.com>]

Hi all,

I'm looking into contributing to the ksmbd crypto module, specifically
around crypto handling in crypto_ctx.c. I wanted to send this RFC to gauge
interest and get feedback before preparing patches.

First, regarding MD5 support: The current code includes HMAC-MD5
(via crypto_alloc_shash("hmac(md5)")) which appears to be for legacy SMB1
compatibility. SMB1 is widely deprecated due to security issues, and MD5
itself is vulnerable to collision attacks, making it unsuitable for modern
use. I propose deprecating or removing this support entirely, perhaps with
a config option (e.g., CONFIG_KSMBD_LEGACY_SMB1) for those who absolutely
need it, but defaulting to off. This would align ksmbd with security best
practices, similar to how Windows has disabled SMB1 by default.

Second, for SMB 3.1.1 compliance: The code already supports AES-GCM via
crypto_alloc_aead("gcm(aes)"), but to fully adhere to the spec (MS-SMB2),
we should explicitly handle AES-128-GCM as the default cipher, with
AES-256-GCM as an optional stronger variant. AES-256-GCM isn't mandatory
but is recommended for higher security (e.g., in Windows Server 2022+).

This would involve:
 - Adding key length checks and setkey logic in the caller side
   (e.g., negotiate or session setup).
 - Updating the negotiate context to include cipher selection
   (0x0001 for AES-128-GCM, 0x0002 for AES-256-GCM).
 - Potentially separating signing (AES-CMAC) from encryption ciphers for
   clarity.

Is this direction worth pursuing? I'd like to prepare patches for review
if there's consensus. Any thoughts on priorities, potential pitfalls, or
related work in progress?

Thanks for your time.

Yunseong