[RFC] ksmbd: Deprecate MD5 support and enhance AES-GCM for SMB 3.1.1 compliance

[RFC] ksmbd: Deprecate MD5 support and enhance AES-GCM for SMB 3.1.1 compliance

[Yunseong Kim]

Hi all,

I'm looking into contributing to the ksmbd crypto module, specifically
around crypto handling in crypto_ctx.c. I wanted to send this RFC to gauge
interest and get feedback before preparing patches.

First, regarding MD5 support: The current code includes HMAC-MD5
(via crypto_alloc_shash("hmac(md5)")) which appears to be for legacy SMB1
compatibility. SMB1 is widely deprecated due to security issues, and MD5
itself is vulnerable to collision attacks, making it unsuitable for modern
use. I propose deprecating or removing this support entirely, perhaps with
a config option (e.g., CONFIG_KSMBD_LEGACY_SMB1) for those who absolutely
need it, but defaulting to off. This would align ksmbd with security best
practices, similar to how Windows has disabled SMB1 by default.

Second, for SMB 3.1.1 compliance: The code already supports AES-GCM via
crypto_alloc_aead("gcm(aes)"), but to fully adhere to the spec (MS-SMB2),
we should explicitly handle AES-128-GCM as the default cipher, with
AES-256-GCM as an optional stronger variant. AES-256-GCM isn't mandatory
but is recommended for higher security (e.g., in Windows Server 2022+).

This would involve:
 - Adding key length checks and setkey logic in the caller side
   (e.g., negotiate or session setup).
 - Updating the negotiate context to include cipher selection
   (0x0001 for AES-128-GCM, 0x0002 for AES-256-GCM).
 - Potentially separating signing (AES-CMAC) from encryption ciphers for
   clarity.

Is this direction worth pursuing? I'd like to prepare patches for review
if there's consensus. Any thoughts on priorities, potential pitfalls, or
related work in progress?

Thanks for your time.

Yunseong

Re: [RFC] ksmbd: Deprecate MD5 support and enhance AES-GCM for SMB 3.1.1 compliance

[Steve French]

> current code includes HMAC-MD5 (via crypto_alloc_shash("hmac(md5)")) which
> appears to be for legacy SMB1 compatibility.

NTLMv2 uses HMAC-MD5 to compute the challenge response, replacing the
weaker DES algorithm used in NTLMv1.   With long passwords AFAIK there
is little security risk if any in using MD5 in such a narrow case, but
I don't think it could be removed without breaking typical mounts
(with the move to IAKERB and Peer-to-Peer Kerberos, not just domain
joined KRB5 which has been supported for years, this may be less of a
problem in a few years as Macs and soon Samba and Windows will support
IAKERB and Peer-to-Peer krb5 as alternatives to the more common
NTLMV2/NTLMSSP mounts).

On your other question, yes these are worth investigating.  For
example the server should be able to support standard AES-128-GCM
encryption AND "military grade" AES-256-GCM encryption - as most
clients (including LInux) can require mounting with AES-256-GCM in
some cases, so not good enough to just support AES-128-GCM, but I was
more concerned about making sure the faster signing algorithm was
supported on both Linux client and server (today e.g. mounting from
Linux client due to lack of support for faster signing algorithm it is
actually faster to mount with "seal" (encryption) than "sign")


On Mon, Sep 15, 2025 at 6:07 PM Yunseong Kim <ysk@kzalloc.com> wrote:
>
> Hi all,
>
> I'm looking into contributing to the ksmbd crypto module, specifically
> around crypto handling in crypto_ctx.c. I wanted to send this RFC to gauge
> interest and get feedback before preparing patches.
>
> First, regarding MD5 support: The current code includes HMAC-MD5
> (via crypto_alloc_shash("hmac(md5)")) which appears to be for legacy SMB1
> compatibility. SMB1 is widely deprecated due to security issues, and MD5
> itself is vulnerable to collision attacks, making it unsuitable for modern
> use. I propose deprecating or removing this support entirely, perhaps with
> a config option (e.g., CONFIG_KSMBD_LEGACY_SMB1) for those who absolutely
> need it, but defaulting to off. This would align ksmbd with security best
> practices, similar to how Windows has disabled SMB1 by default.
>
> Second, for SMB 3.1.1 compliance: The code already supports AES-GCM via
> crypto_alloc_aead("gcm(aes)"), but to fully adhere to the spec (MS-SMB2),
> we should explicitly handle AES-128-GCM as the default cipher, with
> AES-256-GCM as an optional stronger variant. AES-256-GCM isn't mandatory
> but is recommended for higher security (e.g., in Windows Server 2022+).
>
> This would involve:
>  - Adding key length checks and setkey logic in the caller side
>    (e.g., negotiate or session setup).
>  - Updating the negotiate context to include cipher selection
>    (0x0001 for AES-128-GCM, 0x0002 for AES-256-GCM).
>  - Potentially separating signing (AES-CMAC) from encryption ciphers for
>    clarity.
>
> Is this direction worth pursuing? I'd like to prepare patches for review
> if there's consensus. Any thoughts on priorities, potential pitfalls, or
> related work in progress?
>
> Thanks for your time.
>
> Yunseong



--
Thanks,

Steve

Re: [RFC] ksmbd: Deprecate MD5 support and enhance AES-GCM for SMB 3.1.1 compliance

[Namjae Jeon]

On Tue, Sep 16, 2025 at 8:07 AM Yunseong Kim <ysk@kzalloc.com> wrote:
>
> Hi all,
Hi Yunseong,
>
> I'm looking into contributing to the ksmbd crypto module, specifically
> around crypto handling in crypto_ctx.c. I wanted to send this RFC to gauge
> interest and get feedback before preparing patches.
>
> First, regarding MD5 support: The current code includes HMAC-MD5
> (via crypto_alloc_shash("hmac(md5)")) which appears to be for legacy SMB1
> compatibility. SMB1 is widely deprecated due to security issues, and MD5
> itself is vulnerable to collision attacks, making it unsuitable for modern
> use. I propose deprecating or removing this support entirely, perhaps with
> a config option (e.g., CONFIG_KSMBD_LEGACY_SMB1) for those who absolutely
> need it, but defaulting to off. This would align ksmbd with security best
> practices, similar to how Windows has disabled SMB1 by default.
Steve answered it.
>
> Second, for SMB 3.1.1 compliance: The code already supports AES-GCM via
> crypto_alloc_aead("gcm(aes)"), but to fully adhere to the spec (MS-SMB2),
> we should explicitly handle AES-128-GCM as the default cipher, with
> AES-256-GCM as an optional stronger variant. AES-256-GCM isn't mandatory
> but is recommended for higher security (e.g., in Windows Server 2022+).
This cipher array in SMB2_ENCRYPTION_CAPABILITIES is ordered by
the client's preference, with the most preferred cipher at the beginning and
the least preferred at the end. This allows the client to signal its
ideal choice
to the server.
The server chooses the first cipher in the client's array that it also supports.
Are you saying that the ksmbd server does have to choose AES-256-GCM
if it is not the first cipher in the client's array ?
>
> This would involve:
>  - Adding key length checks and setkey logic in the caller side
>    (e.g., negotiate or session setup).
>  - Updating the negotiate context to include cipher selection
>    (0x0001 for AES-128-GCM, 0x0002 for AES-256-GCM).
I'm not sure what you're trying to change. Are you trying to change a macro
defined in smb2pdu.h?

>  - Potentially separating signing (AES-CMAC) from encryption ciphers for
>    clarity.
>
> Is this direction worth pursuing? I'd like to prepare patches for review
> if there's consensus. Any thoughts on priorities, potential pitfalls, or
> related work in progress?
Could you elaborate more about the 3 items you suggested ?
>
> Thanks for your time.
Thanks!
>
> Yunseong