Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

[Kepplinger-Novakovic Martin <Martin.Kepplinger-Novakovic@ginzinger.com>]

Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> On Thu, Feb 26, 2026 at 11:41:56AM +0000, Kepplinger-Novakovic Martin wrote:
> > [    2.272135] PKEY: ==>public_key_verify_signature()
> > [    2.272165] CAAM rsa init start
> > [    2.272180] CAAM rsa init done
> > [    2.272191] caam_rsa_pub_key: free old key in ctx
> > [    2.272201] caam_rsa_pub_key: write rsa_key->e
> > [    2.272210] caam_rsa_pub_key: write rsa_key->n
> > [    2.272220] start rsassa_pkcs1_verify
> > [    2.272228] slen: 256
> > [    2.272238] child_req address: 1d64b62a full size: 64 + 48 + 256 = 368
> > [    2.272274] out_buf1:00000000: 00000000 00000000 00000000 00000000  ................
> > [    2.272298] out_buf1:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.272322] SRC BUF in out_buf1 CRC: 969ee858
> > [    2.272335] start caam_rsa_enc
> > [    2.272352] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> > [    2.272377] key:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.272413] edesc:00000000: 00000001 00000001 00000000 00000000  ................
> > [    2.272438] edesc:00000010: 00000000 00000000 00000000 cf533d6c  ............l=S.
> > [    2.272466] req:00000000: 00000000 00000000 c02e2f68 d083dcb4  ........h/......
> > [    2.272491] req:00000010: cf60a540 00000200 d083dc94 d083dca4  @.`.............
> > [    2.272509] CAAM: calling caam_jr_enqueue
> > [    2.272524] key:00000000: cf60a600 cf4d1240 00000000 00000000  ..`.@.M.........
> > [    2.272546] key:00000010: 00000000 00000000 00000000 00000000  ................
> > [    2.277444] CAAM: completion callback
> > [    2.424765] OUT BUF in out_buf2 CRC: fd0eef11
> > [    2.424799] out_buf2:00000000: 00000000 00000000 00000000 00000000  ................
> > [    2.424827] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424853] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424878] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424902] out_buf2:00000040: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424926] out_buf2:00000050: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424949] out_buf2:00000060: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424973] out_buf2:00000070: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.424996] out_buf2:00000080: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425020] out_buf2:00000090: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425043] out_buf2:000000a0: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425068] out_buf2:000000b0: ffffffff ffffffff ffffffff ffffffff  ................
> > [    2.425095] out_buf2:000000c0: ffffffff ffffffff ffffffff 30313000  .............010
> > [    2.425123] out_buf2:000000d0: 6009060d 65014886 01020403 20040005  ...`.H.e....... 
> > [    2.425148] out_buf2:000000e0: 6155a84e 7aa089cb 7540e613 f28b9a30  N.Ua...z..@u0...
> > [    2.425172] out_buf2:000000f0: 1e98ec34 cecb0e0f 9ee8951a ad8baec3  4...............
> 
> There's an endianness issue here:  30313000 is the zero byte prescribed
> by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> 
> Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> in reverse order.  And so on until 20040005, which are the last four
> bytes of the prefix in reverse order.
> 
> How are you generating that hexdump?  What's the CPU's endianness?
> Is the caam RSA accelerator using a different endianness?


imx6ul is armv7, little endian byte order and the following returns 1 which supports that:
echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6

I always print the hex dump in the following way (here "out_buf" at line
https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247 )
print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);

I don't find anything about imx6ul's CAAM internal endianness in the ref.man.
and can't say much about that.

I simply run mainline, CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API=n and "crypto: rsassa-pkcs1 - Copy source data for SG list"
reverted. 

Again, with this revert, the problem seems to be the same, only that the data that rsassa_pkcs1_verify() is
starting to check here https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266 is still
"old" but now zeroes, not the input-data, thus failing with -EBADMSG instead of -EINVAL.

My feeling is that endianness is not the issue here. I see what you mean, kind of, but let's look at a success-case.
"out_buf" printed here: https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L264

[    3.863485] out_buf2:00000000: ffff0100 ffffffff ffffffff ffffffff  ................
[    3.863516] out_buf2:00000010: ffffffff ffffffff ffffffff ffffffff  ................
[    3.863542] out_buf2:00000020: ffffffff ffffffff ffffffff ffffffff  ................
[    3.863567] out_buf2:00000030: ffffffff ffffffff ffffffff ffffffff  ................

so out_buf[0] is 0x00, out_buf[1] is 0x01, "seeking" forward until !0xff, all succeeds in the lines below.

and sorry for my late response here. I'd be *very* happy to test any ideas!

                               martin