From: Kepplinger-Novakovic Martin <Martin.Kepplinger-Novakovic@ginzinger.com>
To: Lukas Wunner <lukas@wunner.de>
Cc: "ebiggers@google.com" <ebiggers@google.com>,
"horia.geanta@nxp.com" <horia.geanta@nxp.com>,
"pankaj.gupta@nxp.com" <pankaj.gupta@nxp.com>,
"gaurav.jain@nxp.com" <gaurav.jain@nxp.com>,
"herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
"davem@davemloft.net" <davem@davemloft.net>,
"ignat@cloudflare.com" <ignat@cloudflare.com>,
"linux-crypto@vger.kernel.org" <linux-crypto@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf
Date: Tue, 17 Mar 2026 11:45:41 +0000 [thread overview]
Message-ID: <66e3ca0f49e73d76419502241b7d0ce6e84ae741.camel@ginzinger.com> (raw)
In-Reply-To: <abPWcqeDo549UhIk@wunner.de>
Am Freitag, dem 13.03.2026 um 10:18 +0100 schrieb Lukas Wunner:
> On Tue, Mar 10, 2026 at 08:57:36AM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> > > There's an endianness issue here: 30313000 is the zero byte prescribed
> > > by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> > > followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> > >
> > > Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> > > in reverse order. And so on until 20040005, which are the last four
> > > bytes of the prefix in reverse order.
> > >
> > > How are you generating that hexdump? What's the CPU's endianness?
> > > Is the caam RSA accelerator using a different endianness?
> >
> > imx6ul is armv7, little endian byte order and the following returns 1
> > which supports that:
> > echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6
>
> Please double-check whether your .config enables CONFIG_CPU_BIG_ENDIAN
> or CONFIG_CPU_LITTLE_ENDIAN, just to cover all bases.
CONFIG_CPU_LITTLE_ENDIAN is enabled, not CPU_BIG_ENDIAN. thanks for the hint.
>
> > I always print the hex dump in the following way (here "out_buf" at line
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247
> > )
> > print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);
>
> Please use 1 instead of 4 as 5th parameter of print_hex_dump().
> Using 4 only makes sense if the memory location you want to dump
> contains 32-bit values. That's not the case here as the signature
> is a bytestream.
>
> I guess if you use 4, print_hex_dump() dumps the 32-bit values
> in big endian order for human readability, but that's confusing
> if the memory location actually contains a bytestream.
true, thanks, there would have been other debug examples I should have looked at. Changed this, see below.
>
> > Again, with this revert, the problem seems to be the same, only that
> > the data that rsassa_pkcs1_verify() is starting to check here
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266
> > is still "old" but now zeroes, not the input-data, thus failing with
> > -EBADMSG instead of -EINVAL.
>
> Actually the "out_buf2" that you've included in this message...
>
> https://lore.kernel.org/all/1a65ac92579fadb4bfc76b32a3a4f1c6df022801.camel@ginzinger.com/
>
> ...looks like a valid verified (i.e. encrypted) signature,
> the only thing that's weird is the endianness issue and
> that there's a bunch of zero bytes at the beginning of
> the buffer.
yes. As mentioned in this thread's very first email, always only the first 16 bytes of out_buf stay old, thus wrong, after crypto_req_done(). With
commit 8552cb04e083 reverted, it's 16 bytes of zeroes, but otherwise no change, see the dumps below.
>
> Please re-generate the hexdump of "out_buf" after the call
> to crypto_wait_req(), once with a stock kernel and once with
> 8552cb04e083 reverted, and use 1 as 5th argument to
> print_hex_dump().
so, nothing new, but a failure-case dumped as byte-stream running a stock kernel (the "usual" caam config without ahash api...)
[ 2.339406] start rsassa_pkcs1_verify
[ 2.339413] slen: 256
[ 2.339425] child_req address: bc6b492f full size: 64 + 48 + 256 = 368
[ 2.339455] out_buf1:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93 ........|D/.._L.
[ 2.339476] out_buf1:00000010: 48 b9 fe 07 17 bb 21 f7 25 23 4e aa 22 0c 16 b9 H.....!.%#N."...
[ 2.339500] SRC BUF in out_buf1 CRC: 60791b87
[ 2.339513] start caam_rsa_enc
[ 2.339523] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00 ..b.@tb.........
[ 2.339540] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.339567] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.339584] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 ec 7e 49 cf .............~I.
[ 2.722145] req:00000000: 00 00 00 00 00 00 00 00 88 22 2e c0 a4 dc 83 d0 ........."......
[ 2.722178] req:00000010: 40 c7 62 cf 00 02 00 00 b4 dc 83 d0 b4 dc 83 d0 @.b.............
[ 2.722200] CAAM: calling caam_jr_enqueue
[ 2.722210] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00 ..b.@tb.........
[ 2.722227] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.727096] CAAM: completion callback
[ 2.742603] OUT BUF in out_buf2 CRC: 12298efd
[ 2.742622] out_buf2:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93 ........|D/.._L.
[ 2.742643] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742659] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742675] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742691] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742706] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742722] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742737] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742753] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742769] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742785] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742801] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742818] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30 .............010
[ 2.742833] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 ...`.H.e.......
[ 2.742850] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.742866] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.742883] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.742900] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.742915] Encrypted value had no leading 0 byte.
[ 2.742927] PKEY: crypto_sig_verify error: -22
[ 2.742947] PKEY: <==public_key_verify_signature() = -22
[ 2.742961] X.509: public_key_verify_signature error: -22
[ 2.742970] X.509: <==x509_check_for_self_signed() = -22
[ 2.742983] X.509: x509_check_for_self_signed failed: -22
[ 2.743000] X.509: x509_cert_parse failed: -22
[ 2.743009] Parser recognised the format (ret -22)
[ 2.743022] <==asymmetric_key_preparse() = -22
[ 2.743035] preparse: -22
[ 2.743643] Problem loading in-kernel X.509 certificate (-22)
a failure-case dumped running 8552cb04e083 reverted:
[ 2.307228] start rsassa_pkcs1_verify
[ 2.307236] slen: 256
[ 2.307249] child_req address: f56af4d7 full size: 64 + 48 + 256 = 368
[ 2.307285] out_buf1:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307306] out_buf1:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307330] SRC BUF in out_buf1 CRC: 3e37f250
[ 2.307346] start caam_rsa_enc
[ 2.307355] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00 .._..._.........
[ 2.307372] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307401] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307419] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 6c 7e 3d c1 ............l~=.
[ 2.307440] req:00000000: 00 00 00 00 00 00 00 00 88 32 2e c0 b4 dc 83 d0 .........2......
[ 2.307458] req:00000010: 40 f7 5f cf 00 02 00 00 94 dc 83 d0 a4 dc 83 d0 @._.............
[ 2.307476] CAAM: calling caam_jr_enqueue
[ 2.307485] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00 .._..._.........
[ 2.307502] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.308771] CAAM: completion callback
[ 2.308840] OUT BUF in out_buf2 CRC: 86440841
[ 2.308860] out_buf2:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.308881] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308898] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308914] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308930] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308947] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308963] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308980] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308997] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309013] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309031] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309048] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309064] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30 .............010
[ 2.309082] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 ...`.H.e.......
[ 2.309099] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.309116] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.309133] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.309150] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.309168] PKEY: crypto_sig_verify error: -74
[ 2.309188] PKEY: <==public_key_verify_signature() = -74
[ 2.309201] X.509: public_key_verify_signature error: -74
[ 2.309212] X.509: <==x509_check_for_self_signed() = -74
[ 2.309225] X.509: x509_check_for_self_signed failed: -74
[ 2.309240] X.509: x509_cert_parse failed: -74
[ 2.309252] <==asymmetric_key_preparse() = -74
[ 2.309268] preparse: -74
[ 2.413845] Problem loading in-kernel X.509 certificate (-74)
thank you very much for having a look! I'd be happy to test any thoughts you might have.
martin
next prev parent reply other threads:[~2026-03-17 11:45 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-02-24 14:17 Kepplinger-Novakovic Martin
2026-02-24 15:04 ` Lukas Wunner
2026-02-24 16:09 ` Kepplinger-Novakovic Martin
2026-02-24 16:41 ` Lukas Wunner
2026-02-25 8:02 ` Kepplinger-Novakovic Martin
2026-02-25 8:13 ` Lukas Wunner
2026-02-25 8:47 ` Kepplinger-Novakovic Martin
2026-02-26 7:17 ` Lukas Wunner
2026-02-26 11:41 ` Kepplinger-Novakovic Martin
2026-02-26 13:27 ` Lukas Wunner
2026-03-10 8:57 ` Kepplinger-Novakovic Martin
2026-03-13 9:18 ` Lukas Wunner
2026-03-17 11:45 ` Kepplinger-Novakovic Martin [this message]
2026-03-07 5:32 ` Herbert Xu
2026-03-07 13:31 ` Lukas Wunner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=66e3ca0f49e73d76419502241b7d0ce6e84ae741.camel@ginzinger.com \
--to=martin.kepplinger-novakovic@ginzinger.com \
--cc=davem@davemloft.net \
--cc=ebiggers@google.com \
--cc=gaurav.jain@nxp.com \
--cc=herbert@gondor.apana.org.au \
--cc=horia.geanta@nxp.com \
--cc=ignat@cloudflare.com \
--cc=linux-crypto@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lukas@wunner.de \
--cc=pankaj.gupta@nxp.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®