[PATCH] crypto: rsa: add debug message if leading zero byte is missing

[PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Martin Kepplinger-Novakovic]

When debugging RSA certificate validation it can be valuable to see
why the RSA verify() callback returns -EINVAL.

Signed-off-by: Martin Kepplinger-Novakovic <martin.kepplinger-novakovic@ginzinger.com>
---

hi,

my real issue is: When using a certificate based on an RSA-key,
I sometimes see signature-verify errors and (via dm-verity)
rootfs signature-verify errors, all triggered by "no leading 0 byte".

key/cert generation:
openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com

and simply used as trusted built-in key and rootfs hash sign appended
to rootfs (squashfs).

I'm on imx6ul. The thing is: Using the same certificate/key, works on
old v5.4-based kernels, up to v6.6!

Starting with commit 2f1f34c1bf7b309 ("crypto: ahash - optimize performance
when wrapping shash") it starts to break. it is not a commit on it's own I
can revert and move on.

What happended since v6.6 ? On v6.7 I see
[    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error. Invalid length or parity, or certain other problems.

and later the above -EINVAL from the RSA verify callback, where I add
the debug printing I see.

What's the deal with this "leading 0 byte"?


thank you!

                                    martin



 crypto/rsa-pkcs1pad.c | 5 +++--
 crypto/rsassa-pkcs1.c | 5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
index 50bdb18e7b483..65a4821e9758b 100644
--- a/crypto/rsa-pkcs1pad.c
+++ b/crypto/rsa-pkcs1pad.c
@@ -191,9 +191,10 @@ static int pkcs1pad_decrypt_complete(struct akcipher_request *req, int err)
 
 	out_buf = req_ctx->out_buf;
 	if (dst_len == ctx->key_size) {
-		if (out_buf[0] != 0x00)
-			/* Decrypted value had no leading 0 byte */
+		if (out_buf[0] != 0x00) {
+			pr_debug("Decrypted value had no leading 0 byte\n");
 			goto done;
+		}
 
 		dst_len--;
 		out_buf++;
diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
index 94fa5e9600e79..22919728ea1c8 100644
--- a/crypto/rsassa-pkcs1.c
+++ b/crypto/rsassa-pkcs1.c
@@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct crypto_sig *tfm,
 		return -EINVAL;
 
 	if (dst_len == ctx->key_size) {
-		if (out_buf[0] != 0x00)
-			/* Encrypted value had no leading 0 byte */
+		if (out_buf[0] != 0x00) {
+			pr_debug("Encrypted value had no leading 0 byte\n");
 			return -EINVAL;
+		}
 
 		dst_len--;
 		out_buf++;
-- 
2.47.3

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Ignat Korchagin]

Hi,

On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
<martin.kepplinger-novakovic@ginzinger.com> wrote:
>
> When debugging RSA certificate validation it can be valuable to see
> why the RSA verify() callback returns -EINVAL.

Not sure if this would be some kind of an information leak (depending
on a subsystem using RSA). Also what makes this case so special?
Should we then annotate every other validation check in the code?

> Signed-off-by: Martin Kepplinger-Novakovic <martin.kepplinger-novakovic@ginzinger.com>
> ---
>
> hi,
>
> my real issue is: When using a certificate based on an RSA-key,
> I sometimes see signature-verify errors and (via dm-verity)
> rootfs signature-verify errors, all triggered by "no leading 0 byte".
>
> key/cert generation:
> openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> and simply used as trusted built-in key and rootfs hash sign appended
> to rootfs (squashfs).
>
> I'm on imx6ul. The thing is: Using the same certificate/key, works on
> old v5.4-based kernels, up to v6.6!
>
> Starting with commit 2f1f34c1bf7b309 ("crypto: ahash - optimize performance
> when wrapping shash") it starts to break. it is not a commit on it's own I
> can revert and move on.
>
> What happended since v6.6 ? On v6.7 I see
> [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0: Header Error. Invalid length or parity, or certain other problems.
>
> and later the above -EINVAL from the RSA verify callback, where I add
> the debug printing I see.
>
> What's the deal with this "leading 0 byte"?

See RFC 2313, p 8.1

>
> thank you!
>
>                                     martin
>
>
>
>  crypto/rsa-pkcs1pad.c | 5 +++--
>  crypto/rsassa-pkcs1.c | 5 +++--
>  2 files changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> index 50bdb18e7b483..65a4821e9758b 100644
> --- a/crypto/rsa-pkcs1pad.c
> +++ b/crypto/rsa-pkcs1pad.c
> @@ -191,9 +191,10 @@ static int pkcs1pad_decrypt_complete(struct akcipher_request *req, int err)
>
>         out_buf = req_ctx->out_buf;
>         if (dst_len == ctx->key_size) {
> -               if (out_buf[0] != 0x00)
> -                       /* Decrypted value had no leading 0 byte */
> +               if (out_buf[0] != 0x00) {
> +                       pr_debug("Decrypted value had no leading 0 byte\n");
>                         goto done;
> +               }
>
>                 dst_len--;
>                 out_buf++;
> diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> index 94fa5e9600e79..22919728ea1c8 100644
> --- a/crypto/rsassa-pkcs1.c
> +++ b/crypto/rsassa-pkcs1.c
> @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct crypto_sig *tfm,
>                 return -EINVAL;
>
>         if (dst_len == ctx->key_size) {
> -               if (out_buf[0] != 0x00)
> -                       /* Encrypted value had no leading 0 byte */
> +               if (out_buf[0] != 0x00) {
> +                       pr_debug("Encrypted value had no leading 0 byte\n");
>                         return -EINVAL;
> +               }
>
>                 dst_len--;
>                 out_buf++;
> --
> 2.47.3
>

Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Kepplinger-Novakovic Martin]

Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat Korchagin:
> Hi,
> 
> On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > 
> > When debugging RSA certificate validation it can be valuable to see
> > why the RSA verify() callback returns -EINVAL.
> 
> Not sure if this would be some kind of an information leak (depending
> on a subsystem using RSA). Also what makes this case so special?
> Should we then annotate every other validation check in the code?
> 
> > Signed-off-by: Martin Kepplinger-Novakovic
> > <martin.kepplinger-novakovic@ginzinger.com>
> > ---
> > 
> > hi,
> > 
> > my real issue is: When using a certificate based on an RSA-key,
> > I sometimes see signature-verify errors and (via dm-verity)
> > rootfs signature-verify errors, all triggered by "no leading 0
> > byte".
> > 
> > key/cert generation:
> > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -
> > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > and simply used as trusted built-in key and rootfs hash sign
> > appended
> > to rootfs (squashfs).
> > 
> > I'm on imx6ul. The thing is: Using the same certificate/key, works
> > on
> > old v5.4-based kernels, up to v6.6!
> > 
> > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash - optimize
> > performance
> > when wrapping shash") it starts to break. it is not a commit on
> > it's own I
> > can revert and move on.
> > 
> > What happended since v6.6 ? On v6.7 I see
> > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0:
> > Header Error. Invalid length or parity, or certain other problems.
> > 
> > and later the above -EINVAL from the RSA verify callback, where I
> > add
> > the debug printing I see.
> > 
> > What's the deal with this "leading 0 byte"?
> 
> See RFC 2313, p 8.1

hi Ignat,

thanks for your time, the problem is *sometimes* rsa verify fails.
there seems to be a race condition:

in the failure-case after crypto_akcipher_encrypt() and
crypto_wait_req() the *same* data as before is still at out_buf! that
has not yet been written to.

It's not that obvious to be yet because msleep(1000) doesn't change
much and 00, 01, ff, ff... is *still* not yet written to out_buf!

is there a reason why crypto_akcipher_sync_{en,de}crypt() is not used?
Can you imagine what could go wrong here?

*maybe* commit 1e562deacecca1f1bec7d23da526904a1e87525e that did a lot
of things in parallel (in order to keep functionality similar) got
something wrong?

sidenote: when I use an elliptic curve key instead of rsa, everything
works.

also, the auto-free for child_req looks a bit dangerous when using
out_buf, but ok :)

maybe this rings a bell, I'll keep debugging,

                            martin


> 
> > 
> > thank you!
> > 
> >                                     martin
> > 
> > 
> > 
> >  crypto/rsa-pkcs1pad.c | 5 +++--
> >  crypto/rsassa-pkcs1.c | 5 +++--
> >  2 files changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> > index 50bdb18e7b483..65a4821e9758b 100644
> > --- a/crypto/rsa-pkcs1pad.c
> > +++ b/crypto/rsa-pkcs1pad.c
> > @@ -191,9 +191,10 @@ static int pkcs1pad_decrypt_complete(struct
> > akcipher_request *req, int err)
> > 
> >         out_buf = req_ctx->out_buf;
> >         if (dst_len == ctx->key_size) {
> > -               if (out_buf[0] != 0x00)
> > -                       /* Decrypted value had no leading 0 byte */
> > +               if (out_buf[0] != 0x00) {
> > +                       pr_debug("Decrypted value had no leading 0
> > byte\n");
> >                         goto done;
> > +               }
> > 
> >                 dst_len--;
> >                 out_buf++;
> > diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> > index 94fa5e9600e79..22919728ea1c8 100644
> > --- a/crypto/rsassa-pkcs1.c
> > +++ b/crypto/rsassa-pkcs1.c
> > @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct
> > crypto_sig *tfm,
> >                 return -EINVAL;
> > 
> >         if (dst_len == ctx->key_size) {
> > -               if (out_buf[0] != 0x00)
> > -                       /* Encrypted value had no leading 0 byte */
> > +               if (out_buf[0] != 0x00) {
> > +                       pr_debug("Encrypted value had no leading 0
> > byte\n");
> >                         return -EINVAL;
> > +               }
> > 
> >                 dst_len--;
> >                 out_buf++;
> > --
> > 2.47.3
> > 
> 
> Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Ignat Korchagin]

On Wed, Feb 18, 2026 at 9:36 AM Kepplinger-Novakovic Martin
<Martin.Kepplinger-Novakovic@ginzinger.com> wrote:
>
> Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat Korchagin:
> > Hi,
> >
> > On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> > <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > >
> > > When debugging RSA certificate validation it can be valuable to see
> > > why the RSA verify() callback returns -EINVAL.
> >
> > Not sure if this would be some kind of an information leak (depending
> > on a subsystem using RSA). Also what makes this case so special?
> > Should we then annotate every other validation check in the code?
> >
> > > Signed-off-by: Martin Kepplinger-Novakovic
> > > <martin.kepplinger-novakovic@ginzinger.com>
> > > ---
> > >
> > > hi,
> > >
> > > my real issue is: When using a certificate based on an RSA-key,
> > > I sometimes see signature-verify errors and (via dm-verity)
> > > rootfs signature-verify errors, all triggered by "no leading 0
> > > byte".
> > >
> > > key/cert generation:
> > > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -
> > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > > and simply used as trusted built-in key and rootfs hash sign
> > > appended
> > > to rootfs (squashfs).
> > >
> > > I'm on imx6ul. The thing is: Using the same certificate/key, works
> > > on
> > > old v5.4-based kernels, up to v6.6!
> > >
> > > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash - optimize
> > > performance
> > > when wrapping shash") it starts to break. it is not a commit on
> > > it's own I
> > > can revert and move on.
> > >
> > > What happended since v6.6 ? On v6.7 I see
> > > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0:
> > > Header Error. Invalid length or parity, or certain other problems.
> > >
> > > and later the above -EINVAL from the RSA verify callback, where I
> > > add
> > > the debug printing I see.
> > >
> > > What's the deal with this "leading 0 byte"?
> >
> > See RFC 2313, p 8.1
>
> hi Ignat,
>
> thanks for your time, the problem is *sometimes* rsa verify fails.
> there seems to be a race condition:

Can you clarify the failure case a bit? Is this the same signature
that fails? (That is, you just verify a fixed signature in a loop) Or
are these different signatures? (some reliably verify and some
reliably fail)

> in the failure-case after crypto_akcipher_encrypt() and
> crypto_wait_req() the *same* data as before is still at out_buf! that
> has not yet been written to.
>
> It's not that obvious to be yet because msleep(1000) doesn't change
> much and 00, 01, ff, ff... is *still* not yet written to out_buf!
>
> is there a reason why crypto_akcipher_sync_{en,de}crypt() is not used?
> Can you imagine what could go wrong here?
>
> *maybe* commit 1e562deacecca1f1bec7d23da526904a1e87525e that did a lot
> of things in parallel (in order to keep functionality similar) got
> something wrong?
>
> sidenote: when I use an elliptic curve key instead of rsa, everything
> works.
>
> also, the auto-free for child_req looks a bit dangerous when using
> out_buf, but ok :)
>
> maybe this rings a bell, I'll keep debugging,
>
>                             martin
>
>
> >
> > >
> > > thank you!
> > >
> > >                                     martin
> > >
> > >
> > >
> > >  crypto/rsa-pkcs1pad.c | 5 +++--
> > >  crypto/rsassa-pkcs1.c | 5 +++--
> > >  2 files changed, 6 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> > > index 50bdb18e7b483..65a4821e9758b 100644
> > > --- a/crypto/rsa-pkcs1pad.c
> > > +++ b/crypto/rsa-pkcs1pad.c
> > > @@ -191,9 +191,10 @@ static int pkcs1pad_decrypt_complete(struct
> > > akcipher_request *req, int err)
> > >
> > >         out_buf = req_ctx->out_buf;
> > >         if (dst_len == ctx->key_size) {
> > > -               if (out_buf[0] != 0x00)
> > > -                       /* Decrypted value had no leading 0 byte */
> > > +               if (out_buf[0] != 0x00) {
> > > +                       pr_debug("Decrypted value had no leading 0
> > > byte\n");
> > >                         goto done;
> > > +               }
> > >
> > >                 dst_len--;
> > >                 out_buf++;
> > > diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> > > index 94fa5e9600e79..22919728ea1c8 100644
> > > --- a/crypto/rsassa-pkcs1.c
> > > +++ b/crypto/rsassa-pkcs1.c
> > > @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct
> > > crypto_sig *tfm,
> > >                 return -EINVAL;
> > >
> > >         if (dst_len == ctx->key_size) {
> > > -               if (out_buf[0] != 0x00)
> > > -                       /* Encrypted value had no leading 0 byte */
> > > +               if (out_buf[0] != 0x00) {
> > > +                       pr_debug("Encrypted value had no leading 0
> > > byte\n");
> > >                         return -EINVAL;
> > > +               }
> > >
> > >                 dst_len--;
> > >                 out_buf++;
> > > --
> > > 2.47.3
> > >
> >
> > Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Kepplinger-Novakovic Martin]

Am Mittwoch, dem 18.02.2026 um 10:06 +0100 schrieb Ignat Korchagin:
> On Wed, Feb 18, 2026 at 9:36 AM Kepplinger-Novakovic Martin
> <Martin.Kepplinger-Novakovic@ginzinger.com> wrote:
> > 
> > Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat
> > Korchagin:
> > > Hi,
> > > 
> > > On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> > > <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > > > 
> > > > When debugging RSA certificate validation it can be valuable to
> > > > see
> > > > why the RSA verify() callback returns -EINVAL.
> > > 
> > > Not sure if this would be some kind of an information leak
> > > (depending
> > > on a subsystem using RSA). Also what makes this case so special?
> > > Should we then annotate every other validation check in the code?
> > > 
> > > > Signed-off-by: Martin Kepplinger-Novakovic
> > > > <martin.kepplinger-novakovic@ginzinger.com>
> > > > ---
> > > > 
> > > > hi,
> > > > 
> > > > my real issue is: When using a certificate based on an RSA-key,
> > > > I sometimes see signature-verify errors and (via dm-verity)
> > > > rootfs signature-verify errors, all triggered by "no leading 0
> > > > byte".
> > > > 
> > > > key/cert generation:
> > > > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out
> > > > ca.pem -
> > > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > > > and simply used as trusted built-in key and rootfs hash sign
> > > > appended
> > > > to rootfs (squashfs).
> > > > 
> > > > I'm on imx6ul. The thing is: Using the same certificate/key,
> > > > works
> > > > on
> > > > old v5.4-based kernels, up to v6.6!
> > > > 
> > > > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash - optimize
> > > > performance
> > > > when wrapping shash") it starts to break. it is not a commit on
> > > > it's own I
> > > > can revert and move on.
> > > > 
> > > > What happended since v6.6 ? On v6.7 I see
> > > > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx 0:
> > > > Header Error. Invalid length or parity, or certain other
> > > > problems.
> > > > 
> > > > and later the above -EINVAL from the RSA verify callback, where
> > > > I
> > > > add
> > > > the debug printing I see.
> > > > 
> > > > What's the deal with this "leading 0 byte"?
> > > 
> > > See RFC 2313, p 8.1
> > 
> > hi Ignat,
> > 
> > thanks for your time, the problem is *sometimes* rsa verify fails.
> > there seems to be a race condition:
> 
> Can you clarify the failure case a bit? Is this the same signature
> that fails? (That is, you just verify a fixed signature in a loop) Or
> are these different signatures? (some reliably verify and some
> reliably fail)
> 

different signuatures but nothing special: I add ca.pem (output of
"openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -
nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com") to
CONFIG_SYSTEM_TRUSTED_KEYS

during boot, asymmetric_key_preparse() is called, first on this, and
after that, "cfg80211: Loading compiled-in X.509 certificates for
regulatory database" does the same thing for Chen-Yu, Seth's keys in
mainline net/wireless/certs where I also added Ben's Debian
certificate.

The above verifications of 5 keys fail randomly.

In the end I (want to) use my own cert for dm-verity rootfs loading
(which also randomly fails).

on old kernels, most likely up to v6.6, there was no problem.

thank you for asking,

                                martin


> > in the failure-case after crypto_akcipher_encrypt() and
> > crypto_wait_req() the *same* data as before is still at out_buf!
> > that
> > has not yet been written to.
> > 
> > It's not that obvious to be yet because msleep(1000) doesn't change
> > much and 00, 01, ff, ff... is *still* not yet written to out_buf!
> > 
> > is there a reason why crypto_akcipher_sync_{en,de}crypt() is not
> > used?
> > Can you imagine what could go wrong here?
> > 
> > *maybe* commit 1e562deacecca1f1bec7d23da526904a1e87525e that did a
> > lot
> > of things in parallel (in order to keep functionality similar) got
> > something wrong?
> > 
> > sidenote: when I use an elliptic curve key instead of rsa,
> > everything
> > works.
> > 
> > also, the auto-free for child_req looks a bit dangerous when using
> > out_buf, but ok :)
> > 
> > maybe this rings a bell, I'll keep debugging,
> > 
> >                             martin
> > 
> > 
> > > 
> > > > 
> > > > thank you!
> > > > 
> > > >                                     martin
> > > > 
> > > > 
> > > > 
> > > >  crypto/rsa-pkcs1pad.c | 5 +++--
> > > >  crypto/rsassa-pkcs1.c | 5 +++--
> > > >  2 files changed, 6 insertions(+), 4 deletions(-)
> > > > 
> > > > diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> > > > index 50bdb18e7b483..65a4821e9758b 100644
> > > > --- a/crypto/rsa-pkcs1pad.c
> > > > +++ b/crypto/rsa-pkcs1pad.c
> > > > @@ -191,9 +191,10 @@ static int
> > > > pkcs1pad_decrypt_complete(struct
> > > > akcipher_request *req, int err)
> > > > 
> > > >         out_buf = req_ctx->out_buf;
> > > >         if (dst_len == ctx->key_size) {
> > > > -               if (out_buf[0] != 0x00)
> > > > -                       /* Decrypted value had no leading 0
> > > > byte */
> > > > +               if (out_buf[0] != 0x00) {
> > > > +                       pr_debug("Decrypted value had no
> > > > leading 0
> > > > byte\n");
> > > >                         goto done;
> > > > +               }
> > > > 
> > > >                 dst_len--;
> > > >                 out_buf++;
> > > > diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> > > > index 94fa5e9600e79..22919728ea1c8 100644
> > > > --- a/crypto/rsassa-pkcs1.c
> > > > +++ b/crypto/rsassa-pkcs1.c
> > > > @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct
> > > > crypto_sig *tfm,
> > > >                 return -EINVAL;
> > > > 
> > > >         if (dst_len == ctx->key_size) {
> > > > -               if (out_buf[0] != 0x00)
> > > > -                       /* Encrypted value had no leading 0
> > > > byte */
> > > > +               if (out_buf[0] != 0x00) {
> > > > +                       pr_debug("Encrypted value had no
> > > > leading 0
> > > > byte\n");
> > > >                         return -EINVAL;
> > > > +               }
> > > > 
> > > >                 dst_len--;
> > > >                 out_buf++;
> > > > --
> > > > 2.47.3
> > > > 
> > > 
> > > Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Kepplinger-Novakovic Martin]

Am Mittwoch, dem 18.02.2026 um 09:22 +0000 schrieb Kepplinger-Novakovic
Martin:
> Am Mittwoch, dem 18.02.2026 um 10:06 +0100 schrieb Ignat Korchagin:
> > On Wed, Feb 18, 2026 at 9:36 AM Kepplinger-Novakovic Martin
> > <Martin.Kepplinger-Novakovic@ginzinger.com> wrote:
> > > 
> > > Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat
> > > Korchagin:
> > > > Hi,
> > > > 
> > > > On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> > > > <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > > > > 
> > > > > When debugging RSA certificate validation it can be valuable
> > > > > to
> > > > > see
> > > > > why the RSA verify() callback returns -EINVAL.
> > > > 
> > > > Not sure if this would be some kind of an information leak
> > > > (depending
> > > > on a subsystem using RSA). Also what makes this case so
> > > > special?
> > > > Should we then annotate every other validation check in the
> > > > code?
> > > > 
> > > > > Signed-off-by: Martin Kepplinger-Novakovic
> > > > > <martin.kepplinger-novakovic@ginzinger.com>
> > > > > ---
> > > > > 
> > > > > hi,
> > > > > 
> > > > > my real issue is: When using a certificate based on an RSA-
> > > > > key,
> > > > > I sometimes see signature-verify errors and (via dm-verity)
> > > > > rootfs signature-verify errors, all triggered by "no leading
> > > > > 0
> > > > > byte".
> > > > > 
> > > > > key/cert generation:
> > > > > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out
> > > > > ca.pem -
> > > > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > > > > and simply used as trusted built-in key and rootfs hash sign
> > > > > appended
> > > > > to rootfs (squashfs).
> > > > > 
> > > > > I'm on imx6ul. The thing is: Using the same certificate/key,
> > > > > works
> > > > > on
> > > > > old v5.4-based kernels, up to v6.6!
> > > > > 
> > > > > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash -
> > > > > optimize
> > > > > performance
> > > > > when wrapping shash") it starts to break. it is not a commit
> > > > > on
> > > > > it's own I
> > > > > can revert and move on.
> > > > > 
> > > > > What happended since v6.6 ? On v6.7 I see
> > > > > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx
> > > > > 0:
> > > > > Header Error. Invalid length or parity, or certain other
> > > > > problems.
> > > > > 
> > > > > and later the above -EINVAL from the RSA verify callback,
> > > > > where
> > > > > I
> > > > > add
> > > > > the debug printing I see.
> > > > > 
> > > > > What's the deal with this "leading 0 byte"?
> > > > 
> > > > See RFC 2313, p 8.1
> > > 
> > > hi Ignat,
> > > 
> > > thanks for your time, the problem is *sometimes* rsa verify
> > > fails.
> > > there seems to be a race condition:
> > 
> > Can you clarify the failure case a bit? Is this the same signature
> > that fails? (That is, you just verify a fixed signature in a loop)
> > Or
> > are these different signatures? (some reliably verify and some
> > reliably fail)
> > 
> 
> different signuatures but nothing special: I add ca.pem (output of
> "openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem -
> nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com") to
> CONFIG_SYSTEM_TRUSTED_KEYS
> 
> during boot, asymmetric_key_preparse() is called, first on this, and
> after that, "cfg80211: Loading compiled-in X.509 certificates for
> regulatory database" does the same thing for Chen-Yu, Seth's keys in
> mainline net/wireless/certs where I also added Ben's Debian
> certificate.
> 
> The above verifications of 5 keys fail randomly.
> 

to clarify: no verification reliably fails or succeeds. the first one,
my own cert, mostly (always?) succeeds, for the 4 regdb certs I see no
pattern at all. Chen-Yu's "wens" cert kind of fails more often that
than the others maybe.

There is almost never a boot where all certs verifications succeed,
although I've seen at least one already.


> In the end I (want to) use my own cert for dm-verity rootfs loading
> (which also randomly fails).
> 
> on old kernels, most likely up to v6.6, there was no problem.
> 
> thank you for asking,
> 
>                                 martin
> 
> 
> > > in the failure-case after crypto_akcipher_encrypt() and
> > > crypto_wait_req() the *same* data as before is still at out_buf!
> > > that
> > > has not yet been written to.
> > > 
> > > It's not that obvious to be yet because msleep(1000) doesn't
> > > change
> > > much and 00, 01, ff, ff... is *still* not yet written to out_buf!
> > > 
> > > is there a reason why crypto_akcipher_sync_{en,de}crypt() is not
> > > used?
> > > Can you imagine what could go wrong here?
> > > 
> > > *maybe* commit 1e562deacecca1f1bec7d23da526904a1e87525e that did
> > > a
> > > lot
> > > of things in parallel (in order to keep functionality similar)
> > > got
> > > something wrong?
> > > 
> > > sidenote: when I use an elliptic curve key instead of rsa,
> > > everything
> > > works.
> > > 
> > > also, the auto-free for child_req looks a bit dangerous when
> > > using
> > > out_buf, but ok :)
> > > 
> > > maybe this rings a bell, I'll keep debugging,
> > > 
> > >                             martin
> > > 
> > > 
> > > > 
> > > > > 
> > > > > thank you!
> > > > > 
> > > > >                                     martin
> > > > > 
> > > > > 
> > > > > 
> > > > >  crypto/rsa-pkcs1pad.c | 5 +++--
> > > > >  crypto/rsassa-pkcs1.c | 5 +++--
> > > > >  2 files changed, 6 insertions(+), 4 deletions(-)
> > > > > 
> > > > > diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> > > > > index 50bdb18e7b483..65a4821e9758b 100644
> > > > > --- a/crypto/rsa-pkcs1pad.c
> > > > > +++ b/crypto/rsa-pkcs1pad.c
> > > > > @@ -191,9 +191,10 @@ static int
> > > > > pkcs1pad_decrypt_complete(struct
> > > > > akcipher_request *req, int err)
> > > > > 
> > > > >         out_buf = req_ctx->out_buf;
> > > > >         if (dst_len == ctx->key_size) {
> > > > > -               if (out_buf[0] != 0x00)
> > > > > -                       /* Decrypted value had no leading 0
> > > > > byte */
> > > > > +               if (out_buf[0] != 0x00) {
> > > > > +                       pr_debug("Decrypted value had no
> > > > > leading 0
> > > > > byte\n");
> > > > >                         goto done;
> > > > > +               }
> > > > > 
> > > > >                 dst_len--;
> > > > >                 out_buf++;
> > > > > diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> > > > > index 94fa5e9600e79..22919728ea1c8 100644
> > > > > --- a/crypto/rsassa-pkcs1.c
> > > > > +++ b/crypto/rsassa-pkcs1.c
> > > > > @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct
> > > > > crypto_sig *tfm,
> > > > >                 return -EINVAL;
> > > > > 
> > > > >         if (dst_len == ctx->key_size) {
> > > > > -               if (out_buf[0] != 0x00)
> > > > > -                       /* Encrypted value had no leading 0
> > > > > byte */
> > > > > +               if (out_buf[0] != 0x00) {
> > > > > +                       pr_debug("Encrypted value had no
> > > > > leading 0
> > > > > byte\n");
> > > > >                         return -EINVAL;
> > > > > +               }
> > > > > 
> > > > >                 dst_len--;
> > > > >                 out_buf++;
> > > > > --
> > > > > 2.47.3
> > > > > 
> > > > 
> > > > Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Kepplinger-Novakovic Martin]

Am Mittwoch, dem 18.02.2026 um 09:42 +0000 schrieb Kepplinger-Novakovic
Martin:
> Am Mittwoch, dem 18.02.2026 um 09:22 +0000 schrieb Kepplinger-
> Novakovic
> Martin:
> > Am Mittwoch, dem 18.02.2026 um 10:06 +0100 schrieb Ignat Korchagin:
> > > On Wed, Feb 18, 2026 at 9:36 AM Kepplinger-Novakovic Martin
> > > <Martin.Kepplinger-Novakovic@ginzinger.com> wrote:
> > > > 
> > > > Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat
> > > > Korchagin:
> > > > > Hi,
> > > > > 
> > > > > On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> > > > > <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > > > > > 
> > > > > > When debugging RSA certificate validation it can be
> > > > > > valuable
> > > > > > to
> > > > > > see
> > > > > > why the RSA verify() callback returns -EINVAL.
> > > > > 
> > > > > Not sure if this would be some kind of an information leak
> > > > > (depending
> > > > > on a subsystem using RSA). Also what makes this case so
> > > > > special?
> > > > > Should we then annotate every other validation check in the
> > > > > code?
> > > > > 
> > > > > > Signed-off-by: Martin Kepplinger-Novakovic
> > > > > > <martin.kepplinger-novakovic@ginzinger.com>
> > > > > > ---
> > > > > > 
> > > > > > hi,
> > > > > > 
> > > > > > my real issue is: When using a certificate based on an RSA-
> > > > > > key,
> > > > > > I sometimes see signature-verify errors and (via dm-verity)
> > > > > > rootfs signature-verify errors, all triggered by "no
> > > > > > leading
> > > > > > 0
> > > > > > byte".
> > > > > > 
> > > > > > key/cert generation:
> > > > > > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out
> > > > > > ca.pem -
> > > > > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > > > > > and simply used as trusted built-in key and rootfs hash
> > > > > > sign
> > > > > > appended
> > > > > > to rootfs (squashfs).
> > > > > > 
> > > > > > I'm on imx6ul. The thing is: Using the same
> > > > > > certificate/key,
> > > > > > works
> > > > > > on
> > > > > > old v5.4-based kernels, up to v6.6!
> > > > > > 
> > > > > > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash -
> > > > > > optimize
> > > > > > performance
> > > > > > when wrapping shash") it starts to break. it is not a
> > > > > > commit
> > > > > > on
> > > > > > it's own I
> > > > > > can revert and move on.
> > > > > > 
> > > > > > What happended since v6.6 ? On v6.7 I see
> > > > > > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx
> > > > > > 0:
> > > > > > Header Error. Invalid length or parity, or certain other
> > > > > > problems.
> > > > > > 
> > > > > > and later the above -EINVAL from the RSA verify callback,
> > > > > > where
> > > > > > I
> > > > > > add
> > > > > > the debug printing I see.
> > > > > > 
> > > > > > What's the deal with this "leading 0 byte"?
> > > > > 
> > > > > See RFC 2313, p 8.1
> > > > 
> > > > hi Ignat,
> > > > 
> > > > thanks for your time, the problem is *sometimes* rsa verify
> > > > fails.
> > > > there seems to be a race condition:
> > > 
> > > Can you clarify the failure case a bit? Is this the same
> > > signature
> > > that fails? (That is, you just verify a fixed signature in a
> > > loop)
> > > Or
> > > are these different signatures? (some reliably verify and some
> > > reliably fail)
> > > 
> > 
> > different signuatures but nothing special: I add ca.pem (output of
> > "openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem
> > -
> > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com") to
> > CONFIG_SYSTEM_TRUSTED_KEYS
> > 
> > during boot, asymmetric_key_preparse() is called, first on this,
> > and
> > after that, "cfg80211: Loading compiled-in X.509 certificates for
> > regulatory database" does the same thing for Chen-Yu, Seth's keys
> > in
> > mainline net/wireless/certs where I also added Ben's Debian
> > certificate.
> > 
> > The above verifications of 5 keys fail randomly.
> > 
> 
> to clarify: no verification reliably fails or succeeds. the first
> one,
> my own cert, mostly (always?) succeeds, for the 4 regdb certs I see
> no
> pattern at all. Chen-Yu's "wens" cert kind of fails more often that
> than the others maybe.
> 
> There is almost never a boot where all certs verifications succeed,
> although I've seen at least one already.
> 
> 
> > In the end I (want to) use my own cert for dm-verity rootfs loading
> > (which also randomly fails).
> > 
> > on old kernels, most likely up to v6.6, there was no problem.
> > 
> > thank you for asking,
> > 
> >                                 martin
> > 
> > 
> > > > in the failure-case after crypto_akcipher_encrypt() and
> > > > crypto_wait_req() the *same* data as before is still at
> > > > out_buf!
> > > > that
> > > > has not yet been written to.
> > > > 
> > > > It's not that obvious to be yet because msleep(1000) doesn't
> > > > change
> > > > much and 00, 01, ff, ff... is *still* not yet written to
> > > > out_buf!

oh, I might have been on a slightly wrong path: I'm on imx6ul and
disabling CONFIG_CRYPTO_DEV_FSL_CAAM indeed is a workaround, so it's
probably drivers/crypto/caam/ where enqueue + dequeue properly run, but
still the CPU doesn't see updated data.

I added Horia, Pankaj and Gaurav and with luck they see what could go
wrong here with CAAM-DMA/CPU syncing.

> > > > 
> > > > is there a reason why crypto_akcipher_sync_{en,de}crypt() is
> > > > not
> > > > used?
> > > > Can you imagine what could go wrong here?
> > > > 
> > > > *maybe* commit 1e562deacecca1f1bec7d23da526904a1e87525e that
> > > > did
> > > > a
> > > > lot
> > > > of things in parallel (in order to keep functionality similar)
> > > > got
> > > > something wrong?
> > > > 
> > > > sidenote: when I use an elliptic curve key instead of rsa,
> > > > everything
> > > > works.
> > > > 
> > > > also, the auto-free for child_req looks a bit dangerous when
> > > > using
> > > > out_buf, but ok :)
> > > > 
> > > > maybe this rings a bell, I'll keep debugging,
> > > > 
> > > >                             martin
> > > > 
> > > > 
> > > > > 
> > > > > > 
> > > > > > thank you!
> > > > > > 
> > > > > >                                     martin
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  crypto/rsa-pkcs1pad.c | 5 +++--
> > > > > >  crypto/rsassa-pkcs1.c | 5 +++--
> > > > > >  2 files changed, 6 insertions(+), 4 deletions(-)
> > > > > > 
> > > > > > diff --git a/crypto/rsa-pkcs1pad.c b/crypto/rsa-pkcs1pad.c
> > > > > > index 50bdb18e7b483..65a4821e9758b 100644
> > > > > > --- a/crypto/rsa-pkcs1pad.c
> > > > > > +++ b/crypto/rsa-pkcs1pad.c
> > > > > > @@ -191,9 +191,10 @@ static int
> > > > > > pkcs1pad_decrypt_complete(struct
> > > > > > akcipher_request *req, int err)
> > > > > > 
> > > > > >         out_buf = req_ctx->out_buf;
> > > > > >         if (dst_len == ctx->key_size) {
> > > > > > -               if (out_buf[0] != 0x00)
> > > > > > -                       /* Decrypted value had no leading 0
> > > > > > byte */
> > > > > > +               if (out_buf[0] != 0x00) {
> > > > > > +                       pr_debug("Decrypted value had no
> > > > > > leading 0
> > > > > > byte\n");
> > > > > >                         goto done;
> > > > > > +               }
> > > > > > 
> > > > > >                 dst_len--;
> > > > > >                 out_buf++;
> > > > > > diff --git a/crypto/rsassa-pkcs1.c b/crypto/rsassa-pkcs1.c
> > > > > > index 94fa5e9600e79..22919728ea1c8 100644
> > > > > > --- a/crypto/rsassa-pkcs1.c
> > > > > > +++ b/crypto/rsassa-pkcs1.c
> > > > > > @@ -263,9 +263,10 @@ static int rsassa_pkcs1_verify(struct
> > > > > > crypto_sig *tfm,
> > > > > >                 return -EINVAL;
> > > > > > 
> > > > > >         if (dst_len == ctx->key_size) {
> > > > > > -               if (out_buf[0] != 0x00)
> > > > > > -                       /* Encrypted value had no leading 0
> > > > > > byte */
> > > > > > +               if (out_buf[0] != 0x00) {
> > > > > > +                       pr_debug("Encrypted value had no
> > > > > > leading 0
> > > > > > byte\n");
> > > > > >                         return -EINVAL;
> > > > > > +               }
> > > > > > 
> > > > > >                 dst_len--;
> > > > > >                 out_buf++;
> > > > > > --
> > > > > > 2.47.3
> > > > > > 
> > > > > 
> > > > > Ignat

Re: [PATCH] crypto: rsa: add debug message if leading zero byte is missing

[Kepplinger-Novakovic Martin]

Am Mittwoch, dem 18.02.2026 um 13:53 +0000 schrieb Kepplinger-Novakovic Martin:
> Am Mittwoch, dem 18.02.2026 um 09:42 +0000 schrieb Kepplinger-Novakovic
> Martin:
> > Am Mittwoch, dem 18.02.2026 um 09:22 +0000 schrieb Kepplinger-
> > Novakovic
> > Martin:
> > > Am Mittwoch, dem 18.02.2026 um 10:06 +0100 schrieb Ignat Korchagin:
> > > > On Wed, Feb 18, 2026 at 9:36 AM Kepplinger-Novakovic Martin
> > > > <Martin.Kepplinger-Novakovic@ginzinger.com> wrote:
> > > > > 
> > > > > Am Donnerstag, dem 12.02.2026 um 11:15 +0000 schrieb Ignat
> > > > > Korchagin:
> > > > > > Hi,
> > > > > > 
> > > > > > On Thu, Feb 12, 2026 at 10:39 AM Martin Kepplinger-Novakovic
> > > > > > <martin.kepplinger-novakovic@ginzinger.com> wrote:
> > > > > > > 
> > > > > > > When debugging RSA certificate validation it can be
> > > > > > > valuable
> > > > > > > to
> > > > > > > see
> > > > > > > why the RSA verify() callback returns -EINVAL.
> > > > > > 
> > > > > > Not sure if this would be some kind of an information leak
> > > > > > (depending
> > > > > > on a subsystem using RSA). Also what makes this case so
> > > > > > special?
> > > > > > Should we then annotate every other validation check in the
> > > > > > code?
> > > > > > 
> > > > > > > Signed-off-by: Martin Kepplinger-Novakovic
> > > > > > > <martin.kepplinger-novakovic@ginzinger.com>
> > > > > > > ---
> > > > > > > 
> > > > > > > hi,
> > > > > > > 
> > > > > > > my real issue is: When using a certificate based on an RSA-
> > > > > > > key,
> > > > > > > I sometimes see signature-verify errors and (via dm-verity)
> > > > > > > rootfs signature-verify errors, all triggered by "no
> > > > > > > leading
> > > > > > > 0
> > > > > > > byte".
> > > > > > > 
> > > > > > > key/cert generation:
> > > > > > > openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out
> > > > > > > ca.pem -
> > > > > > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com
> > > > > > > and simply used as trusted built-in key and rootfs hash
> > > > > > > sign
> > > > > > > appended
> > > > > > > to rootfs (squashfs).
> > > > > > > 
> > > > > > > I'm on imx6ul. The thing is: Using the same
> > > > > > > certificate/key,
> > > > > > > works
> > > > > > > on
> > > > > > > old v5.4-based kernels, up to v6.6!
> > > > > > > 
> > > > > > > Starting with commit 2f1f34c1bf7b309 ("crypto: ahash -
> > > > > > > optimize
> > > > > > > performance
> > > > > > > when wrapping shash") it starts to break. it is not a
> > > > > > > commit
> > > > > > > on
> > > > > > > it's own I
> > > > > > > can revert and move on.
> > > > > > > 
> > > > > > > What happended since v6.6 ? On v6.7 I see
> > > > > > > [    2.978722] caam_jr 2142000.jr: 40000013: DECO: desc idx
> > > > > > > 0:
> > > > > > > Header Error. Invalid length or parity, or certain other
> > > > > > > problems.
> > > > > > > 
> > > > > > > and later the above -EINVAL from the RSA verify callback,
> > > > > > > where
> > > > > > > I
> > > > > > > add
> > > > > > > the debug printing I see.
> > > > > > > 
> > > > > > > What's the deal with this "leading 0 byte"?
> > > > > > 
> > > > > > See RFC 2313, p 8.1
> > > > > 
> > > > > hi Ignat,
> > > > > 
> > > > > thanks for your time, the problem is *sometimes* rsa verify
> > > > > fails.
> > > > > there seems to be a race condition:
> > > > 
> > > > Can you clarify the failure case a bit? Is this the same
> > > > signature
> > > > that fails? (That is, you just verify a fixed signature in a
> > > > loop)
> > > > Or
> > > > are these different signatures? (some reliably verify and some
> > > > reliably fail)
> > > > 
> > > 
> > > different signuatures but nothing special: I add ca.pem (output of
> > > "openssl req -x509 -newkey rsa:4096 -keyout ca_key.pem -out ca.pem
> > > -
> > > nodes -days 365 -set_serial 01 -subj /CN=ginzinger.com") to
> > > CONFIG_SYSTEM_TRUSTED_KEYS
> > > 
> > > during boot, asymmetric_key_preparse() is called, first on this,
> > > and
> > > after that, "cfg80211: Loading compiled-in X.509 certificates for
> > > regulatory database" does the same thing for Chen-Yu, Seth's keys
> > > in
> > > mainline net/wireless/certs where I also added Ben's Debian
> > > certificate.
> > > 
> > > The above verifications of 5 keys fail randomly.
> > > 
> > 
> > to clarify: no verification reliably fails or succeeds. the first
> > one,
> > my own cert, mostly (always?) succeeds, for the 4 regdb certs I see
> > no
> > pattern at all. Chen-Yu's "wens" cert kind of fails more often that
> > than the others maybe.
> > 
> > There is almost never a boot where all certs verifications succeed,
> > although I've seen at least one already.
> > 
> > 
> > > In the end I (want to) use my own cert for dm-verity rootfs loading
> > > (which also randomly fails).
> > > 
> > > on old kernels, most likely up to v6.6, there was no problem.
> > > 
> > > thank you for asking,
> > > 
> > >                                 martin
> > > 
> > > 
> > > > > in the failure-case after crypto_akcipher_encrypt() and
> > > > > crypto_wait_req() the *same* data as before is still at
> > > > > out_buf!
> > > > > that
> > > > > has not yet been written to.
> > > > > 
> > > > > It's not that obvious to be yet because msleep(1000) doesn't
> > > > > change
> > > > > much and 00, 01, ff, ff... is *still* not yet written to
> > > > > out_buf!
> 
> oh, I might have been on a slightly wrong path: I'm on imx6ul and
> disabling CONFIG_CRYPTO_DEV_FSL_CAAM indeed is a workaround, so it's
> probably drivers/crypto/caam/ where enqueue + dequeue properly run, but
> still the CPU doesn't see updated data.
> 
> I added Horia, Pankaj and Gaurav and with luck they see what could go
> wrong here with CAAM-DMA/CPU syncing.


I wrote a more accurate and detailed bugreport for this, see
https://lore.kernel.org/linux-crypto/6029acc0f0ddfe25e2537c2866d54fd7f54bc182.camel@ginzinger.com/T/#u

thanks,

                                   martin