Issue using faddr2line on kernel modules

Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

Hi there,

Am researching using the cool faddr2line script to help debug Oops'es
from kernel modules..
I find it works just fine when used against the unstripped vmlinux
with debug symbols.

My use case is for a kernel module which Oopses, though. Here's my scenario:
I built a module on a custom debug kernel (5.10.60) with most debug
options enabled...
KASLR is enabled by default as well.

A test kernel module Oopses on my x86_64 guest running this kernel with:
RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]

So, i try this:

$ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
$

(It works fine with addr2line though!).
Now I think I've traced the faddr2line script's failure to locate
anything down to this:
...
done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
print line, "0x"$1 } END {if (found == 1) print     line, end; }')

The nm output is:
$ nm -n ./oops_tryv2.ko |grep -i do_the_work
0000000000000000 t do_the_work
$

nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
with objdump -d on the module).
Am I missing something? Any suggestions as to what I can try, to get
faddr2line working?

TIA,
Regards,
Kaiwan.

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> Hi there,
> 
> Am researching using the cool faddr2line script to help debug Oops'es
> from kernel modules..
> I find it works just fine when used against the unstripped vmlinux
> with debug symbols.
> 
> My use case is for a kernel module which Oopses, though. Here's my scenario:
> I built a module on a custom debug kernel (5.10.60) with most debug
> options enabled...
> KASLR is enabled by default as well.
> 
> A test kernel module Oopses on my x86_64 guest running this kernel with:
> RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> 
> So, i try this:
> 
> $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> $
> 
> (It works fine with addr2line though!).
> Now I think I've traced the faddr2line script's failure to locate
> anything down to this:
> ...
> done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
> print line, "0x"$1 } END {if (found == 1) print     line, end; }')
> 
> The nm output is:
> $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> 0000000000000000 t do_the_work
> $
> 
> nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> with objdump -d on the module).
> Am I missing something? Any suggestions as to what I can try, to get
> faddr2line working?

Hi Kaiwan,

Thanks for reporting this issue.  The module text address of 0x0 is not
necessarily incorrect, as the address is relative the the module, where
all text usually starts at zero.

I was able to recreate this problem using a module which only has a
single function in .text.  Does this fix it?

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..cd6e812b9d06 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -111,21 +111,11 @@ __faddr2line() {
 
 	# Go through each of the object's symbols which match the func name.
 	# In rare cases there might be duplicates.
-	file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
 	while read symbol; do
 		local fields=($symbol)
 		local sym_base=0x${fields[0]}
-		local sym_type=${fields[1]}
-		local sym_end=${fields[3]}
-
-		# calculate the size
-		local sym_size=$(($sym_end - $sym_base))
-		if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
-			warn "bad symbol size: base: $sym_base end: $sym_end"
-			DONE=1
-			return
-		fi
-		sym_size=0x$(printf %x $sym_size)
+		local sym_size=0x${fields[1]}
+		local sym_type=${fields[2]}
 
 		# calculate the address
 		local addr=$(($sym_base + $offset))
@@ -189,7 +179,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+	done < <(${NM} -nS $objfile | awk -v fn=$func '$4 == fn')
 }
 
 [[ $# -lt 2 ]] && usage

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > Hi there,
> > 
> > Am researching using the cool faddr2line script to help debug Oops'es
> > from kernel modules..
> > I find it works just fine when used against the unstripped vmlinux
> > with debug symbols.
> > 
> > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > I built a module on a custom debug kernel (5.10.60) with most debug
> > options enabled...
> > KASLR is enabled by default as well.
> > 
> > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > 
> > So, i try this:
> > 
> > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > $
> > 
> > (It works fine with addr2line though!).
> > Now I think I've traced the faddr2line script's failure to locate
> > anything down to this:
> > ...
> > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
> > print line, "0x"$1 } END {if (found == 1) print     line, end; }')
> > 
> > The nm output is:
> > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > 0000000000000000 t do_the_work
> > $
> > 
> > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > with objdump -d on the module).
> > Am I missing something? Any suggestions as to what I can try, to get
> > faddr2line working?
> 
> Hi Kaiwan,
> 
> Thanks for reporting this issue.  The module text address of 0x0 is not
> necessarily incorrect, as the address is relative the the module, where
> all text usually starts at zero.
> 
> I was able to recreate this problem using a module which only has a
> single function in .text.  Does this fix it?

Actually, that patch has other problems.  Try this one?

----

From: Josh Poimboeuf <jpoimboe@redhat.com>
Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
 calculating function size

With the following commit:

  efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")

... it was discovered that faddr2line can't just read a function's ELF
size, because that wouldn't match the kallsyms function size which is
printed in the stack trace.  The kallsyms size includes any padding
after the function, whereas the ELF size does not.

So faddr2line has to manually calculate the size of a function similar
to how kallsyms does.  It does so by starting with a sorted list of
symbols and subtracting the function address from the subsequent
symbol's address.

That calculation is broken in the case where the function is the last
(or only) symbol in the .text section.  The next symbol in the sorted
list might actually be a data symbol, which can break the function size
detection:

  $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
  bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000

Similar breakage can occur when reading from a .o file.

Fix it by only looking for text symbols.

Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
---
 scripts/faddr2line | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..2a130134f1e6 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -189,7 +189,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
 }
 
 [[ $# -lt 2 ]] && usage
-- 
2.31.1

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

Hi Josh,

Actually your first patch - the one you mentioned had other issues -
worked perfectly when applied:

scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
do_the_work+0x16f/0x0000000000000194:
do_the_work at <...>/oops_tryv2/oops_tryv2.c:62

The second one still failed in the same manner:

scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000

So, is it possible to fixup issues with the first version?
What are these issues?

Thanks,
Kaiwan.



On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > Hi there,
> > >
> > > Am researching using the cool faddr2line script to help debug Oops'es
> > > from kernel modules..
> > > I find it works just fine when used against the unstripped vmlinux
> > > with debug symbols.
> > >
> > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > options enabled...
> > > KASLR is enabled by default as well.
> > >
> > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > >
> > > So, i try this:
> > >
> > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > $
> > >
> > > (It works fine with addr2line though!).
> > > Now I think I've traced the faddr2line script's failure to locate
> > > anything down to this:
> > > ...
> > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
> > > print line, "0x"$1 } END {if (found == 1) print     line, end; }')
> > >
> > > The nm output is:
> > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > 0000000000000000 t do_the_work
> > > $
> > >
> > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > with objdump -d on the module).
> > > Am I missing something? Any suggestions as to what I can try, to get
> > > faddr2line working?
> >
> > Hi Kaiwan,
> >
> > Thanks for reporting this issue.  The module text address of 0x0 is not
> > necessarily incorrect, as the address is relative the the module, where
> > all text usually starts at zero.
> >
> > I was able to recreate this problem using a module which only has a
> > single function in .text.  Does this fix it?
>
> Actually, that patch has other problems.  Try this one?
>
> ----
>
> From: Josh Poimboeuf <jpoimboe@redhat.com>
> Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
>  calculating function size
>
> With the following commit:
>
>   efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
>
> ... it was discovered that faddr2line can't just read a function's ELF
> size, because that wouldn't match the kallsyms function size which is
> printed in the stack trace.  The kallsyms size includes any padding
> after the function, whereas the ELF size does not.
>
> So faddr2line has to manually calculate the size of a function similar
> to how kallsyms does.  It does so by starting with a sorted list of
> symbols and subtracting the function address from the subsequent
> symbol's address.
>
> That calculation is broken in the case where the function is the last
> (or only) symbol in the .text section.  The next symbol in the sorted
> list might actually be a data symbol, which can break the function size
> detection:
>
>   $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
>   bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
>
> Similar breakage can occur when reading from a .o file.
>
> Fix it by only looking for text symbols.
>
> Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> ---
>  scripts/faddr2line | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..2a130134f1e6 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -189,7 +189,7 @@ __faddr2line() {
>
>                 DONE=1
>
> -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> +       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
>  }
>
>  [[ $# -lt 2 ]] && usage
> --
> 2.31.1
>

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> Hi Josh,
> 
> Actually your first patch - the one you mentioned had other issues -
> worked perfectly when applied:
> 
> scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> do_the_work+0x16f/0x0000000000000194:
> do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
> 
> The second one still failed in the same manner:
> 
> scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> 
> So, is it possible to fixup issues with the first version?
> What are these issues?

The first patch basically reverts the fix in commit efdb4167e676
("scripts/faddr2line: Fix "size mismatch" error").  That would be nice
as it's simpler and more robust, but unfortunately it would cause a lot
of "size mismatch" errors with vmlinux symbols.

Can you give the output of 'nm -n ./oops_tryv2.ko'?  There must be some
text symbol immediately after the do_the_work() symbol which is either
out of order, or part of another section.

Is do_the_work() in the .text section?

> On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > > Hi there,
> > > >
> > > > Am researching using the cool faddr2line script to help debug Oops'es
> > > > from kernel modules..
> > > > I find it works just fine when used against the unstripped vmlinux
> > > > with debug symbols.
> > > >
> > > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > > options enabled...
> > > > KASLR is enabled by default as well.
> > > >
> > > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > > >
> > > > So, i try this:
> > > >
> > > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > > $
> > > >
> > > > (It works fine with addr2line though!).
> > > > Now I think I've traced the faddr2line script's failure to locate
> > > > anything down to this:
> > > > ...
> > > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > > fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
> > > > print line, "0x"$1 } END {if (found == 1) print     line, end; }')
> > > >
> > > > The nm output is:
> > > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > > 0000000000000000 t do_the_work
> > > > $
> > > >
> > > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > > with objdump -d on the module).
> > > > Am I missing something? Any suggestions as to what I can try, to get
> > > > faddr2line working?
> > >
> > > Hi Kaiwan,
> > >
> > > Thanks for reporting this issue.  The module text address of 0x0 is not
> > > necessarily incorrect, as the address is relative the the module, where
> > > all text usually starts at zero.
> > >
> > > I was able to recreate this problem using a module which only has a
> > > single function in .text.  Does this fix it?
> >
> > Actually, that patch has other problems.  Try this one?
> >
> > ----
> >
> > From: Josh Poimboeuf <jpoimboe@redhat.com>
> > Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
> >  calculating function size
> >
> > With the following commit:
> >
> >   efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> >
> > ... it was discovered that faddr2line can't just read a function's ELF
> > size, because that wouldn't match the kallsyms function size which is
> > printed in the stack trace.  The kallsyms size includes any padding
> > after the function, whereas the ELF size does not.
> >
> > So faddr2line has to manually calculate the size of a function similar
> > to how kallsyms does.  It does so by starting with a sorted list of
> > symbols and subtracting the function address from the subsequent
> > symbol's address.
> >
> > That calculation is broken in the case where the function is the last
> > (or only) symbol in the .text section.  The next symbol in the sorted
> > list might actually be a data symbol, which can break the function size
> > detection:
> >
> >   $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
> >   bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> >
> > Similar breakage can occur when reading from a .o file.
> >
> > Fix it by only looking for text symbols.
> >
> > Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> > Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > ---
> >  scripts/faddr2line | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > index 6c6439f69a72..2a130134f1e6 100755
> > --- a/scripts/faddr2line
> > +++ b/scripts/faddr2line
> > @@ -189,7 +189,7 @@ __faddr2line() {
> >
> >                 DONE=1
> >
> > -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > +       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> >  }
> >
> >  [[ $# -lt 2 ]] && usage
> > --
> > 2.31.1
> >
> 

-- 
Josh

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

On Tue, Jan 18, 2022 at 11:22 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> > Hi Josh,
> >
> > Actually your first patch - the one you mentioned had other issues -
> > worked perfectly when applied:
> >
> > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > do_the_work+0x16f/0x0000000000000194:
> > do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
> >
> > The second one still failed in the same manner:
> >
> > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> >
> > So, is it possible to fixup issues with the first version?
> > What are these issues?
>
> The first patch basically reverts the fix in commit efdb4167e676
> ("scripts/faddr2line: Fix "size mismatch" error").  That would be nice
> as it's simpler and more robust, but unfortunately it would cause a lot
> of "size mismatch" errors with vmlinux symbols.

I see... if it comes to it, could there be a workaround where the
script detects a vmlinux image and works in a given manner,
else in this manner for other images - in effect, for modules?

>
> Can you give the output of 'nm -n ./oops_tryv2.ko'?  There must be some
> text symbol immediately after the do_the_work() symbol which is either
> out of order, or part of another section.
>
> Is do_the_work() in the .text section?
Yes...
Here's the output I get:

$ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
0000000000000000 r __func__.24215
0000000000000000 r __param_bug_in_workq
0000000000000000 D __this_module
0000000000000000 r _note_7
0000000000000000 T cleanup_module
0000000000000000 t do_the_work
0000000000000000 t do_the_work.cold
0000000000000000 b gctx
0000000000000000 T init_module
0000000000000000 t try_oops_exit
0000000000000000 t try_oops_init
0000000000000008 b t1
$

BTW, here's the code:
https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2

Thanks,
Kaiwan.

>
> > On Tue, Jan 18, 2022 at 1:57 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > >
> > > On Mon, Jan 17, 2022 at 11:48:39AM -0800, Josh Poimboeuf wrote:
> > > > On Mon, Jan 17, 2022 at 10:27:14AM +0530, Kaiwan N Billimoria wrote:
> > > > > Hi there,
> > > > >
> > > > > Am researching using the cool faddr2line script to help debug Oops'es
> > > > > from kernel modules..
> > > > > I find it works just fine when used against the unstripped vmlinux
> > > > > with debug symbols.
> > > > >
> > > > > My use case is for a kernel module which Oopses, though. Here's my scenario:
> > > > > I built a module on a custom debug kernel (5.10.60) with most debug
> > > > > options enabled...
> > > > > KASLR is enabled by default as well.
> > > > >
> > > > > A test kernel module Oopses on my x86_64 guest running this kernel with:
> > > > > RIP: 0010:do_the_work+0x15b/0x174 [oops_tryv2]
> > > > >
> > > > > So, i try this:
> > > > >
> > > > > $ <...>/linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x15b/0x174
> > > > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > > > > $
> > > > >
> > > > > (It works fine with addr2line though!).
> > > > > Now I think I've traced the faddr2line script's failure to locate
> > > > > anything down to this:
> > > > > ...
> > > > > done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 ==
> > > > > fn { found=1; line    =$0; start=$1; next } found == 1 { found=0;
> > > > > print line, "0x"$1 } END {if (found == 1) print     line, end; }')
> > > > >
> > > > > The nm output is:
> > > > > $ nm -n ./oops_tryv2.ko |grep -i do_the_work
> > > > > 0000000000000000 t do_the_work
> > > > > $
> > > > >
> > > > > nm shows the text addr as 0x0; this is obviously incorrect (same 0x0
> > > > > with objdump -d on the module).
> > > > > Am I missing something? Any suggestions as to what I can try, to get
> > > > > faddr2line working?
> > > >
> > > > Hi Kaiwan,
> > > >
> > > > Thanks for reporting this issue.  The module text address of 0x0 is not
> > > > necessarily incorrect, as the address is relative the the module, where
> > > > all text usually starts at zero.
> > > >
> > > > I was able to recreate this problem using a module which only has a
> > > > single function in .text.  Does this fix it?
> > >
> > > Actually, that patch has other problems.  Try this one?
> > >
> > > ----
> > >
> > > From: Josh Poimboeuf <jpoimboe@redhat.com>
> > > Subject: [PATCH] scripts/faddr2line: Only look for text symbols when
> > >  calculating function size
> > >
> > > With the following commit:
> > >
> > >   efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > >
> > > ... it was discovered that faddr2line can't just read a function's ELF
> > > size, because that wouldn't match the kallsyms function size which is
> > > printed in the stack trace.  The kallsyms size includes any padding
> > > after the function, whereas the ELF size does not.
> > >
> > > So faddr2line has to manually calculate the size of a function similar
> > > to how kallsyms does.  It does so by starting with a sorted list of
> > > symbols and subtracting the function address from the subsequent
> > > symbol's address.
> > >
> > > That calculation is broken in the case where the function is the last
> > > (or only) symbol in the .text section.  The next symbol in the sorted
> > > list might actually be a data symbol, which can break the function size
> > > detection:
> > >
> > >   $ scripts/faddr2line sound/soundcore.ko sound_devnode+0x5/0x35
> > >   bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > >
> > > Similar breakage can occur when reading from a .o file.
> > >
> > > Fix it by only looking for text symbols.
> > >
> > > Fixes: efdb4167e676 ("scripts/faddr2line: Fix "size mismatch" error")
> > > Reported-by: Kaiwan N Billimoria <kaiwan.billimoria@gmail.com>
> > > Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > > ---
> > >  scripts/faddr2line | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > > index 6c6439f69a72..2a130134f1e6 100755
> > > --- a/scripts/faddr2line
> > > +++ b/scripts/faddr2line
> > > @@ -189,7 +189,7 @@ __faddr2line() {
> > >
> > >                 DONE=1
> > >
> > > -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > > +       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$2 !~ /[Tt]/ {next} $3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > >  }
> > >
> > >  [[ $# -lt 2 ]] && usage
> > > --
> > > 2.31.1
> > >
> >
>
> --
> Josh
>

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Wed, Jan 19, 2022 at 09:27:18AM +0530, Kaiwan N Billimoria wrote:
> On Tue, Jan 18, 2022 at 11:22 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Tue, Jan 18, 2022 at 08:10:28AM +0530, Kaiwan N Billimoria wrote:
> > > Hi Josh,
> > >
> > > Actually your first patch - the one you mentioned had other issues -
> > > worked perfectly when applied:
> > >
> > > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > > do_the_work+0x16f/0x0000000000000194:
> > > do_the_work at <...>/oops_tryv2/oops_tryv2.c:62
> > >
> > > The second one still failed in the same manner:
> > >
> > > scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
> > > bad symbol size: base: 0x0000000000000000 end: 0x0000000000000000
> > >
> > > So, is it possible to fixup issues with the first version?
> > > What are these issues?
> >
> > The first patch basically reverts the fix in commit efdb4167e676
> > ("scripts/faddr2line: Fix "size mismatch" error").  That would be nice
> > as it's simpler and more robust, but unfortunately it would cause a lot
> > of "size mismatch" errors with vmlinux symbols.
> 
> I see... if it comes to it, could there be a workaround where the
> script detects a vmlinux image and works in a given manner,
> else in this manner for other images - in effect, for modules?

I think I misspoke about the vmlinux behavior being different, so I
don't think that would work.

> > Can you give the output of 'nm -n ./oops_tryv2.ko'?  There must be some
> > text symbol immediately after the do_the_work() symbol which is either
> > out of order, or part of another section.
> >
> > Is do_the_work() in the .text section?
> Yes...
> Here's the output I get:
> 
> $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> 0000000000000000 r __func__.24215
> 0000000000000000 r __param_bug_in_workq
> 0000000000000000 D __this_module
> 0000000000000000 r _note_7
> 0000000000000000 T cleanup_module
> 0000000000000000 t do_the_work
> 0000000000000000 t do_the_work.cold
> 0000000000000000 b gctx
> 0000000000000000 T init_module
> 0000000000000000 t try_oops_exit
> 0000000000000000 t try_oops_init
> 0000000000000008 b t1
> $
> 
> BTW, here's the code:
> https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2

Ok, it looks like the symbols aren't sorted like the code expects.  I
need to do a more robust fix.

-- 
Josh

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > 0000000000000000 r __func__.24215
> > 0000000000000000 r __param_bug_in_workq
> > 0000000000000000 D __this_module
> > 0000000000000000 r _note_7
> > 0000000000000000 T cleanup_module
> > 0000000000000000 t do_the_work
> > 0000000000000000 t do_the_work.cold
> > 0000000000000000 b gctx
> > 0000000000000000 T init_module
> > 0000000000000000 t try_oops_exit
> > 0000000000000000 t try_oops_init
> > 0000000000000008 b t1
> > $
> > 
> > BTW, here's the code:
> > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> 
> Ok, it looks like the symbols aren't sorted like the code expects.  I
> need to do a more robust fix.

Ok, please try this instead.  This takes a much more robust approach to
the function size calculation, using readelf to confine the symbol
search to the section matching the original symbol.

This actually has multiple fixes and cleanups, so it'll eventually be
split up into a patch set.

----

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..1acb68927977 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,10 +97,11 @@ __faddr2line() {
 	local dir_prefix=$3
 	local print_warnings=$4
 
+	local first=1
 	local func=${func_addr%+*}
 	local offset=${func_addr#*+}
 	offset=${offset%/*}
-	local size=
+	local user_size=
 	[[ $func_addr =~ "/" ]] && size=${func_addr#*/}
 
 	if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
@@ -110,73 +111,87 @@ __faddr2line() {
 	fi
 
 	# Go through each of the object's symbols which match the func name.
-	# In rare cases there might be duplicates.
-	file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
+	# In rare cases there might be duplicates, in which case we print both.
 	while read symbol; do
 		local fields=($symbol)
-		local sym_base=0x${fields[0]}
-		local sym_type=${fields[1]}
-		local sym_end=${fields[3]}
-
-		# calculate the size
-		local sym_size=$(($sym_end - $sym_base))
-		if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
-			warn "bad symbol size: base: $sym_base end: $sym_end"
+		local sym_addr=0x${fields[1]}
+		local sym_size=${fields[2]}
+		local sym_sec=${fields[6]}
+
+		# Get the section size:
+		local sec_size=$(${READELF} --section-headers --wide $objfile |
+			sed 's/\[ /\[/' |
+			awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+		if [[ -z $sec_size ]]; then
+			warn "bad section size: section: $sym_sec"
 			DONE=1
 			return
 		fi
-		sym_size=0x$(printf %x $sym_size)
 
-		# calculate the address
-		local addr=$(($sym_base + $offset))
-		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
-			warn "bad address: $sym_base + $offset"
+		# Calculate the symbol size:
+		#
+		# We can't use the ELF size, because kallsyms also includes the
+		# padding bytes in its size calculation.  For kallsyms, the
+		# size calculation is the distance between the symbol and the
+		# next symbol in a sorted list.
+		local size=$(${READELF} --symbols --wide $objfile |
+			awk -v sec=$sym_sec '$7 == sec' |
+			sort --key=2 |
+			awk --bignum -v sym_addr=${sym_addr#0x} -v sym_size=${sym_size} -v sym_name=${func} -v sec_size=${sec_size} \
+				'$2 == sym_addr && $3 == sym_size && $8 == sym_name { found = 1; next } \
+				found == 1 { size = strtonum("0x" $2) - strtonum("0x" sym_addr); if (size < sym_size) next; found = 2; print size; exit } \
+				END { if (found == 1) print strtonum(sec_size) }')
+
+		if [[ -z $size ]]; then
+			warn "bad symbol size: addr: $sym_addr"
 			DONE=1
 			return
 		fi
-		addr=0x$(printf %x $addr)
 
-		# weed out non-function symbols
-		if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
-			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
-			continue
+		# Calculate the specified address:
+		local addr=$(($sym_addr + $offset))
+		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+			warn "bad address: $sym_addr + $offset"
+			DONE=1
+			return
 		fi
+		addr=0x$(printf %x $addr)
 
-		# if the user provided a size, make sure it matches the symbol's size
-		if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+		# If the user provided a size, make sure it matches the symbol's size:
+		if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+				echo "skipping $func address at $addr due to size mismatch ($user_size != $size)"
 			continue;
 		fi
 
-		# make sure the provided offset is within the symbol's range
-		if [[ $offset -gt $sym_size ]]; then
+		# Make sure the provided offset is within the symbol's range:
+		if [[ $offset -gt $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+				echo "skipping $func address at $addr due to size mismatch ($offset > $size)"
 			continue
 		fi
 
-		# separate multiple entries with a blank line
-		[[ $FIRST = 0 ]] && echo
-		FIRST=0
+		# In case of duplicates, separate multiple entries with a blank line:
+		[[ $first = 0 ]] && echo
+		first=0
 
-		# pass real address to addr2line
-		echo "$func+$offset/$sym_size:"
-		local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
-		[[ -z $file_lines ]] && return
+		# Pass full address to addr2line:
+		echo "$func+$offset/$size:"
+		local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+		[[ -z $output ]] && continue
 
 		if [[ $LIST = 0 ]]; then
-			echo "$file_lines" | while read -r line
+			echo "$output" | while read -r line
 			do
 				echo $line
 			done
 			DONE=1;
-			return
+			continue
 		fi
 
-		# show each line with context
-		echo "$file_lines" | while read -r line
+		# If --list was specified, show each line with context:
+		echo "$output" | while read -r line
 		do
 			echo
 			echo $line
@@ -189,7 +204,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+	done < <(${READELF} --symbols --wide $objfile | awk -v fn=$func '$4 == "FUNC" && $8 == fn')
 }
 
 [[ $# -lt 2 ]] && usage

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > 0000000000000000 r __func__.24215
> > > 0000000000000000 r __param_bug_in_workq
> > > 0000000000000000 D __this_module
> > > 0000000000000000 r _note_7
> > > 0000000000000000 T cleanup_module
> > > 0000000000000000 t do_the_work
> > > 0000000000000000 t do_the_work.cold
> > > 0000000000000000 b gctx
> > > 0000000000000000 T init_module
> > > 0000000000000000 t try_oops_exit
> > > 0000000000000000 t try_oops_init
> > > 0000000000000008 b t1
> > > $
> > >
> > > BTW, here's the code:
> > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> >
> > Ok, it looks like the symbols aren't sorted like the code expects.  I
> > need to do a more robust fix.
>
> Ok, please try this instead.  This takes a much more robust approach to
> the function size calculation, using readelf to confine the symbol
> search to the section matching the original symbol.
>
> This actually has multiple fixes and cleanups, so it'll eventually be
> split up into a patch set.

Great. Am facing a few issues though:

1. Not sure if it's me but am having issues applying your patches; it
seems to fail:

$ patch -p1 --dry-run < ./fa1.patch
checking file scripts/faddr2line
Hunk #1 FAILED at 97.
Hunk #2 FAILED at 110.
Hunk #3 FAILED at 189.
3 out of 3 hunks FAILED
$

Against which ver are you generating the patch?
(Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
in Nov 2018, so vanilla 5.10.60 should be fine, no?).

2. I manually applied your patch :)   This is the output I then get:

$ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x16f/0x194
awk: not an option: --bignum
bad symbol size: addr: 0x0000000000000000
$

(Not sure if this helps but am running on Ubuntu 20.04; my awk is mawk
v 1.3.4.20200120-2).

Thanks,
Kaiwan.

>
> ----
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..1acb68927977 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -97,10 +97,11 @@ __faddr2line() {
>         local dir_prefix=$3
>         local print_warnings=$4
>
> +       local first=1
>         local func=${func_addr%+*}
>         local offset=${func_addr#*+}
>         offset=${offset%/*}
> -       local size=
> +       local user_size=
>         [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
>
>         if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> @@ -110,73 +111,87 @@ __faddr2line() {
>         fi
>
>         # Go through each of the object's symbols which match the func name.
> -       # In rare cases there might be duplicates.
> -       file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> +       # In rare cases there might be duplicates, in which case we print both.
>         while read symbol; do
>                 local fields=($symbol)
> -               local sym_base=0x${fields[0]}
> -               local sym_type=${fields[1]}
> -               local sym_end=${fields[3]}
> -
> -               # calculate the size
> -               local sym_size=$(($sym_end - $sym_base))
> -               if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> -                       warn "bad symbol size: base: $sym_base end: $sym_end"
> +               local sym_addr=0x${fields[1]}
> +               local sym_size=${fields[2]}
> +               local sym_sec=${fields[6]}
> +
> +               # Get the section size:
> +               local sec_size=$(${READELF} --section-headers --wide $objfile |
> +                       sed 's/\[ /\[/' |
> +                       awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> +
> +               if [[ -z $sec_size ]]; then
> +                       warn "bad section size: section: $sym_sec"
>                         DONE=1
>                         return
>                 fi
> -               sym_size=0x$(printf %x $sym_size)
>
> -               # calculate the address
> -               local addr=$(($sym_base + $offset))
> -               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> -                       warn "bad address: $sym_base + $offset"
> +               # Calculate the symbol size:
> +               #
> +               # We can't use the ELF size, because kallsyms also includes the
> +               # padding bytes in its size calculation.  For kallsyms, the
> +               # size calculation is the distance between the symbol and the
> +               # next symbol in a sorted list.
> +               local size=$(${READELF} --symbols --wide $objfile |
> +                       awk -v sec=$sym_sec '$7 == sec' |
> +                       sort --key=2 |
> +                       awk --bignum -v sym_addr=${sym_addr#0x} -v sym_size=${sym_size} -v sym_name=${func} -v sec_size=${sec_size} \
> +                               '$2 == sym_addr && $3 == sym_size && $8 == sym_name { found = 1; next } \
> +                               found == 1 { size = strtonum("0x" $2) - strtonum("0x" sym_addr); if (size < sym_size) next; found = 2; print size; exit } \
> +                               END { if (found == 1) print strtonum(sec_size) }')
> +
> +               if [[ -z $size ]]; then
> +                       warn "bad symbol size: addr: $sym_addr"
>                         DONE=1
>                         return
>                 fi
> -               addr=0x$(printf %x $addr)
>
> -               # weed out non-function symbols
> -               if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> -                       [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> -                       continue
> +               # Calculate the specified address:
> +               local addr=$(($sym_addr + $offset))
> +               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> +                       warn "bad address: $sym_addr + $offset"
> +                       DONE=1
> +                       return
>                 fi
> +               addr=0x$(printf %x $addr)
>
> -               # if the user provided a size, make sure it matches the symbol's size
> -               if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> +               # If the user provided a size, make sure it matches the symbol's size:
> +               if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
>                         [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> +                               echo "skipping $func address at $addr due to size mismatch ($user_size != $size)"
>                         continue;
>                 fi
>
> -               # make sure the provided offset is within the symbol's range
> -               if [[ $offset -gt $sym_size ]]; then
> +               # Make sure the provided offset is within the symbol's range:
> +               if [[ $offset -gt $size ]]; then
>                         [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> +                               echo "skipping $func address at $addr due to size mismatch ($offset > $size)"
>                         continue
>                 fi
>
> -               # separate multiple entries with a blank line
> -               [[ $FIRST = 0 ]] && echo
> -               FIRST=0
> +               # In case of duplicates, separate multiple entries with a blank line:
> +               [[ $first = 0 ]] && echo
> +               first=0
>
> -               # pass real address to addr2line
> -               echo "$func+$offset/$sym_size:"
> -               local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> -               [[ -z $file_lines ]] && return
> +               # Pass full address to addr2line:
> +               echo "$func+$offset/$size:"
> +               local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> +               [[ -z $output ]] && continue
>
>                 if [[ $LIST = 0 ]]; then
> -                       echo "$file_lines" | while read -r line
> +                       echo "$output" | while read -r line
>                         do
>                                 echo $line
>                         done
>                         DONE=1;
> -                       return
> +                       continue
>                 fi
>
> -               # show each line with context
> -               echo "$file_lines" | while read -r line
> +               # If --list was specified, show each line with context:
> +               echo "$output" | while read -r line
>                 do
>                         echo
>                         echo $line
> @@ -189,7 +204,7 @@ __faddr2line() {
>
>                 DONE=1
>
> -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> +       done < <(${READELF} --symbols --wide $objfile | awk -v fn=$func '$4 == "FUNC" && $8 == fn')
>  }
>
>  [[ $# -lt 2 ]] && usage
>

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

[-- Attachment #1: Type: text/plain, Size: 8639 bytes --]

On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > 0000000000000000 r __func__.24215
> > > > 0000000000000000 r __param_bug_in_workq
> > > > 0000000000000000 D __this_module
> > > > 0000000000000000 r _note_7
> > > > 0000000000000000 T cleanup_module
> > > > 0000000000000000 t do_the_work
> > > > 0000000000000000 t do_the_work.cold
> > > > 0000000000000000 b gctx
> > > > 0000000000000000 T init_module
> > > > 0000000000000000 t try_oops_exit
> > > > 0000000000000000 t try_oops_init
> > > > 0000000000000008 b t1
> > > > $
> > > >
> > > > BTW, here's the code:
> > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > >
> > > Ok, it looks like the symbols aren't sorted like the code expects.  I
> > > need to do a more robust fix.
> >
> > Ok, please try this instead.  This takes a much more robust approach to
> > the function size calculation, using readelf to confine the symbol
> > search to the section matching the original symbol.
> >
> > This actually has multiple fixes and cleanups, so it'll eventually be
> > split up into a patch set.
> 
> Great. Am facing a few issues though:
> 
> 1. Not sure if it's me but am having issues applying your patches; it
> seems to fail:
> 
> $ patch -p1 --dry-run < ./fa1.patch
> checking file scripts/faddr2line
> Hunk #1 FAILED at 97.
> Hunk #2 FAILED at 110.
> Hunk #3 FAILED at 189.
> 3 out of 3 hunks FAILED
> $

Hm, did you happen to have the older patch already applied?  This should
be applied instead of that one.  Though it's definitely possible I
messed the patch up somehow.

When saving the patch, make sure to preserve the original whitespace.

I'm adding the new patch here as an attachment, if that helps.

> Against which ver are you generating the patch?
> (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> in Nov 2018, so vanilla 5.10.60 should be fine, no?).

It was against a newer kernel, but yes, applying it against 5.10.60
should also work.
> 
> 2. I manually applied your patch :)   This is the output I then get:
> 
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work+0x16f/0x194 awk: not an option: --bignum bad symbol size:
> addr: 0x0000000000000000 $
> 
> (Not sure if this helps but am running on Ubuntu 20.04; my awk is mawk
> v 1.3.4.20200120-2).

Sorry, I didn't realize mawk was still a thing :-)

Try this one?

----

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..01f28e52f3fc 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,86 +97,127 @@ __faddr2line() {
 	local dir_prefix=$3
 	local print_warnings=$4
 
-	local func=${func_addr%+*}
+	local first=1
+	local sym_name=${func_addr%+*}
 	local offset=${func_addr#*+}
 	offset=${offset%/*}
-	local size=
-	[[ $func_addr =~ "/" ]] && size=${func_addr#*/}
+	local user_size=
+	[[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
 
-	if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
+	if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
 		warn "bad func+offset $func_addr"
 		DONE=1
 		return
 	fi
 
 	# Go through each of the object's symbols which match the func name.
-	# In rare cases there might be duplicates.
-	file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
-	while read symbol; do
-		local fields=($symbol)
-		local sym_base=0x${fields[0]}
-		local sym_type=${fields[1]}
-		local sym_end=${fields[3]}
-
-		# calculate the size
-		local sym_size=$(($sym_end - $sym_base))
-		if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
-			warn "bad symbol size: base: $sym_base end: $sym_end"
+	# In rare cases there might be duplicates, in which case we print all
+	# matches.
+	while read line; do
+		local fields=($line)
+		local sym_addr=0x${fields[1]}
+		local sym_size=${fields[2]}
+		local sym_sec=${fields[6]}
+
+		# Get the section size:
+		local sec_size=$(${READELF} --section-headers --wide $objfile |
+			sed 's/\[ /\[/' |
+			awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+		if [[ -z $sec_size ]]; then
+			warn "bad section size: section: $sym_sec"
 			DONE=1
 			return
 		fi
-		sym_size=0x$(printf %x $sym_size)
 
-		# calculate the address
-		local addr=$(($sym_base + $offset))
-		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
-			warn "bad address: $sym_base + $offset"
+		# Calculate the symbol size:
+		#
+		# We can't use the ELF size, because kallsyms also includes the
+		# padding bytes in its size calculation.  For kallsyms, the
+		# size calculation is the distance between the symbol and the
+		# next symbol in a sorted list.
+		local size
+		local next_sym_addr
+		local found=0
+		while read line; do
+			local fields=($line)
+			next_sym_addr=0x${fields[1]}
+			local next_sym_size=${fields[2]}
+			local next_sym_name=${fields[7]:-}
+
+			if [[ $next_sym_addr = $sym_addr ]] &&
+			   [[ $next_sym_size = $sym_size ]] &&
+			   [[ $next_sym_name = $sym_name ]]; then
+				found=1
+				continue
+			fi
+
+			if [[ $found = 1 ]]; then
+				size=$(($next_sym_addr - $sym_addr))
+				[[ $size -lt $sym_size ]] && continue;
+				found=2
+				break
+			fi
+		done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
+
+		if [[ $found = 0 ]]; then
+			warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
 			DONE=1
 			return
 		fi
-		addr=0x$(printf %x $addr)
 
-		# weed out non-function symbols
-		if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
-			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
-			continue
+		[[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
+
+		if [[ -z $size ]] || [[ $size -le 0 ]]; then
+			warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
+			DONE=1
+			return
 		fi
+		size=0x$(printf %x $size)
 
-		# if the user provided a size, make sure it matches the symbol's size
-		if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+		# Calculate the specified address:
+		local addr=$(($sym_addr + $offset))
+		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+			warn "bad address: $sym_addr + $offset"
+			DONE=1
+			return
+		fi
+		addr=0x$(printf %x $addr)
+
+		# If the user provided a size, make sure it matches the symbol's size:
+		if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+				echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
 			continue;
 		fi
 
-		# make sure the provided offset is within the symbol's range
-		if [[ $offset -gt $sym_size ]]; then
+		# Make sure the provided offset is within the symbol's range:
+		if [[ $offset -gt $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+				echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
 			continue
 		fi
 
-		# separate multiple entries with a blank line
-		[[ $FIRST = 0 ]] && echo
-		FIRST=0
+		# In case of duplicates, separate multiple entries with a blank line:
+		[[ $first = 0 ]] && echo
+		first=0
 
-		# pass real address to addr2line
-		echo "$func+$offset/$sym_size:"
-		local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
-		[[ -z $file_lines ]] && return
+		# Pass full address to addr2line:
+		echo "$sym_name+$offset/$size:"
+		local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+		[[ -z $output ]] && continue
 
 		if [[ $LIST = 0 ]]; then
-			echo "$file_lines" | while read -r line
+			echo "$output" | while read -r line
 			do
 				echo $line
 			done
 			DONE=1;
-			return
+			continue
 		fi
 
-		# show each line with context
-		echo "$file_lines" | while read -r line
+		# If --list was specified, show each line with context:
+		echo "$output" | while read -r line
 		do
 			echo
 			echo $line
@@ -189,7 +230,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+	done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
 }
 
 [[ $# -lt 2 ]] && usage

[-- Attachment #2: faddr2line-fix.patch --]
[-- Type: text/plain, Size: 6025 bytes --]

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 6c6439f69a72..01f28e52f3fc 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -97,86 +97,127 @@ __faddr2line() {
 	local dir_prefix=$3
 	local print_warnings=$4
 
-	local func=${func_addr%+*}
+	local first=1
+	local sym_name=${func_addr%+*}
 	local offset=${func_addr#*+}
 	offset=${offset%/*}
-	local size=
-	[[ $func_addr =~ "/" ]] && size=${func_addr#*/}
+	local user_size=
+	[[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
 
-	if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
+	if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
 		warn "bad func+offset $func_addr"
 		DONE=1
 		return
 	fi
 
 	# Go through each of the object's symbols which match the func name.
-	# In rare cases there might be duplicates.
-	file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
-	while read symbol; do
-		local fields=($symbol)
-		local sym_base=0x${fields[0]}
-		local sym_type=${fields[1]}
-		local sym_end=${fields[3]}
-
-		# calculate the size
-		local sym_size=$(($sym_end - $sym_base))
-		if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
-			warn "bad symbol size: base: $sym_base end: $sym_end"
+	# In rare cases there might be duplicates, in which case we print all
+	# matches.
+	while read line; do
+		local fields=($line)
+		local sym_addr=0x${fields[1]}
+		local sym_size=${fields[2]}
+		local sym_sec=${fields[6]}
+
+		# Get the section size:
+		local sec_size=$(${READELF} --section-headers --wide $objfile |
+			sed 's/\[ /\[/' |
+			awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
+
+		if [[ -z $sec_size ]]; then
+			warn "bad section size: section: $sym_sec"
 			DONE=1
 			return
 		fi
-		sym_size=0x$(printf %x $sym_size)
 
-		# calculate the address
-		local addr=$(($sym_base + $offset))
-		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
-			warn "bad address: $sym_base + $offset"
+		# Calculate the symbol size:
+		#
+		# We can't use the ELF size, because kallsyms also includes the
+		# padding bytes in its size calculation.  For kallsyms, the
+		# size calculation is the distance between the symbol and the
+		# next symbol in a sorted list.
+		local size
+		local next_sym_addr
+		local found=0
+		while read line; do
+			local fields=($line)
+			next_sym_addr=0x${fields[1]}
+			local next_sym_size=${fields[2]}
+			local next_sym_name=${fields[7]:-}
+
+			if [[ $next_sym_addr = $sym_addr ]] &&
+			   [[ $next_sym_size = $sym_size ]] &&
+			   [[ $next_sym_name = $sym_name ]]; then
+				found=1
+				continue
+			fi
+
+			if [[ $found = 1 ]]; then
+				size=$(($next_sym_addr - $sym_addr))
+				[[ $size -lt $sym_size ]] && continue;
+				found=2
+				break
+			fi
+		done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
+
+		if [[ $found = 0 ]]; then
+			warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
 			DONE=1
 			return
 		fi
-		addr=0x$(printf %x $addr)
 
-		# weed out non-function symbols
-		if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
-			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
-			continue
+		[[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
+
+		if [[ -z $size ]] || [[ $size -le 0 ]]; then
+			warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
+			DONE=1
+			return
 		fi
+		size=0x$(printf %x $size)
 
-		# if the user provided a size, make sure it matches the symbol's size
-		if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
+		# Calculate the specified address:
+		local addr=$(($sym_addr + $offset))
+		if [[ -z $addr ]] || [[ $addr = 0 ]]; then
+			warn "bad address: $sym_addr + $offset"
+			DONE=1
+			return
+		fi
+		addr=0x$(printf %x $addr)
+
+		# If the user provided a size, make sure it matches the symbol's size:
+		if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
+				echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
 			continue;
 		fi
 
-		# make sure the provided offset is within the symbol's range
-		if [[ $offset -gt $sym_size ]]; then
+		# Make sure the provided offset is within the symbol's range:
+		if [[ $offset -gt $size ]]; then
 			[[ $print_warnings = 1 ]] &&
-				echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
+				echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
 			continue
 		fi
 
-		# separate multiple entries with a blank line
-		[[ $FIRST = 0 ]] && echo
-		FIRST=0
+		# In case of duplicates, separate multiple entries with a blank line:
+		[[ $first = 0 ]] && echo
+		first=0
 
-		# pass real address to addr2line
-		echo "$func+$offset/$sym_size:"
-		local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
-		[[ -z $file_lines ]] && return
+		# Pass full address to addr2line:
+		echo "$sym_name+$offset/$size:"
+		local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
+		[[ -z $output ]] && continue
 
 		if [[ $LIST = 0 ]]; then
-			echo "$file_lines" | while read -r line
+			echo "$output" | while read -r line
 			do
 				echo $line
 			done
 			DONE=1;
-			return
+			continue
 		fi
 
-		# show each line with context
-		echo "$file_lines" | while read -r line
+		# If --list was specified, show each line with context:
+		echo "$output" | while read -r line
 		do
 			echo
 			echo $line
@@ -189,7 +230,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
+	done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
 }
 
 [[ $# -lt 2 ]] && usage

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

On Thu, Jan 20, 2022 at 10:48 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> > On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > >
> > > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > > 0000000000000000 r __func__.24215
> > > > > 0000000000000000 r __param_bug_in_workq
> > > > > 0000000000000000 D __this_module
> > > > > 0000000000000000 r _note_7
> > > > > 0000000000000000 T cleanup_module
> > > > > 0000000000000000 t do_the_work
> > > > > 0000000000000000 t do_the_work.cold
> > > > > 0000000000000000 b gctx
> > > > > 0000000000000000 T init_module
> > > > > 0000000000000000 t try_oops_exit
> > > > > 0000000000000000 t try_oops_init
> > > > > 0000000000000008 b t1
> > > > > $
> > > > >
> > > > > BTW, here's the code:
> > > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > > >
> > > > Ok, it looks like the symbols aren't sorted like the code expects.  I
> > > > need to do a more robust fix.
> > >
> > > Ok, please try this instead.  This takes a much more robust approach to
> > > the function size calculation, using readelf to confine the symbol
> > > search to the section matching the original symbol.
> > >
> > > This actually has multiple fixes and cleanups, so it'll eventually be
> > > split up into a patch set.
> >
> > Great. Am facing a few issues though:
> >
> > 1. Not sure if it's me but am having issues applying your patches; it
> > seems to fail:
> >
> > $ patch -p1 --dry-run < ./fa1.patch
> > checking file scripts/faddr2line
> > Hunk #1 FAILED at 97.
> > Hunk #2 FAILED at 110.
> > Hunk #3 FAILED at 189.
> > 3 out of 3 hunks FAILED
> > $
>
> Hm, did you happen to have the older patch already applied?  This should
> be applied instead of that one.  Though it's definitely possible I
> messed the patch up somehow.
>
> When saving the patch, make sure to preserve the original whitespace.
>
> I'm adding the new patch here as an attachment, if that helps.

That helped! I think my stubbornly using Gmail perhaps caused issues
when i copy-pasted the earlier patches... thanks.
>
> > Against which ver are you generating the patch?
> > (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> > in Nov 2018, so vanilla 5.10.60 should be fine, no?).
>
> It was against a newer kernel, but yes, applying it against 5.10.60
> should also work.
Cool.
>...
> Sorry, I didn't realize mawk was still a thing :-)
>
> Try this one?

Definitely better !
This is the o/p i now get:

$ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
do_the_work.cold+0x68/0x126
do_the_work.cold+0x68/0x126:
delay_sec at <...>/oops_tryv2/../../convenient.h:279
$

The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
(I'm unsure why the line delay_sec at <...>convenient.h:279   shows up
; i don't call this delay_sec() func directly... anyway, that's my
problem.)

Thanks.
>
> ----
>
> diff --git a/scripts/faddr2line b/scripts/faddr2line
> index 6c6439f69a72..01f28e52f3fc 100755
> --- a/scripts/faddr2line
> +++ b/scripts/faddr2line
> @@ -97,86 +97,127 @@ __faddr2line() {
>         local dir_prefix=$3
>         local print_warnings=$4
>
> -       local func=${func_addr%+*}
> +       local first=1
> +       local sym_name=${func_addr%+*}
>         local offset=${func_addr#*+}
>         offset=${offset%/*}
> -       local size=
> -       [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
> +       local user_size=
> +       [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
>
> -       if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> +       if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
>                 warn "bad func+offset $func_addr"
>                 DONE=1
>                 return
>         fi
>
>         # Go through each of the object's symbols which match the func name.
> -       # In rare cases there might be duplicates.
> -       file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> -       while read symbol; do
> -               local fields=($symbol)
> -               local sym_base=0x${fields[0]}
> -               local sym_type=${fields[1]}
> -               local sym_end=${fields[3]}
> -
> -               # calculate the size
> -               local sym_size=$(($sym_end - $sym_base))
> -               if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> -                       warn "bad symbol size: base: $sym_base end: $sym_end"
> +       # In rare cases there might be duplicates, in which case we print all
> +       # matches.
> +       while read line; do
> +               local fields=($line)
> +               local sym_addr=0x${fields[1]}
> +               local sym_size=${fields[2]}
> +               local sym_sec=${fields[6]}
> +
> +               # Get the section size:
> +               local sec_size=$(${READELF} --section-headers --wide $objfile |
> +                       sed 's/\[ /\[/' |
> +                       awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> +
> +               if [[ -z $sec_size ]]; then
> +                       warn "bad section size: section: $sym_sec"
>                         DONE=1
>                         return
>                 fi
> -               sym_size=0x$(printf %x $sym_size)
>
> -               # calculate the address
> -               local addr=$(($sym_base + $offset))
> -               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> -                       warn "bad address: $sym_base + $offset"
> +               # Calculate the symbol size:
> +               #
> +               # We can't use the ELF size, because kallsyms also includes the
> +               # padding bytes in its size calculation.  For kallsyms, the
> +               # size calculation is the distance between the symbol and the
> +               # next symbol in a sorted list.
> +               local size
> +               local next_sym_addr
> +               local found=0
> +               while read line; do
> +                       local fields=($line)
> +                       next_sym_addr=0x${fields[1]}
> +                       local next_sym_size=${fields[2]}
> +                       local next_sym_name=${fields[7]:-}
> +
> +                       if [[ $next_sym_addr = $sym_addr ]] &&
> +                          [[ $next_sym_size = $sym_size ]] &&
> +                          [[ $next_sym_name = $sym_name ]]; then
> +                               found=1
> +                               continue
> +                       fi
> +
> +                       if [[ $found = 1 ]]; then
> +                               size=$(($next_sym_addr - $sym_addr))
> +                               [[ $size -lt $sym_size ]] && continue;
> +                               found=2
> +                               break
> +                       fi
> +               done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
> +
> +               if [[ $found = 0 ]]; then
> +                       warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
>                         DONE=1
>                         return
>                 fi
> -               addr=0x$(printf %x $addr)
>
> -               # weed out non-function symbols
> -               if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> -                       [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> -                       continue
> +               [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
> +
> +               if [[ -z $size ]] || [[ $size -le 0 ]]; then
> +                       warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
> +                       DONE=1
> +                       return
>                 fi
> +               size=0x$(printf %x $size)
>
> -               # if the user provided a size, make sure it matches the symbol's size
> -               if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> +               # Calculate the specified address:
> +               local addr=$(($sym_addr + $offset))
> +               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> +                       warn "bad address: $sym_addr + $offset"
> +                       DONE=1
> +                       return
> +               fi
> +               addr=0x$(printf %x $addr)
> +
> +               # If the user provided a size, make sure it matches the symbol's size:
> +               if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
>                         [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> +                               echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
>                         continue;
>                 fi
>
> -               # make sure the provided offset is within the symbol's range
> -               if [[ $offset -gt $sym_size ]]; then
> +               # Make sure the provided offset is within the symbol's range:
> +               if [[ $offset -gt $size ]]; then
>                         [[ $print_warnings = 1 ]] &&
> -                               echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> +                               echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
>                         continue
>                 fi
>
> -               # separate multiple entries with a blank line
> -               [[ $FIRST = 0 ]] && echo
> -               FIRST=0
> +               # In case of duplicates, separate multiple entries with a blank line:
> +               [[ $first = 0 ]] && echo
> +               first=0
>
> -               # pass real address to addr2line
> -               echo "$func+$offset/$sym_size:"
> -               local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> -               [[ -z $file_lines ]] && return
> +               # Pass full address to addr2line:
> +               echo "$sym_name+$offset/$size:"
> +               local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> +               [[ -z $output ]] && continue
>
>                 if [[ $LIST = 0 ]]; then
> -                       echo "$file_lines" | while read -r line
> +                       echo "$output" | while read -r line
>                         do
>                                 echo $line
>                         done
>                         DONE=1;
> -                       return
> +                       continue
>                 fi
>
> -               # show each line with context
> -               echo "$file_lines" | while read -r line
> +               # If --list was specified, show each line with context:
> +               echo "$output" | while read -r line
>                 do
>                         echo
>                         echo $line
> @@ -189,7 +230,7 @@ __faddr2line() {
>
>                 DONE=1
>
> -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> +       done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
>  }
>
>  [[ $# -lt 2 ]] && usage

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

Hi Josh,

<gentle ping>
I understand you must be busy, just checking...
any update on the faddr2line script?

Thanks,
Kaiwan.

On Fri, Jan 21, 2022 at 1:35 AM Kaiwan N Billimoria
<kaiwan.billimoria@gmail.com> wrote:
>
> On Thu, Jan 20, 2022 at 10:48 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Thu, Jan 20, 2022 at 10:44:44AM +0530, Kaiwan N Billimoria wrote:
> > > On Thu, Jan 20, 2022 at 5:13 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > > >
> > > > On Wed, Jan 19, 2022 at 01:21:29PM -0800, Josh Poimboeuf wrote:
> > > > > > $ nm -n ./oops_tryv2.ko |grep -C5 do_the_work
> > > > > > 0000000000000000 r __func__.24215
> > > > > > 0000000000000000 r __param_bug_in_workq
> > > > > > 0000000000000000 D __this_module
> > > > > > 0000000000000000 r _note_7
> > > > > > 0000000000000000 T cleanup_module
> > > > > > 0000000000000000 t do_the_work
> > > > > > 0000000000000000 t do_the_work.cold
> > > > > > 0000000000000000 b gctx
> > > > > > 0000000000000000 T init_module
> > > > > > 0000000000000000 t try_oops_exit
> > > > > > 0000000000000000 t try_oops_init
> > > > > > 0000000000000008 b t1
> > > > > > $
> > > > > >
> > > > > > BTW, here's the code:
> > > > > > https://github.com/PacktPublishing/Linux-Kernel-Debugging/tree/main/ch7/oops_tryv2
> > > > >
> > > > > Ok, it looks like the symbols aren't sorted like the code expects.  I
> > > > > need to do a more robust fix.
> > > >
> > > > Ok, please try this instead.  This takes a much more robust approach to
> > > > the function size calculation, using readelf to confine the symbol
> > > > search to the section matching the original symbol.
> > > >
> > > > This actually has multiple fixes and cleanups, so it'll eventually be
> > > > split up into a patch set.
> > >
> > > Great. Am facing a few issues though:
> > >
> > > 1. Not sure if it's me but am having issues applying your patches; it
> > > seems to fail:
> > >
> > > $ patch -p1 --dry-run < ./fa1.patch
> > > checking file scripts/faddr2line
> > > Hunk #1 FAILED at 97.
> > > Hunk #2 FAILED at 110.
> > > Hunk #3 FAILED at 189.
> > > 3 out of 3 hunks FAILED
> > > $
> >
> > Hm, did you happen to have the older patch already applied?  This should
> > be applied instead of that one.  Though it's definitely possible I
> > messed the patch up somehow.
> >
> > When saving the patch, make sure to preserve the original whitespace.
> >
> > I'm adding the new patch here as an attachment, if that helps.
>
> That helped! I think my stubbornly using Gmail perhaps caused issues
> when i copy-pasted the earlier patches... thanks.
> >
> > > Against which ver are you generating the patch?
> > > (Am using 5.10.60; i noticed the last commit in mainline was f5f67cc
> > > in Nov 2018, so vanilla 5.10.60 should be fine, no?).
> >
> > It was against a newer kernel, but yes, applying it against 5.10.60
> > should also work.
> Cool.
> >...
> > Sorry, I didn't realize mawk was still a thing :-)
> >
> > Try this one?
>
> Definitely better !
> This is the o/p i now get:
>
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work.cold+0x68/0x126
> do_the_work.cold+0x68/0x126:
> delay_sec at <...>/oops_tryv2/../../convenient.h:279
> $
>
> The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> (I'm unsure why the line delay_sec at <...>convenient.h:279   shows up
> ; i don't call this delay_sec() func directly... anyway, that's my
> problem.)
>
> Thanks.
> >
> > ----
> >
> > diff --git a/scripts/faddr2line b/scripts/faddr2line
> > index 6c6439f69a72..01f28e52f3fc 100755
> > --- a/scripts/faddr2line
> > +++ b/scripts/faddr2line
> > @@ -97,86 +97,127 @@ __faddr2line() {
> >         local dir_prefix=$3
> >         local print_warnings=$4
> >
> > -       local func=${func_addr%+*}
> > +       local first=1
> > +       local sym_name=${func_addr%+*}
> >         local offset=${func_addr#*+}
> >         offset=${offset%/*}
> > -       local size=
> > -       [[ $func_addr =~ "/" ]] && size=${func_addr#*/}
> > +       local user_size=
> > +       [[ $func_addr =~ "/" ]] && user_size=${func_addr#*/}
> >
> > -       if [[ -z $func ]] || [[ -z $offset ]] || [[ $func = $func_addr ]]; then
> > +       if [[ -z $sym_name ]] || [[ -z $offset ]] || [[ $sym_name = $func_addr ]]; then
> >                 warn "bad func+offset $func_addr"
> >                 DONE=1
> >                 return
> >         fi
> >
> >         # Go through each of the object's symbols which match the func name.
> > -       # In rare cases there might be duplicates.
> > -       file_end=$(${SIZE} -Ax $objfile | awk '$1 == ".text" {print $2}')
> > -       while read symbol; do
> > -               local fields=($symbol)
> > -               local sym_base=0x${fields[0]}
> > -               local sym_type=${fields[1]}
> > -               local sym_end=${fields[3]}
> > -
> > -               # calculate the size
> > -               local sym_size=$(($sym_end - $sym_base))
> > -               if [[ -z $sym_size ]] || [[ $sym_size -le 0 ]]; then
> > -                       warn "bad symbol size: base: $sym_base end: $sym_end"
> > +       # In rare cases there might be duplicates, in which case we print all
> > +       # matches.
> > +       while read line; do
> > +               local fields=($line)
> > +               local sym_addr=0x${fields[1]}
> > +               local sym_size=${fields[2]}
> > +               local sym_sec=${fields[6]}
> > +
> > +               # Get the section size:
> > +               local sec_size=$(${READELF} --section-headers --wide $objfile |
> > +                       sed 's/\[ /\[/' |
> > +                       awk -v sec=$sym_sec '$1 == "[" sec "]" { print "0x" $6; exit }')
> > +
> > +               if [[ -z $sec_size ]]; then
> > +                       warn "bad section size: section: $sym_sec"
> >                         DONE=1
> >                         return
> >                 fi
> > -               sym_size=0x$(printf %x $sym_size)
> >
> > -               # calculate the address
> > -               local addr=$(($sym_base + $offset))
> > -               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> > -                       warn "bad address: $sym_base + $offset"
> > +               # Calculate the symbol size:
> > +               #
> > +               # We can't use the ELF size, because kallsyms also includes the
> > +               # padding bytes in its size calculation.  For kallsyms, the
> > +               # size calculation is the distance between the symbol and the
> > +               # next symbol in a sorted list.
> > +               local size
> > +               local next_sym_addr
> > +               local found=0
> > +               while read line; do
> > +                       local fields=($line)
> > +                       next_sym_addr=0x${fields[1]}
> > +                       local next_sym_size=${fields[2]}
> > +                       local next_sym_name=${fields[7]:-}
> > +
> > +                       if [[ $next_sym_addr = $sym_addr ]] &&
> > +                          [[ $next_sym_size = $sym_size ]] &&
> > +                          [[ $next_sym_name = $sym_name ]]; then
> > +                               found=1
> > +                               continue
> > +                       fi
> > +
> > +                       if [[ $found = 1 ]]; then
> > +                               size=$(($next_sym_addr - $sym_addr))
> > +                               [[ $size -lt $sym_size ]] && continue;
> > +                               found=2
> > +                               break
> > +                       fi
> > +               done < <(${READELF} --symbols --wide $objfile | awk -v sec=$sym_sec '$7 == sec' | sort --key=2)
> > +
> > +               if [[ $found = 0 ]]; then
> > +                       warn "can't find symbol: sym_name: $sym_name sym_sec=$sym_sec sym_addr: $sym_addr sym_size: $sym_size"
> >                         DONE=1
> >                         return
> >                 fi
> > -               addr=0x$(printf %x $addr)
> >
> > -               # weed out non-function symbols
> > -               if [[ $sym_type != t ]] && [[ $sym_type != T ]]; then
> > -                       [[ $print_warnings = 1 ]] &&
> > -                               echo "skipping $func address at $addr due to non-function symbol of type '$sym_type'"
> > -                       continue
> > +               [[ $found = 1 ]] && size=$(($sec_size - $sym_addr))
> > +
> > +               if [[ -z $size ]] || [[ $size -le 0 ]]; then
> > +                       warn "bad symbol size: sym_addr: $sym_addr next_sym_addr: $next_sym_addr"
> > +                       DONE=1
> > +                       return
> >                 fi
> > +               size=0x$(printf %x $size)
> >
> > -               # if the user provided a size, make sure it matches the symbol's size
> > -               if [[ -n $size ]] && [[ $size -ne $sym_size ]]; then
> > +               # Calculate the specified address:
> > +               local addr=$(($sym_addr + $offset))
> > +               if [[ -z $addr ]] || [[ $addr = 0 ]]; then
> > +                       warn "bad address: $sym_addr + $offset"
> > +                       DONE=1
> > +                       return
> > +               fi
> > +               addr=0x$(printf %x $addr)
> > +
> > +               # If the user provided a size, make sure it matches the symbol's size:
> > +               if [[ -n $user_size ]] && [[ $user_size -ne $size ]]; then
> >                         [[ $print_warnings = 1 ]] &&
> > -                               echo "skipping $func address at $addr due to size mismatch ($size != $sym_size)"
> > +                               echo "skipping $sym_name address at $addr due to size mismatch ($user_size != $size)"
> >                         continue;
> >                 fi
> >
> > -               # make sure the provided offset is within the symbol's range
> > -               if [[ $offset -gt $sym_size ]]; then
> > +               # Make sure the provided offset is within the symbol's range:
> > +               if [[ $offset -gt $size ]]; then
> >                         [[ $print_warnings = 1 ]] &&
> > -                               echo "skipping $func address at $addr due to size mismatch ($offset > $sym_size)"
> > +                               echo "skipping $sym_name address at $addr due to size mismatch ($offset > $size)"
> >                         continue
> >                 fi
> >
> > -               # separate multiple entries with a blank line
> > -               [[ $FIRST = 0 ]] && echo
> > -               FIRST=0
> > +               # In case of duplicates, separate multiple entries with a blank line:
> > +               [[ $first = 0 ]] && echo
> > +               first=0
> >
> > -               # pass real address to addr2line
> > -               echo "$func+$offset/$sym_size:"
> > -               local file_lines=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> > -               [[ -z $file_lines ]] && return
> > +               # Pass full address to addr2line:
> > +               echo "$sym_name+$offset/$size:"
> > +               local output=$(${ADDR2LINE} -fpie $objfile $addr | sed "s; $dir_prefix\(\./\)*; ;")
> > +               [[ -z $output ]] && continue
> >
> >                 if [[ $LIST = 0 ]]; then
> > -                       echo "$file_lines" | while read -r line
> > +                       echo "$output" | while read -r line
> >                         do
> >                                 echo $line
> >                         done
> >                         DONE=1;
> > -                       return
> > +                       continue
> >                 fi
> >
> > -               # show each line with context
> > -               echo "$file_lines" | while read -r line
> > +               # If --list was specified, show each line with context:
> > +               echo "$output" | while read -r line
> >                 do
> >                         echo
> >                         echo $line
> > @@ -189,7 +230,7 @@ __faddr2line() {
> >
> >                 DONE=1
> >
> > -       done < <(${NM} -n $objfile | awk -v fn=$func -v end=$file_end '$3 == fn { found=1; line=$0; start=$1; next } found == 1 { found=0; print line, "0x"$1 } END {if (found == 1) print line, end; }')
> > +       done < <(${READELF} --symbols --wide $objfile | awk -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
> >  }
> >
> >  [[ $# -lt 2 ]] && usage

Re: Issue using faddr2line on kernel modules

[Josh Poimboeuf]

On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > Try this one?
> 
> Definitely better !
> This is the o/p i now get:
> 
> $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> do_the_work.cold+0x68/0x126
> do_the_work.cold+0x68/0x126:
> delay_sec at <...>/oops_tryv2/../../convenient.h:279
> $
> 
> The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> (I'm unsure why the line delay_sec at <...>convenient.h:279   shows up
> ; i don't call this delay_sec() func directly... anyway, that's my
> problem.)

Hm... do you get the same result with addr2line?

-- 
Josh

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

On Mon, Feb 7, 2022 at 1:59 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > > Try this one?
> >
> > Definitely better !
> > This is the o/p i now get:
> >
> > $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> > do_the_work.cold+0x68/0x126
> > do_the_work.cold+0x68/0x126:
> > delay_sec at <...>/oops_tryv2/../../convenient.h:279
> > $
> >
> > The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> > (I'm unsure why the line delay_sec at <...>convenient.h:279   shows up
> > ; i don't call this delay_sec() func directly... anyway, that's my
> > problem.)
>
> Hm... do you get the same result with addr2line?

addr2line works correctly.
And, so does the patched faddr2line, when i tried it out now on
another VM! Lemme check some more and get back, thanks.
- Kaiwan.
>
>
> --
> Josh
>

Re: Issue using faddr2line on kernel modules

[Kaiwan N Billimoria]

It works !

$ .../linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko do_the_work+0x124
do_the_work+0x124/0x15e:
do_the_work at .../oops_tryv2.c:62
$

Will you now upstream this patch?

Thanks!

On Mon, Feb 7, 2022 at 7:53 AM Kaiwan N Billimoria
<kaiwan.billimoria@gmail.com> wrote:
>
> On Mon, Feb 7, 2022 at 1:59 AM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> >
> > On Fri, Jan 21, 2022 at 01:35:13AM +0530, Kaiwan N Billimoria wrote:
> > > > Try this one?
> > >
> > > Definitely better !
> > > This is the o/p i now get:
> > >
> > > $ <...>linux-5.10.60/scripts/faddr2line ./oops_tryv2.ko
> > > do_the_work.cold+0x68/0x126
> > > do_the_work.cold+0x68/0x126:
> > > delay_sec at <...>/oops_tryv2/../../convenient.h:279
> > > $
> > >
> > > The source file+line-no doesn't show up here though... (as it's a foo.cold()?).
> > > (I'm unsure why the line delay_sec at <...>convenient.h:279   shows up
> > > ; i don't call this delay_sec() func directly... anyway, that's my
> > > problem.)
> >
> > Hm... do you get the same result with addr2line?
>
> addr2line works correctly.
> And, so does the patched faddr2line, when i tried it out now on
> another VM! Lemme check some more and get back, thanks.
> - Kaiwan.
> >
> >
> > --
> > Josh
> >