why can't i use gdb to dump module variables anymore?

why can't i use gdb to dump module variables anymore?

[Robert P. J. Day]

  trying very hard to make a long story short, documenting how to use
gdb to poke around in a loaded module.  the important part of the
module (to demonstrate the visibility of different types of
variables):

static int rpjday_1;
int rpjday_2 = 20;
int rpjday_3 = 30;

EXPORT_SYMBOL(rpjday_3);


  once i load that module, i can see the good stuff in /proc/kallsyms:

$ grep rpjday /proc/kallsyms
ffffffffa007c090 r __ksymtab_rpjday_3	[crash_gdb]
ffffffffa007c0a8 r __kstrtab_rpjday_3	[crash_gdb]
ffffffffa007c0a0 r __kcrctab_rpjday_3	[crash_gdb]
ffffffffa007c0b4 d rpjday_2	[crash_gdb]
ffffffffa007c0b8 D rpjday_3	[crash_gdb]
$

  i can examine the salient files under
/sys/module/crash_gdb/sections/, like .data

0xffffffffa007c0b4

so here's a couple questions.

  if i want to add that module's symbol table info to my running gdb
session, i would use "add-symbol-file" but the first arg *must* be the
address of the text section and this module has *no* .text section.
so what do i use for that value?  the address of the .exit.text
section?

  and if i try to load:

(gdb) add-symbol-file /tmp/crash_gdb.ko 0xffffffffa007c000 \
-s .data 0xffffffffa007c0b4
add symbol table from file "/tmp/crash_gdb.ko" at
	.text_addr = 0xffffffffa007c000
	.data_addr = 0xffffffffa007c0b4
(y or n) y
Reading symbols from /tmp/crash_gdb.ko...done.
(gdb)

  it certainly *looks* like that worked, and:

(gdb) whatis rpjday_2
type = int
(gdb) whatis rpjday_3
type = int

  looks good, but here's where things go wrong:

(gdb) p rpjday_2
Cannot access memory at address 0xffffffffa007c0b4
(gdb) p rpjday_3
Cannot access memory at address 0xffffffffa007c0b8

  that sort of thing used to work when i was testing this on a 32-bit
fedora system, but i'm now on 64-but ubuntu and i don't see why those
last couple commands no longer work.  the addresses certainly match
what's in /proc/kallsyms, and i don't recall having to do anything
special last time.  have i omitted a critical compile-time option for
this?  thanks.

rday

-- 

========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Top-notch, inexpensive online Linux/OSS/kernel courses
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: why can't i use gdb to dump module variables anymore?

[Robert P. J. Day]

On Sun, 8 Aug 2010, Jon Masters wrote:

>
> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
> >
> > (gdb) p rpjday_2
> > Cannot access memory at address 0xffffffffa007c0b4
> > (gdb) p rpjday_3
> > Cannot access memory at address 0xffffffffa007c0b8
>
> Did you ever get a reply to this one? Were you trying to do this
> locally, by loading a kernel core or using kgdb?

  never figured out what was happening here, used to work just fine
for simple gdb debugging of the kernel; nowadays (at least on ubuntu),
not so much.  anyway, i'm in bahstahn for linuxcon, staying at the
onyx hotel for first 3 days.  my girlfriend is doing a bofs session:

http://events.linuxfoundation.org/linuxcon2010/mckinnon

  anyway, regarding gdb, this was based on a column i wrote last
year:

http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb

where it worked just fine -- it's the technique described in LDD3.
but now, on ubuntu 10.04, weirdness.  i haven't looked at it closely
again for a while, but if you have any ideas, i'm open to suggestions.
maybe some odd security/selinux/??? setting?

rday


-- 

========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Top-notch, inexpensive online Linux/OSS/kernel courses
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: why can't i use gdb to dump module variables anymore?

[Jason Wessel]

On 08/09/2010 11:06 AM, Robert P. J. Day wrote:
> On Sun, 8 Aug 2010, Jon Masters wrote:
>
>   
>> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
>>     
>>> (gdb) p rpjday_2
>>> Cannot access memory at address 0xffffffffa007c0b4
>>> (gdb) p rpjday_3
>>> Cannot access memory at address 0xffffffffa007c0b8
>>>       
>> Did you ever get a reply to this one? Were you trying to do this
>> locally, by loading a kernel core or using kgdb?
>>     
>
>   never figured out what was happening here, used to work just fine
> for simple gdb debugging of the kernel; nowadays (at least on ubuntu),
> not so much.  anyway, i'm in bahstahn for linuxcon, staying at the
> onyx hotel for first 3 days.  my girlfriend is doing a bofs session:
>
> http://events.linuxfoundation.org/linuxcon2010/mckinnon
>
>   anyway, regarding gdb, this was based on a column i wrote last
> year:
>
> http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb
>
> where it worked just fine -- it's the technique described in LDD3.
> but now, on ubuntu 10.04, weirdness.  i haven't looked at it closely
> again for a while, but if you have any ideas, i'm open to suggestions.
> maybe some odd security/selinux/??? setting?
>   

I took a brief look at the instructions you had a the URL.  It seemed
like something that should probably work ok unless that memory address
is really not accessible via /proc/kcore.

Depending on the age of the ubuntu system and if you have a serial port
or not, kgdb is actually enabled in the kernel and you could certainly
try the same test there.

It should also be possible to try out a gdb with kernel module awareness
on the /proc/kcore.  If you want to try and debug it at some point
perhaps we'll cross paths at LinuxCON.

Jason

Re: why can't i use gdb to dump module variables anymore?

[Robert P. J. Day]

On Tue, 10 Aug 2010, Jason Wessel wrote:

> On 08/09/2010 11:06 AM, Robert P. J. Day wrote:
> > On Sun, 8 Aug 2010, Jon Masters wrote:
> >
> >
> >> On Jul 8, 2010, at 7:51 AM, Robert P. J. Day wrote:
> >>
> >>> (gdb) p rpjday_2
> >>> Cannot access memory at address 0xffffffffa007c0b4
> >>> (gdb) p rpjday_3
> >>> Cannot access memory at address 0xffffffffa007c0b8
> >>>
> >> Did you ever get a reply to this one? Were you trying to do this
> >> locally, by loading a kernel core or using kgdb?
> >>
> >
> >   never figured out what was happening here, used to work just
> > fine for simple gdb debugging of the kernel; nowadays (at least on
> > ubuntu), not so much.  anyway, i'm in bahstahn for linuxcon,
> > staying at the onyx hotel for first 3 days.  my girlfriend is
> > doing a bofs session:
> >
> > http://events.linuxfoundation.org/linuxcon2010/mckinnon
> >
> >   anyway, regarding gdb, this was based on a column i wrote last
> > year:
> >
> > http://www.linux.com/learn/linux-training/33991-the-kernel-newbie-corner-kernel-and-module-debugging-with-gdb
> >
> > where it worked just fine -- it's the technique described in LDD3.
> > but now, on ubuntu 10.04, weirdness.  i haven't looked at it
> > closely again for a while, but if you have any ideas, i'm open to
> > suggestions. maybe some odd security/selinux/??? setting?
> >
>
> I took a brief look at the instructions you had a the URL.  It
> seemed like something that should probably work ok unless that
> memory address is really not accessible via /proc/kcore.

  i've used that technique before as a simple way to examine kernel
data on a running system, and it always worked fine before.

> Depending on the age of the ubuntu system and if you have a serial
> port or not, kgdb is actually enabled in the kernel and you could
> certainly try the same test there.

  i can give that a shot later, certainly.

> It should also be possible to try out a gdb with kernel module
> awareness on the /proc/kcore.  If you want to try and debug it at
> some point perhaps we'll cross paths at LinuxCON.

  i haven't looked at this issue for a while, but i'd certainly like
to figure out why what used to work before has stopped working.  this
is something i like to show my beginner kernel programming students,
and it's moderately frustrating that i can't get it to work anymore.

  i'm sure it's something idiotic i'm doing.  and i'm around linuxcon
all week, so yes, let's bump into each other.

rday


-- 

========================================================================
Robert P. J. Day                               Waterloo, Ontario, CANADA

        Top-notch, inexpensive online Linux/OSS/kernel courses
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================