* Re: [Kgdb-bugreport] compiling kernel with -O0 flag (For optimal debugging with kgdb and/or crash).
[not found] <2cf1ee820609211354m3ae2ec5btc7274402d84b5400@mail.gmail.com>
@ 2006-09-22 22:40 ` Piet Delaney
[not found] ` <1159219251.32095.143.camel@piet2.bluelane.com>
1 sibling, 0 replies; 2+ messages in thread
From: Piet Delaney @ 2006-09-22 22:40 UTC (permalink / raw)
To: emin ak
Cc: Piet Delaney, Kgdb-bugreport, Andrew Morton, netdev,
linux-kernel, crash-utility, Subhachandra Chandra
[-- Attachment #1.1: Type: text/plain, Size: 10350 bytes --]
On Thu, 2006-09-21 at 23:54 +0300, emin ak wrote:
> Dear All;
> Firstly thank you very much for your great effort for kgdb that makes
> kernel much understandable.
> I'am using kgdb to debug tcp-ip stack but I have experienced serious
> difficulties while debugging inline functions.
Hi Emin:
Yep, I edit "static inline" to "static_inline" and then define
static_inline as static for KGDB kernels.
In include/linux/compiler-gcc3.h and include/linux/compiler-gcc4.h
I added:
------------------------------------------------------------------
#if defined(CONFIG_KGDB) || defined(CONFIG_KEXEC)
# define static_inline static __attribute__ ((__unused__))
# define static__inline__ static __attribute__ ((__unused__))
# define INLINE __attribute__ ((__unused__))
# define __INLINE__ __attribute__ ((__unused__))
#else
# define static_inline static inline
# define static__inline__ static __inline__
# define INLINE inline
# define __INLINE__ __inline__
#endif
----------------------------------------------------------------------
I'm using it today to understand the device mapping and encryption code.
It's great! Inline's make skipping over code with the gdb 'next'
instruction impossible and you can't see the local variables.
I like having a large stack, compiling -O0 and without inlines
can increase the stack size. I think I notices more stability
by adding this to include/asm-i386/thread_info.h:
----------------------------------------------------------------------
#if defined(CONFIG_DEBUG_PREEMPT_AUDIT) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
#define THREAD_SIZE (8192 * 2)
#else
#ifdef CONFIG_4KSTACKS
#define THREAD_SIZE (4096)
#else
#define THREAD_SIZE (8192)
#endif
#endif
-----------------------------------------------------------------------
Without OPTIMIZATION I found the MMU code needs a tweak
in../linux-4/mm/memory.c:
------------------------------------------------------------------------
#if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
/*
* Allocate page upper directory.
*
* We've already handled the fast-path in-line, and we own the
* page table lock.
*/
pud_t fastcall *__pud_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned
long address)
{
.
.
.
}
#if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
/*
* Allocate page middle directory.
*
* We've already handled the fast-path in-line, and we own the
* page table lock.
*/
pmd_t fastcall *__pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned
long address)
{
.
.
}
---------------------------------------------------------------------------
Maybe I should have used #if !defined(__OPTIMIZE__)
in ../linux-4/mm/memory.c. Another change is I needed
to define a few network byte swapping functions. I currently
define them in ../linux-4/net/core/sock.c but I'm not
resistant to putting it in a better place:
----------------------------------------------------------------------------
/*
* If compiling -O0 we need to define
* these functions somewhere.
*/
#if !defined(__OPTIMIZE__)
#define ___htonl(x) __cpu_to_be32(x)
#define ___htons(x) __cpu_to_be16(x)
#define ___ntohl(x) __be32_to_cpu(x)
#define ___ntohs(x) __be16_to_cpu(x)
__u32 htonl(__be32 x) { return(___htonl(x)); }
__u32 ntohl(__be32 x) { return(___ntohl(x)); }
__be16 htons(__u16 x) { return(___htons(x)); }
__u16 ntohs(__be16 x) { return(___ntohs(x)); }
EXPORT_SYMBOL(htonl);
EXPORT_SYMBOL(ntohl);
EXPORT_SYMBOL(htons);
EXPORT_SYMBOL(ntohs);
#endif
------------------------------------------------------------------------------
Sometimes I only want to compile the tcp/ip code -O0, so I modified the
networking Makefiles and added:
-------------------------------------------------------------------------------
ifdef CONFIG_KGDB
CFLAGS += -gdwarf-2 -O0
else
ifdef CONFIG_KEXEC
CFLAGS += -gdwarf-2 -O0
endif
endif
------------------------------------------------------------------------------
In the top level kernel Makefile I have:
------------------------------------------------------------------------------
ifdef CONFIG_FRAME_POINTER
CFLAGS += -fno-omit-frame-pointer
else
CFLAGS += -fomit-frame-pointer
endif
#
# Compiling the complete kernel without optimization (-O0) for enhanced
debugging
# with kgdb/kdump requires ./mm/memory.c to have:
#
# if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
# and
# if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
#
# A less invasive procedure is to use -O1 and only use -O0 for
networking code.
# The networking Makefiles have been setup to support this. So just
change
# -O0 to -O1 below and back out the kgdb change in ./mm/memory.c for a
# less invasive change. Compiling -O0 also required increasing
ROUNDUP_WAIT in
# linux/kernel/kgdb.c; value in 2.6.12 patch was way to low and value in
2.6.16
# is marginal and frequently causes lead CPU to times out prematurely
waiting for
# other CPU's to stop.
#
ifdef CONFIG_DEBUG_INFO
ifdef CONFIG_KGDB
CFLAGS += -gdwarf-2 -O0
else
ifdef CONFIG_KEXEC
CFLAGS += -gdwarf-2 -O1
else
CFLAGS += -g
endif
endif
endif
------------------------------------------------------------------------------------
> I know this is not a
> bug but with -O optimizations and inlines on tcp-ip stack, program
> counter goes everywhere madly even with step or next command and this
> makes debugging incomprehensible.
Yep, I don't understand why everyone else doesn't. It's also
like using debug printf's, I like being able to trace the code
to get the big picture and then a -O0 to look at details with kgdb.
Some believe doing this kind of stuff is blasphemy. The
Bible says I should be killed for working on Sunday; I
happen to disagree.
> At this point I have two questions:
> 1- Is there any way to compile kernel with -O0 flag and if it's
> possible, may it cause any problems?
I offered to post them to Amit back on Sept 06(2:45 PM) but I don't
think I ever heard back. I'd prefer to see the -O0 and KGDB_DEBUG
code for tracing the kgdb stub assimilated. If they would be accepted
I could make a patch to Tom's git repository...
> 2- Why does kernel fail while compiling with O0 flag and why does
> linux kernel depends on inline functions so much?
I think it's an obsession with performance. As long as I/we can map
"static inline" to "static" it's not a big deal.
> Is there anyone
> whoever uses kgdb for debugging linux tcp-ip stack or any effort to
> compile kernel with no optimization?
I'm using it every day; works great. I also recommend by SOCK_DEBUG,
SKB_DEBUG, and TCP_DEBUG macros to trace the TCP code. I also indent
the trace to make it easy to read.
function1() {
function2() {
function3() {
function4();
}
}
}
The brackets make it easy to see the scope of the trace with vi.
I like tracing with 'C' syntax since it what the reader is use to.
If folks are interested I could also add that to the git diff, but
I think that likely belongs else where and isn't likely the current
dogma. See snippet from attached network trace. I gave a talk
at a UNENIX conference back in the 1980 recommending a common UNIX
tracing paradigm and a few liked it. The director of Siemens,
Struck Zimmerman, didn't; you can't please everyone, so I just do what
I think is best and live with the world not being as I'd expect it to
be.
For TCP I'm using the attached sock.h fragment which has a
backward compatible SOCK_DEBUG() macro. I used the same paradigm in
skbuff.h; see attachment. Likewise I'm doing the same in kgdb.h; also
attached.
In printk I added:
--------------------------------------------------------------------
for (tp = tbuf; tp < tbuf + tlen; tp++)
emit_log_char(*tp);
printed_len += tlen - 3;
#ifdef CONFIG_PRINTK_INDENT
if (!in_interrupt()) {
int depth = stack_depth();
int i;
if ((depth > 0) && (depth < 120)) {
for(i = 0; i < depth; i++) {
emit_log_char(' ');
printed_len++;
}
}
}
#endif
-----------------------------------------------------------
and I added stack_depth() function
to ../linux-4/arch/i386/kernel/traps.c
-----------------------------------------------------------
int stack_depth(void)
{
struct thread_info *tinfo;
unsigned long ebp;
int depth = 0;
#ifdef CONFIG_FRAME_POINTER
asm("andl %%esp,%0; ":"=r" (tinfo) : "0" (~(THREAD_SIZE - 1)));
asm ("movl %%ebp, %0" : "=r" (ebp) : );
while (valid_stack_ptr(tinfo, (void *)ebp)) {
ebp = *(unsigned long *)ebp;
if (depth++ > 100) {
break;
}
}
#endif
return(depth);
}
---------------------------------------------------------------------------
Let me know if you you have any questions. Sounds like your on the right
track; IMHO.
-piet
>
> Thanks alot.
> Emin
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Kgdb-bugreport mailing list
> Kgdb-bugreport@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
--
Piet Delaney Phone: (408) 200-5256
Blue Lane Technologies Fax: (408) 200-5299
10450 Bubb Rd.
Cupertino, Ca. 95014 Email: piet@bluelane.com
[-- Attachment #1.2: kgdb.h --]
[-- Type: text/x-chdr, Size: 11156 bytes --]
/*
* include/linux/kgdb.h
*
* This provides the hooks and functions that KGDB needs to share between
* the core, I/O and arch-specific portions.
*
* Author: Amit Kale <amitkale@linsyssoft.com> and
* Tom Rini <trini@kernel.crashing.org>
*
* 2001-2004 (c) Amit S. Kale and 2003-2004 (c) MontaVista Software, Inc.
* This file is licensed under the terms of the GNU General Public License
* version 2. This program is licensed "as is" without any warranty of any
* kind, whether express or implied.
*/
#ifdef __KERNEL__
#ifndef _KGDB_H_
#define _KGDB_H_
#include <asm/atomic.h>
#ifdef CONFIG_KGDB
#include <asm/kgdb.h>
#include <linux/serial_8250.h>
#include <linux/linkage.h>
struct tasklet_struct;
struct pt_regs;
struct task_struct;
struct uart_port;
/* To enter the debugger explicitly. */
#if defined(CONFIG_VPROXY) || defined(CONFIG_CRYPTO_DEV_CAVIUM_NITROX)
extern void kgdb_bp(void);
extern void bp(void);
#endif
static inline int kgdb_function_entry(const char *fmt, ...)
{
return(fmt[0] == '(');
}
#define KGDB_DEBUGING
#ifdef KGDB_DEBUGING
# define KGDB_MAX_LEVEL 10 /* Max Level to compile in */
# define KGDB_DEBUG(level, msg...) \
do { \
if (level <= KGDB_MAX_LEVEL && level <= kgdb_debug) { \
int cpu = smp_processor_id(); \
\
spin_lock(&kgdb_printf_lock); \
if (kgdb_function_entry(msg)) { \
printk(KERN_ERR "<cpu:%d> %s", cpu, __func__); \
} else { \
printk(KERN_ERR "<cpu:%d> %s: ", cpu, __func__); \
} \
printk(msg); \
spin_unlock(&kgdb_printf_lock); \
} \
} while (0)
extern volatile int kgdb_debug; /* Debug level; like gdb set debug remote 'n' */
extern spinlock_t kgdb_printf_lock;
#else
# define KGDB_DEBUG(level, msg...) do { } while (0)
#endif
extern void breakpoint(void);
extern int kgdb_connected;
extern int kgdb_may_fault;
extern struct tasklet_struct kgdb_tasklet_breakpoint;
extern atomic_t kgdb_setting_breakpoint;
extern atomic_t cpu_doing_single_step;
extern struct task_struct *kgdb_usethread, *kgdb_contthread;
enum kgdb_bptype {
bp_breakpoint = '0',
bp_hardware_breakpoint,
bp_write_watchpoint,
bp_read_watchpoint,
bp_access_watchpoint
};
enum kgdb_bpstate {
bp_disabled,
bp_enabled
};
struct kgdb_bkpt {
unsigned long bpt_addr;
unsigned char saved_instr[BREAK_INSTR_SIZE];
enum kgdb_bptype type;
enum kgdb_bpstate state;
};
/* The maximum number of KGDB I/O modules that can be loaded */
#define MAX_KGDB_IO_HANDLERS 3
#ifndef MAX_BREAKPOINTS
#define MAX_BREAKPOINTS 16
#endif
#define KGDB_HW_BREAKPOINT 1
/* Required functions. */
/**
* regs_to_gdb_regs - Convert ptrace regs to GDB regs
* @gdb_regs: A pointer to hold the registers in the order GDB wants.
* @regs: The &struct pt_regs of the current process.
*
* Convert the pt_regs in @regs into the format for registers that
* GDB expects, stored in @gdb_regs.
*/
extern void regs_to_gdb_regs(unsigned long *gdb_regs, struct pt_regs *regs);
/**
* sleeping_regs_to_gdb_regs - Convert ptrace regs to GDB regs
* @gdb_regs: A pointer to hold the registers in the order GDB wants.
* @p: The &struct task_struct of the desired process.
*
* Convert the register values of the sleeping process in @p to
* the format that GDB expects.
* This function is called when kgdb does not have access to the
* &struct pt_regs and therefore it should fill the gdb registers
* @gdb_regs with what has been saved in &struct thread_struct
* thread field during switch_to.
*/
extern void sleeping_thread_to_gdb_regs(unsigned long *gdb_regs,
struct task_struct *p);
/**
* gdb_regs_to_regs - Convert GDB regs to ptrace regs.
* @gdb_regs: A pointer to hold the registers we've recieved from GDB.
* @regs: A pointer to a &struct pt_regs to hold these values in.
*
* Convert the GDB regs in @gdb_regs into the pt_regs, and store them
* in @regs.
*/
extern void gdb_regs_to_regs(unsigned long *gdb_regs, struct pt_regs *regs);
/**
* kgdb_arch_handle_exception - Handle architecture specific GDB packets.
* @vector: The error vector of the exception that happened.
* @signo: The signal number of the exception that happened.
* @err_code: The error code of the exception that happened.
* @remcom_in_buffer: The buffer of the packet we have read.
* @remcom_out_buffer: The buffer, of %BUFMAX to write a packet into.
* @regs: The &struct pt_regs of the current process.
*
* This function MUST handle the 'c' and 's' command packets,
* as well packets to set / remove a hardware breakpoint, if used.
* If there are additional packets which the hardware needs to handle,
* they are handled here. The code should return -1 if it wants to
* process more packets, and a %0 or %1 if it wants to exit from the
* kgdb hook.
*/
extern int kgdb_arch_handle_exception(int vector, int signo, int err_code,
char *remcom_in_buffer,
char *remcom_out_buffer,
struct pt_regs *regs);
#ifndef JMP_REGS_ALIGNMENT
#define JMP_REGS_ALIGNMENT
#endif
extern unsigned long kgdb_fault_jmp_regs[];
/**
* kgdb_fault_setjmp - Store state in case we fault.
* @curr_context: An array to store state into.
*
* Certain functions may try and access memory, and in doing so may
* cause a fault. When this happens, we trap it, restore state to
* this call, and let ourself know that something bad has happened.
*/
extern asmlinkage int kgdb_fault_setjmp(unsigned long *curr_context);
/**
* kgdb_fault_longjmp - Restore state when we have faulted.
* @curr_context: The previously stored state.
*
* When something bad does happen, this function is called to
* restore the known good state, and set the return value to 1, so
* we know something bad happened.
*/
extern asmlinkage void kgdb_fault_longjmp(unsigned long *curr_context);
/* Optional functions. */
extern int kgdb_arch_init(void);
extern void kgdb_disable_hw_debug(struct pt_regs *regs);
extern void kgdb_post_master_code(struct pt_regs *regs, int e_vector,
int err_code);
extern void kgdb_roundup_cpus(unsigned long flags);
extern int kgdb_set_hw_break(unsigned long addr);
extern int kgdb_remove_hw_break(unsigned long addr);
extern void kgdb_remove_all_hw_break(void);
extern void kgdb_correct_hw_break(void);
extern void kgdb_shadowinfo(struct pt_regs *regs, char *buffer,
unsigned threadid);
extern struct task_struct *kgdb_get_shadow_thread(struct pt_regs *regs,
int threadid);
extern struct pt_regs *kgdb_shadow_regs(struct pt_regs *regs, int threadid);
/**
* struct kgdb_arch - Desribe architecture specific values.
* @gdb_bpt_instr: The instruction to trigger a breakpoint.
* @flags: Flags for the breakpoint, currently just %KGDB_HW_BREAKPOINT.
* @shadowth: A value of %1 indicates we shadow information on processes.
* @set_breakpoint: Allow an architecture to specify how to set a software
* breakpoint.
* @remove_breakpoint: Allow an architecture to specify how to remove a
* software breakpoint.
* @set_hw_breakpoint: Allow an architecture to specify how to set a hardware
* breakpoint.
* @remove_hw_breakpoint: Allow an architecture to specify how to remove a
* hardware breakpoint.
*
* The @shadowth flag is an option to shadow information not retrievable by
* gdb otherwise. This is deprecated in favor of a binutils which supports
* CFI macros.
*/
struct kgdb_arch {
unsigned char gdb_bpt_instr[BREAK_INSTR_SIZE];
unsigned long flags;
unsigned shadowth;
int (*set_breakpoint) (unsigned long, char *);
int (*remove_breakpoint)(unsigned long, char *);
int (*set_hw_breakpoint)(unsigned long, int, enum kgdb_bptype);
int (*remove_hw_breakpoint)(unsigned long, int, enum kgdb_bptype);
};
/* Thread reference */
typedef unsigned char threadref[8];
/**
* struct kgdb_io - Desribe the interface for an I/O driver to talk with KGDB.
* @read_char: Pointer to a function that will return one char.
* @write_char: Pointer to a function that will write one char.
* @flush: Pointer to a function that will flush any pending writes.
* @init: Pointer to a function that will initialize the device.
* @late_init: Pointer to a function that will do any setup that has
* other dependencies.
* @pre_exception: Pointer to a function that will do any prep work for
* the I/O driver.
* @post_exception: Pointer to a function that will do any cleanup work
* for the I/O driver.
*
* The @init and @late_init function pointers allow for an I/O driver
* such as a serial driver to fully initialize the port with @init and
* be called very early, yet safely call request_irq() later in the boot
* sequence.
*
* @init is allowed to return a non-0 return value to indicate failure.
* If this is called early on, then KGDB will try again when it would call
* @late_init. If it has failed later in boot as well, the user will be
* notified.
*/
struct kgdb_io {
int (*read_char) (void);
void (*write_char) (u8);
void (*flush) (void);
int (*init) (void);
void (*late_init) (void);
void (*pre_exception) (void);
void (*post_exception) (void);
};
extern struct kgdb_io kgdb_io_ops;
extern struct kgdb_arch arch_kgdb_ops;
extern int kgdb_initialized;
extern int kgdb_register_io_module(struct kgdb_io *local_kgdb_io_ops);
extern void kgdb_unregister_io_module(struct kgdb_io *local_kgdb_io_ops);
extern void kgdb8250_add_port(int i, struct uart_port *serial_req);
extern void kgdb8250_add_platform_port(int i, struct plat_serial8250_port *serial_req);
extern int kgdb8250_get_ttyS(void);
extern int kgdb_hex2long(char **ptr, long *long_val);
extern char *kgdb_mem2hex(char *mem, char *buf, int count);
extern char *kgdb_hex2mem(char *buf, char *mem, int count);
extern int kgdb_get_mem(char *addr, unsigned char *buf, int count);
extern int kgdb_set_mem(char *addr, unsigned char *buf, int count);
extern int kgdb_handle_exception(int ex_vector, int signo, int err_code,
struct pt_regs *regs);
extern void kgdb_nmihook(int cpu, void *regs);
extern int debugger_step;
extern atomic_t debugger_active;
#else
/* Stubs for when KGDB is not set. */
static const atomic_t debugger_active = ATOMIC_INIT(0);
#endif /* CONFIG_KGDB */
#endif /* _KGDB_H_ */
#endif /* __KERNEL__ */
[-- Attachment #1.3: sock.h.frag --]
[-- Type: text/plain, Size: 6941 bytes --]
# ifdef SA_TCP_DEBUG
/*
* A similar socket debug macro exist is net/core/skbuff.h
*/
extern unsigned char sa_debug_global_flag;
#if 0
# define SA_DEBUG(msg...) \
do { \
if (sa_debug_global_flag) { \
printk(KERN_DEBUG "%s: ", __func__); \
printk(msg); \
} \
} while (0)
# endif
# define SOCK_DEBUG(sk, msg...) \
do { \
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# define SOCK_ERR(sk, msg...) \
do { \
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
printk(KERN_ERR "%s: ", __func__); \
printk(msg); \
} \
} \
} while (0)
# define TCP_SOCK_DEBUG(tp, msg...) \
do { \
struct sock *sk = &tp->inet.sk; \
\
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# define TW_SOCK_DEBUG(tw, msg...) \
do { \
if (sa_debug_global_flag && (tw) != NULL) { \
if ((tw)->socket_type || (tw)->vproxy_id) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# else /* !SA_TCP_DEBUG */
# define SOCK_DEBUG(sk, msg...) \
do { if ((sk) && sock_flag((sk), SOCK_DBG)) \
printk(KERN_DEBUG msg); \
} while (0)
# define SOCK_ERR(sk, msg...) do { } while (0)
// # define SA_DEBUG(tp, msg...) do { } while (0)
# define TCP_SOCK_DEBUG(tp, msg...) do { } while (0)
# define TW_SOCK_DEBUG(tw, msg...) do { } while (0)
# endif /* SA_TCP_DEBUG */
#else /* !SOCK_DEBUGGING */
// # define SA_DEBUG(sk, msg...) do { } while (0)
# define SOCK_DEBUG(sk, msg...) do { } while (0)
# define SOCK_ERR(sk, msg...) do { } while (0)
# define TCP_SOCK_DEBUG(tp, msg...) do { } while (0)
# define TW_SOCK_DEBUG(tw, msg...) do { } while (0)
#endif /* SOCK_DEBUGGING */
[-- Attachment #1.4: skbuff.h.frag --]
[-- Type: text/plain, Size: 1606 bytes --]
#ifdef CONFIG_VPROXY
#include <net/vproxy.h>
# if defined(SA_TCP_DEBUG) && !defined(CONFIG_ATM)
extern unsigned char sa_debug_global_flag;
/*
* A similar socket debug macro exist is net/sock.h
* SKB_DEBUG() unfortunatly is also defined by some
* CONFIG_ATM code; might switch to SA_SKB_DEBUG().
*/
# define SKB_DEBUG(skb, msg...) \
do { \
if (sa_debug_global_flag && (skb) != NULL) { \
if ((skb)->vdata.vproxy_flags & PROXY_PACKET ) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# else /* !SA_TCP_DEBUG */
# else /* !CONFIG_VPROXY */
# define SKB_DEBUG(sk, msg...) do { } while (0)
#endif /* CONFIG_VPROXY */
[-- Attachment #1.5: trace_example --]
[-- Type: text/plain, Size: 7526 bytes --]
Jun 1 19:17:59 localhost kernel: [ 3180.034711] vproxy_ip_local_deliver_finish(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034714] ip_local_deliver_finish(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034718] tcp_v4_rcv(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034721] sa_tcp_v4_rcv(skb:dfbf3c80) {
Jun 1 19:17:59 localhost kernel: [ 3180.034726] sa_tcp_v4_rcv: SA_TCP_INC_STATS(SA_TCP_STATS_PktReceived:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034731] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034739] sa_tcp_v4_rcv: skb:dfbf3c80->{dev->name:'u0', vdata.{outdev->name:'u0', indev->name:'p0'} }
Jun 1 19:17:59 localhost kernel: [ 3180.034741] ... TCP Packet with src:(192.168.136.103:32989) dest:(192.168.135.102:5001)
Jun 1 19:17:59 localhost kernel: [ 3180.034743] ... TCP FLAGS:' S ' Seq:214233949, End_seq:214233950, Ack_seq:0, Window:5840
Jun 1 19:17:59 localhost kernel: [ 3180.034745] ... skb->ip_summed:2, skb->csum:0x0, th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034749] sa_tcp_v4_rcv: 'Client' packet; sk:00000000 = __new_vproxy_tcp_v4_lookup_established();
Jun 1 19:17:59 localhost kernel: [ 3180.034753] sa_tcp_v4_rcv: tmp_checksum_received:2, skb:dfbf3c80->{ip_summed:2, csum:0x0} th->check 0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034756] sa_tcp_v4_rcv: Listening socket 0xde319080 state 10:Listen
Jun 1 19:17:59 localhost kernel: [ 3180.034761] sa_tcp_v4_rcv: Taking lock on client/listen sk:0xde319080
Jun 1 19:17:59 localhost kernel: [ 3180.034764] sa_tcp_v4_rcv: bh_lock_sock(sk:0xde319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034767] sa_tcp_v4_rcv: SA_TCP_INC_STATS(SA_TCP_STATS_PktReceivedForListeningSocket:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034771] sa_tcp_v4_rcv: Client/listen sock pkt: Got the locks
Jun 1 19:17:59 localhost kernel: [ 3180.034775] sa_tcp_v4_do_rcv(sk:0xde319080, skb:0xdfbf3c80) sk->sk_state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034779] sa_client_tcp_v4_hnd_req(sk:de319080, skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034784] vproxy_tcp_v4_search_req(tp:0xde319080, prevp:c0613a40, rport:56704, lport:35091, raddr:6788a8c0, laddr:6687a8c0) {
Jun 1 19:17:59 localhost kernel: [ 3180.034790] vproxy_tcp_v4_search_req: return(req:00000000);
Jun 1 19:17:59 localhost kernel: [ 3180.034792] }
Jun 1 19:17:59 localhost kernel: [ 3180.034794] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034799] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034802] sa_client_tcp_v4_hnd_req: return(sk:de319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034806] sa_tcp_v4_do_rcv: LISTENING SOCKET; sk_state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034809] sa_tcp_rcv_state_process(sk:0xde319080, skb 0xdfbf3c80, th:0xf7efe074, len:40) sk->state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034816] sa_tcp_v4_conn_request(sk:0xde319080, skb:0xdfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034819] sa_tcp_v4_conn_request: SA_TCP_INC_STATS(SA_TCP_STATS_NewSynReceived:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034827] sa_tcp_v4_conn_request: SA_TCP_INC_STATS(SA_TCP_STATS_OpenreqCreated:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034833] sa_server_create_peer_request(sk:de319080, skb:dfbf3c80, rx_opts:c06137e8, client_req:e6622800, server_req:e6622780)
Jun 1 19:17:59 localhost kernel: [ 3180.034839] sa_server_create_peer_request: server_tcp_req:e6622780->snt_isn:0x0 = client_tcp_req:e6622800->rcv_isn:0xcc4f35d;
Jun 1 19:17:59 localhost kernel: [ 3180.034845] sa_server_tcp_v4_send_syn: Sending out the SYN packet without modifying ANY data.
Jun 1 19:17:59 localhost kernel: [ 3180.034849] sa_server_tcp_v4_send_syn: buff:dfbf3c80->{ip_summed:0, csum:0, h.th->check:0x6ef0}
Jun 1 19:17:59 localhost kernel: [ 3180.034853] sa_server_tcp_v4_send_syn: ntohl(buff:dfbf3c80->h.th->seq:0x5df3c40c):0xcc4f35d [NEW]
Jun 1 19:17:59 localhost kernel: [ 3180.034857] sa_server_tcp_v4_send_syn: tcp_req:e6622800->{rcv_isn:0xcc4f35d, snt_isn:0x0}
Jun 1 19:17:59 localhost kernel: [ 3180.034860] sa_server_tcp_v4_send_syn: peer_tcp_req:e6622780->{rcv_isn:0xcc4f35d, snt_isn:0xcc4f35d}
Jun 1 19:17:59 localhost kernel: [ 3180.034865] sa_tcp_forward_L4_packet(skb:0xdfbf3c80) skb->{ip:summed:0, csum:0, checksum:0x6ef0}
Jun 1 19:17:59 localhost kernel: [ 3180.034869] sa_ip_fast_output(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034876] sa_ip_fast_output: skb:dfbf3c80->{dev->name:'<<No Dev>>', vdata.{outdev->name:'u0', indev->name:'p0'} }
Jun 1 19:17:59 localhost kernel: [ 3180.034878] ... TCP Packet with src:(192.168.136.103:32989) dest:(192.168.135.102:5001)
Jun 1 19:17:59 localhost kernel: [ 3180.034880] ... TCP FLAGS:' S ' Seq:214233949, End_seq:214233950, Ack_seq:0, Window:5840
Jun 1 19:17:59 localhost kernel: [ 3180.034882] ... skb->ip_summed:0, skb->csum:0x0, th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034886] sa_xmit_ip_pkts(skb:dfbf3c80) skb->vdata.{l2type:0, l2addlen:0}, skb->h.th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034891] __skb_queue_tail(list:c2516314, newsk:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034895] __skb_dequeue: return(result:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034899] sa_ip_fast_output: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034902] tcp_reset_keepalive_timer(sk:de319080, len:2ee)
Jun 1 19:17:59 localhost kernel: [ 3180.034906] sa_tcp_v4_conn_request: return(0);
Jun 1 19:17:59 localhost kernel: [ 3180.034909] sa_tcp_v4_do_rcv: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034912] sa_tcp_v4_rcv: Release client/listen sock lock sk:0xde319080
Jun 1 19:17:59 localhost kernel: [ 3180.034916] sa_tcp_v4_rcv: bh_unlock_sock(sk:0xde319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034919] sa_tcp_v4_rcv: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034921] }
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [Kgdb-bugreport] compiling kernel with -O0 flag - Lets make it a config option!
[not found] ` <200609261426.16481.amitkale@linsyssoft.com>
@ 2006-09-26 21:48 ` Piet Delaney
0 siblings, 0 replies; 2+ messages in thread
From: Piet Delaney @ 2006-09-26 21:48 UTC (permalink / raw)
To: Amit S. Kale, Andrew Morton; +Cc: Piet Delaney, kgdb-bugreport, emin ak
[-- Attachment #1: Type: text/plain, Size: 15627 bytes --]
On Tue, 2006-09-26 at 14:26 +0530, Amit S. Kale wrote:
> That's great! Let's include this in kgdb documentation.
Why not make it a config option to compile -O0 and
leave in the static_inline --> inline declarations so developers
can easily just change "static inline" to "static_inline"
in the areas of the kernel that they are working on.
There are already so many things that should be included
in the kgdb documentation that aren't. For example using
kgdboe and NAPI. Seems to me NAPI should be disabled by
our config menu until the problems are resolved. Leaving
land mines laying around just frustrates the user community.
Same for debugging -O0; it should be configurable.
Andrew: would including these kinda of changes in the
the kgdb patch be a problem for eventual integration?
IMHO it just makes the kgdb patch that more worthy
of integration. Makes debugging it at least twice
as enlightening. I'd still like to find a
fix for the kgdb list problem and get code to
stop using enpty structures instead of correctly
including the structure declarations; for example
the device mapping code. I'd like to see kernel
code written with debug-ability a major consideration.
-piet
>
> Thanks, Piet.
> -Amit
>
> On Tuesday 26 September 2006 14:20, emin ak wrote:
> > Hi Piet;
> > Firstly thank you very much for your detailed answer. I'am happy to
> > say that I have achieved to debug tcp-ip stack succesfully with you
> > hints. It's nice to see next command working properly, and backtraces
> > look good. Because of our architecture (powerpc), I can't apply all
> > the hints especially which one includes asm inst. I'll try scripts
> > also, it seems they'll be very helpfull.
> >
> > Most of the features of linux kernel are lack of documentation,
> > without a reliable debug facility, even the sources are opened, it's
> > hard to understand what's happing on the code especially for complex
> > ones like tcp-ip stack. I believe that, one day the maintainers of
> > linux kernel will notice old man's printk is not enough all debugging
> > purposes and one day, we'll debug all around the code without
> > modifications
> >
> >
> > Thanks again.
> > Emin
> >
> > 2006/9/26, Piet Delaney <piet@bluelane.com>:
> > > On Thu, 2006-09-21 at 23:54 +0300, emin ak wrote:
> > > > Dear All;
> > > > Firstly thank you very much for your great effort for kgdb that makes
> > > > kernel much understandable.
> > > > I'am using kgdb to debug tcp-ip stack but I have experienced serious
> > > > difficulties while debugging inline functions.
> > >
> > > Hi Emin:
> > >
> > > [Looks like my posting to Kgdb-bugreport@lists.sourceforge.net got
> > > dropped but I didn't receive any mail back from sourceforge on why;
> > > trying again. It's less that the 100k limit.]
> > >
> > > Yep, I edit "static inline" to "static_inline" and then define
> > > static_inline as static for KGDB kernels.
> > >
> > > In include/linux/compiler-gcc3.h and include/linux/compiler-gcc4.h
> > > I added:
> > > ------------------------------------------------------------------
> > > #if defined(CONFIG_KGDB) || defined(CONFIG_KEXEC)
> > > # define static_inline static __attribute__ ((__unused__))
> > > # define static__inline__ static __attribute__ ((__unused__))
> > > # define INLINE __attribute__ ((__unused__))
> > > # define __INLINE__ __attribute__ ((__unused__))
> > > #else
> > > # define static_inline static inline
> > > # define static__inline__ static __inline__
> > > # define INLINE inline
> > > # define __INLINE__ __inline__
> > > #endif
> > > ----------------------------------------------------------------------
> > > I'm using it today to understand the device mapping and encryption code.
> > > It's great! Inline's make skipping over code with the gdb 'next'
> > > instruction impossible and you can't see the local variables.
> > >
> > > I like having a large stack, compiling -O0 and without inlines
> > > can increase the stack size. I think I notices more stability
> > > by adding this to include/asm-i386/thread_info.h:
> > > ----------------------------------------------------------------------
> > > #if defined(CONFIG_DEBUG_PREEMPT_AUDIT) || defined(CONFIG_KGDB) ||
> > > defined(CONFIG_KEXEC)
> > > #define THREAD_SIZE (8192 * 2)
> > > #else
> > > #ifdef CONFIG_4KSTACKS
> > > #define THREAD_SIZE (4096)
> > > #else
> > > #define THREAD_SIZE (8192)
> > > #endif
> > > #endif
> > > -----------------------------------------------------------------------
> > >
> > > Without OPTIMIZATION I found the MMU code needs a tweak
> > > in../linux-4/mm/memory.c:
> > > ------------------------------------------------------------------------
> > > #if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
> > > defined(CONFIG_KEXEC)
> > > /*
> > > * Allocate page upper directory.
> > > *
> > > * We've already handled the fast-path in-line, and we own the
> > > * page table lock.
> > > */
> > > pud_t fastcall *__pud_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned
> > > long address)
> > > {
> > > .
> > > .
> > > .
> > > }
> > > #if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
> > > defined(CONFIG_KEXEC)
> > > /*
> > > * Allocate page middle directory.
> > > *
> > > * We've already handled the fast-path in-line, and we own the
> > > * page table lock.
> > > */
> > > pmd_t fastcall *__pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned
> > > long address)
> > > {
> > > .
> > > .
> > > }
> > > -------------------------------------------------------------------------
> > >--
> > >
> > > Maybe I should have used #if !defined(__OPTIMIZE__)
> > > in ../linux-4/mm/memory.c. Another change is I needed
> > > to define a few network byte swapping functions. I currently
> > > define them in ../linux-4/net/core/sock.c but I'm not
> > > resistant to putting it in a better place:
> > > -------------------------------------------------------------------------
> > >--- /*
> > > * If compiling -O0 we need to define
> > > * these functions somewhere.
> > > */
> > > #if !defined(__OPTIMIZE__)
> > > #define ___htonl(x) __cpu_to_be32(x)
> > > #define ___htons(x) __cpu_to_be16(x)
> > > #define ___ntohl(x) __be32_to_cpu(x)
> > > #define ___ntohs(x) __be16_to_cpu(x)
> > >
> > > __u32 htonl(__be32 x) { return(___htonl(x)); }
> > > __u32 ntohl(__be32 x) { return(___ntohl(x)); }
> > > __be16 htons(__u16 x) { return(___htons(x)); }
> > > __u16 ntohs(__be16 x) { return(___ntohs(x)); }
> > >
> > > EXPORT_SYMBOL(htonl);
> > > EXPORT_SYMBOL(ntohl);
> > > EXPORT_SYMBOL(htons);
> > > EXPORT_SYMBOL(ntohs);
> > > #endif
> > > -------------------------------------------------------------------------
> > >-----
> > >
> > > Sometimes I only want to compile the tcp/ip code -O0, so I modified the
> > > networking Makefiles and added:
> > >
> > > -------------------------------------------------------------------------
> > >------ ifdef CONFIG_KGDB
> > > CFLAGS += -gdwarf-2 -O0
> > > else
> > > ifdef CONFIG_KEXEC
> > > CFLAGS += -gdwarf-2 -O0
> > > endif
> > > endif
> > > -------------------------------------------------------------------------
> > >-----
> > >
> > > In the top level kernel Makefile I have:
> > > -------------------------------------------------------------------------
> > >----- ifdef CONFIG_FRAME_POINTER
> > > CFLAGS += -fno-omit-frame-pointer
> > > else
> > > CFLAGS += -fomit-frame-pointer
> > > endif
> > >
> > > #
> > > # Compiling the complete kernel without optimization (-O0) for enhanced
> > > debugging
> > > # with kgdb/kdump requires ./mm/memory.c to have:
> > > #
> > > # if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
> > > defined(CONFIG_KEXEC)
> > > # and
> > > # if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
> > > defined(CONFIG_KEXEC)
> > > #
> > > # A less invasive procedure is to use -O1 and only use -O0 for
> > > networking code.
> > > # The networking Makefiles have been setup to support this. So just
> > > change
> > > # -O0 to -O1 below and back out the kgdb change in ./mm/memory.c for a
> > > # less invasive change. Compiling -O0 also required increasing
> > > ROUNDUP_WAIT in
> > > # linux/kernel/kgdb.c; value in 2.6.12 patch was way to low and value in
> > > 2.6.16
> > > # is marginal and frequently causes lead CPU to times out prematurely
> > > waiting for
> > > # other CPU's to stop.
> > > #
> > > ifdef CONFIG_DEBUG_INFO
> > > ifdef CONFIG_KGDB
> > > CFLAGS += -gdwarf-2 -O0
> > > else
> > > ifdef CONFIG_KEXEC
> > > CFLAGS += -gdwarf-2 -O1
> > > else
> > > CFLAGS += -g
> > > endif
> > > endif
> > > endif
> > > -------------------------------------------------------------------------
> > >-----------
> > >
> > > > I know this is not a
> > > > bug but with -O optimizations and inlines on tcp-ip stack, program
> > > > counter goes everywhere madly even with step or next command and this
> > > > makes debugging incomprehensible.
> > >
> > > Yep, I don't understand why everyone else doesn't. It's also
> > > like using debug printf's, I like being able to trace the code
> > > to get the big picture and then a -O0 to look at details with kgdb.
> > >
> > > Some believe doing this kind of stuff is blasphemy. The
> > > Bible says I should be killed for working on Sunday; I
> > > happen to disagree.
> > >
> > > > At this point I have two questions:
> > > > 1- Is there any way to compile kernel with -O0 flag and if it's
> > > > possible, may it cause any problems?
> > >
> > > I offered to post them to Amit back on Sept 06(2:45 PM) but I don't
> > > think I ever heard back. I'd prefer to see the -O0 and KGDB_DEBUG
> > > code for tracing the kgdb stub assimilated. If they would be accepted
> > > I could make a patch to Tom's git repository...
> > >
> > > > 2- Why does kernel fail while compiling with O0 flag and why does
> > > > linux kernel depends on inline functions so much?
> > >
> > > I think it's an obsession with performance. As long as I/we can map
> > > "static inline" to "static" it's not a big deal.
> > >
> > > > Is there anyone
> > > > whoever uses kgdb for debugging linux tcp-ip stack or any effort to
> > > > compile kernel with no optimization?
> > >
> > > I'm using it every day; works great. I also recommend by SOCK_DEBUG,
> > > SKB_DEBUG, and TCP_DEBUG macros to trace the TCP code. I also indent
> > > the trace to make it easy to read.
> > >
> > > function1() {
> > > function2() {
> > > function3() {
> > > function4();
> > > }
> > > }
> > > }
> > >
> > > The brackets make it easy to see the scope of the trace with vi.
> > > I like tracing with 'C' syntax since it what the reader is use to.
> > > If folks are interested I could also add that to the git diff, but
> > > I think that likely belongs else where and isn't likely the current
> > > dogma. See snippet from attached network trace. I gave a talk
> > > at a UNENIX conference back in the 1980 recommending a common UNIX
> > > tracing paradigm and a few liked it. The director of Siemens,
> > > Struck Zimmerman, didn't; you can't please everyone, so I just do what
> > > I think is best and live with the world not being as I'd expect it to
> > > be.
> > >
> > > For TCP I'm using the attached sock.h fragment which has a
> > > backward compatible SOCK_DEBUG() macro. I used the same paradigm in
> > > skbuff.h; see attachment. Likewise I'm doing the same in kgdb.h; also
> > > attached.
> > >
> > > In printk I added:
> > > --------------------------------------------------------------------
> > >
> > > for (tp = tbuf; tp < tbuf + tlen; tp++)
> > > emit_log_char(*tp);
> > > printed_len += tlen - 3;
> > > #ifdef CONFIG_PRINTK_INDENT
> > > if (!in_interrupt()) {
> > > int depth = stack_depth();
> > > int i;
> > >
> > > if ((depth > 0) && (depth < 120)) {
> > > for(i = 0; i < depth; i++) {
> > > emit_log_char(' ');
> > > printed_len++;
> > > }
> > > }
> > > }
> > > #endif
> > > -----------------------------------------------------------
> > >
> > > and I added stack_depth() function
> > > to ../linux-4/arch/i386/kernel/traps.c
> > > -----------------------------------------------------------
> > > int stack_depth(void)
> > > {
> > > struct thread_info *tinfo;
> > > unsigned long ebp;
> > > int depth = 0;
> > >
> > > #ifdef CONFIG_FRAME_POINTER
> > > asm("andl %%esp,%0; ":"=r" (tinfo) : "0" (~(THREAD_SIZE - 1)));
> > > asm ("movl %%ebp, %0" : "=r" (ebp) : );
> > >
> > > while (valid_stack_ptr(tinfo, (void *)ebp)) {
> > > ebp = *(unsigned long *)ebp;
> > > if (depth++ > 100) {
> > > break;
> > > }
> > > }
> > > #endif
> > > return(depth);
> > > }
> > > -------------------------------------------------------------------------
> > >--
> > >
> > > Let me know if you you have any questions. Sounds like your on the right
> > > track; IMHO.
> > >
> > > -piet
> > >
> > > > Thanks alot.
> > > > Emin
> > >
> > > -------------------------------------------------------------------------
> > >
> > > > Take Surveys. Earn Cash. Influence the Future of IT
> > > > Join SourceForge.net's Techsay panel and you'll get the chance to
> > >
> > > share your
> > >
> > > > opinions on IT & business topics through brief surveys -- and earn
> > >
> > > cash
> > >
> > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > >
> > > > _______________________________________________
> > > > Kgdb-bugreport mailing list
> > > > Kgdb-bugreport@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
> > >
> > > --
> > > Piet Delaney Phone: (408) 200-5256
> > > Blue Lane Technologies Fax: (408) 200-5299
> > > 10450 Bubb Rd.
> > > Cupertino, Ca. 95014 Email: piet@bluelane.com
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share
> > your opinions on IT & business topics through brief surveys -- and earn
> > cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Kgdb-bugreport mailing list
> > Kgdb-bugreport@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
--
Piet Delaney Phone: (408) 200-5256
Blue Lane Technologies Fax: (408) 200-5299
10450 Bubb Rd.
Cupertino, Ca. 95014 Email: piet@bluelane.com
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2006-09-26 21:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <2cf1ee820609211354m3ae2ec5btc7274402d84b5400@mail.gmail.com>
2006-09-22 22:40 ` [Kgdb-bugreport] compiling kernel with -O0 flag (For optimal debugging with kgdb and/or crash) Piet Delaney
[not found] ` <1159219251.32095.143.camel@piet2.bluelane.com>
[not found] ` <2cf1ee820609260150o77d7759did8d38937669b356f@mail.gmail.com>
[not found] ` <200609261426.16481.amitkale@linsyssoft.com>
2006-09-26 21:48 ` [Kgdb-bugreport] compiling kernel with -O0 flag - Lets make it a config option! Piet Delaney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®