[RFC 0/3] x86_64,tracing: multiplexing function tracer

[RFC 0/3] x86_64,tracing: multiplexing function tracer

[Jiri Olsa]

hi, me again ;)

from last RFC email it turned out it'd be good to have support
for more than 1 currently running function tracers. With
the possibility to specify separate filter for each of them.

I thought about one way how to do that and put it to the
code for considering. It seems to work properly, though
I was fighting with set_ftrace_filter for a while.. and
it is still not ideal ;)


How does it work?

 - when the function tracer is registered it got assigned unique ID
 - 'mcount callback' beside ip and parent_ip provides bitmask
   set with tracers IDs which are interested in the current function
 - 'mcount callback' calls tracers callbacks as set in the bitmask


How did that bitfield get to the 'mcount callback'?

 - several 'mcount callbacks' are statically generated with code
   providing unique bitmask
 - each combination of tracers (IDs) map to a single 'mcount callback'
   providing bitmask of this combination
 - function pointers of these callbacks are stored and used
   to patch the function 'mcount call' instruction

 To have all bitmask possibilities covered it is generated
 2^(allowed tracers) number of 'mcount callbacks'. As this
 statical part is relativelly small it's probably ok.

 I guess it's possible to use dynamic approach and allocate
 only as many 'mcount callbacks' as needed (similar as
 for optimized kprobes).


How does filtering work (interface) ?

 - each tracer got unique name
 - added file function_tracers providing list of registered tracers
 - one tracer is allways default, and this one will be changed
   via set_ftrace_filter interface
 - default tracer can be changed by write the name to the function_tracers
   file


How does filtering work (code) ?

 - each 'struct dyn_ftrace' record now holds 2 bitmasks:
   filter and notrace
 - those 2 bitmask are updated by the set_ftrace_filter code
   to carry bitmask of tracers interested in the function
 - this info is processed during the 'mcount callback' assigning


Example session

	# echo function > ./current_tracer 
	# cat function_tracers 
	*trace
	# echo 1 > ./function_profile_enabled 
	# cat function_tracers 
	 trace
	*trace_profile
	# cat set_ftrace_filter 
	[               trace] #### all functions enabled ####
	[       trace_profile] #### all functions enabled ####
	# echo sys_read > ./set_ftrace_filter 
	# cat set_ftrace_filter 
	[               trace] #### all functions enabled ####
	[       trace_profile] sys_read
	# echo trace > ./function_tracers 
	# cat function_tracers 
	*trace
	 trace_profile
	# echo sys_write > ./set_ftrace_filter 
	# cat set_ftrace_filter 
	[               trace] sys_write
	[       trace_profile] sys_read


attached patches:
- 1/3 tracing: function tracer registration
- 2/3 tracing: adding statical callers
- 3/3 tracing: set_ftrace_filter support


This is by no means complete solution, and there are
many leftovers.. just wanted to try this way ;)

plz let me know what you think

wbr,
jirka
---
 arch/x86/include/asm/ftrace.h     |    4 +
 arch/x86/kernel/entry_64.S        |   22 ++-
 arch/x86/kernel/ftrace.c          |   50 ++--
 include/linux/ftrace.h            |   16 +-
 kernel/trace/ftrace.c             |  555 +++++++++++++++++++++----------------
 kernel/trace/trace_events.c       |    1 +
 kernel/trace/trace_functions.c    |    2 +
 kernel/trace/trace_irqsoff.c      |    1 +
 kernel/trace/trace_sched_wakeup.c |    1 +
 kernel/trace/trace_stack.c        |    1 +
 10 files changed, 383 insertions(+), 270 deletions(-)

[RFC 1/3] tracing: function tracer registration

[Jiri Olsa]

adding ID and default tracer support for function
tracer registration

---
 arch/x86/include/asm/ftrace.h     |    2 +
 arch/x86/kernel/entry_64.S        |    6 +-
 arch/x86/kernel/ftrace.c          |   13 --
 include/linux/ftrace.h            |    9 +-
 kernel/trace/ftrace.c             |  290 +++++++++++++++----------------------
 kernel/trace/trace_events.c       |    1 +
 kernel/trace/trace_functions.c    |    2 +
 kernel/trace/trace_irqsoff.c      |    1 +
 kernel/trace/trace_sched_wakeup.c |    1 +
 kernel/trace/trace_stack.c        |    1 +
 10 files changed, 136 insertions(+), 190 deletions(-)

diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h
index db24c22..45ae1fa 100644
--- a/arch/x86/include/asm/ftrace.h
+++ b/arch/x86/include/asm/ftrace.h
@@ -1,6 +1,8 @@
 #ifndef _ASM_X86_FTRACE_H
 #define _ASM_X86_FTRACE_H
 
+#define FTRACE_TRACERS_MAX 4
+
 #ifdef __ASSEMBLY__
 
 	.macro MCOUNT_SAVE_FRAME
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 8a445a0..e0ba7d9 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -79,8 +79,10 @@ ENTRY(ftrace_caller)
 	movq 8(%rbp), %rsi
 	subq $MCOUNT_INSN_SIZE, %rdi
 
-GLOBAL(ftrace_call)
-	call ftrace_stub
+	/* by default call all registered handlers */
+	movq $-1, %rdx
+
+	call ftrace_call
 
 	MCOUNT_RESTORE_FRAME
 
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 0ba15a6..dc92378 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -321,19 +321,6 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
 	return ftrace_modify_code(rec->ip, old, new);
 }
 
-int ftrace_update_ftrace_func(ftrace_func_t func)
-{
-	unsigned long ip = (unsigned long)(&ftrace_call);
-	unsigned char old[MCOUNT_INSN_SIZE], *new;
-	int ret;
-
-	memcpy(old, &ftrace_call, MCOUNT_INSN_SIZE);
-	new = ftrace_call_replace(ip, (unsigned long)func);
-	ret = ftrace_modify_code(ip, old, new);
-
-	return ret;
-}
-
 int __init ftrace_dyn_arch_init(void *data)
 {
 	/* The return code is retured via data */
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index ca29e03..b7ab09d 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -30,8 +30,9 @@ ftrace_enable_sysctl(struct ctl_table *table, int write,
 typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip);
 
 struct ftrace_ops {
-	ftrace_func_t	  func;
-	struct ftrace_ops *next;
+	char *name;
+	int   id;
+	ftrace_func_t func;
 };
 
 extern int function_trace_stop;
@@ -175,9 +176,9 @@ int unregister_ftrace_command(struct ftrace_func_command *cmd);
 /* defined in arch */
 extern int ftrace_ip_converted(unsigned long ip);
 extern int ftrace_dyn_arch_init(void *data);
-extern int ftrace_update_ftrace_func(ftrace_func_t func);
 extern void ftrace_caller(void);
-extern void ftrace_call(void);
+extern void ftrace_call(unsigned long ip, unsigned long parent_ip,
+			unsigned long flags);
 extern void mcount_call(void);
 
 #ifndef FTRACE_ADDR
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index ee24fa1..f1eedda 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -76,62 +76,12 @@ static int ftrace_disabled __read_mostly;
 
 static DEFINE_MUTEX(ftrace_lock);
 
-static struct ftrace_ops ftrace_list_end __read_mostly =
-{
-	.func		= ftrace_stub,
-};
-
-static struct ftrace_ops *ftrace_list __read_mostly = &ftrace_list_end;
-ftrace_func_t ftrace_trace_function __read_mostly = ftrace_stub;
-ftrace_func_t __ftrace_trace_function __read_mostly = ftrace_stub;
-ftrace_func_t ftrace_pid_function __read_mostly = ftrace_stub;
-
-/*
- * Traverse the ftrace_list, invoking all entries.  The reason that we
- * can use rcu_dereference_raw() is that elements removed from this list
- * are simply leaked, so there is no need to interact with a grace-period
- * mechanism.  The rcu_dereference_raw() calls are needed to handle
- * concurrent insertions into the ftrace_list.
- *
- * Silly Alpha and silly pointer-speculation compiler optimizations!
- */
-static void ftrace_list_func(unsigned long ip, unsigned long parent_ip)
-{
-	struct ftrace_ops *op = rcu_dereference_raw(ftrace_list); /*see above*/
-
-	while (op != &ftrace_list_end) {
-		op->func(ip, parent_ip);
-		op = rcu_dereference_raw(op->next); /*see above*/
-	};
-}
-
-static void ftrace_pid_func(unsigned long ip, unsigned long parent_ip)
-{
-	if (!test_tsk_trace_trace(current))
-		return;
-
-	ftrace_pid_function(ip, parent_ip);
-}
+static struct ftrace_ops *ftrace_tracers[FTRACE_TRACERS_MAX];
+static DECLARE_BITMAP(ftrace_tracers_bm, FTRACE_TRACERS_MAX);
+static int ftrace_tracers_cur = -1;
 
-static void set_ftrace_pid_function(ftrace_func_t func)
-{
-	/* do not set ftrace_pid_function to itself! */
-	if (func != ftrace_pid_func)
-		ftrace_pid_function = func;
-}
-
-/**
- * clear_ftrace_function - reset the ftrace function
- *
- * This NULLs the ftrace function and in essence stops
- * tracing.  There may be lag
- */
-void clear_ftrace_function(void)
-{
-	ftrace_trace_function = ftrace_stub;
-	__ftrace_trace_function = ftrace_stub;
-	ftrace_pid_function = ftrace_stub;
-}
+#define for_each_tracer(id, bm) \
+	for_each_set_bit(id, bm, FTRACE_TRACERS_MAX)
 
 #ifndef CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST
 /*
@@ -147,114 +97,82 @@ static void ftrace_test_stop_func(unsigned long ip, unsigned long parent_ip)
 }
 #endif
 
-static int __register_ftrace_function(struct ftrace_ops *ops)
-{
-	ops->next = ftrace_list;
-	/*
-	 * We are entering ops into the ftrace_list but another
-	 * CPU might be walking that list. We need to make sure
-	 * the ops->next pointer is valid before another CPU sees
-	 * the ops pointer included into the ftrace_list.
-	 */
-	rcu_assign_pointer(ftrace_list, ops);
 
-	if (ftrace_enabled) {
-		ftrace_func_t func;
+void ftrace_call(unsigned long ip, unsigned long parent_ip,
+		 unsigned long flags)
+{
+	int id;
 
-		if (ops->next == &ftrace_list_end)
-			func = ops->func;
-		else
-			func = ftrace_list_func;
+	if (!list_empty(&ftrace_pids) &&
+	    !test_tsk_trace_trace(current))
+		return;
 
-		if (!list_empty(&ftrace_pids)) {
-			set_ftrace_pid_function(func);
-			func = ftrace_pid_func;
-		}
+	rcu_read_lock();
+	for_each_tracer(id, &flags) {
+		struct ftrace_ops *ops = rcu_dereference(ftrace_tracers[id]);
+		if (!ops)
+			continue;
 
-		/*
-		 * For one func, simply call it directly.
-		 * For more than one func, call the chain.
-		 */
-#ifdef CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST
-		ftrace_trace_function = func;
-#else
-		__ftrace_trace_function = func;
-		ftrace_trace_function = ftrace_test_stop_func;
-#endif
+		ops->func(ip, parent_ip);
 	}
-
-	return 0;
+	rcu_read_unlock();
 }
 
-static int __unregister_ftrace_function(struct ftrace_ops *ops)
+static int __register_ftrace_function(struct ftrace_ops *ops)
 {
-	struct ftrace_ops **p;
+	int id;
 
-	/*
-	 * If we are removing the last function, then simply point
-	 * to the ftrace_stub.
-	 */
-	if (ftrace_list == ops && ops->next == &ftrace_list_end) {
-		ftrace_trace_function = ftrace_stub;
-		ftrace_list = &ftrace_list_end;
-		return 0;
-	}
+	id = find_first_zero_bit(ftrace_tracers_bm, FTRACE_TRACERS_MAX);
+	if (id >= FTRACE_TRACERS_MAX)
+		return -ENOSPC;
 
-	for (p = &ftrace_list; *p != &ftrace_list_end; p = &(*p)->next)
-		if (*p == ops)
-			break;
+	ops->id = id;
+	set_bit(id, ftrace_tracers_bm);
+	ftrace_tracers_cur = id;
+	rcu_assign_pointer(ftrace_tracers[id], ops);
+	return 0;
+}
 
-	if (*p != ops)
-		return -1;
+static int __unregister_ftrace_function(struct ftrace_ops *ops)
+{
+	int id = ops->id;
 
-	*p = (*p)->next;
+	clear_bit(id, ftrace_tracers_bm);
+	BUG_ON(ops != ftrace_tracers[id]);
+	rcu_assign_pointer(ftrace_tracers[id], NULL);
 
-	if (ftrace_enabled) {
-		/* If we only have one func left, then call that directly */
-		if (ftrace_list->next == &ftrace_list_end) {
-			ftrace_func_t func = ftrace_list->func;
+	if (ftrace_tracers_cur != id)
+		return 0;
 
-			if (!list_empty(&ftrace_pids)) {
-				set_ftrace_pid_function(func);
-				func = ftrace_pid_func;
-			}
-#ifdef CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST
-			ftrace_trace_function = func;
-#else
-			__ftrace_trace_function = func;
-#endif
-		}
-	}
+	/* find another default tracer */
+	ftrace_tracers_cur = find_first_bit(ftrace_tracers_bm,
+					    FTRACE_TRACERS_MAX);
+	if (ftrace_tracers_cur >= FTRACE_TRACERS_MAX)
+		ftrace_tracers_cur = -1;
 
 	return 0;
 }
 
-static void ftrace_update_pid_func(void)
+static int ftrace_tracer_default(char *name)
 {
-	ftrace_func_t func;
+	int id, ret = -EINVAL;
 
-	if (ftrace_trace_function == ftrace_stub)
-		return;
+	mutex_lock(&ftrace_lock);
+	for_each_tracer(id, ftrace_tracers_bm) {
+		struct ftrace_ops *ops = ftrace_tracers[id];
+		BUG_ON(!ops);
 
-#ifdef CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST
-	func = ftrace_trace_function;
-#else
-	func = __ftrace_trace_function;
-#endif
+		if (!strcmp(name, ops->name))
+			break;
+	}
 
-	if (!list_empty(&ftrace_pids)) {
-		set_ftrace_pid_function(func);
-		func = ftrace_pid_func;
-	} else {
-		if (func == ftrace_pid_func)
-			func = ftrace_pid_function;
+	if (id < FTRACE_TRACERS_MAX) {
+		ftrace_tracers_cur = id;
+		ret = 0;
 	}
 
-#ifdef CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST
-	ftrace_trace_function = func;
-#else
-	__ftrace_trace_function = func;
-#endif
+	mutex_unlock(&ftrace_lock);
+	return ret;
 }
 
 #ifdef CONFIG_FUNCTION_PROFILER
@@ -717,6 +635,7 @@ static void unregister_ftrace_profiler(void)
 #else
 static struct ftrace_ops ftrace_profile_ops __read_mostly =
 {
+	.name		= "trace_profile",
 	.func		= function_profile_call,
 };
 
@@ -1143,9 +1062,6 @@ static int __ftrace_modify_code(void *data)
 	else if (*command & FTRACE_DISABLE_CALLS)
 		ftrace_replace_code(0);
 
-	if (*command & FTRACE_UPDATE_TRACE_FUNC)
-		ftrace_update_ftrace_func(ftrace_trace_function);
-
 	if (*command & FTRACE_START_FUNC_RET)
 		ftrace_enable_ftrace_graph_caller();
 	else if (*command & FTRACE_STOP_FUNC_RET)
@@ -1174,11 +1090,6 @@ static int ftrace_start_up;
 
 static void ftrace_startup_enable(int command)
 {
-	if (saved_ftrace_func != ftrace_trace_function) {
-		saved_ftrace_func = ftrace_trace_function;
-		command |= FTRACE_UPDATE_TRACE_FUNC;
-	}
-
 	if (!command || !ftrace_enabled)
 		return;
 
@@ -1212,11 +1123,6 @@ static void ftrace_shutdown(int command)
 	if (!ftrace_start_up)
 		command |= FTRACE_DISABLE_CALLS;
 
-	if (saved_ftrace_func != ftrace_trace_function) {
-		saved_ftrace_func = ftrace_trace_function;
-		command |= FTRACE_UPDATE_TRACE_FUNC;
-	}
-
 	if (!command || !ftrace_enabled)
 		return;
 
@@ -1732,6 +1638,53 @@ ftrace_regex_lseek(struct file *file, loff_t offset, int origin)
 	return ret;
 }
 
+static int ftrace_tracers_show(struct seq_file *seq, void *v)
+{
+	int id;
+
+	mutex_lock(&ftrace_lock);
+	for_each_tracer(id, ftrace_tracers_bm) {
+		struct ftrace_ops *ops = ftrace_tracers[id];
+
+		BUG_ON(!ops);
+		seq_printf(seq, "%s%s\n",
+			ftrace_tracers_cur == id ? "*" : " ",
+			ops->name);
+	}
+
+	mutex_unlock(&ftrace_lock);
+	return 0;
+}
+
+static int
+ftrace_tracers_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, ftrace_tracers_show, NULL);
+}
+
+static ssize_t
+ftrace_tracers_write(struct file *filp, const char __user *ubuf, size_t cnt,
+		     loff_t *ppos)
+{
+	char buf[64];
+	int i;
+
+	if (cnt >= sizeof(buf))
+		return -EINVAL;
+
+	if (copy_from_user(&buf, ubuf, cnt))
+		return -EFAULT;
+
+	buf[cnt] = 0;
+	for (i = cnt - 1; i > 0 && isspace(buf[i]); i--)
+		buf[i] = 0;
+
+	if (ftrace_tracer_default(buf))
+		return -EINVAL;
+
+	return cnt;
+}
+
 static int ftrace_match(char *str, char *regex, int len, int type)
 {
 	int matched = 0;
@@ -1952,6 +1905,7 @@ function_trace_probe_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_probe_ops __read_mostly =
 {
+	.name		= "trace_probe",
 	.func		= function_trace_probe_call,
 };
 
@@ -2467,6 +2421,14 @@ static const struct file_operations ftrace_notrace_fops = {
 	.release = ftrace_notrace_release,
 };
 
+static const struct file_operations ftrace_tracers_fops = {
+	.open		= ftrace_tracers_open,
+	.read		= seq_read,
+	.release	= single_release,
+	.llseek		= seq_lseek,
+	.write          = ftrace_tracers_write,
+};
+
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 
 static DEFINE_MUTEX(graph_lock);
@@ -2688,6 +2650,9 @@ static __init int ftrace_init_dyn_debugfs(struct dentry *d_tracer)
 	trace_create_file("set_ftrace_notrace", 0644, d_tracer,
 				    NULL, &ftrace_notrace_fops);
 
+	trace_create_file("function_tracers", 0644, d_tracer,
+				    NULL, &ftrace_tracers_fops);
+
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 	trace_create_file("set_graph_function", 0444, d_tracer,
 				    NULL,
@@ -2950,7 +2915,6 @@ static int ftrace_pid_add(int p)
 
 	set_ftrace_pid_task(pid);
 
-	ftrace_update_pid_func();
 	ftrace_startup_enable(0);
 
 	mutex_unlock(&ftrace_lock);
@@ -2979,7 +2943,6 @@ static void ftrace_pid_reset(void)
 		kfree(fpid);
 	}
 
-	ftrace_update_pid_func();
 	ftrace_startup_enable(0);
 
 	mutex_unlock(&ftrace_lock);
@@ -3127,7 +3090,6 @@ void ftrace_kill(void)
 {
 	ftrace_disabled = 1;
 	ftrace_enabled = 0;
-	clear_ftrace_function();
 }
 
 /**
@@ -3194,24 +3156,10 @@ ftrace_enable_sysctl(struct ctl_table *table, int write,
 
 	last_ftrace_enabled = !!ftrace_enabled;
 
-	if (ftrace_enabled) {
-
+	if (ftrace_enabled)
 		ftrace_startup_sysctl();
-
-		/* we are starting ftrace again */
-		if (ftrace_list != &ftrace_list_end) {
-			if (ftrace_list->next == &ftrace_list_end)
-				ftrace_trace_function = ftrace_list->func;
-			else
-				ftrace_trace_function = ftrace_list_func;
-		}
-
-	} else {
-		/* stopping ftrace calls (just send to ftrace_stub) */
-		ftrace_trace_function = ftrace_stub;
-
+	else
 		ftrace_shutdown_sysctl();
-	}
 
  out:
 	mutex_unlock(&ftrace_lock);
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index e88f74f..43600a1 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -1651,6 +1651,7 @@ function_test_events_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_ops __initdata  =
 {
+	.name = "trace_event",
 	.func = function_test_events_call,
 };
 
diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c
index 16aee4d..a991743 100644
--- a/kernel/trace/trace_functions.c
+++ b/kernel/trace/trace_functions.c
@@ -148,11 +148,13 @@ function_stack_trace_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_ops __read_mostly =
 {
+	.name = "trace",
 	.func = function_trace_call,
 };
 
 static struct ftrace_ops trace_stack_ops __read_mostly =
 {
+	.name = "trace_stack",
 	.func = function_stack_trace_call,
 };
 
diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c
index a4969b4..6983594 100644
--- a/kernel/trace/trace_irqsoff.c
+++ b/kernel/trace/trace_irqsoff.c
@@ -152,6 +152,7 @@ irqsoff_tracer_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_ops __read_mostly =
 {
+	.name = "trace_irqs",
 	.func = irqsoff_tracer_call,
 };
 #endif /* CONFIG_FUNCTION_TRACER */
diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c
index 7319559..f8b4622 100644
--- a/kernel/trace/trace_sched_wakeup.c
+++ b/kernel/trace/trace_sched_wakeup.c
@@ -128,6 +128,7 @@ wakeup_tracer_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_ops __read_mostly =
 {
+	.name = "trace_wakeup",
 	.func = wakeup_tracer_call,
 };
 #endif /* CONFIG_FUNCTION_TRACER */
diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c
index 4c5dead..31f361e 100644
--- a/kernel/trace/trace_stack.c
+++ b/kernel/trace/trace_stack.c
@@ -132,6 +132,7 @@ stack_trace_call(unsigned long ip, unsigned long parent_ip)
 
 static struct ftrace_ops trace_ops __read_mostly =
 {
+	.name = "trace_stack",
 	.func = stack_trace_call,
 };
 
-- 
1.7.1

[RFC 2/3] tracing: adding statical callers

[Jiri Olsa]

adding statically generated code for 'mcount callback' providing
unique bitmask

---
 arch/x86/include/asm/ftrace.h |    2 ++
 arch/x86/kernel/entry_64.S    |   22 +++++++++++++++++-----
 arch/x86/kernel/ftrace.c      |   13 +++++++++++++
 include/linux/ftrace.h        |    4 +---
 kernel/trace/ftrace.c         |    2 +-
 5 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h
index 45ae1fa..6ef436b 100644
--- a/arch/x86/include/asm/ftrace.h
+++ b/arch/x86/include/asm/ftrace.h
@@ -2,6 +2,7 @@
 #define _ASM_X86_FTRACE_H
 
 #define FTRACE_TRACERS_MAX 4
+#define FTRACE_CALLERS_MAX ((1 << FTRACE_TRACERS_MAX) - 1)
 
 #ifdef __ASSEMBLY__
 
@@ -36,6 +37,7 @@
 
 #ifndef __ASSEMBLY__
 extern void mcount(void);
+extern unsigned long ftrace_callers[FTRACE_CALLERS_MAX];
 
 static inline unsigned long ftrace_call_adjust(unsigned long addr)
 {
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index e0ba7d9..1ec1668 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -69,19 +69,31 @@ ENTRY(mcount)
 	retq
 END(mcount)
 
-ENTRY(ftrace_caller)
+	.section .rodata,"a"
+ENTRY(ftrace_callers)
+	.section .entry.text
+.macro FTRACE_GEN_CALLERS num
+i=1
+.rept (1 << \num) - 1
+1:
 	cmpl $0, function_trace_stop
 	jne  ftrace_stub
-
 	MCOUNT_SAVE_FRAME
+	movq $i, %rdx
+	jmp ftrace_caller
+	.previous
+	.quad 1b
+	.section .entry.text
+i=i+1
+.endr
+.endm
+	FTRACE_GEN_CALLERS FTRACE_TRACERS_MAX
 
+ENTRY(ftrace_caller)
 	movq 0x38(%rsp), %rdi
 	movq 8(%rbp), %rsi
 	subq $MCOUNT_INSN_SIZE, %rdi
 
-	/* by default call all registered handlers */
-	movq $-1, %rdx
-
 	call ftrace_call
 
 	MCOUNT_RESTORE_FRAME
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index dc92378..5a8e679 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -328,6 +328,19 @@ int __init ftrace_dyn_arch_init(void *data)
 
 	return 0;
 }
+
+unsigned long ftrace_arch_tracer_addr(int mask)
+{
+	if (!mask)
+		return 0L;
+
+	/* tracer 1 is index 0 in ftrace_callers */
+	--mask;
+
+	BUG_ON(mask >= FTRACE_CALLERS_MAX);
+	return ftrace_callers[mask];
+}
+
 #endif
 
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index b7ab09d..b12f5df 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -180,10 +180,8 @@ extern void ftrace_caller(void);
 extern void ftrace_call(unsigned long ip, unsigned long parent_ip,
 			unsigned long flags);
 extern void mcount_call(void);
+extern unsigned long ftrace_arch_tracer_addr(int mask);
 
-#ifndef FTRACE_ADDR
-#define FTRACE_ADDR ((unsigned long)ftrace_caller)
-#endif
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 extern void ftrace_graph_caller(void);
 extern int ftrace_enable_ftrace_graph_caller(void);
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index f1eedda..b39dd52 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -963,7 +963,7 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable)
 	unsigned long ftrace_addr;
 	unsigned long flag = 0UL;
 
-	ftrace_addr = (unsigned long)FTRACE_ADDR;
+	ftrace_addr = ftrace_arch_tracer_addr(FTRACE_CALLERS_MAX);
 
 	/*
 	 * If this record is not to be traced or we want to disable it,
-- 
1.7.1

[RFC 3/3] tracing: set_ftrace_filter support

[Jiri Olsa]

adding support to use set_ftrace_filter for multiple tracers

---
 arch/x86/kernel/ftrace.c |   24 +++--
 include/linux/ftrace.h   |    3 +
 kernel/trace/ftrace.c    |  269 ++++++++++++++++++++++++++++++++++------------
 3 files changed, 219 insertions(+), 77 deletions(-)

diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 5a8e679..5fdb40e 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -76,18 +76,20 @@ static int ftrace_calc_offset(long ip, long addr)
 	return (int)(addr - ip);
 }
 
-static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr)
+static unsigned char *ftrace_call_replace(unsigned long ip, unsigned long addr, int old)
 {
-	static union ftrace_code_union calc;
+	static union ftrace_code_union calc_new;
+	static union ftrace_code_union calc_old;
+	union ftrace_code_union *calc = old ? &calc_old : &calc_new;
 
-	calc.e8		= 0xe8;
-	calc.offset	= ftrace_calc_offset(ip + MCOUNT_INSN_SIZE, addr);
+	calc->e8	= 0xe8;
+	calc->offset	= ftrace_calc_offset(ip + MCOUNT_INSN_SIZE, addr);
 
 	/*
 	 * No locking needed, this must be called via kstop_machine
 	 * which in essence is like running on a uniprocessor machine.
 	 */
-	return calc.code;
+	return calc->code;
 }
 
 /*
@@ -304,9 +306,10 @@ int ftrace_make_nop(struct module *mod,
 	unsigned char *new, *old;
 	unsigned long ip = rec->ip;
 
-	old = ftrace_call_replace(ip, addr);
+	old = ftrace_call_replace(ip, addr, 1);
 	new = ftrace_nop_replace();
 
+	rec->addr = 0L;
 	return ftrace_modify_code(rec->ip, old, new);
 }
 
@@ -315,9 +318,14 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
 	unsigned char *new, *old;
 	unsigned long ip = rec->ip;
 
-	old = ftrace_nop_replace();
-	new = ftrace_call_replace(ip, addr);
+	if (rec->addr)
+		old = ftrace_call_replace(ip, rec->addr, 1);
+	else
+		old = ftrace_nop_replace();
 
+	new = ftrace_call_replace(ip, addr, 0);
+
+	rec->addr = addr;
 	return ftrace_modify_code(rec->ip, old, new);
 }
 
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index b12f5df..9ce3484 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -164,6 +164,9 @@ struct dyn_ftrace {
 		unsigned long		flags;
 		struct dyn_ftrace	*newlist;
 	};
+	unsigned long			filter;
+	unsigned long			no_trace;
+	unsigned long			addr;
 	struct dyn_arch_ftrace		arch;
 };
 
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index b39dd52..bbffb05 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -80,6 +80,8 @@ static struct ftrace_ops *ftrace_tracers[FTRACE_TRACERS_MAX];
 static DECLARE_BITMAP(ftrace_tracers_bm, FTRACE_TRACERS_MAX);
 static int ftrace_tracers_cur = -1;
 
+static void __ftrace_tracer_clear(int id, int type);
+
 #define for_each_tracer(id, bm) \
 	for_each_set_bit(id, bm, FTRACE_TRACERS_MAX)
 
@@ -141,6 +143,8 @@ static int __unregister_ftrace_function(struct ftrace_ops *ops)
 	BUG_ON(ops != ftrace_tracers[id]);
 	rcu_assign_pointer(ftrace_tracers[id], NULL);
 
+	__ftrace_tracer_clear(id, FTRACE_FL_FILTER|FTRACE_FL_NOTRACE);
+
 	if (ftrace_tracers_cur != id)
 		return 0;
 
@@ -175,6 +179,11 @@ static int ftrace_tracer_default(char *name)
 	return ret;
 }
 
+static int __ftrace_get_next_tracer(int idx)
+{
+	return find_next_bit(ftrace_tracers_bm, FTRACE_TRACERS_MAX, idx);
+}
+
 #ifdef CONFIG_FUNCTION_PROFILER
 struct ftrace_profile {
 	struct hlist_node		node;
@@ -808,7 +817,7 @@ enum {
 	FTRACE_STOP_FUNC_RET		= (1 << 4),
 };
 
-static int ftrace_filtered;
+static unsigned long ftrace_tracer_filtered;
 
 static struct dyn_ftrace *ftrace_new_addrs;
 
@@ -845,6 +854,20 @@ static struct dyn_ftrace *ftrace_free_records;
 		}				\
 	}
 
+static void __ftrace_tracer_clear(int id, int type)
+{
+	struct ftrace_page *pg;
+	struct dyn_ftrace *rec;
+	unsigned long mask = ~(1 << id);
+
+	do_for_each_ftrace_rec(pg, rec) {
+		if (type & FTRACE_FL_FILTER)
+			rec->filter  &= mask;
+		if (type & FTRACE_FL_NOTRACE)
+			rec->no_trace &= mask;
+	} while_for_each_ftrace_rec();
+}
+
 static void ftrace_free_rec(struct dyn_ftrace *rec)
 {
 	rec->freelist = ftrace_free_records;
@@ -956,31 +979,52 @@ int ftrace_text_reserved(void *start, void *end)
 	return 0;
 }
 
+unsigned long ftrace_tracer_flag_addr(struct dyn_ftrace *rec,
+				      unsigned long *ftrace_addr, int enable)
+{
+	unsigned long flag = 0L;
+	int id, mask = 0;
+
+	for_each_tracer(id, ftrace_tracers_bm) {
+#define SET(var) (var & (1 << id))
+		int ftrace_filtered = SET(ftrace_tracer_filtered);
+
+		/*
+		 * If this record is not to be traced or we want to disable it,
+		 * then disable it.
+		 *
+		 * If we want to enable it and filtering is off, then enable
+		 * it.
+		 *
+		 * If we want to enable it and filtering is on, enable it
+		 * only if it's filtered
+		 */
+		if (enable && !(SET(rec->no_trace)))
+			if (!ftrace_filtered || (SET(rec->filter)))
+				mask |= (1 << id);
+#undef SET
+	}
+
+	if (mask)
+		flag = FTRACE_FL_ENABLED;
+
+	*ftrace_addr = ftrace_arch_tracer_addr(mask);
+	return flag;
+}
 
 static int
 __ftrace_replace_code(struct dyn_ftrace *rec, int enable)
 {
+	unsigned long flag;
 	unsigned long ftrace_addr;
-	unsigned long flag = 0UL;
 
-	ftrace_addr = ftrace_arch_tracer_addr(FTRACE_CALLERS_MAX);
+	flag = ftrace_tracer_flag_addr(rec, &ftrace_addr, enable);
 
-	/*
-	 * If this record is not to be traced or we want to disable it,
-	 * then disable it.
-	 *
-	 * If we want to enable it and filtering is off, then enable it.
-	 *
-	 * If we want to enable it and filtering is on, enable it only if
-	 * it's filtered
-	 */
-	if (enable && !(rec->flags & FTRACE_FL_NOTRACE)) {
-		if (!ftrace_filtered || (rec->flags & FTRACE_FL_FILTER))
-			flag = FTRACE_FL_ENABLED;
-	}
+	BUG_ON(flag && !ftrace_addr);
 
 	/* If the state of this record hasn't changed, then do nothing */
-	if ((rec->flags & FTRACE_FL_ENABLED) == flag)
+	if (((rec->flags & FTRACE_FL_ENABLED) == flag) &&
+	    (rec->addr == ftrace_addr))
 		return 0;
 
 	if (flag) {
@@ -989,7 +1033,7 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable)
 	}
 
 	rec->flags &= ~FTRACE_FL_ENABLED;
-	return ftrace_make_nop(NULL, rec, ftrace_addr);
+	return ftrace_make_nop(NULL, rec, rec->addr);
 }
 
 static void ftrace_replace_code(int enable)
@@ -1273,6 +1317,7 @@ struct ftrace_iterator {
 	struct trace_parser		parser;
 	int				hidx;
 	int				idx;
+	int				tracer;
 	unsigned			flags;
 };
 
@@ -1363,6 +1408,76 @@ t_hash_show(struct seq_file *m, struct ftrace_iterator *iter)
 	return 0;
 }
 
+static void reset_iter_read(struct ftrace_iterator *iter)
+{
+	iter->pos = 0;
+	iter->func_pos = 0;
+	iter->flags &= ~(FTRACE_ITER_PRINTALL & FTRACE_ITER_HASH);
+}
+
+static void *t_start(struct seq_file *m, loff_t *pos);
+
+#define TRACER_ISSET(var) (var & (1 << iter->tracer))
+
+static struct dyn_ftrace*
+t_next_entry(struct ftrace_iterator *iter)
+{
+	struct dyn_ftrace *rec = NULL;
+
+	while(!rec) {
+		if (iter->idx >= iter->pg->index) {
+			if (!iter->pg->next)
+				break;
+
+			iter->pg = iter->pg->next;
+			iter->idx = 0;
+
+		} else {
+			rec = &iter->pg->records[iter->idx++];
+
+			if ((rec->flags & FTRACE_FL_FREE) ||
+
+			    (!(iter->flags & FTRACE_ITER_FAILURES) &&
+			     (rec->flags & FTRACE_FL_FAILED)) ||
+
+			    ((iter->flags & FTRACE_ITER_FAILURES) &&
+			     !(rec->flags & FTRACE_FL_FAILED)) ||
+
+			    ((iter->flags & FTRACE_ITER_FILTER) &&
+			     !(TRACER_ISSET(rec->filter))) ||
+
+			    ((iter->flags & FTRACE_ITER_NOTRACE) &&
+			     !(TRACER_ISSET(rec->no_trace)))) {
+				rec = NULL;
+			}
+		}
+	}
+
+	return rec;
+}
+
+static int t_next_new_tracer(struct ftrace_iterator *iter)
+{
+	int id = __ftrace_get_next_tracer(++iter->tracer);
+	if (id >= FTRACE_TRACERS_MAX)
+		return -1;
+
+	iter->tracer = id;
+	iter->pg = ftrace_pages_start;
+	iter->idx = 0;
+
+	if (iter->flags & FTRACE_ITER_FILTER) {
+		if (!TRACER_ISSET(ftrace_tracer_filtered))
+			iter->flags |= FTRACE_ITER_PRINTALL;
+		else
+			iter->flags &= ~FTRACE_ITER_PRINTALL;
+		/* reset in case of seek/pread */
+		iter->flags &= ~FTRACE_ITER_HASH;
+	}
+
+	return id;
+}
+
 static void *
 t_next(struct seq_file *m, void *v, loff_t *pos)
 {
@@ -1375,32 +1490,21 @@ t_next(struct seq_file *m, void *v, loff_t *pos)
 	(*pos)++;
 	iter->pos = iter->func_pos = *pos;
 
+	if (iter->flags & FTRACE_ITER_PRINTALL) {
+		if (t_next_new_tracer(iter) > 0)
+			if (iter->flags & FTRACE_ITER_PRINTALL)
+				return iter;
+	}
+
 	if (iter->flags & FTRACE_ITER_PRINTALL)
 		return t_hash_start(m, pos);
 
- retry:
-	if (iter->idx >= iter->pg->index) {
-		if (iter->pg->next) {
-			iter->pg = iter->pg->next;
-			iter->idx = 0;
-			goto retry;
-		}
-	} else {
-		rec = &iter->pg->records[iter->idx++];
-		if ((rec->flags & FTRACE_FL_FREE) ||
-
-		    (!(iter->flags & FTRACE_ITER_FAILURES) &&
-		     (rec->flags & FTRACE_FL_FAILED)) ||
-
-		    ((iter->flags & FTRACE_ITER_FAILURES) &&
-		     !(rec->flags & FTRACE_FL_FAILED)) ||
-
-		    ((iter->flags & FTRACE_ITER_FILTER) &&
-		     !(rec->flags & FTRACE_FL_FILTER)) ||
-
-		    ((iter->flags & FTRACE_ITER_NOTRACE) &&
-		     !(rec->flags & FTRACE_FL_NOTRACE))) {
-			rec = NULL;
+retry:
+	rec = t_next_entry(iter);
+	if (!rec) {
+		if (t_next_new_tracer(iter) > 0) {
+			if (iter->flags & FTRACE_ITER_PRINTALL)
+				return iter;
 			goto retry;
 		}
 	}
@@ -1409,17 +1513,9 @@ t_next(struct seq_file *m, void *v, loff_t *pos)
 		return t_hash_start(m, pos);
 
 	iter->func = rec;
-
 	return iter;
 }
 
-static void reset_iter_read(struct ftrace_iterator *iter)
-{
-	iter->pos = 0;
-	iter->func_pos = 0;
-	iter->flags &= ~(FTRACE_ITER_PRINTALL & FTRACE_ITER_HASH);
-}
-
 static void *t_start(struct seq_file *m, loff_t *pos)
 {
 	struct ftrace_iterator *iter = m->private;
@@ -1430,15 +1526,17 @@ static void *t_start(struct seq_file *m, loff_t *pos)
 	/*
 	 * If an lseek was done, then reset and start from beginning.
 	 */
-	if (*pos < iter->pos)
+	if (*pos < iter->pos) {
 		reset_iter_read(iter);
+		iter->tracer = 0;
+	}
 
 	/*
 	 * For set_ftrace_filter reading, if we have the filter
 	 * off, we can short cut and just print out that all
 	 * functions are enabled.
 	 */
-	if (iter->flags & FTRACE_ITER_FILTER && !ftrace_filtered) {
+	if (iter->flags & FTRACE_ITER_FILTER && !TRACER_ISSET(ftrace_tracer_filtered)) {
 		if (*pos > 0)
 			return t_hash_start(m, pos);
 		iter->flags |= FTRACE_ITER_PRINTALL;
@@ -1472,6 +1570,7 @@ static void *t_start(struct seq_file *m, loff_t *pos)
 
 	return iter;
 }
+#undef TRACER_ISSET
 
 static void t_stop(struct seq_file *m, void *p)
 {
@@ -1482,12 +1581,14 @@ static int t_show(struct seq_file *m, void *v)
 {
 	struct ftrace_iterator *iter = m->private;
 	struct dyn_ftrace *rec;
+	char *tracer = iter->tracer < FTRACE_TRACERS_MAX ?
+		       ftrace_tracers[iter->tracer]->name : NULL;
 
 	if (iter->flags & FTRACE_ITER_HASH)
 		return t_hash_show(m, iter);
 
 	if (iter->flags & FTRACE_ITER_PRINTALL) {
-		seq_printf(m, "#### all functions enabled ####\n");
+		seq_printf(m, "[%20s] #### all functions enabled ####\n", tracer);
 		return 0;
 	}
 
@@ -1496,7 +1597,7 @@ static int t_show(struct seq_file *m, void *v)
 	if (!rec)
 		return 0;
 
-	seq_printf(m, "%ps\n", (void *)rec->ip);
+	seq_printf(m, "[%20s] %ps\n", tracer, (void *)rec->ip);
 
 	return 0;
 }
@@ -1552,21 +1653,15 @@ ftrace_failures_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
-
 static void ftrace_filter_reset(int enable)
 {
-	struct ftrace_page *pg;
-	struct dyn_ftrace *rec;
 	unsigned long type = enable ? FTRACE_FL_FILTER : FTRACE_FL_NOTRACE;
 
 	mutex_lock(&ftrace_lock);
 	if (enable)
-		ftrace_filtered = 0;
-	do_for_each_ftrace_rec(pg, rec) {
-		if (rec->flags & FTRACE_FL_FAILED)
-			continue;
-		rec->flags &= ~type;
-	} while_for_each_ftrace_rec();
+		ftrace_tracer_filtered &= ~(1 << ftrace_tracers_cur);
+
+	__ftrace_tracer_clear(ftrace_tracers_cur, type);
 	mutex_unlock(&ftrace_lock);
 }
 
@@ -1722,6 +1817,22 @@ ftrace_match_record(struct dyn_ftrace *rec, char *regex, int len, int type)
 	return ftrace_match(str, regex, len, type);
 }
 
+#define TRACER_SET(enable) \
+do { \
+	if (enable) \
+		rec->filter   |= (1 << ftrace_tracers_cur); \
+	else \
+		rec->no_trace |= (1 << ftrace_tracers_cur); \
+} while (0)
+
+#define TRACER_UNSET(enable) \
+do { \
+	if (enable) \
+		rec->filter   &= ~(1 << ftrace_tracers_cur); \
+	else \
+		rec->no_trace &= ~(1 << ftrace_tracers_cur); \
+} while (0)
+
 static int ftrace_match_records(char *buff, int len, int enable)
 {
 	unsigned int search_len;
@@ -1739,16 +1850,23 @@ static int ftrace_match_records(char *buff, int len, int enable)
 	search_len = strlen(search);
 
 	mutex_lock(&ftrace_lock);
+
+	if (ftrace_tracers_cur < 0)
+		goto out_unlock;
+
 	do_for_each_ftrace_rec(pg, rec) {
 
 		if (rec->flags & FTRACE_FL_FAILED)
 			continue;
 
 		if (ftrace_match_record(rec, search, search_len, type)) {
-			if (not)
+			if (not) {
 				rec->flags &= ~flag;
-			else
+				TRACER_UNSET(enable);
+			} else {
 				rec->flags |= flag;
+				TRACER_SET(enable);
+			}
 			found = 1;
 		}
 		/*
@@ -1756,10 +1874,12 @@ static int ftrace_match_records(char *buff, int len, int enable)
 		 * is filtered on.
 		 */
 		if (enable && (rec->flags & FTRACE_FL_FILTER))
-			ftrace_filtered = 1;
+			ftrace_tracer_filtered |= (1 << ftrace_tracers_cur);
+
 	} while_for_each_ftrace_rec();
-	mutex_unlock(&ftrace_lock);
 
+ out_unlock:
+	mutex_unlock(&ftrace_lock);
 	return found;
 }
 
@@ -1811,6 +1931,10 @@ static int ftrace_match_module_records(char *buff, char *mod, int enable)
 	}
 
 	mutex_lock(&ftrace_lock);
+
+	if (ftrace_tracers_cur < 0)
+		goto out_unlock;
+
 	do_for_each_ftrace_rec(pg, rec) {
 
 		if (rec->flags & FTRACE_FL_FAILED)
@@ -1818,20 +1942,27 @@ static int ftrace_match_module_records(char *buff, char *mod, int enable)
 
 		if (ftrace_match_module_record(rec, mod,
 					       search, search_len, type)) {
-			if (not)
+			if (not) {
 				rec->flags &= ~flag;
-			else
+				TRACER_UNSET(enable);
+			} else {
 				rec->flags |= flag;
+				TRACER_SET(enable);
+			}
 			found = 1;
 		}
 		if (enable && (rec->flags & FTRACE_FL_FILTER))
-			ftrace_filtered = 1;
+			ftrace_tracer_filtered |= (1 << ftrace_tracers_cur);
 
 	} while_for_each_ftrace_rec();
+
+ out_unlock:
 	mutex_unlock(&ftrace_lock);
 
 	return found;
 }
+#undef TRACER_SET
+#undef TRACER_UNSET
 
 /*
  * We register the module command as a template to show others how
-- 
1.7.1

Re: [RFC 0/3] x86_64,tracing: multiplexing function tracer

[Steven Rostedt]

On Fri, 2011-04-29 at 16:43 +0200, Jiri Olsa wrote:
> hi, me again ;)
> 
> from last RFC email it turned out it'd be good to have support
> for more than 1 currently running function tracers. With
> the possibility to specify separate filter for each of them.

Jiri, I thought I told you to wait, as I'm basically doing the same
thing. Except that I'm gutting the heck out of ftrace.c.

I've been working the last several days rewriting this code.

> 
> I thought about one way how to do that and put it to the
> code for considering. It seems to work properly, though
> I was fighting with set_ftrace_filter for a while.. and
> it is still not ideal ;)

Soon, set_ftrace_filter will only affect the function tracer and not any
other user of function tracing.

I'm almost done. I was hoping to release something today, but it will
probably have to wait till next week.

-- Steve

Re: [RFC 0/3] x86_64,tracing: multiplexing function tracer

[Jiri Olsa]

On Fri, Apr 29, 2011 at 10:57:10AM -0400, Steven Rostedt wrote:
> On Fri, 2011-04-29 at 16:43 +0200, Jiri Olsa wrote:
> > hi, me again ;)
> > 
> > from last RFC email it turned out it'd be good to have support
> > for more than 1 currently running function tracers. With
> > the possibility to specify separate filter for each of them.
> 
> Jiri, I thought I told you to wait, as I'm basically doing the same
> thing. Except that I'm gutting the heck out of ftrace.c.

yep, I was just keeping busy.. ;)
looking forward to see your change

jirka

> 
> I've been working the last several days rewriting this code.
> 
> > 
> > I thought about one way how to do that and put it to the
> > code for considering. It seems to work properly, though
> > I was fighting with set_ftrace_filter for a while.. and
> > it is still not ideal ;)
> 
> Soon, set_ftrace_filter will only affect the function tracer and not any
> other user of function tracing.
> 
> I'm almost done. I was hoping to release something today, but it will
> probably have to wait till next week.
> 
> -- Steve
> 
> 

Re: [RFC 3/3] tracing: set_ftrace_filter support

[Steven Rostedt]

On Fri, 2011-04-29 at 16:43 +0200, Jiri Olsa wrote:
> diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
> index b12f5df..9ce3484 100644
> --- a/include/linux/ftrace.h
> +++ b/include/linux/ftrace.h
> @@ -164,6 +164,9 @@ struct dyn_ftrace {
>                 unsigned long           flags;
>                 struct dyn_ftrace       *newlist;
>         };
> +       unsigned long                   filter;
> +       unsigned long                   no_trace;
> +       unsigned long                   addr;
>         struct dyn_arch_ftrace          arch;
>  };
>   

Big Fricken NACK!

Absolutely not! The dyn_ftrace structure is allocated for almost every
single function in the kernel. Any function that can be traced has this
structure allocated.

On my box I have:

# cat /debug/tracing/dyn_ftrace_total_info 
32605 0 0

That's 32,605 functions that can be traced, which turns to be 32,605
dyn_ftrace structs allocated. Now this number is after all modules have
been loaded. You added 3 longs, on 64 bit boxes that's 24 bytes, you
just increased the function tracer footprint by 782,520 bytes!

I'm working around not touching that structure, and I will continue to
NAK any changes that increases its size. Actually, I did touch that
structure, but I did not increase the size of it.

-- Steve